├── README.md ├── C1_W3_Lab07_Scikit_Learn_Soln.ipynb ├── C1_W3_Lab01_Classification_Soln.ipynb ├── C1_W3_Lab03_Decision_Boundary_Soln.ipynb ├── C1_W3_Lab05_Cost_Function_Soln.ipynb ├── C1_W3_Lab02_Sigmoid_function_Soln.ipynb ├── C1_W3_Lab04_LogisticLoss_Soln.ipynb ├── C1_W3_Lab06_Gradient_Descent_Soln.ipynb ├── C1_W3_Lab09_Regularization_Soln.ipynb ├── C1_W2_Lab01_Python_Numpy_Vectorization_Soln.ipynb └── C1_W2_Lab05_Sklearn_GD_Soln.ipynb /README.md: -------------------------------------------------------------------------------- 1 | # DeepLearningAi-Machine-Learning-Specialization 2 | Jupyter Notebooks of the Andrew Ng's Machine Learning Specialization. 3 | 4 | Particularly, these are the notebooks of the Supervised Machine Learning: Linear regression and logistic regression. 5 | -------------------------------------------------------------------------------- /C1_W3_Lab07_Scikit_Learn_Soln.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Ungraded Lab: Logistic Regression using Scikit-Learn\n", 8 | "\n", 9 | "\n" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "## Goals\n", 17 | "In this lab you will:\n", 18 | "- Train a logistic regression model using scikit-learn.\n" 19 | ] 20 | }, 21 | { 22 | "cell_type": "markdown", 23 | "metadata": {}, 24 | "source": [ 25 | "## Dataset \n", 26 | "Let's start with the same dataset as before." 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": 1, 32 | "metadata": {}, 33 | "outputs": [], 34 | "source": [ 35 | "import numpy as np\n", 36 | "\n", 37 | "X = np.array([[0.5, 1.5], [1,1], [1.5, 0.5], [3, 0.5], [2, 2], [1, 2.5]])\n", 38 | "y = np.array([0, 0, 0, 1, 1, 1])" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": {}, 44 | "source": [ 45 | "## Fit the model\n", 46 | "\n", 47 | "The code below imports the [logistic regression model](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression) from scikit-learn. You can fit this model on the training data by calling `fit` function." 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": 2, 53 | "metadata": {}, 54 | "outputs": [ 55 | { 56 | "data": { 57 | "text/plain": [ 58 | "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n", 59 | " intercept_scaling=1, l1_ratio=None, max_iter=100,\n", 60 | " multi_class='auto', n_jobs=None, penalty='l2',\n", 61 | " random_state=None, solver='lbfgs', tol=0.0001, verbose=0,\n", 62 | " warm_start=False)" 63 | ] 64 | }, 65 | "execution_count": 2, 66 | "metadata": {}, 67 | "output_type": "execute_result" 68 | } 69 | ], 70 | "source": [ 71 | "from sklearn.linear_model import LogisticRegression\n", 72 | "\n", 73 | "lr_model = LogisticRegression()\n", 74 | "lr_model.fit(X, y)" 75 | ] 76 | }, 77 | { 78 | "cell_type": "markdown", 79 | "metadata": {}, 80 | "source": [ 81 | "## Make Predictions\n", 82 | "\n", 83 | "You can see the predictions made by this model by calling the `predict` function." 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": 3, 89 | "metadata": {}, 90 | "outputs": [ 91 | { 92 | "name": "stdout", 93 | "output_type": "stream", 94 | "text": [ 95 | "Prediction on training set: [0 0 0 1 1 1]\n" 96 | ] 97 | } 98 | ], 99 | "source": [ 100 | "y_pred = lr_model.predict(X)\n", 101 | "\n", 102 | "print(\"Prediction on training set:\", y_pred)" 103 | ] 104 | }, 105 | { 106 | "cell_type": "markdown", 107 | "metadata": {}, 108 | "source": [ 109 | "## Calculate accuracy\n", 110 | "\n", 111 | "You can calculate this accuracy of this model by calling the `score` function." 112 | ] 113 | }, 114 | { 115 | "cell_type": "code", 116 | "execution_count": 4, 117 | "metadata": {}, 118 | "outputs": [ 119 | { 120 | "name": "stdout", 121 | "output_type": "stream", 122 | "text": [ 123 | "Accuracy on training set: 1.0\n" 124 | ] 125 | } 126 | ], 127 | "source": [ 128 | "print(\"Accuracy on training set:\", lr_model.score(X, y))" 129 | ] 130 | }, 131 | { 132 | "cell_type": "code", 133 | "execution_count": null, 134 | "metadata": {}, 135 | "outputs": [], 136 | "source": [] 137 | } 138 | ], 139 | "metadata": { 140 | "kernelspec": { 141 | "display_name": "Python 3", 142 | "language": "python", 143 | "name": "python3" 144 | }, 145 | "language_info": { 146 | "codemirror_mode": { 147 | "name": "ipython", 148 | "version": 3 149 | }, 150 | "file_extension": ".py", 151 | "mimetype": "text/x-python", 152 | "name": "python", 153 | "nbconvert_exporter": "python", 154 | "pygments_lexer": "ipython3", 155 | "version": "3.7.6" 156 | } 157 | }, 158 | "nbformat": 4, 159 | "nbformat_minor": 5 160 | } 161 | -------------------------------------------------------------------------------- /C1_W3_Lab01_Classification_Soln.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Optional Lab: Classification\n", 8 | "\n", 9 | "In this lab, you will contrast regression and classification." 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 1, 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "import numpy as np\n", 19 | "%matplotlib widget\n", 20 | "import matplotlib.pyplot as plt\n", 21 | "from lab_utils_common import dlc, plot_data\n", 22 | "from plt_one_addpt_onclick import plt_one_addpt_onclick\n", 23 | "plt.style.use('./deeplearning.mplstyle')" 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "metadata": {}, 29 | "source": [ 30 | "## Classification Problems\n", 31 | " Examples of classification problems are things like: identifying email as Spam or Not Spam or determining if a tumor is malignant or benign. In particular, these are examples of *binary* classification where there are two possible outcomes. Outcomes can be described in pairs of 'positive'/'negative' such as 'yes'/'no, 'true'/'false' or '1'/'0'. \n", 32 | "\n", 33 | "Plots of classification data sets often use symbols to indicate the outcome of an example. In the plots below, 'X' is used to represent the positive values while 'O' represents negative outcomes. " 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": 2, 39 | "metadata": {}, 40 | "outputs": [], 41 | "source": [ 42 | "x_train = np.array([0., 1, 2, 3, 4, 5])\n", 43 | "y_train = np.array([0, 0, 0, 1, 1, 1])\n", 44 | "X_train2 = np.array([[0.5, 1.5], [1,1], [1.5, 0.5], [3, 0.5], [2, 2], [1, 2.5]])\n", 45 | "y_train2 = np.array([0, 0, 0, 1, 1, 1])" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 3, 51 | "metadata": {}, 52 | "outputs": [ 53 | { 54 | "data": { 55 | "application/vnd.jupyter.widget-view+json": { 56 | "model_id": "bbf868844bbf4331a2bd7d4327cc1b5e", 57 | "version_major": 2, 58 | "version_minor": 0 59 | }, 60 | "text/plain": [ 61 | "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …" 62 | ] 63 | }, 64 | "metadata": {}, 65 | "output_type": "display_data" 66 | } 67 | ], 68 | "source": [ 69 | "pos = y_train == 1\n", 70 | "neg = y_train == 0\n", 71 | "\n", 72 | "fig,ax = plt.subplots(1,2,figsize=(8,3))\n", 73 | "#plot 1, single variable\n", 74 | "ax[0].scatter(x_train[pos], y_train[pos], marker='x', s=80, c = 'red', label=\"y=1\")\n", 75 | "ax[0].scatter(x_train[neg], y_train[neg], marker='o', s=100, label=\"y=0\", facecolors='none', \n", 76 | " edgecolors=dlc[\"dlblue\"],lw=3)\n", 77 | "\n", 78 | "ax[0].set_ylim(-0.08,1.1)\n", 79 | "ax[0].set_ylabel('y', fontsize=12)\n", 80 | "ax[0].set_xlabel('x', fontsize=12)\n", 81 | "ax[0].set_title('one variable plot')\n", 82 | "ax[0].legend()\n", 83 | "\n", 84 | "#plot 2, two variables\n", 85 | "plot_data(X_train2, y_train2, ax[1])\n", 86 | "ax[1].axis([0, 4, 0, 4])\n", 87 | "ax[1].set_ylabel('$x_1$', fontsize=12)\n", 88 | "ax[1].set_xlabel('$x_0$', fontsize=12)\n", 89 | "ax[1].set_title('two variable plot')\n", 90 | "ax[1].legend()\n", 91 | "plt.tight_layout()\n", 92 | "plt.show()\n" 93 | ] 94 | }, 95 | { 96 | "cell_type": "markdown", 97 | "metadata": {}, 98 | "source": [ 99 | "Note in the plots above:\n", 100 | "- In the single variable plot, positive results are shown both a red 'X's and as y=1. Negative results are blue 'O's and are located at y=0.\n", 101 | " - Recall in the case of linear regression, y would not have been limited to two values but could have been any value.\n", 102 | "- In the two-variable plot, the y axis is not available. Positive results are shown as red 'X's, while negative results use the blue 'O' symbol.\n", 103 | " - Recall in the case of linear regression with multiple variables, y would not have been limited to two values and a similar plot would have been three-dimensional." 104 | ] 105 | }, 106 | { 107 | "cell_type": "markdown", 108 | "metadata": {}, 109 | "source": [ 110 | "## Linear Regression approach\n", 111 | "In the previous week, you applied linear regression to build a prediction model. Let's try that approach here using the simple example that was described in the lecture. The model will predict if a tumor is benign or malignant based on tumor size. Try the following:\n", 112 | "- Click on 'Run Linear Regression' to find the best linear regression model for the given data.\n", 113 | " - Note the resulting linear model does **not** match the data well. \n", 114 | "One option to improve the results is to apply a *threshold*. \n", 115 | "- Tick the box on the 'Toggle 0.5 threshold' to show the predictions if a threshold is applied.\n", 116 | " - These predictions look good, the predictions match the data\n", 117 | "- *Important*: Now, add further 'malignant' data points on the far right, in the large tumor size range (near 10), and re-run linear regression.\n", 118 | " - Now, the model predicts the larger tumor, but data point at x=3 is being incorrectly predicted!\n", 119 | "- to clear/renew the plot, rerun the cell containing the plot command." 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": 5, 125 | "metadata": {}, 126 | "outputs": [ 127 | { 128 | "data": { 129 | "application/vnd.jupyter.widget-view+json": { 130 | "model_id": "84441d0beb5c4ea6b875f6a6614fdf5e", 131 | "version_major": 2, 132 | "version_minor": 0 133 | }, 134 | "text/plain": [ 135 | "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …" 136 | ] 137 | }, 138 | "metadata": {}, 139 | "output_type": "display_data" 140 | } 141 | ], 142 | "source": [ 143 | "w_in = np.zeros((1))\n", 144 | "b_in = 0\n", 145 | "plt.close('all') \n", 146 | "addpt = plt_one_addpt_onclick( x_train,y_train, w_in, b_in, logistic=False)" 147 | ] 148 | }, 149 | { 150 | "cell_type": "markdown", 151 | "metadata": {}, 152 | "source": [ 153 | "The example above demonstrates that the linear model is insufficient to model categorical data. The model can be extended as described in the following lab." 154 | ] 155 | }, 156 | { 157 | "cell_type": "markdown", 158 | "metadata": {}, 159 | "source": [ 160 | "## Congratulations!\n", 161 | "In this lab you:\n", 162 | "- explored categorical data sets and plotting\n", 163 | "- determined that linear regression was insufficient for a classification problem." 164 | ] 165 | }, 166 | { 167 | "cell_type": "code", 168 | "execution_count": null, 169 | "metadata": {}, 170 | "outputs": [], 171 | "source": [] 172 | }, 173 | { 174 | "cell_type": "code", 175 | "execution_count": null, 176 | "metadata": {}, 177 | "outputs": [], 178 | "source": [] 179 | } 180 | ], 181 | "metadata": { 182 | "kernelspec": { 183 | "display_name": "Python 3", 184 | "language": "python", 185 | "name": "python3" 186 | }, 187 | "language_info": { 188 | "codemirror_mode": { 189 | "name": "ipython", 190 | "version": 3 191 | }, 192 | "file_extension": ".py", 193 | "mimetype": "text/x-python", 194 | "name": "python", 195 | "nbconvert_exporter": "python", 196 | "pygments_lexer": "ipython3", 197 | "version": "3.7.6" 198 | } 199 | }, 200 | "nbformat": 4, 201 | "nbformat_minor": 5 202 | } 203 | -------------------------------------------------------------------------------- /C1_W3_Lab03_Decision_Boundary_Soln.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Optional Lab: Logistic Regression, Decision Boundary\n" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## Goals\n", 15 | "In this lab, you will:\n", 16 | "- Plot the decision boundary for a logistic regression model. This will give you a better sense of what the model is predicting.\n" 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": 1, 22 | "metadata": {}, 23 | "outputs": [], 24 | "source": [ 25 | "import numpy as np\n", 26 | "%matplotlib widget\n", 27 | "import matplotlib.pyplot as plt\n", 28 | "from lab_utils_common import plot_data, sigmoid, draw_vthresh\n", 29 | "plt.style.use('./deeplearning.mplstyle')" 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "metadata": {}, 35 | "source": [ 36 | "## Dataset\n", 37 | "\n", 38 | "Let's suppose you have following training dataset\n", 39 | "- The input variable `X` is a numpy array which has 6 training examples, each with two features\n", 40 | "- The output variable `y` is also a numpy array with 6 examples, and `y` is either `0` or `1`" 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": 2, 46 | "metadata": {}, 47 | "outputs": [], 48 | "source": [ 49 | "X = np.array([[0.5, 1.5], [1,1], [1.5, 0.5], [3, 0.5], [2, 2], [1, 2.5]])\n", 50 | "y = np.array([0, 0, 0, 1, 1, 1]).reshape(-1,1) " 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": {}, 56 | "source": [ 57 | "### Plot data \n", 58 | "\n", 59 | "Let's use a helper function to plot this data. The data points with label $y=1$ are shown as red crosses, while the data points with label $y=0$ are shown as blue circles. " 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 3, 65 | "metadata": {}, 66 | "outputs": [ 67 | { 68 | "data": { 69 | "application/vnd.jupyter.widget-view+json": { 70 | "model_id": "b388e05cf87d4d25a154cab77c2df72c", 71 | "version_major": 2, 72 | "version_minor": 0 73 | }, 74 | "text/plain": [ 75 | "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …" 76 | ] 77 | }, 78 | "metadata": {}, 79 | "output_type": "display_data" 80 | } 81 | ], 82 | "source": [ 83 | "fig,ax = plt.subplots(1,1,figsize=(4,4))\n", 84 | "plot_data(X, y, ax)\n", 85 | "\n", 86 | "ax.axis([0, 4, 0, 3.5])\n", 87 | "ax.set_ylabel('$x_1$')\n", 88 | "ax.set_xlabel('$x_0$')\n", 89 | "plt.show()" 90 | ] 91 | }, 92 | { 93 | "cell_type": "markdown", 94 | "metadata": {}, 95 | "source": [ 96 | "## Logistic regression model\n", 97 | "\n", 98 | "\n", 99 | "* Suppose you'd like to train a logistic regression model on this data which has the form \n", 100 | "\n", 101 | " $f(x) = g(w_0x_0+w_1x_1 + b)$\n", 102 | " \n", 103 | " where $g(z) = \\frac{1}{1+e^{-z}}$, which is the sigmoid function\n", 104 | "\n", 105 | "\n", 106 | "* Let's say that you trained the model and get the parameters as $b = -3, w_0 = 1, w_1 = 1$. That is,\n", 107 | "\n", 108 | " $f(x) = g(x_0+x_1-3)$\n", 109 | "\n", 110 | " (You'll learn how to fit these parameters to the data further in the course)\n", 111 | " \n", 112 | " \n", 113 | "Let's try to understand what this trained model is predicting by plotting its decision boundary" 114 | ] 115 | }, 116 | { 117 | "cell_type": "markdown", 118 | "metadata": {}, 119 | "source": [ 120 | "### Refresher on logistic regression and decision boundary\n", 121 | "\n", 122 | "* Recall that for logistic regression, the model is represented as \n", 123 | "\n", 124 | " $$f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}) = g(\\mathbf{w} \\cdot \\mathbf{x}^{(i)} + b) \\tag{1}$$\n", 125 | "\n", 126 | " where $g(z)$ is known as the sigmoid function and it maps all input values to values between 0 and 1:\n", 127 | "\n", 128 | " $g(z) = \\frac{1}{1+e^{-z}}\\tag{2}$\n", 129 | " and $\\mathbf{w} \\cdot \\mathbf{x}$ is the vector dot product:\n", 130 | " \n", 131 | " $$\\mathbf{w} \\cdot \\mathbf{x} = w_0 x_0 + w_1 x_1$$\n", 132 | " \n", 133 | " \n", 134 | " * We interpret the output of the model ($f_{\\mathbf{w},b}(x)$) as the probability that $y=1$ given $\\mathbf{x}$ and parameterized by $\\mathbf{w}$ and $b$.\n", 135 | "* Therefore, to get a final prediction ($y=0$ or $y=1$) from the logistic regression model, we can use the following heuristic -\n", 136 | "\n", 137 | " if $f_{\\mathbf{w},b}(x) >= 0.5$, predict $y=1$\n", 138 | " \n", 139 | " if $f_{\\mathbf{w},b}(x) < 0.5$, predict $y=0$\n", 140 | " \n", 141 | " \n", 142 | "* Let's plot the sigmoid function to see where $g(z) >= 0.5$" 143 | ] 144 | }, 145 | { 146 | "cell_type": "code", 147 | "execution_count": 4, 148 | "metadata": {}, 149 | "outputs": [ 150 | { 151 | "data": { 152 | "application/vnd.jupyter.widget-view+json": { 153 | "model_id": "4cc4cc9bb4a244e68e6ca7a89763ceae", 154 | "version_major": 2, 155 | "version_minor": 0 156 | }, 157 | "text/plain": [ 158 | "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …" 159 | ] 160 | }, 161 | "metadata": {}, 162 | "output_type": "display_data" 163 | } 164 | ], 165 | "source": [ 166 | "# Plot sigmoid(z) over a range of values from -10 to 10\n", 167 | "z = np.arange(-10,11)\n", 168 | "\n", 169 | "fig,ax = plt.subplots(1,1,figsize=(5,3))\n", 170 | "# Plot z vs sigmoid(z)\n", 171 | "ax.plot(z, sigmoid(z), c=\"b\")\n", 172 | "\n", 173 | "ax.set_title(\"Sigmoid function\")\n", 174 | "ax.set_ylabel('sigmoid(z)')\n", 175 | "ax.set_xlabel('z')\n", 176 | "draw_vthresh(ax,0)" 177 | ] 178 | }, 179 | { 180 | "cell_type": "markdown", 181 | "metadata": {}, 182 | "source": [ 183 | "* As you can see, $g(z) >= 0.5$ for $z >=0$\n", 184 | "\n", 185 | "* For a logistic regression model, $z = \\mathbf{w} \\cdot \\mathbf{x} + b$. Therefore,\n", 186 | "\n", 187 | " if $\\mathbf{w} \\cdot \\mathbf{x} + b >= 0$, the model predicts $y=1$\n", 188 | " \n", 189 | " if $\\mathbf{w} \\cdot \\mathbf{x} + b < 0$, the model predicts $y=0$\n", 190 | " \n", 191 | " \n", 192 | " \n", 193 | "### Plotting decision boundary\n", 194 | "\n", 195 | "Now, let's go back to our example to understand how the logistic regression model is making predictions.\n", 196 | "\n", 197 | "* Our logistic regression model has the form\n", 198 | "\n", 199 | " $f(\\mathbf{x}) = g(-3 + x_0+x_1)$\n", 200 | "\n", 201 | "\n", 202 | "* From what you've learnt above, you can see that this model predicts $y=1$ if $-3 + x_0+x_1 >= 0$\n", 203 | "\n", 204 | "Let's see what this looks like graphically. We'll start by plotting $-3 + x_0+x_1 = 0$, which is equivalent to $x_1 = 3 - x_0$.\n" 205 | ] 206 | }, 207 | { 208 | "cell_type": "code", 209 | "execution_count": 5, 210 | "metadata": {}, 211 | "outputs": [ 212 | { 213 | "data": { 214 | "application/vnd.jupyter.widget-view+json": { 215 | "model_id": "c00e4ac749fb4cc6b218ef2f9fce23a7", 216 | "version_major": 2, 217 | "version_minor": 0 218 | }, 219 | "text/plain": [ 220 | "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …" 221 | ] 222 | }, 223 | "metadata": {}, 224 | "output_type": "display_data" 225 | } 226 | ], 227 | "source": [ 228 | "# Choose values between 0 and 6\n", 229 | "x0 = np.arange(0,6)\n", 230 | "\n", 231 | "x1 = 3 - x0\n", 232 | "fig,ax = plt.subplots(1,1,figsize=(5,4))\n", 233 | "# Plot the decision boundary\n", 234 | "ax.plot(x0,x1, c=\"b\")\n", 235 | "ax.axis([0, 4, 0, 3.5])\n", 236 | "\n", 237 | "# Fill the region below the line\n", 238 | "ax.fill_between(x0,x1, alpha=0.2)\n", 239 | "\n", 240 | "# Plot the original data\n", 241 | "plot_data(X,y,ax)\n", 242 | "ax.set_ylabel(r'$x_1$')\n", 243 | "ax.set_xlabel(r'$x_0$')\n", 244 | "plt.show()" 245 | ] 246 | }, 247 | { 248 | "cell_type": "markdown", 249 | "metadata": {}, 250 | "source": [ 251 | "* In the plot above, the blue line represents the line $x_0 + x_1 - 3 = 0$ and it should intersect the x1 axis at 3 (if we set $x_1$ = 3, $x_0$ = 0) and the x0 axis at 3 (if we set $x_1$ = 0, $x_0$ = 3). \n", 252 | "\n", 253 | "\n", 254 | "* The shaded region represents $-3 + x_0+x_1 < 0$. The region above the line is $-3 + x_0+x_1 > 0$.\n", 255 | "\n", 256 | "\n", 257 | "* Any point in the shaded region (under the line) is classified as $y=0$. Any point on or above the line is classified as $y=1$. This line is known as the \"decision boundary\".\n", 258 | "\n", 259 | "As we've seen in the lectures, by using higher order polynomial terms (eg: $f(x) = g( x_0^2 + x_1 -1)$, we can come up with more complex non-linear boundaries." 260 | ] 261 | }, 262 | { 263 | "cell_type": "markdown", 264 | "metadata": {}, 265 | "source": [ 266 | "## Congratulations!\n", 267 | "You have explored the decision boundary in the context of logistic regression." 268 | ] 269 | }, 270 | { 271 | "cell_type": "code", 272 | "execution_count": null, 273 | "metadata": {}, 274 | "outputs": [], 275 | "source": [] 276 | }, 277 | { 278 | "cell_type": "code", 279 | "execution_count": null, 280 | "metadata": {}, 281 | "outputs": [], 282 | "source": [] 283 | } 284 | ], 285 | "metadata": { 286 | "kernelspec": { 287 | "display_name": "Python 3", 288 | "language": "python", 289 | "name": "python3" 290 | }, 291 | "language_info": { 292 | "codemirror_mode": { 293 | "name": "ipython", 294 | "version": 3 295 | }, 296 | "file_extension": ".py", 297 | "mimetype": "text/x-python", 298 | "name": "python", 299 | "nbconvert_exporter": "python", 300 | "pygments_lexer": "ipython3", 301 | "version": "3.7.6" 302 | } 303 | }, 304 | "nbformat": 4, 305 | "nbformat_minor": 5 306 | } 307 | -------------------------------------------------------------------------------- /C1_W3_Lab05_Cost_Function_Soln.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Optional Lab: Cost Function for Logistic Regression\n", 8 | "\n", 9 | "## Goals\n", 10 | "In this lab, you will:\n", 11 | "- examine the implementation and utilize the cost function for logistic regression." 12 | ] 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": 1, 17 | "metadata": {}, 18 | "outputs": [], 19 | "source": [ 20 | "import numpy as np\n", 21 | "%matplotlib widget\n", 22 | "import matplotlib.pyplot as plt\n", 23 | "from lab_utils_common import plot_data, sigmoid, dlc\n", 24 | "plt.style.use('./deeplearning.mplstyle')" 25 | ] 26 | }, 27 | { 28 | "cell_type": "markdown", 29 | "metadata": {}, 30 | "source": [ 31 | "## Dataset \n", 32 | "Let's start with the same dataset as was used in the decision boundary lab." 33 | ] 34 | }, 35 | { 36 | "cell_type": "code", 37 | "execution_count": 2, 38 | "metadata": { 39 | "tags": [] 40 | }, 41 | "outputs": [], 42 | "source": [ 43 | "X_train = np.array([[0.5, 1.5], [1,1], [1.5, 0.5], [3, 0.5], [2, 2], [1, 2.5]]) #(m,n)\n", 44 | "y_train = np.array([0, 0, 0, 1, 1, 1]) #(m,)" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": {}, 50 | "source": [ 51 | "We will use a helper function to plot this data. The data points with label $y=1$ are shown as red crosses, while the data points with label $y=0$ are shown as blue circles." 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": 3, 57 | "metadata": {}, 58 | "outputs": [ 59 | { 60 | "data": { 61 | "application/vnd.jupyter.widget-view+json": { 62 | "model_id": "3185aacae1dc46ec864ac6810c03e755", 63 | "version_major": 2, 64 | "version_minor": 0 65 | }, 66 | "text/plain": [ 67 | "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …" 68 | ] 69 | }, 70 | "metadata": {}, 71 | "output_type": "display_data" 72 | } 73 | ], 74 | "source": [ 75 | "fig,ax = plt.subplots(1,1,figsize=(4,4))\n", 76 | "plot_data(X_train, y_train, ax)\n", 77 | "\n", 78 | "# Set both axes to be from 0-4\n", 79 | "ax.axis([0, 4, 0, 3.5])\n", 80 | "ax.set_ylabel('$x_1$', fontsize=12)\n", 81 | "ax.set_xlabel('$x_0$', fontsize=12)\n", 82 | "plt.show()" 83 | ] 84 | }, 85 | { 86 | "cell_type": "markdown", 87 | "metadata": {}, 88 | "source": [ 89 | "## Cost function\n", 90 | "\n", 91 | "In a previous lab, you developed the *logistic loss* function. Recall, loss is defined to apply to one example. Here you combine the losses to form the **cost**, which includes all the examples.\n", 92 | "\n", 93 | "\n", 94 | "Recall that for logistic regression, the cost function is of the form \n", 95 | "\n", 96 | "$$ J(\\mathbf{w},b) = \\frac{1}{m} \\sum_{i=0}^{m-1} \\left[ loss(f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}), y^{(i)}) \\right] \\tag{1}$$\n", 97 | "\n", 98 | "where\n", 99 | "* $loss(f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}), y^{(i)})$ is the cost for a single data point, which is:\n", 100 | "\n", 101 | " $$loss(f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}), y^{(i)}) = -y^{(i)} \\log\\left(f_{\\mathbf{w},b}\\left( \\mathbf{x}^{(i)} \\right) \\right) - \\left( 1 - y^{(i)}\\right) \\log \\left( 1 - f_{\\mathbf{w},b}\\left( \\mathbf{x}^{(i)} \\right) \\right) \\tag{2}$$\n", 102 | " \n", 103 | "* where m is the number of training examples in the data set and:\n", 104 | "$$\n", 105 | "\\begin{align}\n", 106 | " f_{\\mathbf{w},b}(\\mathbf{x^{(i)}}) &= g(z^{(i)})\\tag{3} \\\\\n", 107 | " z^{(i)} &= \\mathbf{w} \\cdot \\mathbf{x}^{(i)}+ b\\tag{4} \\\\\n", 108 | " g(z^{(i)}) &= \\frac{1}{1+e^{-z^{(i)}}}\\tag{5} \n", 109 | "\\end{align}\n", 110 | "$$\n", 111 | " " 112 | ] 113 | }, 114 | { 115 | "cell_type": "markdown", 116 | "metadata": {}, 117 | "source": [ 118 | "\n", 119 | "#### Code Description\n", 120 | "\n", 121 | "The algorithm for `compute_cost_logistic` loops over all the examples calculating the loss for each example and accumulating the total.\n", 122 | "\n", 123 | "Note that the variables X and y are not scalar values but matrices of shape ($m, n$) and ($𝑚$,) respectively, where $𝑛$ is the number of features and $𝑚$ is the number of training examples.\n" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": 4, 129 | "metadata": {}, 130 | "outputs": [], 131 | "source": [ 132 | "def compute_cost_logistic(X, y, w, b):\n", 133 | " \"\"\"\n", 134 | " Computes cost\n", 135 | "\n", 136 | " Args:\n", 137 | " X (ndarray (m,n)): Data, m examples with n features\n", 138 | " y (ndarray (m,)) : target values\n", 139 | " w (ndarray (n,)) : model parameters \n", 140 | " b (scalar) : model parameter\n", 141 | " \n", 142 | " Returns:\n", 143 | " cost (scalar): cost\n", 144 | " \"\"\"\n", 145 | "\n", 146 | " m = X.shape[0]\n", 147 | " cost = 0.0\n", 148 | " for i in range(m):\n", 149 | " z_i = np.dot(X[i],w) + b\n", 150 | " f_wb_i = sigmoid(z_i)\n", 151 | " cost += -y[i]*np.log(f_wb_i) - (1-y[i])*np.log(1-f_wb_i)\n", 152 | " \n", 153 | " cost = cost / m\n", 154 | " return cost\n" 155 | ] 156 | }, 157 | { 158 | "cell_type": "markdown", 159 | "metadata": {}, 160 | "source": [ 161 | "Check the implementation of the cost function using the cell below." 162 | ] 163 | }, 164 | { 165 | "cell_type": "code", 166 | "execution_count": 5, 167 | "metadata": {}, 168 | "outputs": [ 169 | { 170 | "name": "stdout", 171 | "output_type": "stream", 172 | "text": [ 173 | "0.36686678640551745\n" 174 | ] 175 | } 176 | ], 177 | "source": [ 178 | "w_tmp = np.array([1,1])\n", 179 | "b_tmp = -3\n", 180 | "print(compute_cost_logistic(X_train, y_train, w_tmp, b_tmp))" 181 | ] 182 | }, 183 | { 184 | "cell_type": "markdown", 185 | "metadata": {}, 186 | "source": [ 187 | "**Expected output**: 0.3668667864055175" 188 | ] 189 | }, 190 | { 191 | "cell_type": "markdown", 192 | "metadata": {}, 193 | "source": [ 194 | "## Example\n", 195 | "Now, let's see what the cost function output is for a different value of $w$. \n", 196 | "\n", 197 | "* In a previous lab, you plotted the decision boundary for $b = -3, w_0 = 1, w_1 = 1$. That is, you had `b = -3, w = np.array([1,1])`.\n", 198 | "\n", 199 | "* Let's say you want to see if $b = -4, w_0 = 1, w_1 = 1$, or `b = -4, w = np.array([1,1])` provides a better model.\n", 200 | "\n", 201 | "Let's first plot the decision boundary for these two different $b$ values to see which one fits the data better.\n", 202 | "\n", 203 | "* For $b = -3, w_0 = 1, w_1 = 1$, we'll plot $-3 + x_0+x_1 = 0$ (shown in blue)\n", 204 | "* For $b = -4, w_0 = 1, w_1 = 1$, we'll plot $-4 + x_0+x_1 = 0$ (shown in magenta)" 205 | ] 206 | }, 207 | { 208 | "cell_type": "code", 209 | "execution_count": 6, 210 | "metadata": {}, 211 | "outputs": [ 212 | { 213 | "data": { 214 | "application/vnd.jupyter.widget-view+json": { 215 | "model_id": "7efc6d6d108d489ebc7fb8cb7b1927d7", 216 | "version_major": 2, 217 | "version_minor": 0 218 | }, 219 | "text/plain": [ 220 | "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …" 221 | ] 222 | }, 223 | "metadata": {}, 224 | "output_type": "display_data" 225 | } 226 | ], 227 | "source": [ 228 | "import matplotlib.pyplot as plt\n", 229 | "\n", 230 | "# Choose values between 0 and 6\n", 231 | "x0 = np.arange(0,6)\n", 232 | "\n", 233 | "# Plot the two decision boundaries\n", 234 | "x1 = 3 - x0\n", 235 | "x1_other = 4 - x0\n", 236 | "\n", 237 | "fig,ax = plt.subplots(1, 1, figsize=(4,4))\n", 238 | "# Plot the decision boundary\n", 239 | "ax.plot(x0,x1, c=dlc[\"dlblue\"], label=\"$b$=-3\")\n", 240 | "ax.plot(x0,x1_other, c=dlc[\"dlmagenta\"], label=\"$b$=-4\")\n", 241 | "ax.axis([0, 4, 0, 4])\n", 242 | "\n", 243 | "# Plot the original data\n", 244 | "plot_data(X_train,y_train,ax)\n", 245 | "ax.axis([0, 4, 0, 4])\n", 246 | "ax.set_ylabel('$x_1$', fontsize=12)\n", 247 | "ax.set_xlabel('$x_0$', fontsize=12)\n", 248 | "plt.legend(loc=\"upper right\")\n", 249 | "plt.title(\"Decision Boundary\")\n", 250 | "plt.show()" 251 | ] 252 | }, 253 | { 254 | "cell_type": "markdown", 255 | "metadata": {}, 256 | "source": [ 257 | "You can see from this plot that `b = -4, w = np.array([1,1])` is a worse model for the training data. Let's see if the cost function implementation reflects this." 258 | ] 259 | }, 260 | { 261 | "cell_type": "code", 262 | "execution_count": 7, 263 | "metadata": {}, 264 | "outputs": [ 265 | { 266 | "name": "stdout", 267 | "output_type": "stream", 268 | "text": [ 269 | "Cost for b = -3 : 0.36686678640551745\n", 270 | "Cost for b = -4 : 0.5036808636748461\n" 271 | ] 272 | } 273 | ], 274 | "source": [ 275 | "w_array1 = np.array([1,1])\n", 276 | "b_1 = -3\n", 277 | "w_array2 = np.array([1,1])\n", 278 | "b_2 = -4\n", 279 | "\n", 280 | "print(\"Cost for b = -3 : \", compute_cost_logistic(X_train, y_train, w_array1, b_1))\n", 281 | "print(\"Cost for b = -4 : \", compute_cost_logistic(X_train, y_train, w_array2, b_2))" 282 | ] 283 | }, 284 | { 285 | "cell_type": "markdown", 286 | "metadata": {}, 287 | "source": [ 288 | "**Expected output**\n", 289 | "\n", 290 | "Cost for b = -3 : 0.3668667864055175\n", 291 | "\n", 292 | "Cost for b = -4 : 0.5036808636748461\n", 293 | "\n", 294 | "\n", 295 | "You can see the cost function behaves as expected and the cost for `b = -4, w = np.array([1,1])` is indeed higher than the cost for `b = -3, w = np.array([1,1])`" 296 | ] 297 | }, 298 | { 299 | "cell_type": "markdown", 300 | "metadata": {}, 301 | "source": [ 302 | "## Congratulations!\n", 303 | "In this lab you examined and utilized the cost function for logistic regression." 304 | ] 305 | }, 306 | { 307 | "cell_type": "code", 308 | "execution_count": null, 309 | "metadata": {}, 310 | "outputs": [], 311 | "source": [] 312 | } 313 | ], 314 | "metadata": { 315 | "kernelspec": { 316 | "display_name": "Python 3", 317 | "language": "python", 318 | "name": "python3" 319 | }, 320 | "language_info": { 321 | "codemirror_mode": { 322 | "name": "ipython", 323 | "version": 3 324 | }, 325 | "file_extension": ".py", 326 | "mimetype": "text/x-python", 327 | "name": "python", 328 | "nbconvert_exporter": "python", 329 | "pygments_lexer": "ipython3", 330 | "version": "3.7.6" 331 | } 332 | }, 333 | "nbformat": 4, 334 | "nbformat_minor": 5 335 | } 336 | -------------------------------------------------------------------------------- /C1_W3_Lab02_Sigmoid_function_Soln.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Optional Lab: Logistic Regression\n", 8 | "\n", 9 | "In this ungraded lab, you will \n", 10 | "- explore the sigmoid function (also known as the logistic function)\n", 11 | "- explore logistic regression; which uses the sigmoid function" 12 | ] 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": 1, 17 | "metadata": {}, 18 | "outputs": [], 19 | "source": [ 20 | "import numpy as np\n", 21 | "%matplotlib widget\n", 22 | "import matplotlib.pyplot as plt\n", 23 | "from plt_one_addpt_onclick import plt_one_addpt_onclick\n", 24 | "from lab_utils_common import draw_vthresh\n", 25 | "plt.style.use('./deeplearning.mplstyle')" 26 | ] 27 | }, 28 | { 29 | "cell_type": "markdown", 30 | "metadata": {}, 31 | "source": [ 32 | "## Sigmoid or Logistic Function\n", 33 | "As discussed in the lecture videos, for a classification task, we can start by using our linear regression model, $f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}) = \\mathbf{w} \\cdot \\mathbf{x}^{(i)} + b$, to predict $y$ given $x$. \n", 34 | "- However, we would like the predictions of our classification model to be between 0 and 1 since our output variable $y$ is either 0 or 1. \n", 35 | "- This can be accomplished by using a \"sigmoid function\" which maps all input values to values between 0 and 1. \n", 36 | "\n", 37 | "\n", 38 | "Let's implement the sigmoid function and see this for ourselves.\n", 39 | "\n", 40 | "## Formula for Sigmoid function\n", 41 | "\n", 42 | "The formula for a sigmoid function is as follows - \n", 43 | "\n", 44 | "$g(z) = \\frac{1}{1+e^{-z}}\\tag{1}$\n", 45 | "\n", 46 | "In the case of logistic regression, z (the input to the sigmoid function), is the output of a linear regression model. \n", 47 | "- In the case of a single example, $z$ is scalar.\n", 48 | "- in the case of multiple examples, $z$ may be a vector consisting of $m$ values, one for each example. \n", 49 | "- The implementation of the sigmoid function should cover both of these potential input formats.\n", 50 | "Let's implement this in Python." 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": {}, 56 | "source": [ 57 | "NumPy has a function called [`exp()`](https://numpy.org/doc/stable/reference/generated/numpy.exp.html), which offers a convenient way to calculate the exponential ( $e^{z}$) of all elements in the input array (`z`).\n", 58 | " \n", 59 | "It also works with a single number as an input, as shown below." 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 2, 65 | "metadata": {}, 66 | "outputs": [ 67 | { 68 | "name": "stdout", 69 | "output_type": "stream", 70 | "text": [ 71 | "Input to exp: [1 2 3]\n", 72 | "Output of exp: [ 2.72 7.39 20.09]\n", 73 | "Input to exp: 1\n", 74 | "Output of exp: 2.718281828459045\n" 75 | ] 76 | } 77 | ], 78 | "source": [ 79 | "# Input is an array. \n", 80 | "input_array = np.array([1,2,3])\n", 81 | "exp_array = np.exp(input_array)\n", 82 | "\n", 83 | "print(\"Input to exp:\", input_array)\n", 84 | "print(\"Output of exp:\", exp_array)\n", 85 | "\n", 86 | "# Input is a single number\n", 87 | "input_val = 1 \n", 88 | "exp_val = np.exp(input_val)\n", 89 | "\n", 90 | "print(\"Input to exp:\", input_val)\n", 91 | "print(\"Output of exp:\", exp_val)" 92 | ] 93 | }, 94 | { 95 | "cell_type": "markdown", 96 | "metadata": {}, 97 | "source": [ 98 | "The `sigmoid` function is implemented in python as shown in the cell below." 99 | ] 100 | }, 101 | { 102 | "cell_type": "code", 103 | "execution_count": 3, 104 | "metadata": {}, 105 | "outputs": [], 106 | "source": [ 107 | "def sigmoid(z):\n", 108 | " \"\"\"\n", 109 | " Compute the sigmoid of z\n", 110 | "\n", 111 | " Args:\n", 112 | " z (ndarray): A scalar, numpy array of any size.\n", 113 | "\n", 114 | " Returns:\n", 115 | " g (ndarray): sigmoid(z), with the same shape as z\n", 116 | " \n", 117 | " \"\"\"\n", 118 | "\n", 119 | " g = 1/(1+np.exp(-z))\n", 120 | " \n", 121 | " return g" 122 | ] 123 | }, 124 | { 125 | "cell_type": "markdown", 126 | "metadata": {}, 127 | "source": [ 128 | "Let's see what the output of this function is for various value of `z`" 129 | ] 130 | }, 131 | { 132 | "cell_type": "code", 133 | "execution_count": 4, 134 | "metadata": {}, 135 | "outputs": [ 136 | { 137 | "name": "stdout", 138 | "output_type": "stream", 139 | "text": [ 140 | "Input (z), Output (sigmoid(z))\n", 141 | "[[-1.000e+01 4.540e-05]\n", 142 | " [-9.000e+00 1.234e-04]\n", 143 | " [-8.000e+00 3.354e-04]\n", 144 | " [-7.000e+00 9.111e-04]\n", 145 | " [-6.000e+00 2.473e-03]\n", 146 | " [-5.000e+00 6.693e-03]\n", 147 | " [-4.000e+00 1.799e-02]\n", 148 | " [-3.000e+00 4.743e-02]\n", 149 | " [-2.000e+00 1.192e-01]\n", 150 | " [-1.000e+00 2.689e-01]\n", 151 | " [ 0.000e+00 5.000e-01]\n", 152 | " [ 1.000e+00 7.311e-01]\n", 153 | " [ 2.000e+00 8.808e-01]\n", 154 | " [ 3.000e+00 9.526e-01]\n", 155 | " [ 4.000e+00 9.820e-01]\n", 156 | " [ 5.000e+00 9.933e-01]\n", 157 | " [ 6.000e+00 9.975e-01]\n", 158 | " [ 7.000e+00 9.991e-01]\n", 159 | " [ 8.000e+00 9.997e-01]\n", 160 | " [ 9.000e+00 9.999e-01]\n", 161 | " [ 1.000e+01 1.000e+00]]\n" 162 | ] 163 | } 164 | ], 165 | "source": [ 166 | "# Generate an array of evenly spaced values between -10 and 10\n", 167 | "z_tmp = np.arange(-10,11)\n", 168 | "\n", 169 | "# Use the function implemented above to get the sigmoid values\n", 170 | "y = sigmoid(z_tmp)\n", 171 | "\n", 172 | "# Code for pretty printing the two arrays next to each other\n", 173 | "np.set_printoptions(precision=3) \n", 174 | "print(\"Input (z), Output (sigmoid(z))\")\n", 175 | "print(np.c_[z_tmp, y])" 176 | ] 177 | }, 178 | { 179 | "cell_type": "markdown", 180 | "metadata": {}, 181 | "source": [ 182 | "The values in the left column are `z`, and the values in the right column are `sigmoid(z)`. As you can see, the input values to the sigmoid range from -10 to 10, and the output values range from 0 to 1. \n", 183 | "\n", 184 | "Now, let's try to plot this function using the `matplotlib` library." 185 | ] 186 | }, 187 | { 188 | "cell_type": "code", 189 | "execution_count": 5, 190 | "metadata": {}, 191 | "outputs": [ 192 | { 193 | "data": { 194 | "application/vnd.jupyter.widget-view+json": { 195 | "model_id": "29b854cb9fc64c1985a63732afc5d34d", 196 | "version_major": 2, 197 | "version_minor": 0 198 | }, 199 | "text/plain": [ 200 | "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …" 201 | ] 202 | }, 203 | "metadata": {}, 204 | "output_type": "display_data" 205 | } 206 | ], 207 | "source": [ 208 | "# Plot z vs sigmoid(z)\n", 209 | "fig,ax = plt.subplots(1,1,figsize=(5,3))\n", 210 | "ax.plot(z_tmp, y, c=\"b\")\n", 211 | "\n", 212 | "ax.set_title(\"Sigmoid function\")\n", 213 | "ax.set_ylabel('sigmoid(z)')\n", 214 | "ax.set_xlabel('z')\n", 215 | "draw_vthresh(ax,0)" 216 | ] 217 | }, 218 | { 219 | "cell_type": "markdown", 220 | "metadata": {}, 221 | "source": [ 222 | "As you can see, the sigmoid function approaches `0` as `z` goes to large negative values and approaches `1` as `z` goes to large positive values.\n" 223 | ] 224 | }, 225 | { 226 | "cell_type": "markdown", 227 | "metadata": {}, 228 | "source": [ 229 | "## Logistic Regression\n", 230 | " A logistic regression model applies the sigmoid to the familiar linear regression model as shown below:\n", 231 | "\n", 232 | "$$ f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}) = g(\\mathbf{w} \\cdot \\mathbf{x}^{(i)} + b ) \\tag{2} $$ \n", 233 | "\n", 234 | " where\n", 235 | "\n", 236 | " $g(z) = \\frac{1}{1+e^{-z}}\\tag{3}$\n" 237 | ] 238 | }, 239 | { 240 | "cell_type": "markdown", 241 | "metadata": {}, 242 | "source": [ 243 | " \n", 244 | "Let's apply logistic regression to the categorical data example of tumor classification. \n", 245 | "First, load the examples and initial values for the parameters.\n", 246 | " \n" 247 | ] 248 | }, 249 | { 250 | "cell_type": "code", 251 | "execution_count": 6, 252 | "metadata": { 253 | "tags": [] 254 | }, 255 | "outputs": [], 256 | "source": [ 257 | "x_train = np.array([0., 1, 2, 3, 4, 5])\n", 258 | "y_train = np.array([0, 0, 0, 1, 1, 1])\n", 259 | "\n", 260 | "w_in = np.zeros((1))\n", 261 | "b_in = 0" 262 | ] 263 | }, 264 | { 265 | "cell_type": "markdown", 266 | "metadata": {}, 267 | "source": [ 268 | "Try the following steps:\n", 269 | "- Click on 'Run Logistic Regression' to find the best logistic regression model for the given training data\n", 270 | " - Note the resulting model fits the data quite well.\n", 271 | " - Note, the orange line is '$z$' or $\\mathbf{w} \\cdot \\mathbf{x}^{(i)} + b$ above. It does not match the line in a linear regression model.\n", 272 | "Further improve these results by applying a *threshold*. \n", 273 | "- Tick the box on the 'Toggle 0.5 threshold' to show the predictions if a threshold is applied.\n", 274 | " - These predictions look good. The predictions match the data\n", 275 | " - Now, add further data points in the large tumor size range (near 10), and re-run logistic regression.\n", 276 | " - unlike the linear regression model, this model continues to make correct predictions" 277 | ] 278 | }, 279 | { 280 | "cell_type": "code", 281 | "execution_count": 7, 282 | "metadata": {}, 283 | "outputs": [ 284 | { 285 | "data": { 286 | "application/vnd.jupyter.widget-view+json": { 287 | "model_id": "55e52cd307c14a0892f07625ac72bfc0", 288 | "version_major": 2, 289 | "version_minor": 0 290 | }, 291 | "text/plain": [ 292 | "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …" 293 | ] 294 | }, 295 | "metadata": {}, 296 | "output_type": "display_data" 297 | } 298 | ], 299 | "source": [ 300 | "plt.close('all') \n", 301 | "addpt = plt_one_addpt_onclick( x_train,y_train, w_in, b_in, logistic=True)" 302 | ] 303 | }, 304 | { 305 | "cell_type": "markdown", 306 | "metadata": {}, 307 | "source": [ 308 | "## Congratulations!\n", 309 | "You have explored the use of the sigmoid function in logistic regression." 310 | ] 311 | }, 312 | { 313 | "cell_type": "code", 314 | "execution_count": null, 315 | "metadata": {}, 316 | "outputs": [], 317 | "source": [] 318 | } 319 | ], 320 | "metadata": { 321 | "kernelspec": { 322 | "display_name": "Python 3", 323 | "language": "python", 324 | "name": "python3" 325 | }, 326 | "language_info": { 327 | "codemirror_mode": { 328 | "name": "ipython", 329 | "version": 3 330 | }, 331 | "file_extension": ".py", 332 | "mimetype": "text/x-python", 333 | "name": "python", 334 | "nbconvert_exporter": "python", 335 | "pygments_lexer": "ipython3", 336 | "version": "3.7.6" 337 | } 338 | }, 339 | "nbformat": 4, 340 | "nbformat_minor": 5 341 | } 342 | -------------------------------------------------------------------------------- /C1_W3_Lab04_LogisticLoss_Soln.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Optional Lab: Logistic Regression, Logistic Loss\n", 8 | "\n", 9 | "In this ungraded lab, you will:\n", 10 | "- explore the reason the squared error loss is not appropriate for logistic regression\n", 11 | "- explore the logistic loss function" 12 | ] 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": 1, 17 | "metadata": {}, 18 | "outputs": [], 19 | "source": [ 20 | "import numpy as np\n", 21 | "%matplotlib widget\n", 22 | "import matplotlib.pyplot as plt\n", 23 | "from plt_logistic_loss import plt_logistic_cost, plt_two_logistic_loss_curves, plt_simple_example\n", 24 | "from plt_logistic_loss import soup_bowl, plt_logistic_squared_error\n", 25 | "plt.style.use('./deeplearning.mplstyle')" 26 | ] 27 | }, 28 | { 29 | "cell_type": "markdown", 30 | "metadata": {}, 31 | "source": [ 32 | "## Squared error for logistic regression?\n", 33 | " Recall for **Linear** Regression we have used the **squared error cost function**:\n", 34 | "The equation for the squared error cost with one variable is:\n", 35 | " $$J(w,b) = \\frac{1}{2m} \\sum\\limits_{i = 0}^{m-1} (f_{w,b}(x^{(i)}) - y^{(i)})^2 \\tag{1}$$ \n", 36 | " \n", 37 | "where \n", 38 | " $$f_{w,b}(x^{(i)}) = wx^{(i)} + b \\tag{2}$$\n" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": {}, 44 | "source": [ 45 | "Recall, the squared error cost had the nice property that following the derivative of the cost leads to the minimum." 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 2, 51 | "metadata": {}, 52 | "outputs": [ 53 | { 54 | "data": { 55 | "application/vnd.jupyter.widget-view+json": { 56 | "model_id": "e95faeb9cd7d48469cc1bbd1cd299521", 57 | "version_major": 2, 58 | "version_minor": 0 59 | }, 60 | "text/plain": [ 61 | "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …" 62 | ] 63 | }, 64 | "metadata": {}, 65 | "output_type": "display_data" 66 | } 67 | ], 68 | "source": [ 69 | "soup_bowl()" 70 | ] 71 | }, 72 | { 73 | "cell_type": "markdown", 74 | "metadata": {}, 75 | "source": [ 76 | "This cost function worked well for linear regression, it is natural to consider it for logistic regression as well. However, as the slide above points out, $f_{wb}(x)$ now has a non-linear component, the sigmoid function: $f_{w,b}(x^{(i)}) = sigmoid(wx^{(i)} + b )$. Let's try a squared error cost on the example from an earlier lab, now including the sigmoid.\n", 77 | "\n", 78 | "Here is our training data:" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": 3, 84 | "metadata": {}, 85 | "outputs": [ 86 | { 87 | "data": { 88 | "application/vnd.jupyter.widget-view+json": { 89 | "model_id": "5bc9cae7aa224fb9b995c3ec90dc4826", 90 | "version_major": 2, 91 | "version_minor": 0 92 | }, 93 | "text/plain": [ 94 | "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …" 95 | ] 96 | }, 97 | "metadata": {}, 98 | "output_type": "display_data" 99 | } 100 | ], 101 | "source": [ 102 | "x_train = np.array([0., 1, 2, 3, 4, 5],dtype=np.longdouble)\n", 103 | "y_train = np.array([0, 0, 0, 1, 1, 1],dtype=np.longdouble)\n", 104 | "plt_simple_example(x_train, y_train)" 105 | ] 106 | }, 107 | { 108 | "cell_type": "markdown", 109 | "metadata": {}, 110 | "source": [ 111 | "Now, let's get a surface plot of the cost using a *squared error cost*:\n", 112 | " $$J(w,b) = \\frac{1}{2m} \\sum\\limits_{i = 0}^{m-1} (f_{w,b}(x^{(i)}) - y^{(i)})^2 $$ \n", 113 | " \n", 114 | "where \n", 115 | " $$f_{w,b}(x^{(i)}) = sigmoid(wx^{(i)} + b )$$\n" 116 | ] 117 | }, 118 | { 119 | "cell_type": "code", 120 | "execution_count": 4, 121 | "metadata": {}, 122 | "outputs": [ 123 | { 124 | "data": { 125 | "application/vnd.jupyter.widget-view+json": { 126 | "model_id": "fa9a9a4a9e7f49d7b922749d9fe9a296", 127 | "version_major": 2, 128 | "version_minor": 0 129 | }, 130 | "text/plain": [ 131 | "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …" 132 | ] 133 | }, 134 | "metadata": {}, 135 | "output_type": "display_data" 136 | } 137 | ], 138 | "source": [ 139 | "plt.close('all')\n", 140 | "plt_logistic_squared_error(x_train,y_train)\n", 141 | "plt.show()" 142 | ] 143 | }, 144 | { 145 | "cell_type": "markdown", 146 | "metadata": {}, 147 | "source": [ 148 | "While this produces a pretty interesting plot, the surface above not nearly as smooth as the 'soup bowl' from linear regression! \n", 149 | "\n", 150 | "Logistic regression requires a cost function more suitable to its non-linear nature. This starts with a Loss function. This is described below." 151 | ] 152 | }, 153 | { 154 | "cell_type": "markdown", 155 | "metadata": {}, 156 | "source": [ 157 | "## Logistic Loss Function\n", 158 | "\n", 159 | "\n", 160 | " " 161 | ] 162 | }, 163 | { 164 | "cell_type": "markdown", 165 | "metadata": {}, 166 | "source": [ 167 | "Logistic Regression uses a loss function more suited to the task of categorization where the target is 0 or 1 rather than any number. \n", 168 | "\n", 169 | ">**Definition Note:** In this course, these definitions are used: \n", 170 | "**Loss** is a measure of the difference of a single example to its target value while the \n", 171 | "**Cost** is a measure of the losses over the training set\n", 172 | "\n", 173 | "\n", 174 | "This is defined: \n", 175 | "* $loss(f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}), y^{(i)})$ is the cost for a single data point, which is:\n", 176 | "\n", 177 | "\\begin{equation}\n", 178 | " loss(f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}), y^{(i)}) = \\begin{cases}\n", 179 | " - \\log\\left(f_{\\mathbf{w},b}\\left( \\mathbf{x}^{(i)} \\right) \\right) & \\text{if $y^{(i)}=1$}\\\\\n", 180 | " - \\log \\left( 1 - f_{\\mathbf{w},b}\\left( \\mathbf{x}^{(i)} \\right) \\right) & \\text{if $y^{(i)}=0$}\n", 181 | " \\end{cases}\n", 182 | "\\end{equation}\n", 183 | "\n", 184 | "\n", 185 | "* $f_{\\mathbf{w},b}(\\mathbf{x}^{(i)})$ is the model's prediction, while $y^{(i)}$ is the target value.\n", 186 | "\n", 187 | "* $f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}) = g(\\mathbf{w} \\cdot\\mathbf{x}^{(i)}+b)$ where function $g$ is the sigmoid function.\n", 188 | "\n", 189 | "The defining feature of this loss function is the fact that it uses two separate curves. One for the case when the target is zero or ($y=0$) and another for when the target is one ($y=1$). Combined, these curves provide the behavior useful for a loss function, namely, being zero when the prediction matches the target and rapidly increasing in value as the prediction differs from the target. Consider the curves below:" 190 | ] 191 | }, 192 | { 193 | "cell_type": "code", 194 | "execution_count": 5, 195 | "metadata": {}, 196 | "outputs": [ 197 | { 198 | "data": { 199 | "application/vnd.jupyter.widget-view+json": { 200 | "model_id": "c3856c00667b4424b52513d324a1a0ea", 201 | "version_major": 2, 202 | "version_minor": 0 203 | }, 204 | "text/plain": [ 205 | "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …" 206 | ] 207 | }, 208 | "metadata": {}, 209 | "output_type": "display_data" 210 | } 211 | ], 212 | "source": [ 213 | "plt_two_logistic_loss_curves()" 214 | ] 215 | }, 216 | { 217 | "cell_type": "markdown", 218 | "metadata": {}, 219 | "source": [ 220 | "Combined, the curves are similar to the quadratic curve of the squared error loss. Note, the x-axis is $f_{\\mathbf{w},b}$ which is the output of a sigmoid. The sigmoid output is strictly between 0 and 1." 221 | ] 222 | }, 223 | { 224 | "cell_type": "markdown", 225 | "metadata": {}, 226 | "source": [ 227 | "The loss function above can be rewritten to be easier to implement.\n", 228 | " $$loss(f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}), y^{(i)}) = (-y^{(i)} \\log\\left(f_{\\mathbf{w},b}\\left( \\mathbf{x}^{(i)} \\right) \\right) - \\left( 1 - y^{(i)}\\right) \\log \\left( 1 - f_{\\mathbf{w},b}\\left( \\mathbf{x}^{(i)} \\right) \\right)$$\n", 229 | " \n", 230 | "This is a rather formidable-looking equation. It is less daunting when you consider $y^{(i)}$ can have only two values, 0 and 1. One can then consider the equation in two pieces: \n", 231 | "when $ y^{(i)} = 0$, the left-hand term is eliminated:\n", 232 | "$$\n", 233 | "\\begin{align}\n", 234 | "loss(f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}), 0) &= (-(0) \\log\\left(f_{\\mathbf{w},b}\\left( \\mathbf{x}^{(i)} \\right) \\right) - \\left( 1 - 0\\right) \\log \\left( 1 - f_{\\mathbf{w},b}\\left( \\mathbf{x}^{(i)} \\right) \\right) \\\\\n", 235 | "&= -\\log \\left( 1 - f_{\\mathbf{w},b}\\left( \\mathbf{x}^{(i)} \\right) \\right)\n", 236 | "\\end{align}\n", 237 | "$$\n", 238 | "and when $ y^{(i)} = 1$, the right-hand term is eliminated:\n", 239 | "$$\n", 240 | "\\begin{align}\n", 241 | " loss(f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}), 1) &= (-(1) \\log\\left(f_{\\mathbf{w},b}\\left( \\mathbf{x}^{(i)} \\right) \\right) - \\left( 1 - 1\\right) \\log \\left( 1 - f_{\\mathbf{w},b}\\left( \\mathbf{x}^{(i)} \\right) \\right)\\\\\n", 242 | " &= -\\log\\left(f_{\\mathbf{w},b}\\left( \\mathbf{x}^{(i)} \\right) \\right)\n", 243 | "\\end{align}\n", 244 | "$$\n", 245 | "\n", 246 | "OK, with this new logistic loss function, a cost function can be produced that incorporates the loss from all the examples. This will be the topic of the next lab. For now, let's take a look at the cost vs parameters curve for the simple example we considered above:" 247 | ] 248 | }, 249 | { 250 | "cell_type": "code", 251 | "execution_count": 6, 252 | "metadata": {}, 253 | "outputs": [ 254 | { 255 | "data": { 256 | "application/vnd.jupyter.widget-view+json": { 257 | "model_id": "4eb4e64ed7e04318975920ee0ddc8981", 258 | "version_major": 2, 259 | "version_minor": 0 260 | }, 261 | "text/plain": [ 262 | "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …" 263 | ] 264 | }, 265 | "metadata": {}, 266 | "output_type": "display_data" 267 | } 268 | ], 269 | "source": [ 270 | "plt.close('all')\n", 271 | "cst = plt_logistic_cost(x_train,y_train)" 272 | ] 273 | }, 274 | { 275 | "cell_type": "markdown", 276 | "metadata": {}, 277 | "source": [ 278 | "This curve is well suited to gradient descent! It does not have plateaus, local minima, or discontinuities. Note, it is not a bowl as in the case of squared error. Both the cost and the log of the cost are plotted to illuminate the fact that the curve, when the cost is small, has a slope and continues to decline. Reminder: you can rotate the above plots using your mouse." 279 | ] 280 | }, 281 | { 282 | "cell_type": "markdown", 283 | "metadata": {}, 284 | "source": [ 285 | "## Congratulation! \n", 286 | "You have:\n", 287 | " - determined a squared error loss function is not suitable for classification tasks\n", 288 | " - developed and examined the logistic loss function which **is** suitable for classification tasks.\n", 289 | "\n" 290 | ] 291 | }, 292 | { 293 | "cell_type": "code", 294 | "execution_count": null, 295 | "metadata": {}, 296 | "outputs": [], 297 | "source": [] 298 | } 299 | ], 300 | "metadata": { 301 | "kernelspec": { 302 | "display_name": "Python 3", 303 | "language": "python", 304 | "name": "python3" 305 | }, 306 | "language_info": { 307 | "codemirror_mode": { 308 | "name": "ipython", 309 | "version": 3 310 | }, 311 | "file_extension": ".py", 312 | "mimetype": "text/x-python", 313 | "name": "python", 314 | "nbconvert_exporter": "python", 315 | "pygments_lexer": "ipython3", 316 | "version": "3.7.6" 317 | }, 318 | "toc-showcode": true 319 | }, 320 | "nbformat": 4, 321 | "nbformat_minor": 5 322 | } 323 | -------------------------------------------------------------------------------- /C1_W3_Lab06_Gradient_Descent_Soln.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Optional Lab: Gradient Descent for Logistic Regression" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## Goals\n", 15 | "In this lab, you will:\n", 16 | "- update gradient descent for logistic regression.\n", 17 | "- explore gradient descent on a familiar data set" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 1, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "import copy, math\n", 27 | "import numpy as np\n", 28 | "%matplotlib widget\n", 29 | "import matplotlib.pyplot as plt\n", 30 | "from lab_utils_common import dlc, plot_data, plt_tumor_data, sigmoid, compute_cost_logistic\n", 31 | "from plt_quad_logistic import plt_quad_logistic, plt_prob\n", 32 | "plt.style.use('./deeplearning.mplstyle')" 33 | ] 34 | }, 35 | { 36 | "cell_type": "markdown", 37 | "metadata": {}, 38 | "source": [ 39 | "## Data set \n", 40 | "Let's start with the same two feature data set used in the decision boundary lab." 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": 2, 46 | "metadata": {}, 47 | "outputs": [], 48 | "source": [ 49 | "X_train = np.array([[0.5, 1.5], [1,1], [1.5, 0.5], [3, 0.5], [2, 2], [1, 2.5]])\n", 50 | "y_train = np.array([0, 0, 0, 1, 1, 1])" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": {}, 56 | "source": [ 57 | "As before, we'll use a helper function to plot this data. The data points with label $y=1$ are shown as red crosses, while the data points with label $y=0$ are shown as blue circles." 58 | ] 59 | }, 60 | { 61 | "cell_type": "code", 62 | "execution_count": 3, 63 | "metadata": {}, 64 | "outputs": [ 65 | { 66 | "data": { 67 | "application/vnd.jupyter.widget-view+json": { 68 | "model_id": "a946866652fd48d5b2914550250ed084", 69 | "version_major": 2, 70 | "version_minor": 0 71 | }, 72 | "text/plain": [ 73 | "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …" 74 | ] 75 | }, 76 | "metadata": {}, 77 | "output_type": "display_data" 78 | } 79 | ], 80 | "source": [ 81 | "fig,ax = plt.subplots(1,1,figsize=(4,4))\n", 82 | "plot_data(X_train, y_train, ax)\n", 83 | "\n", 84 | "ax.axis([0, 4, 0, 3.5])\n", 85 | "ax.set_ylabel('$x_1$', fontsize=12)\n", 86 | "ax.set_xlabel('$x_0$', fontsize=12)\n", 87 | "plt.show()" 88 | ] 89 | }, 90 | { 91 | "cell_type": "markdown", 92 | "metadata": {}, 93 | "source": [ 94 | "## Logistic Gradient Descent\n", 95 | "\n", 96 | "\n", 97 | "Recall the gradient descent algorithm utilizes the gradient calculation:\n", 98 | "$$\\begin{align*}\n", 99 | "&\\text{repeat until convergence:} \\; \\lbrace \\\\\n", 100 | "& \\; \\; \\;w_j = w_j - \\alpha \\frac{\\partial J(\\mathbf{w},b)}{\\partial w_j} \\tag{1} \\; & \\text{for j := 0..n-1} \\\\ \n", 101 | "& \\; \\; \\; \\; \\;b = b - \\alpha \\frac{\\partial J(\\mathbf{w},b)}{\\partial b} \\\\\n", 102 | "&\\rbrace\n", 103 | "\\end{align*}$$\n", 104 | "\n", 105 | "Where each iteration performs simultaneous updates on $w_j$ for all $j$, where\n", 106 | "$$\\begin{align*}\n", 107 | "\\frac{\\partial J(\\mathbf{w},b)}{\\partial w_j} &= \\frac{1}{m} \\sum\\limits_{i = 0}^{m-1} (f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}) - y^{(i)})x_{j}^{(i)} \\tag{2} \\\\\n", 108 | "\\frac{\\partial J(\\mathbf{w},b)}{\\partial b} &= \\frac{1}{m} \\sum\\limits_{i = 0}^{m-1} (f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}) - y^{(i)}) \\tag{3} \n", 109 | "\\end{align*}$$\n", 110 | "\n", 111 | "* m is the number of training examples in the data set \n", 112 | "* $f_{\\mathbf{w},b}(x^{(i)})$ is the model's prediction, while $y^{(i)}$ is the target\n", 113 | "* For a logistic regression model \n", 114 | " $z = \\mathbf{w} \\cdot \\mathbf{x} + b$ \n", 115 | " $f_{\\mathbf{w},b}(x) = g(z)$ \n", 116 | " where $g(z)$ is the sigmoid function: \n", 117 | " $g(z) = \\frac{1}{1+e^{-z}}$ \n", 118 | " \n" 119 | ] 120 | }, 121 | { 122 | "cell_type": "markdown", 123 | "metadata": {}, 124 | "source": [ 125 | "### Gradient Descent Implementation\n", 126 | "The gradient descent algorithm implementation has two components: \n", 127 | "- The loop implementing equation (1) above. This is `gradient_descent` below and is generally provided to you in optional and practice labs.\n", 128 | "- The calculation of the current gradient, equations (2,3) above. This is `compute_gradient_logistic` below. You will be asked to implement this week's practice lab.\n", 129 | "\n", 130 | "#### Calculating the Gradient, Code Description\n", 131 | "Implements equation (2),(3) above for all $w_j$ and $b$.\n", 132 | "There are many ways to implement this. Outlined below is this:\n", 133 | "- initialize variables to accumulate `dj_dw` and `dj_db`\n", 134 | "- for each example\n", 135 | " - calculate the error for that example $g(\\mathbf{w} \\cdot \\mathbf{x}^{(i)} + b) - \\mathbf{y}^{(i)}$\n", 136 | " - for each input value $x_{j}^{(i)}$ in this example, \n", 137 | " - multiply the error by the input $x_{j}^{(i)}$, and add to the corresponding element of `dj_dw`. (equation 2 above)\n", 138 | " - add the error to `dj_db` (equation 3 above)\n", 139 | "\n", 140 | "- divide `dj_db` and `dj_dw` by total number of examples (m)\n", 141 | "- note that $\\mathbf{x}^{(i)}$ in numpy `X[i,:]` or `X[i]` and $x_{j}^{(i)}$ is `X[i,j]`" 142 | ] 143 | }, 144 | { 145 | "cell_type": "code", 146 | "execution_count": 4, 147 | "metadata": {}, 148 | "outputs": [], 149 | "source": [ 150 | "def compute_gradient_logistic(X, y, w, b): \n", 151 | " \"\"\"\n", 152 | " Computes the gradient for linear regression \n", 153 | " \n", 154 | " Args:\n", 155 | " X (ndarray (m,n): Data, m examples with n features\n", 156 | " y (ndarray (m,)): target values\n", 157 | " w (ndarray (n,)): model parameters \n", 158 | " b (scalar) : model parameter\n", 159 | " Returns\n", 160 | " dj_dw (ndarray (n,)): The gradient of the cost w.r.t. the parameters w. \n", 161 | " dj_db (scalar) : The gradient of the cost w.r.t. the parameter b. \n", 162 | " \"\"\"\n", 163 | " m,n = X.shape\n", 164 | " dj_dw = np.zeros((n,)) #(n,)\n", 165 | " dj_db = 0.\n", 166 | "\n", 167 | " for i in range(m):\n", 168 | " f_wb_i = sigmoid(np.dot(X[i],w) + b) #(n,)(n,)=scalar\n", 169 | " err_i = f_wb_i - y[i] #scalar\n", 170 | " for j in range(n):\n", 171 | " dj_dw[j] = dj_dw[j] + err_i * X[i,j] #scalar\n", 172 | " dj_db = dj_db + err_i\n", 173 | " dj_dw = dj_dw/m #(n,)\n", 174 | " dj_db = dj_db/m #scalar\n", 175 | " \n", 176 | " return dj_db, dj_dw " 177 | ] 178 | }, 179 | { 180 | "cell_type": "markdown", 181 | "metadata": {}, 182 | "source": [ 183 | "Check the implementation of the gradient function using the cell below." 184 | ] 185 | }, 186 | { 187 | "cell_type": "code", 188 | "execution_count": 5, 189 | "metadata": {}, 190 | "outputs": [ 191 | { 192 | "name": "stdout", 193 | "output_type": "stream", 194 | "text": [ 195 | "dj_db: 0.49861806546328574\n", 196 | "dj_dw: [0.498333393278696, 0.49883942983996693]\n" 197 | ] 198 | } 199 | ], 200 | "source": [ 201 | "X_tmp = np.array([[0.5, 1.5], [1,1], [1.5, 0.5], [3, 0.5], [2, 2], [1, 2.5]])\n", 202 | "y_tmp = np.array([0, 0, 0, 1, 1, 1])\n", 203 | "w_tmp = np.array([2.,3.])\n", 204 | "b_tmp = 1.\n", 205 | "dj_db_tmp, dj_dw_tmp = compute_gradient_logistic(X_tmp, y_tmp, w_tmp, b_tmp)\n", 206 | "print(f\"dj_db: {dj_db_tmp}\" )\n", 207 | "print(f\"dj_dw: {dj_dw_tmp.tolist()}\" )" 208 | ] 209 | }, 210 | { 211 | "cell_type": "markdown", 212 | "metadata": {}, 213 | "source": [ 214 | "**Expected output**\n", 215 | "``` \n", 216 | "dj_db: 0.49861806546328574\n", 217 | "dj_dw: [0.498333393278696, 0.49883942983996693]\n", 218 | "```" 219 | ] 220 | }, 221 | { 222 | "cell_type": "markdown", 223 | "metadata": {}, 224 | "source": [ 225 | "#### Gradient Descent Code \n", 226 | "The code implementing equation (1) above is implemented below. Take a moment to locate and compare the functions in the routine to the equations above." 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": 6, 232 | "metadata": {}, 233 | "outputs": [], 234 | "source": [ 235 | "def gradient_descent(X, y, w_in, b_in, alpha, num_iters): \n", 236 | " \"\"\"\n", 237 | " Performs batch gradient descent\n", 238 | " \n", 239 | " Args:\n", 240 | " X (ndarray (m,n) : Data, m examples with n features\n", 241 | " y (ndarray (m,)) : target values\n", 242 | " w_in (ndarray (n,)): Initial values of model parameters \n", 243 | " b_in (scalar) : Initial values of model parameter\n", 244 | " alpha (float) : Learning rate\n", 245 | " num_iters (scalar) : number of iterations to run gradient descent\n", 246 | " \n", 247 | " Returns:\n", 248 | " w (ndarray (n,)) : Updated values of parameters\n", 249 | " b (scalar) : Updated value of parameter \n", 250 | " \"\"\"\n", 251 | " # An array to store cost J and w's at each iteration primarily for graphing later\n", 252 | " J_history = []\n", 253 | " w = copy.deepcopy(w_in) #avoid modifying global w within function\n", 254 | " b = b_in\n", 255 | " \n", 256 | " for i in range(num_iters):\n", 257 | " # Calculate the gradient and update the parameters\n", 258 | " dj_db, dj_dw = compute_gradient_logistic(X, y, w, b) \n", 259 | "\n", 260 | " # Update Parameters using w, b, alpha and gradient\n", 261 | " w = w - alpha * dj_dw \n", 262 | " b = b - alpha * dj_db \n", 263 | " \n", 264 | " # Save cost J at each iteration\n", 265 | " if i<100000: # prevent resource exhaustion \n", 266 | " J_history.append( compute_cost_logistic(X, y, w, b) )\n", 267 | "\n", 268 | " # Print cost every at intervals 10 times or as many iterations if < 10\n", 269 | " if i% math.ceil(num_iters / 10) == 0:\n", 270 | " print(f\"Iteration {i:4d}: Cost {J_history[-1]} \")\n", 271 | " \n", 272 | " return w, b, J_history #return final w,b and J history for graphing\n" 273 | ] 274 | }, 275 | { 276 | "cell_type": "markdown", 277 | "metadata": {}, 278 | "source": [ 279 | "Let's run gradient descent on our data set." 280 | ] 281 | }, 282 | { 283 | "cell_type": "code", 284 | "execution_count": 7, 285 | "metadata": {}, 286 | "outputs": [ 287 | { 288 | "name": "stdout", 289 | "output_type": "stream", 290 | "text": [ 291 | "Iteration 0: Cost 0.684610468560574 \n", 292 | "Iteration 1000: Cost 0.1590977666870456 \n", 293 | "Iteration 2000: Cost 0.08460064176930081 \n", 294 | "Iteration 3000: Cost 0.05705327279402531 \n", 295 | "Iteration 4000: Cost 0.042907594216820076 \n", 296 | "Iteration 5000: Cost 0.034338477298845684 \n", 297 | "Iteration 6000: Cost 0.028603798022120097 \n", 298 | "Iteration 7000: Cost 0.024501569608793 \n", 299 | "Iteration 8000: Cost 0.02142370332569295 \n", 300 | "Iteration 9000: Cost 0.019030137124109114 \n", 301 | "\n", 302 | "updated parameters: w:[5.28 5.08], b:-14.222409982019837\n" 303 | ] 304 | } 305 | ], 306 | "source": [ 307 | "w_tmp = np.zeros_like(X_train[0])\n", 308 | "b_tmp = 0.\n", 309 | "alph = 0.1\n", 310 | "iters = 10000\n", 311 | "\n", 312 | "w_out, b_out, _ = gradient_descent(X_train, y_train, w_tmp, b_tmp, alph, iters) \n", 313 | "print(f\"\\nupdated parameters: w:{w_out}, b:{b_out}\")" 314 | ] 315 | }, 316 | { 317 | "cell_type": "markdown", 318 | "metadata": {}, 319 | "source": [ 320 | "#### Let's plot the results of gradient descent:" 321 | ] 322 | }, 323 | { 324 | "cell_type": "code", 325 | "execution_count": 8, 326 | "metadata": {}, 327 | "outputs": [ 328 | { 329 | "data": { 330 | "application/vnd.jupyter.widget-view+json": { 331 | "model_id": "e0e86533af8e438f9a171f5a74c47ea2", 332 | "version_major": 2, 333 | "version_minor": 0 334 | }, 335 | "text/plain": [ 336 | "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …" 337 | ] 338 | }, 339 | "metadata": {}, 340 | "output_type": "display_data" 341 | } 342 | ], 343 | "source": [ 344 | "fig,ax = plt.subplots(1,1,figsize=(5,4))\n", 345 | "# plot the probability \n", 346 | "plt_prob(ax, w_out, b_out)\n", 347 | "\n", 348 | "# Plot the original data\n", 349 | "ax.set_ylabel(r'$x_1$')\n", 350 | "ax.set_xlabel(r'$x_0$') \n", 351 | "ax.axis([0, 4, 0, 3.5])\n", 352 | "plot_data(X_train,y_train,ax)\n", 353 | "\n", 354 | "# Plot the decision boundary\n", 355 | "x0 = -b_out/w_out[0]\n", 356 | "x1 = -b_out/w_out[1]\n", 357 | "ax.plot([0,x0],[x1,0], c=dlc[\"dlblue\"], lw=1)\n", 358 | "plt.show()" 359 | ] 360 | }, 361 | { 362 | "cell_type": "markdown", 363 | "metadata": {}, 364 | "source": [ 365 | "In the plot above:\n", 366 | " - the shading reflects the probability y=1 (result prior to decision boundary)\n", 367 | " - the decision boundary is the line at which the probability = 0.5\n", 368 | " " 369 | ] 370 | }, 371 | { 372 | "cell_type": "markdown", 373 | "metadata": {}, 374 | "source": [ 375 | "## Another Data set\n", 376 | "Let's return to a one-variable data set. With just two parameters, $w$, $b$, it is possible to plot the cost function using a contour plot to get a better idea of what gradient descent is up to." 377 | ] 378 | }, 379 | { 380 | "cell_type": "code", 381 | "execution_count": 9, 382 | "metadata": {}, 383 | "outputs": [], 384 | "source": [ 385 | "x_train = np.array([0., 1, 2, 3, 4, 5])\n", 386 | "y_train = np.array([0, 0, 0, 1, 1, 1])" 387 | ] 388 | }, 389 | { 390 | "cell_type": "markdown", 391 | "metadata": {}, 392 | "source": [ 393 | "As before, we'll use a helper function to plot this data. The data points with label $y=1$ are shown as red crosses, while the data points with label $y=0$ are shown as blue circles." 394 | ] 395 | }, 396 | { 397 | "cell_type": "code", 398 | "execution_count": 10, 399 | "metadata": {}, 400 | "outputs": [ 401 | { 402 | "data": { 403 | "application/vnd.jupyter.widget-view+json": { 404 | "model_id": "0cabe350bee94a9fb883514cbebfe6da", 405 | "version_major": 2, 406 | "version_minor": 0 407 | }, 408 | "text/plain": [ 409 | "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …" 410 | ] 411 | }, 412 | "metadata": {}, 413 | "output_type": "display_data" 414 | } 415 | ], 416 | "source": [ 417 | "fig,ax = plt.subplots(1,1,figsize=(4,3))\n", 418 | "plt_tumor_data(x_train, y_train, ax)\n", 419 | "plt.show()" 420 | ] 421 | }, 422 | { 423 | "cell_type": "markdown", 424 | "metadata": {}, 425 | "source": [ 426 | "In the plot below, try:\n", 427 | "- changing $w$ and $b$ by clicking within the contour plot on the upper right.\n", 428 | " - changes may take a second or two\n", 429 | " - note the changing value of cost on the upper left plot.\n", 430 | " - note the cost is accumulated by a loss on each example (vertical dotted lines)\n", 431 | "- run gradient descent by clicking the orange button.\n", 432 | " - note the steadily decreasing cost (contour and cost plot are in log(cost) \n", 433 | " - clicking in the contour plot will reset the model for a new run\n", 434 | "- to reset the plot, rerun the cell" 435 | ] 436 | }, 437 | { 438 | "cell_type": "code", 439 | "execution_count": 11, 440 | "metadata": {}, 441 | "outputs": [ 442 | { 443 | "data": { 444 | "application/vnd.jupyter.widget-view+json": { 445 | "model_id": "476db5c70921467987c323c1cde887b1", 446 | "version_major": 2, 447 | "version_minor": 0 448 | }, 449 | "text/plain": [ 450 | "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …" 451 | ] 452 | }, 453 | "metadata": {}, 454 | "output_type": "display_data" 455 | } 456 | ], 457 | "source": [ 458 | "w_range = np.array([-1, 7])\n", 459 | "b_range = np.array([1, -14])\n", 460 | "quad = plt_quad_logistic( x_train, y_train, w_range, b_range )" 461 | ] 462 | }, 463 | { 464 | "cell_type": "markdown", 465 | "metadata": {}, 466 | "source": [ 467 | "## Congratulations!\n", 468 | "You have:\n", 469 | "- examined the formulas and implementation of calculating the gradient for logistic regression\n", 470 | "- utilized those routines in\n", 471 | " - exploring a single variable data set\n", 472 | " - exploring a two-variable data set" 473 | ] 474 | }, 475 | { 476 | "cell_type": "code", 477 | "execution_count": null, 478 | "metadata": {}, 479 | "outputs": [], 480 | "source": [] 481 | } 482 | ], 483 | "metadata": { 484 | "kernelspec": { 485 | "display_name": "Python 3", 486 | "language": "python", 487 | "name": "python3" 488 | }, 489 | "language_info": { 490 | "codemirror_mode": { 491 | "name": "ipython", 492 | "version": 3 493 | }, 494 | "file_extension": ".py", 495 | "mimetype": "text/x-python", 496 | "name": "python", 497 | "nbconvert_exporter": "python", 498 | "pygments_lexer": "ipython3", 499 | "version": "3.7.6" 500 | } 501 | }, 502 | "nbformat": 4, 503 | "nbformat_minor": 5 504 | } 505 | -------------------------------------------------------------------------------- /C1_W3_Lab09_Regularization_Soln.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Optional Lab - Regularized Cost and Gradient" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## Goals\n", 15 | "In this lab, you will:\n", 16 | "- extend the previous linear and logistic cost functions with a regularization term.\n", 17 | "- rerun the previous example of over-fitting with a regularization term added.\n" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 1, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "import numpy as np\n", 27 | "%matplotlib widget\n", 28 | "import matplotlib.pyplot as plt\n", 29 | "from plt_overfit import overfit_example, output\n", 30 | "from lab_utils_common import sigmoid\n", 31 | "np.set_printoptions(precision=8)" 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "metadata": {}, 37 | "source": [ 38 | "# Adding regularization\n", 39 | "\n", 40 | "\n", 41 | "\n", 42 | "The slides above show the cost and gradient functions for both linear and logistic regression. Note:\n", 43 | "- Cost\n", 44 | " - The cost functions differ significantly between linear and logistic regression, but adding regularization to the equations is the same.\n", 45 | "- Gradient\n", 46 | " - The gradient functions for linear and logistic regression are very similar. They differ only in the implementation of $f_{wb}$." 47 | ] 48 | }, 49 | { 50 | "cell_type": "markdown", 51 | "metadata": {}, 52 | "source": [ 53 | "## Cost functions with regularization\n", 54 | "### Cost function for regularized linear regression\n", 55 | "\n", 56 | "The equation for the cost function regularized linear regression is:\n", 57 | "$$J(\\mathbf{w},b) = \\frac{1}{2m} \\sum\\limits_{i = 0}^{m-1} (f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}) - y^{(i)})^2 + \\frac{\\lambda}{2m} \\sum_{j=0}^{n-1} w_j^2 \\tag{1}$$ \n", 58 | "where:\n", 59 | "$$ f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}) = \\mathbf{w} \\cdot \\mathbf{x}^{(i)} + b \\tag{2} $$ \n", 60 | "\n", 61 | "\n", 62 | "Compare this to the cost function without regularization (which you implemented in a previous lab), which is of the form:\n", 63 | "\n", 64 | "$$J(\\mathbf{w},b) = \\frac{1}{2m} \\sum\\limits_{i = 0}^{m-1} (f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}) - y^{(i)})^2 $$ \n", 65 | "\n", 66 | "The difference is the regularization term, \n", 67 | " $\\frac{\\lambda}{2m} \\sum_{j=0}^{n-1} w_j^2$ \n", 68 | " \n", 69 | "Including this term encourages gradient descent to minimize the size of the parameters. Note, in this example, the parameter $b$ is not regularized. This is standard practice.\n", 70 | "\n", 71 | "Below is an implementation of equations (1) and (2). Note that this uses a *standard pattern for this course*, a `for loop` over all `m` examples." 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": 2, 77 | "metadata": {}, 78 | "outputs": [], 79 | "source": [ 80 | "def compute_cost_linear_reg(X, y, w, b, lambda_ = 1):\n", 81 | " \"\"\"\n", 82 | " Computes the cost over all examples\n", 83 | " Args:\n", 84 | " X (ndarray (m,n): Data, m examples with n features\n", 85 | " y (ndarray (m,)): target values\n", 86 | " w (ndarray (n,)): model parameters \n", 87 | " b (scalar) : model parameter\n", 88 | " lambda_ (scalar): Controls amount of regularization\n", 89 | " Returns:\n", 90 | " total_cost (scalar): cost \n", 91 | " \"\"\"\n", 92 | "\n", 93 | " m = X.shape[0]\n", 94 | " n = len(w)\n", 95 | " cost = 0.\n", 96 | " for i in range(m):\n", 97 | " f_wb_i = np.dot(X[i], w) + b #(n,)(n,)=scalar, see np.dot\n", 98 | " cost = cost + (f_wb_i - y[i])**2 #scalar \n", 99 | " cost = cost / (2 * m) #scalar \n", 100 | " \n", 101 | " reg_cost = 0\n", 102 | " for j in range(n):\n", 103 | " reg_cost += (w[j]**2) #scalar\n", 104 | " reg_cost = (lambda_/(2*m)) * reg_cost #scalar\n", 105 | " \n", 106 | " total_cost = cost + reg_cost #scalar\n", 107 | " return total_cost #scalar" 108 | ] 109 | }, 110 | { 111 | "cell_type": "markdown", 112 | "metadata": {}, 113 | "source": [ 114 | "Run the cell below to see it in action." 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": 3, 120 | "metadata": { 121 | "tags": [] 122 | }, 123 | "outputs": [ 124 | { 125 | "name": "stdout", 126 | "output_type": "stream", 127 | "text": [ 128 | "Regularized cost: 0.07917239320214275\n" 129 | ] 130 | } 131 | ], 132 | "source": [ 133 | "np.random.seed(1)\n", 134 | "X_tmp = np.random.rand(5,6)\n", 135 | "y_tmp = np.array([0,1,0,1,0])\n", 136 | "w_tmp = np.random.rand(X_tmp.shape[1]).reshape(-1,)-0.5\n", 137 | "b_tmp = 0.5\n", 138 | "lambda_tmp = 0.7\n", 139 | "cost_tmp = compute_cost_linear_reg(X_tmp, y_tmp, w_tmp, b_tmp, lambda_tmp)\n", 140 | "\n", 141 | "print(\"Regularized cost:\", cost_tmp)" 142 | ] 143 | }, 144 | { 145 | "cell_type": "markdown", 146 | "metadata": {}, 147 | "source": [ 148 | "**Expected Output**:\n", 149 | "\n", 150 | " \n", 151 | " \n", 152 | " \n", 153 | "
Regularized cost: 0.07917239320214275
" 154 | ] 155 | }, 156 | { 157 | "cell_type": "markdown", 158 | "metadata": {}, 159 | "source": [ 160 | "### Cost function for regularized logistic regression\n", 161 | "For regularized **logistic** regression, the cost function is of the form\n", 162 | "$$J(\\mathbf{w},b) = \\frac{1}{m} \\sum_{i=0}^{m-1} \\left[ -y^{(i)} \\log\\left(f_{\\mathbf{w},b}\\left( \\mathbf{x}^{(i)} \\right) \\right) - \\left( 1 - y^{(i)}\\right) \\log \\left( 1 - f_{\\mathbf{w},b}\\left( \\mathbf{x}^{(i)} \\right) \\right) \\right] + \\frac{\\lambda}{2m} \\sum_{j=0}^{n-1} w_j^2 \\tag{3}$$\n", 163 | "where:\n", 164 | "$$ f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}) = sigmoid(\\mathbf{w} \\cdot \\mathbf{x}^{(i)} + b) \\tag{4} $$ \n", 165 | "\n", 166 | "Compare this to the cost function without regularization (which you implemented in a previous lab):\n", 167 | "\n", 168 | "$$ J(\\mathbf{w},b) = \\frac{1}{m}\\sum_{i=0}^{m-1} \\left[ (-y^{(i)} \\log\\left(f_{\\mathbf{w},b}\\left( \\mathbf{x}^{(i)} \\right) \\right) - \\left( 1 - y^{(i)}\\right) \\log \\left( 1 - f_{\\mathbf{w},b}\\left( \\mathbf{x}^{(i)} \\right) \\right)\\right] $$\n", 169 | "\n", 170 | "As was the case in linear regression above, the difference is the regularization term, which is \n", 171 | " $\\frac{\\lambda}{2m} \\sum_{j=0}^{n-1} w_j^2$ \n", 172 | "\n", 173 | "Including this term encourages gradient descent to minimize the size of the parameters. Note, in this example, the parameter $b$ is not regularized. This is standard practice. " 174 | ] 175 | }, 176 | { 177 | "cell_type": "code", 178 | "execution_count": 4, 179 | "metadata": {}, 180 | "outputs": [], 181 | "source": [ 182 | "def compute_cost_logistic_reg(X, y, w, b, lambda_ = 1):\n", 183 | " \"\"\"\n", 184 | " Computes the cost over all examples\n", 185 | " Args:\n", 186 | " Args:\n", 187 | " X (ndarray (m,n): Data, m examples with n features\n", 188 | " y (ndarray (m,)): target values\n", 189 | " w (ndarray (n,)): model parameters \n", 190 | " b (scalar) : model parameter\n", 191 | " lambda_ (scalar): Controls amount of regularization\n", 192 | " Returns:\n", 193 | " total_cost (scalar): cost \n", 194 | " \"\"\"\n", 195 | "\n", 196 | " m,n = X.shape\n", 197 | " cost = 0.\n", 198 | " for i in range(m):\n", 199 | " z_i = np.dot(X[i], w) + b #(n,)(n,)=scalar, see np.dot\n", 200 | " f_wb_i = sigmoid(z_i) #scalar\n", 201 | " cost += -y[i]*np.log(f_wb_i) - (1-y[i])*np.log(1-f_wb_i) #scalar\n", 202 | " \n", 203 | " cost = cost/m #scalar\n", 204 | "\n", 205 | " reg_cost = 0\n", 206 | " for j in range(n):\n", 207 | " reg_cost += (w[j]**2) #scalar\n", 208 | " reg_cost = (lambda_/(2*m)) * reg_cost #scalar\n", 209 | " \n", 210 | " total_cost = cost + reg_cost #scalar\n", 211 | " return total_cost #scalar" 212 | ] 213 | }, 214 | { 215 | "cell_type": "markdown", 216 | "metadata": {}, 217 | "source": [ 218 | "Run the cell below to see it in action." 219 | ] 220 | }, 221 | { 222 | "cell_type": "code", 223 | "execution_count": 5, 224 | "metadata": { 225 | "tags": [] 226 | }, 227 | "outputs": [ 228 | { 229 | "name": "stdout", 230 | "output_type": "stream", 231 | "text": [ 232 | "Regularized cost: 0.6850849138741673\n" 233 | ] 234 | } 235 | ], 236 | "source": [ 237 | "np.random.seed(1)\n", 238 | "X_tmp = np.random.rand(5,6)\n", 239 | "y_tmp = np.array([0,1,0,1,0])\n", 240 | "w_tmp = np.random.rand(X_tmp.shape[1]).reshape(-1,)-0.5\n", 241 | "b_tmp = 0.5\n", 242 | "lambda_tmp = 0.7\n", 243 | "cost_tmp = compute_cost_logistic_reg(X_tmp, y_tmp, w_tmp, b_tmp, lambda_tmp)\n", 244 | "\n", 245 | "print(\"Regularized cost:\", cost_tmp)" 246 | ] 247 | }, 248 | { 249 | "cell_type": "markdown", 250 | "metadata": {}, 251 | "source": [ 252 | "**Expected Output**:\n", 253 | "\n", 254 | " \n", 255 | " \n", 256 | " \n", 257 | "
Regularized cost: 0.6850849138741673
" 258 | ] 259 | }, 260 | { 261 | "cell_type": "markdown", 262 | "metadata": {}, 263 | "source": [ 264 | "## Gradient descent with regularization\n", 265 | "The basic algorithm for running gradient descent does not change with regularization, it is:\n", 266 | "$$\\begin{align*}\n", 267 | "&\\text{repeat until convergence:} \\; \\lbrace \\\\\n", 268 | "& \\; \\; \\;w_j = w_j - \\alpha \\frac{\\partial J(\\mathbf{w},b)}{\\partial w_j} \\tag{1} \\; & \\text{for j := 0..n-1} \\\\ \n", 269 | "& \\; \\; \\; \\; \\;b = b - \\alpha \\frac{\\partial J(\\mathbf{w},b)}{\\partial b} \\\\\n", 270 | "&\\rbrace\n", 271 | "\\end{align*}$$\n", 272 | "Where each iteration performs simultaneous updates on $w_j$ for all $j$.\n", 273 | "\n", 274 | "What changes with regularization is computing the gradients." 275 | ] 276 | }, 277 | { 278 | "cell_type": "markdown", 279 | "metadata": {}, 280 | "source": [ 281 | "### Computing the Gradient with regularization (both linear/logistic)\n", 282 | "The gradient calculation for both linear and logistic regression are nearly identical, differing only in computation of $f_{\\mathbf{w}b}$.\n", 283 | "$$\\begin{align*}\n", 284 | "\\frac{\\partial J(\\mathbf{w},b)}{\\partial w_j} &= \\frac{1}{m} \\sum\\limits_{i = 0}^{m-1} (f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}) - y^{(i)})x_{j}^{(i)} + \\frac{\\lambda}{m} w_j \\tag{2} \\\\\n", 285 | "\\frac{\\partial J(\\mathbf{w},b)}{\\partial b} &= \\frac{1}{m} \\sum\\limits_{i = 0}^{m-1} (f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}) - y^{(i)}) \\tag{3} \n", 286 | "\\end{align*}$$\n", 287 | "\n", 288 | "* m is the number of training examples in the data set \n", 289 | "* $f_{\\mathbf{w},b}(x^{(i)})$ is the model's prediction, while $y^{(i)}$ is the target\n", 290 | "\n", 291 | " \n", 292 | "* For a **linear** regression model \n", 293 | " $f_{\\mathbf{w},b}(x) = \\mathbf{w} \\cdot \\mathbf{x} + b$ \n", 294 | "* For a **logistic** regression model \n", 295 | " $z = \\mathbf{w} \\cdot \\mathbf{x} + b$ \n", 296 | " $f_{\\mathbf{w},b}(x) = g(z)$ \n", 297 | " where $g(z)$ is the sigmoid function: \n", 298 | " $g(z) = \\frac{1}{1+e^{-z}}$ \n", 299 | " \n", 300 | "The term which adds regularization is the $\\frac{\\lambda}{m} w_j $." 301 | ] 302 | }, 303 | { 304 | "cell_type": "markdown", 305 | "metadata": {}, 306 | "source": [ 307 | "### Gradient function for regularized linear regression" 308 | ] 309 | }, 310 | { 311 | "cell_type": "code", 312 | "execution_count": 6, 313 | "metadata": {}, 314 | "outputs": [], 315 | "source": [ 316 | "def compute_gradient_linear_reg(X, y, w, b, lambda_): \n", 317 | " \"\"\"\n", 318 | " Computes the gradient for linear regression \n", 319 | " Args:\n", 320 | " X (ndarray (m,n): Data, m examples with n features\n", 321 | " y (ndarray (m,)): target values\n", 322 | " w (ndarray (n,)): model parameters \n", 323 | " b (scalar) : model parameter\n", 324 | " lambda_ (scalar): Controls amount of regularization\n", 325 | " \n", 326 | " Returns:\n", 327 | " dj_dw (ndarray (n,)): The gradient of the cost w.r.t. the parameters w. \n", 328 | " dj_db (scalar): The gradient of the cost w.r.t. the parameter b. \n", 329 | " \"\"\"\n", 330 | " m,n = X.shape #(number of examples, number of features)\n", 331 | " dj_dw = np.zeros((n,))\n", 332 | " dj_db = 0.\n", 333 | "\n", 334 | " for i in range(m): \n", 335 | " err = (np.dot(X[i], w) + b) - y[i] \n", 336 | " for j in range(n): \n", 337 | " dj_dw[j] = dj_dw[j] + err * X[i, j] \n", 338 | " dj_db = dj_db + err \n", 339 | " dj_dw = dj_dw / m \n", 340 | " dj_db = dj_db / m \n", 341 | " \n", 342 | " for j in range(n):\n", 343 | " dj_dw[j] = dj_dw[j] + (lambda_/m) * w[j]\n", 344 | "\n", 345 | " return dj_db, dj_dw" 346 | ] 347 | }, 348 | { 349 | "cell_type": "markdown", 350 | "metadata": {}, 351 | "source": [ 352 | "Run the cell below to see it in action." 353 | ] 354 | }, 355 | { 356 | "cell_type": "code", 357 | "execution_count": 7, 358 | "metadata": {}, 359 | "outputs": [ 360 | { 361 | "name": "stdout", 362 | "output_type": "stream", 363 | "text": [ 364 | "dj_db: 0.6648774569425726\n", 365 | "Regularized dj_dw:\n", 366 | " [0.29653214748822276, 0.4911679625918033, 0.21645877535865857]\n" 367 | ] 368 | } 369 | ], 370 | "source": [ 371 | "np.random.seed(1)\n", 372 | "X_tmp = np.random.rand(5,3)\n", 373 | "y_tmp = np.array([0,1,0,1,0])\n", 374 | "w_tmp = np.random.rand(X_tmp.shape[1])\n", 375 | "b_tmp = 0.5\n", 376 | "lambda_tmp = 0.7\n", 377 | "dj_db_tmp, dj_dw_tmp = compute_gradient_linear_reg(X_tmp, y_tmp, w_tmp, b_tmp, lambda_tmp)\n", 378 | "\n", 379 | "print(f\"dj_db: {dj_db_tmp}\", )\n", 380 | "print(f\"Regularized dj_dw:\\n {dj_dw_tmp.tolist()}\", )" 381 | ] 382 | }, 383 | { 384 | "cell_type": "markdown", 385 | "metadata": {}, 386 | "source": [ 387 | "**Expected Output**\n", 388 | "```\n", 389 | "dj_db: 0.6648774569425726\n", 390 | "Regularized dj_dw:\n", 391 | " [0.29653214748822276, 0.4911679625918033, 0.21645877535865857]\n", 392 | " ```" 393 | ] 394 | }, 395 | { 396 | "cell_type": "markdown", 397 | "metadata": {}, 398 | "source": [ 399 | "### Gradient function for regularized logistic regression" 400 | ] 401 | }, 402 | { 403 | "cell_type": "code", 404 | "execution_count": 8, 405 | "metadata": {}, 406 | "outputs": [], 407 | "source": [ 408 | "def compute_gradient_logistic_reg(X, y, w, b, lambda_): \n", 409 | " \"\"\"\n", 410 | " Computes the gradient for linear regression \n", 411 | " \n", 412 | " Args:\n", 413 | " X (ndarray (m,n): Data, m examples with n features\n", 414 | " y (ndarray (m,)): target values\n", 415 | " w (ndarray (n,)): model parameters \n", 416 | " b (scalar) : model parameter\n", 417 | " lambda_ (scalar): Controls amount of regularization\n", 418 | " Returns\n", 419 | " dj_dw (ndarray Shape (n,)): The gradient of the cost w.r.t. the parameters w. \n", 420 | " dj_db (scalar) : The gradient of the cost w.r.t. the parameter b. \n", 421 | " \"\"\"\n", 422 | " m,n = X.shape\n", 423 | " dj_dw = np.zeros((n,)) #(n,)\n", 424 | " dj_db = 0.0 #scalar\n", 425 | "\n", 426 | " for i in range(m):\n", 427 | " f_wb_i = sigmoid(np.dot(X[i],w) + b) #(n,)(n,)=scalar\n", 428 | " err_i = f_wb_i - y[i] #scalar\n", 429 | " for j in range(n):\n", 430 | " dj_dw[j] = dj_dw[j] + err_i * X[i,j] #scalar\n", 431 | " dj_db = dj_db + err_i\n", 432 | " dj_dw = dj_dw/m #(n,)\n", 433 | " dj_db = dj_db/m #scalar\n", 434 | "\n", 435 | " for j in range(n):\n", 436 | " dj_dw[j] = dj_dw[j] + (lambda_/m) * w[j]\n", 437 | "\n", 438 | " return dj_db, dj_dw \n" 439 | ] 440 | }, 441 | { 442 | "cell_type": "markdown", 443 | "metadata": {}, 444 | "source": [ 445 | "Run the cell below to see it in action." 446 | ] 447 | }, 448 | { 449 | "cell_type": "code", 450 | "execution_count": 9, 451 | "metadata": {}, 452 | "outputs": [ 453 | { 454 | "name": "stdout", 455 | "output_type": "stream", 456 | "text": [ 457 | "dj_db: 0.341798994972791\n", 458 | "Regularized dj_dw:\n", 459 | " [0.17380012933994293, 0.32007507881566943, 0.10776313396851499]\n" 460 | ] 461 | } 462 | ], 463 | "source": [ 464 | "np.random.seed(1)\n", 465 | "X_tmp = np.random.rand(5,3)\n", 466 | "y_tmp = np.array([0,1,0,1,0])\n", 467 | "w_tmp = np.random.rand(X_tmp.shape[1])\n", 468 | "b_tmp = 0.5\n", 469 | "lambda_tmp = 0.7\n", 470 | "dj_db_tmp, dj_dw_tmp = compute_gradient_logistic_reg(X_tmp, y_tmp, w_tmp, b_tmp, lambda_tmp)\n", 471 | "\n", 472 | "print(f\"dj_db: {dj_db_tmp}\", )\n", 473 | "print(f\"Regularized dj_dw:\\n {dj_dw_tmp.tolist()}\", )" 474 | ] 475 | }, 476 | { 477 | "cell_type": "markdown", 478 | "metadata": {}, 479 | "source": [ 480 | "**Expected Output**\n", 481 | "```\n", 482 | "dj_db: 0.341798994972791\n", 483 | "Regularized dj_dw:\n", 484 | " [0.17380012933994293, 0.32007507881566943, 0.10776313396851499]\n", 485 | " ```" 486 | ] 487 | }, 488 | { 489 | "cell_type": "markdown", 490 | "metadata": {}, 491 | "source": [ 492 | "## Rerun over-fitting example" 493 | ] 494 | }, 495 | { 496 | "cell_type": "code", 497 | "execution_count": 10, 498 | "metadata": {}, 499 | "outputs": [ 500 | { 501 | "data": { 502 | "application/vnd.jupyter.widget-view+json": { 503 | "model_id": "c00f6d8dc0b74fd78de2650dbc1d8159", 504 | "version_major": 2, 505 | "version_minor": 0 506 | }, 507 | "text/plain": [ 508 | "Output()" 509 | ] 510 | }, 511 | "metadata": {}, 512 | "output_type": "display_data" 513 | }, 514 | { 515 | "data": { 516 | "application/vnd.jupyter.widget-view+json": { 517 | "model_id": "3a227dc1721944ad952522fb536c93e5", 518 | "version_major": 2, 519 | "version_minor": 0 520 | }, 521 | "text/plain": [ 522 | "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …" 523 | ] 524 | }, 525 | "metadata": {}, 526 | "output_type": "display_data" 527 | } 528 | ], 529 | "source": [ 530 | "plt.close(\"all\")\n", 531 | "display(output)\n", 532 | "ofit = overfit_example(True)" 533 | ] 534 | }, 535 | { 536 | "cell_type": "markdown", 537 | "metadata": {}, 538 | "source": [ 539 | "In the plot above, try out regularization on the previous example. In particular:\n", 540 | "- Categorical (logistic regression)\n", 541 | " - set degree to 6, lambda to 0 (no regularization), fit the data\n", 542 | " - now set lambda to 1 (increase regularization), fit the data, notice the difference.\n", 543 | "- Regression (linear regression)\n", 544 | " - try the same procedure." 545 | ] 546 | }, 547 | { 548 | "cell_type": "markdown", 549 | "metadata": {}, 550 | "source": [ 551 | "## Congratulations!\n", 552 | "You have:\n", 553 | "- examples of cost and gradient routines with regularization added for both linear and logistic regression\n", 554 | "- developed some intuition on how regularization can reduce over-fitting" 555 | ] 556 | }, 557 | { 558 | "cell_type": "code", 559 | "execution_count": null, 560 | "metadata": {}, 561 | "outputs": [], 562 | "source": [] 563 | } 564 | ], 565 | "metadata": { 566 | "kernelspec": { 567 | "display_name": "Python 3", 568 | "language": "python", 569 | "name": "python3" 570 | }, 571 | "language_info": { 572 | "codemirror_mode": { 573 | "name": "ipython", 574 | "version": 3 575 | }, 576 | "file_extension": ".py", 577 | "mimetype": "text/x-python", 578 | "name": "python", 579 | "nbconvert_exporter": "python", 580 | "pygments_lexer": "ipython3", 581 | "version": "3.7.6" 582 | } 583 | }, 584 | "nbformat": 4, 585 | "nbformat_minor": 5 586 | } 587 | -------------------------------------------------------------------------------- /C1_W2_Lab01_Python_Numpy_Vectorization_Soln.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Optional Lab: Python, NumPy and Vectorization\n", 8 | "A brief introduction to some of the scientific computing used in this course. In particular the NumPy scientific computing package and its use with python.\n", 9 | "\n", 10 | "# Outline\n", 11 | "- [  1.1 Goals](#toc_40015_1.1)\n", 12 | "- [  1.2 Useful References](#toc_40015_1.2)\n", 13 | "- [2 Python and NumPy ](#toc_40015_2)\n", 14 | "- [3 Vectors](#toc_40015_3)\n", 15 | "- [  3.1 Abstract](#toc_40015_3.1)\n", 16 | "- [  3.2 NumPy Arrays](#toc_40015_3.2)\n", 17 | "- [  3.3 Vector Creation](#toc_40015_3.3)\n", 18 | "- [  3.4 Operations on Vectors](#toc_40015_3.4)\n", 19 | "- [4 Matrices](#toc_40015_4)\n", 20 | "- [  4.1 Abstract](#toc_40015_4.1)\n", 21 | "- [  4.2 NumPy Arrays](#toc_40015_4.2)\n", 22 | "- [  4.3 Matrix Creation](#toc_40015_4.3)\n", 23 | "- [  4.4 Operations on Matrices](#toc_40015_4.4)\n" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": 1, 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "import numpy as np # it is an unofficial standard to use np for numpy\n", 33 | "import time" 34 | ] 35 | }, 36 | { 37 | "cell_type": "markdown", 38 | "metadata": {}, 39 | "source": [ 40 | "\n", 41 | "## 1.1 Goals\n", 42 | "In this lab, you will:\n", 43 | "- Review the features of NumPy and Python that are used in Course 1" 44 | ] 45 | }, 46 | { 47 | "cell_type": "markdown", 48 | "metadata": {}, 49 | "source": [ 50 | "\n", 51 | "## 1.2 Useful References\n", 52 | "- NumPy Documentation including a basic introduction: [NumPy.org](https://NumPy.org/doc/stable/)\n", 53 | "- A challenging feature topic: [NumPy Broadcasting](https://NumPy.org/doc/stable/user/basics.broadcasting.html)\n" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "\n", 61 | "# 2 Python and NumPy \n", 62 | "Python is the programming language we will be using in this course. It has a set of numeric data types and arithmetic operations. NumPy is a library that extends the base capabilities of python to add a richer data set including more numeric types, vectors, matrices, and many matrix functions. NumPy and python work together fairly seamlessly. Python arithmetic operators work on NumPy data types and many NumPy functions will accept python data types.\n" 63 | ] 64 | }, 65 | { 66 | "cell_type": "markdown", 67 | "metadata": {}, 68 | "source": [ 69 | "\n", 70 | "# 3 Vectors\n", 71 | "\n", 72 | "## 3.1 Abstract\n", 73 | "Vectors, as you will use them in this course, are ordered arrays of numbers. In notation, vectors are denoted with lower case bold letters such as $\\mathbf{x}$. The elements of a vector are all the same type. A vector does not, for example, contain both characters and numbers. The number of elements in the array is often referred to as the *dimension* though mathematicians may prefer *rank*. The vector shown has a dimension of $n$. The elements of a vector can be referenced with an index. In math settings, indexes typically run from 1 to n. In computer science and these labs, indexing will typically run from 0 to n-1. In notation, elements of a vector, when referenced individually will indicate the index in a subscript, for example, the $0^{th}$ element, of the vector $\\mathbf{x}$ is $x_0$. Note, the x is not bold in this case. \n" 74 | ] 75 | }, 76 | { 77 | "cell_type": "markdown", 78 | "metadata": {}, 79 | "source": [ 80 | "\n", 81 | "## 3.2 NumPy Arrays\n", 82 | "\n", 83 | "NumPy's basic data structure is an indexable, n-dimensional *array* containing elements of the same type (`dtype`). Right away, you may notice we have overloaded the term 'dimension'. Above, it was the number of elements in the vector, here, dimension refers to the number of indexes of an array. A one-dimensional or 1-D array has one index. In Course 1, we will represent vectors as NumPy 1-D arrays. \n", 84 | "\n", 85 | " - 1-D array, shape (n,): n elements indexed [0] through [n-1]\n", 86 | " " 87 | ] 88 | }, 89 | { 90 | "cell_type": "markdown", 91 | "metadata": {}, 92 | "source": [ 93 | "\n", 94 | "## 3.3 Vector Creation\n" 95 | ] 96 | }, 97 | { 98 | "cell_type": "markdown", 99 | "metadata": {}, 100 | "source": [ 101 | "Data creation routines in NumPy will generally have a first parameter which is the shape of the object. This can either be a single value for a 1-D result or a tuple (n,m,...) specifying the shape of the result. Below are examples of creating vectors using these routines." 102 | ] 103 | }, 104 | { 105 | "cell_type": "code", 106 | "execution_count": 2, 107 | "metadata": {}, 108 | "outputs": [ 109 | { 110 | "name": "stdout", 111 | "output_type": "stream", 112 | "text": [ 113 | "np.zeros(4) : a = [0. 0. 0. 0.], a shape = (4,), a data type = float64\n", 114 | "np.zeros(4,) : a = [0. 0. 0. 0.], a shape = (4,), a data type = float64\n", 115 | "np.random.random_sample(4): a = [0.30397547 0.59294873 0.38950072 0.76822198], a shape = (4,), a data type = float64\n" 116 | ] 117 | } 118 | ], 119 | "source": [ 120 | "# NumPy routines which allocate memory and fill arrays with value\n", 121 | "a = np.zeros(4); print(f\"np.zeros(4) : a = {a}, a shape = {a.shape}, a data type = {a.dtype}\")\n", 122 | "a = np.zeros((4,)); print(f\"np.zeros(4,) : a = {a}, a shape = {a.shape}, a data type = {a.dtype}\")\n", 123 | "a = np.random.random_sample(4); print(f\"np.random.random_sample(4): a = {a}, a shape = {a.shape}, a data type = {a.dtype}\")" 124 | ] 125 | }, 126 | { 127 | "cell_type": "markdown", 128 | "metadata": {}, 129 | "source": [ 130 | "Some data creation routines do not take a shape tuple:" 131 | ] 132 | }, 133 | { 134 | "cell_type": "code", 135 | "execution_count": 3, 136 | "metadata": {}, 137 | "outputs": [ 138 | { 139 | "name": "stdout", 140 | "output_type": "stream", 141 | "text": [ 142 | "np.arange(4.): a = [0. 1. 2. 3.], a shape = (4,), a data type = float64\n", 143 | "np.random.rand(4): a = [0.29686812 0.42095156 0.55338997 0.12826189], a shape = (4,), a data type = float64\n" 144 | ] 145 | } 146 | ], 147 | "source": [ 148 | "# NumPy routines which allocate memory and fill arrays with value but do not accept shape as input argument\n", 149 | "a = np.arange(4.); print(f\"np.arange(4.): a = {a}, a shape = {a.shape}, a data type = {a.dtype}\")\n", 150 | "a = np.random.rand(4); print(f\"np.random.rand(4): a = {a}, a shape = {a.shape}, a data type = {a.dtype}\")" 151 | ] 152 | }, 153 | { 154 | "cell_type": "markdown", 155 | "metadata": {}, 156 | "source": [ 157 | "values can be specified manually as well. " 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "execution_count": 4, 163 | "metadata": {}, 164 | "outputs": [ 165 | { 166 | "name": "stdout", 167 | "output_type": "stream", 168 | "text": [ 169 | "np.array([5,4,3,2]): a = [5 4 3 2], a shape = (4,), a data type = int64\n", 170 | "np.array([5.,4,3,2]): a = [5. 4. 3. 2.], a shape = (4,), a data type = float64\n" 171 | ] 172 | } 173 | ], 174 | "source": [ 175 | "# NumPy routines which allocate memory and fill with user specified values\n", 176 | "a = np.array([5,4,3,2]); print(f\"np.array([5,4,3,2]): a = {a}, a shape = {a.shape}, a data type = {a.dtype}\")\n", 177 | "a = np.array([5.,4,3,2]); print(f\"np.array([5.,4,3,2]): a = {a}, a shape = {a.shape}, a data type = {a.dtype}\")" 178 | ] 179 | }, 180 | { 181 | "cell_type": "markdown", 182 | "metadata": {}, 183 | "source": [ 184 | "These have all created a one-dimensional vector `a` with four elements. `a.shape` returns the dimensions. Here we see a.shape = `(4,)` indicating a 1-d array with 4 elements. " 185 | ] 186 | }, 187 | { 188 | "cell_type": "markdown", 189 | "metadata": {}, 190 | "source": [ 191 | "\n", 192 | "## 3.4 Operations on Vectors\n", 193 | "Let's explore some operations using vectors.\n", 194 | "\n", 195 | "### 3.4.1 Indexing\n", 196 | "Elements of vectors can be accessed via indexing and slicing. NumPy provides a very complete set of indexing and slicing capabilities. We will explore only the basics needed for the course here. Reference [Slicing and Indexing](https://NumPy.org/doc/stable/reference/arrays.indexing.html) for more details. \n", 197 | "**Indexing** means referring to *an element* of an array by its position within the array. \n", 198 | "**Slicing** means getting a *subset* of elements from an array based on their indices. \n", 199 | "NumPy starts indexing at zero so the 3rd element of an vector $\\mathbf{a}$ is `a[2]`." 200 | ] 201 | }, 202 | { 203 | "cell_type": "code", 204 | "execution_count": 5, 205 | "metadata": {}, 206 | "outputs": [ 207 | { 208 | "name": "stdout", 209 | "output_type": "stream", 210 | "text": [ 211 | "[0 1 2 3 4 5 6 7 8 9]\n", 212 | "a[2].shape: () a[2] = 2, Accessing an element returns a scalar\n", 213 | "a[-1] = 9\n", 214 | "The error message you'll see is:\n", 215 | "index 10 is out of bounds for axis 0 with size 10\n" 216 | ] 217 | } 218 | ], 219 | "source": [ 220 | "#vector indexing operations on 1-D vectors\n", 221 | "a = np.arange(10)\n", 222 | "print(a)\n", 223 | "\n", 224 | "#access an element\n", 225 | "print(f\"a[2].shape: {a[2].shape} a[2] = {a[2]}, Accessing an element returns a scalar\")\n", 226 | "\n", 227 | "# access the last element, negative indexes count from the end\n", 228 | "print(f\"a[-1] = {a[-1]}\")\n", 229 | "\n", 230 | "#indexs must be within the range of the vector or they will produce and error\n", 231 | "try:\n", 232 | " c = a[10]\n", 233 | "except Exception as e:\n", 234 | " print(\"The error message you'll see is:\")\n", 235 | " print(e)" 236 | ] 237 | }, 238 | { 239 | "cell_type": "markdown", 240 | "metadata": {}, 241 | "source": [ 242 | "\n", 243 | "### 3.4.2 Slicing\n", 244 | "Slicing creates an array of indices using a set of three values (`start:stop:step`). A subset of values is also valid. Its use is best explained by example:" 245 | ] 246 | }, 247 | { 248 | "cell_type": "code", 249 | "execution_count": 6, 250 | "metadata": {}, 251 | "outputs": [ 252 | { 253 | "name": "stdout", 254 | "output_type": "stream", 255 | "text": [ 256 | "a = [0 1 2 3 4 5 6 7 8 9]\n", 257 | "a[2:7:1] = [2 3 4 5 6]\n", 258 | "a[2:7:2] = [2 4 6]\n", 259 | "a[3:] = [3 4 5 6 7 8 9]\n", 260 | "a[:3] = [0 1 2]\n", 261 | "a[:] = [0 1 2 3 4 5 6 7 8 9]\n" 262 | ] 263 | } 264 | ], 265 | "source": [ 266 | "#vector slicing operations\n", 267 | "a = np.arange(10)\n", 268 | "print(f\"a = {a}\")\n", 269 | "\n", 270 | "#access 5 consecutive elements (start:stop:step)\n", 271 | "c = a[2:7:1]; print(\"a[2:7:1] = \", c)\n", 272 | "\n", 273 | "# access 3 elements separated by two \n", 274 | "c = a[2:7:2]; print(\"a[2:7:2] = \", c)\n", 275 | "\n", 276 | "# access all elements index 3 and above\n", 277 | "c = a[3:]; print(\"a[3:] = \", c)\n", 278 | "\n", 279 | "# access all elements below index 3\n", 280 | "c = a[:3]; print(\"a[:3] = \", c)\n", 281 | "\n", 282 | "# access all elements\n", 283 | "c = a[:]; print(\"a[:] = \", c)" 284 | ] 285 | }, 286 | { 287 | "cell_type": "markdown", 288 | "metadata": {}, 289 | "source": [ 290 | "\n", 291 | "### 3.4.3 Single vector operations\n", 292 | "There are a number of useful operations that involve operations on a single vector." 293 | ] 294 | }, 295 | { 296 | "cell_type": "code", 297 | "execution_count": 7, 298 | "metadata": {}, 299 | "outputs": [ 300 | { 301 | "name": "stdout", 302 | "output_type": "stream", 303 | "text": [ 304 | "a : [1 2 3 4]\n", 305 | "b = -a : [-1 -2 -3 -4]\n", 306 | "b = np.sum(a) : 10\n", 307 | "b = np.mean(a): 2.5\n", 308 | "b = a**2 : [ 1 4 9 16]\n" 309 | ] 310 | } 311 | ], 312 | "source": [ 313 | "a = np.array([1,2,3,4])\n", 314 | "print(f\"a : {a}\")\n", 315 | "# negate elements of a\n", 316 | "b = -a \n", 317 | "print(f\"b = -a : {b}\")\n", 318 | "\n", 319 | "# sum all elements of a, returns a scalar\n", 320 | "b = np.sum(a) \n", 321 | "print(f\"b = np.sum(a) : {b}\")\n", 322 | "\n", 323 | "b = np.mean(a)\n", 324 | "print(f\"b = np.mean(a): {b}\")\n", 325 | "\n", 326 | "b = a**2\n", 327 | "print(f\"b = a**2 : {b}\")" 328 | ] 329 | }, 330 | { 331 | "cell_type": "markdown", 332 | "metadata": {}, 333 | "source": [ 334 | "\n", 335 | "### 3.4.4 Vector Vector element-wise operations\n", 336 | "Most of the NumPy arithmetic, logical and comparison operations apply to vectors as well. These operators work on an element-by-element basis. For example \n", 337 | "$$ \\mathbf{a} + \\mathbf{b} = \\sum_{i=0}^{n-1} a_i + b_i $$" 338 | ] 339 | }, 340 | { 341 | "cell_type": "code", 342 | "execution_count": 8, 343 | "metadata": {}, 344 | "outputs": [ 345 | { 346 | "name": "stdout", 347 | "output_type": "stream", 348 | "text": [ 349 | "Binary operators work element wise: [0 0 6 8]\n" 350 | ] 351 | } 352 | ], 353 | "source": [ 354 | "a = np.array([ 1, 2, 3, 4])\n", 355 | "b = np.array([-1,-2, 3, 4])\n", 356 | "print(f\"Binary operators work element wise: {a + b}\")" 357 | ] 358 | }, 359 | { 360 | "cell_type": "markdown", 361 | "metadata": {}, 362 | "source": [ 363 | "Of course, for this to work correctly, the vectors must be of the same size:" 364 | ] 365 | }, 366 | { 367 | "cell_type": "code", 368 | "execution_count": 9, 369 | "metadata": {}, 370 | "outputs": [ 371 | { 372 | "name": "stdout", 373 | "output_type": "stream", 374 | "text": [ 375 | "The error message you'll see is:\n", 376 | "operands could not be broadcast together with shapes (4,) (2,) \n" 377 | ] 378 | } 379 | ], 380 | "source": [ 381 | "#try a mismatched vector operation\n", 382 | "c = np.array([1, 2])\n", 383 | "try:\n", 384 | " d = a + c\n", 385 | "except Exception as e:\n", 386 | " print(\"The error message you'll see is:\")\n", 387 | " print(e)" 388 | ] 389 | }, 390 | { 391 | "cell_type": "markdown", 392 | "metadata": {}, 393 | "source": [ 394 | "\n", 395 | "### 3.4.5 Scalar Vector operations\n", 396 | "Vectors can be 'scaled' by scalar values. A scalar value is just a number. The scalar multiplies all the elements of the vector." 397 | ] 398 | }, 399 | { 400 | "cell_type": "code", 401 | "execution_count": 10, 402 | "metadata": {}, 403 | "outputs": [ 404 | { 405 | "name": "stdout", 406 | "output_type": "stream", 407 | "text": [ 408 | "b = 5 * a : [ 5 10 15 20]\n" 409 | ] 410 | } 411 | ], 412 | "source": [ 413 | "a = np.array([1, 2, 3, 4])\n", 414 | "\n", 415 | "# multiply a by a scalar\n", 416 | "b = 5 * a \n", 417 | "print(f\"b = 5 * a : {b}\")" 418 | ] 419 | }, 420 | { 421 | "cell_type": "markdown", 422 | "metadata": {}, 423 | "source": [ 424 | "\n", 425 | "### 3.4.6 Vector Vector dot product\n", 426 | "The dot product is a mainstay of Linear Algebra and NumPy. This is an operation used extensively in this course and should be well understood. The dot product is shown below." 427 | ] 428 | }, 429 | { 430 | "cell_type": "markdown", 431 | "metadata": {}, 432 | "source": [ 433 | " " 434 | ] 435 | }, 436 | { 437 | "cell_type": "markdown", 438 | "metadata": {}, 439 | "source": [ 440 | "The dot product multiplies the values in two vectors element-wise and then sums the result.\n", 441 | "Vector dot product requires the dimensions of the two vectors to be the same. " 442 | ] 443 | }, 444 | { 445 | "cell_type": "markdown", 446 | "metadata": {}, 447 | "source": [ 448 | "Let's implement our own version of the dot product below:\n", 449 | "\n", 450 | "**Using a for loop**, implement a function which returns the dot product of two vectors. The function to return given inputs $a$ and $b$:\n", 451 | "$$ x = \\sum_{i=0}^{n-1} a_i b_i $$\n", 452 | "Assume both `a` and `b` are the same shape." 453 | ] 454 | }, 455 | { 456 | "cell_type": "code", 457 | "execution_count": 11, 458 | "metadata": {}, 459 | "outputs": [], 460 | "source": [ 461 | "def my_dot(a, b): \n", 462 | " \"\"\"\n", 463 | " Compute the dot product of two vectors\n", 464 | " \n", 465 | " Args:\n", 466 | " a (ndarray (n,)): input vector \n", 467 | " b (ndarray (n,)): input vector with same dimension as a\n", 468 | " \n", 469 | " Returns:\n", 470 | " x (scalar): \n", 471 | " \"\"\"\n", 472 | " x=0\n", 473 | " for i in range(a.shape[0]):\n", 474 | " x = x + a[i] * b[i]\n", 475 | " return x" 476 | ] 477 | }, 478 | { 479 | "cell_type": "code", 480 | "execution_count": 12, 481 | "metadata": {}, 482 | "outputs": [ 483 | { 484 | "name": "stdout", 485 | "output_type": "stream", 486 | "text": [ 487 | "my_dot(a, b) = 24\n" 488 | ] 489 | } 490 | ], 491 | "source": [ 492 | "# test 1-D\n", 493 | "a = np.array([1, 2, 3, 4])\n", 494 | "b = np.array([-1, 4, 3, 2])\n", 495 | "print(f\"my_dot(a, b) = {my_dot(a, b)}\")" 496 | ] 497 | }, 498 | { 499 | "cell_type": "markdown", 500 | "metadata": {}, 501 | "source": [ 502 | "Note, the dot product is expected to return a scalar value. \n", 503 | "\n", 504 | "Let's try the same operations using `np.dot`. " 505 | ] 506 | }, 507 | { 508 | "cell_type": "code", 509 | "execution_count": 13, 510 | "metadata": {}, 511 | "outputs": [ 512 | { 513 | "name": "stdout", 514 | "output_type": "stream", 515 | "text": [ 516 | "NumPy 1-D np.dot(a, b) = 24, np.dot(a, b).shape = () \n", 517 | "NumPy 1-D np.dot(b, a) = 24, np.dot(a, b).shape = () \n" 518 | ] 519 | } 520 | ], 521 | "source": [ 522 | "# test 1-D\n", 523 | "a = np.array([1, 2, 3, 4])\n", 524 | "b = np.array([-1, 4, 3, 2])\n", 525 | "c = np.dot(a, b)\n", 526 | "print(f\"NumPy 1-D np.dot(a, b) = {c}, np.dot(a, b).shape = {c.shape} \") \n", 527 | "c = np.dot(b, a)\n", 528 | "print(f\"NumPy 1-D np.dot(b, a) = {c}, np.dot(a, b).shape = {c.shape} \")\n" 529 | ] 530 | }, 531 | { 532 | "cell_type": "markdown", 533 | "metadata": {}, 534 | "source": [ 535 | "Above, you will note that the results for 1-D matched our implementation." 536 | ] 537 | }, 538 | { 539 | "cell_type": "markdown", 540 | "metadata": {}, 541 | "source": [ 542 | "\n", 543 | "### 3.4.7 The Need for Speed: vector vs for loop\n", 544 | "We utilized the NumPy library because it improves speed memory efficiency. Let's demonstrate:" 545 | ] 546 | }, 547 | { 548 | "cell_type": "code", 549 | "execution_count": 14, 550 | "metadata": {}, 551 | "outputs": [ 552 | { 553 | "name": "stdout", 554 | "output_type": "stream", 555 | "text": [ 556 | "np.dot(a, b) = 2501072.5817\n", 557 | "Vectorized version duration: 266.8433 ms \n", 558 | "my_dot(a, b) = 2501072.5817\n", 559 | "loop version duration: 8638.0229 ms \n" 560 | ] 561 | } 562 | ], 563 | "source": [ 564 | "np.random.seed(1)\n", 565 | "a = np.random.rand(10000000) # very large arrays\n", 566 | "b = np.random.rand(10000000)\n", 567 | "\n", 568 | "tic = time.time() # capture start time\n", 569 | "c = np.dot(a, b)\n", 570 | "toc = time.time() # capture end time\n", 571 | "\n", 572 | "print(f\"np.dot(a, b) = {c:.4f}\")\n", 573 | "print(f\"Vectorized version duration: {1000*(toc-tic):.4f} ms \")\n", 574 | "\n", 575 | "tic = time.time() # capture start time\n", 576 | "c = my_dot(a,b)\n", 577 | "toc = time.time() # capture end time\n", 578 | "\n", 579 | "print(f\"my_dot(a, b) = {c:.4f}\")\n", 580 | "print(f\"loop version duration: {1000*(toc-tic):.4f} ms \")\n", 581 | "\n", 582 | "del(a);del(b) #remove these big arrays from memory" 583 | ] 584 | }, 585 | { 586 | "cell_type": "markdown", 587 | "metadata": {}, 588 | "source": [ 589 | "So, vectorization provides a large speed up in this example. This is because NumPy makes better use of available data parallelism in the underlying hardware. GPU's and modern CPU's implement Single Instruction, Multiple Data (SIMD) pipelines allowing multiple operations to be issued in parallel. This is critical in Machine Learning where the data sets are often very large." 590 | ] 591 | }, 592 | { 593 | "cell_type": "markdown", 594 | "metadata": {}, 595 | "source": [ 596 | "\n", 597 | "### 3.4.8 Vector Vector operations in Course 1\n", 598 | "Vector Vector operations will appear frequently in course 1. Here is why:\n", 599 | "- Going forward, our examples will be stored in an array, `X_train` of dimension (m,n). This will be explained more in context, but here it is important to note it is a 2 Dimensional array or matrix (see next section on matrices).\n", 600 | "- `w` will be a 1-dimensional vector of shape (n,).\n", 601 | "- we will perform operations by looping through the examples, extracting each example to work on individually by indexing X. For example:`X[i]`\n", 602 | "- `X[i]` returns a value of shape (n,), a 1-dimensional vector. Consequently, operations involving `X[i]` are often vector-vector. \n", 603 | "\n", 604 | "That is a somewhat lengthy explanation, but aligning and understanding the shapes of your operands is important when performing vector operations." 605 | ] 606 | }, 607 | { 608 | "cell_type": "code", 609 | "execution_count": 15, 610 | "metadata": {}, 611 | "outputs": [ 612 | { 613 | "name": "stdout", 614 | "output_type": "stream", 615 | "text": [ 616 | "X[1] has shape (1,)\n", 617 | "w has shape (1,)\n", 618 | "c has shape ()\n" 619 | ] 620 | } 621 | ], 622 | "source": [ 623 | "# show common Course 1 example\n", 624 | "X = np.array([[1],[2],[3],[4]])\n", 625 | "w = np.array([2])\n", 626 | "c = np.dot(X[1], w)\n", 627 | "\n", 628 | "print(f\"X[1] has shape {X[1].shape}\")\n", 629 | "print(f\"w has shape {w.shape}\")\n", 630 | "print(f\"c has shape {c.shape}\")" 631 | ] 632 | }, 633 | { 634 | "cell_type": "markdown", 635 | "metadata": {}, 636 | "source": [ 637 | "\n", 638 | "# 4 Matrices\n" 639 | ] 640 | }, 641 | { 642 | "cell_type": "markdown", 643 | "metadata": {}, 644 | "source": [ 645 | "\n", 646 | "## 4.1 Abstract\n", 647 | "Matrices, are two dimensional arrays. The elements of a matrix are all of the same type. In notation, matrices are denoted with capitol, bold letter such as $\\mathbf{X}$. In this and other labs, `m` is often the number of rows and `n` the number of columns. The elements of a matrix can be referenced with a two dimensional index. In math settings, numbers in the index typically run from 1 to n. In computer science and these labs, indexing will run from 0 to n-1. \n", 648 | "
\n", 649 | "
missing
\n", 650 | "
Generic Matrix Notation, 1st index is row, 2nd is column
\n", 651 | "
" 652 | ] 653 | }, 654 | { 655 | "cell_type": "markdown", 656 | "metadata": {}, 657 | "source": [ 658 | "\n", 659 | "## 4.2 NumPy Arrays\n", 660 | "\n", 661 | "NumPy's basic data structure is an indexable, n-dimensional *array* containing elements of the same type (`dtype`). These were described earlier. Matrices have a two-dimensional (2-D) index [m,n].\n", 662 | "\n", 663 | "In Course 1, 2-D matrices are used to hold training data. Training data is $m$ examples by $n$ features creating an (m,n) array. Course 1 does not do operations directly on matrices but typically extracts an example as a vector and operates on that. Below you will review: \n", 664 | "- data creation\n", 665 | "- slicing and indexing" 666 | ] 667 | }, 668 | { 669 | "cell_type": "markdown", 670 | "metadata": {}, 671 | "source": [ 672 | "\n", 673 | "## 4.3 Matrix Creation\n", 674 | "The same functions that created 1-D vectors will create 2-D or n-D arrays. Here are some examples\n" 675 | ] 676 | }, 677 | { 678 | "cell_type": "markdown", 679 | "metadata": {}, 680 | "source": [ 681 | "Below, the shape tuple is provided to achieve a 2-D result. Notice how NumPy uses brackets to denote each dimension. Notice further than NumPy, when printing, will print one row per line.\n" 682 | ] 683 | }, 684 | { 685 | "cell_type": "code", 686 | "execution_count": 16, 687 | "metadata": {}, 688 | "outputs": [ 689 | { 690 | "name": "stdout", 691 | "output_type": "stream", 692 | "text": [ 693 | "a shape = (1, 5), a = [[0. 0. 0. 0. 0.]]\n", 694 | "a shape = (2, 1), a = [[0.]\n", 695 | " [0.]]\n", 696 | "a shape = (1, 1), a = [[0.44236513]]\n" 697 | ] 698 | } 699 | ], 700 | "source": [ 701 | "a = np.zeros((1, 5)) \n", 702 | "print(f\"a shape = {a.shape}, a = {a}\") \n", 703 | "\n", 704 | "a = np.zeros((2, 1)) \n", 705 | "print(f\"a shape = {a.shape}, a = {a}\") \n", 706 | "\n", 707 | "a = np.random.random_sample((1, 1)) \n", 708 | "print(f\"a shape = {a.shape}, a = {a}\") " 709 | ] 710 | }, 711 | { 712 | "cell_type": "markdown", 713 | "metadata": {}, 714 | "source": [ 715 | "One can also manually specify data. Dimensions are specified with additional brackets matching the format in the printing above." 716 | ] 717 | }, 718 | { 719 | "cell_type": "code", 720 | "execution_count": 17, 721 | "metadata": {}, 722 | "outputs": [ 723 | { 724 | "name": "stdout", 725 | "output_type": "stream", 726 | "text": [ 727 | " a shape = (3, 1), np.array: a = [[5]\n", 728 | " [4]\n", 729 | " [3]]\n", 730 | " a shape = (3, 1), np.array: a = [[5]\n", 731 | " [4]\n", 732 | " [3]]\n" 733 | ] 734 | } 735 | ], 736 | "source": [ 737 | "# NumPy routines which allocate memory and fill with user specified values\n", 738 | "a = np.array([[5], [4], [3]]); print(f\" a shape = {a.shape}, np.array: a = {a}\")\n", 739 | "a = np.array([[5], # One can also\n", 740 | " [4], # separate values\n", 741 | " [3]]); #into separate rows\n", 742 | "print(f\" a shape = {a.shape}, np.array: a = {a}\")" 743 | ] 744 | }, 745 | { 746 | "cell_type": "markdown", 747 | "metadata": {}, 748 | "source": [ 749 | "\n", 750 | "## 4.4 Operations on Matrices\n", 751 | "Let's explore some operations using matrices." 752 | ] 753 | }, 754 | { 755 | "cell_type": "markdown", 756 | "metadata": {}, 757 | "source": [ 758 | "\n", 759 | "### 4.4.1 Indexing\n" 760 | ] 761 | }, 762 | { 763 | "cell_type": "markdown", 764 | "metadata": {}, 765 | "source": [ 766 | "Matrices include a second index. The two indexes describe [row, column]. Access can either return an element or a row/column. See below:" 767 | ] 768 | }, 769 | { 770 | "cell_type": "code", 771 | "execution_count": 18, 772 | "metadata": {}, 773 | "outputs": [ 774 | { 775 | "name": "stdout", 776 | "output_type": "stream", 777 | "text": [ 778 | "a.shape: (3, 2), \n", 779 | "a= [[0 1]\n", 780 | " [2 3]\n", 781 | " [4 5]]\n", 782 | "\n", 783 | "a[2,0].shape: (), a[2,0] = 4, type(a[2,0]) = Accessing an element returns a scalar\n", 784 | "\n", 785 | "a[2].shape: (2,), a[2] = [4 5], type(a[2]) = \n" 786 | ] 787 | } 788 | ], 789 | "source": [ 790 | "#vector indexing operations on matrices\n", 791 | "a = np.arange(6).reshape(-1, 2) #reshape is a convenient way to create matrices\n", 792 | "print(f\"a.shape: {a.shape}, \\na= {a}\")\n", 793 | "\n", 794 | "#access an element\n", 795 | "print(f\"\\na[2,0].shape: {a[2, 0].shape}, a[2,0] = {a[2, 0]}, type(a[2,0]) = {type(a[2, 0])} Accessing an element returns a scalar\\n\")\n", 796 | "\n", 797 | "#access a row\n", 798 | "print(f\"a[2].shape: {a[2].shape}, a[2] = {a[2]}, type(a[2]) = {type(a[2])}\")" 799 | ] 800 | }, 801 | { 802 | "cell_type": "markdown", 803 | "metadata": {}, 804 | "source": [ 805 | "It is worth drawing attention to the last example. Accessing a matrix by just specifying the row will return a *1-D vector*." 806 | ] 807 | }, 808 | { 809 | "cell_type": "markdown", 810 | "metadata": {}, 811 | "source": [ 812 | "**Reshape** \n", 813 | "The previous example used [reshape](https://numpy.org/doc/stable/reference/generated/numpy.reshape.html) to shape the array. \n", 814 | "`a = np.arange(6).reshape(-1, 2) ` \n", 815 | "This line of code first created a *1-D Vector* of six elements. It then reshaped that vector into a *2-D* array using the reshape command. This could have been written: \n", 816 | "`a = np.arange(6).reshape(3, 2) ` \n", 817 | "To arrive at the same 3 row, 2 column array.\n", 818 | "The -1 argument tells the routine to compute the number of rows given the size of the array and the number of columns.\n" 819 | ] 820 | }, 821 | { 822 | "cell_type": "markdown", 823 | "metadata": {}, 824 | "source": [ 825 | "\n", 826 | "### 4.4.2 Slicing\n", 827 | "Slicing creates an array of indices using a set of three values (`start:stop:step`). A subset of values is also valid. Its use is best explained by example:" 828 | ] 829 | }, 830 | { 831 | "cell_type": "code", 832 | "execution_count": 19, 833 | "metadata": {}, 834 | "outputs": [ 835 | { 836 | "name": "stdout", 837 | "output_type": "stream", 838 | "text": [ 839 | "a = \n", 840 | "[[ 0 1 2 3 4 5 6 7 8 9]\n", 841 | " [10 11 12 13 14 15 16 17 18 19]]\n", 842 | "a[0, 2:7:1] = [2 3 4 5 6] , a[0, 2:7:1].shape = (5,) a 1-D array\n", 843 | "a[:, 2:7:1] = \n", 844 | " [[ 2 3 4 5 6]\n", 845 | " [12 13 14 15 16]] , a[:, 2:7:1].shape = (2, 5) a 2-D array\n", 846 | "a[:,:] = \n", 847 | " [[ 0 1 2 3 4 5 6 7 8 9]\n", 848 | " [10 11 12 13 14 15 16 17 18 19]] , a[:,:].shape = (2, 10)\n", 849 | "a[1,:] = [10 11 12 13 14 15 16 17 18 19] , a[1,:].shape = (10,) a 1-D array\n", 850 | "a[1] = [10 11 12 13 14 15 16 17 18 19] , a[1].shape = (10,) a 1-D array\n" 851 | ] 852 | } 853 | ], 854 | "source": [ 855 | "#vector 2-D slicing operations\n", 856 | "a = np.arange(20).reshape(-1, 10)\n", 857 | "print(f\"a = \\n{a}\")\n", 858 | "\n", 859 | "#access 5 consecutive elements (start:stop:step)\n", 860 | "print(\"a[0, 2:7:1] = \", a[0, 2:7:1], \", a[0, 2:7:1].shape =\", a[0, 2:7:1].shape, \"a 1-D array\")\n", 861 | "\n", 862 | "#access 5 consecutive elements (start:stop:step) in two rows\n", 863 | "print(\"a[:, 2:7:1] = \\n\", a[:, 2:7:1], \", a[:, 2:7:1].shape =\", a[:, 2:7:1].shape, \"a 2-D array\")\n", 864 | "\n", 865 | "# access all elements\n", 866 | "print(\"a[:,:] = \\n\", a[:,:], \", a[:,:].shape =\", a[:,:].shape)\n", 867 | "\n", 868 | "# access all elements in one row (very common usage)\n", 869 | "print(\"a[1,:] = \", a[1,:], \", a[1,:].shape =\", a[1,:].shape, \"a 1-D array\")\n", 870 | "# same as\n", 871 | "print(\"a[1] = \", a[1], \", a[1].shape =\", a[1].shape, \"a 1-D array\")\n" 872 | ] 873 | }, 874 | { 875 | "cell_type": "markdown", 876 | "metadata": {}, 877 | "source": [ 878 | "\n", 879 | "## Congratulations!\n", 880 | "In this lab you mastered the features of Python and NumPy that are needed for Course 1." 881 | ] 882 | }, 883 | { 884 | "cell_type": "code", 885 | "execution_count": null, 886 | "metadata": {}, 887 | "outputs": [], 888 | "source": [] 889 | } 890 | ], 891 | "metadata": { 892 | "dl_toc_settings": { 893 | "rndtag": "40015" 894 | }, 895 | "kernelspec": { 896 | "display_name": "Python 3", 897 | "language": "python", 898 | "name": "python3" 899 | }, 900 | "language_info": { 901 | "codemirror_mode": { 902 | "name": "ipython", 903 | "version": 3 904 | }, 905 | "file_extension": ".py", 906 | "mimetype": "text/x-python", 907 | "name": "python", 908 | "nbconvert_exporter": "python", 909 | "pygments_lexer": "ipython3", 910 | "version": "3.7.6" 911 | }, 912 | "toc-autonumbering": false 913 | }, 914 | "nbformat": 4, 915 | "nbformat_minor": 4 916 | } 917 | -------------------------------------------------------------------------------- /C1_W2_Lab05_Sklearn_GD_Soln.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Optional Lab: Linear Regression using Scikit-Learn" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "There is an open-source, commercially usable machine learning toolkit called [scikit-learn](https://scikit-learn.org/stable/index.html). This toolkit contains implementations of many of the algorithms that you will work with in this course.\n", 15 | "\n" 16 | ] 17 | }, 18 | { 19 | "cell_type": "markdown", 20 | "metadata": {}, 21 | "source": [ 22 | "## Goals\n", 23 | "In this lab you will:\n", 24 | "- Utilize scikit-learn to implement linear regression using Gradient Descent" 25 | ] 26 | }, 27 | { 28 | "cell_type": "markdown", 29 | "metadata": {}, 30 | "source": [ 31 | "## Tools\n", 32 | "You will utilize functions from scikit-learn as well as matplotlib and NumPy. " 33 | ] 34 | }, 35 | { 36 | "cell_type": "code", 37 | "execution_count": 1, 38 | "metadata": {}, 39 | "outputs": [], 40 | "source": [ 41 | "import numpy as np\n", 42 | "import matplotlib.pyplot as plt\n", 43 | "from sklearn.linear_model import SGDRegressor\n", 44 | "from sklearn.preprocessing import StandardScaler\n", 45 | "from lab_utils_multi import load_house_data\n", 46 | "from lab_utils_common import dlc\n", 47 | "np.set_printoptions(precision=2)\n", 48 | "plt.style.use('./deeplearning.mplstyle')" 49 | ] 50 | }, 51 | { 52 | "cell_type": "markdown", 53 | "metadata": {}, 54 | "source": [ 55 | "# Gradient Descent\n", 56 | "Scikit-learn has a gradient descent regression model [sklearn.linear_model.SGDRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDRegressor.html#examples-using-sklearn-linear-model-sgdregressor). Like your previous implementation of gradient descent, this model performs best with normalized inputs. [sklearn.preprocessing.StandardScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html#sklearn.preprocessing.StandardScaler) will perform z-score normalization as in a previous lab. Here it is referred to as 'standard score'." 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "metadata": {}, 62 | "source": [ 63 | "### Load the data set" 64 | ] 65 | }, 66 | { 67 | "cell_type": "code", 68 | "execution_count": 2, 69 | "metadata": {}, 70 | "outputs": [], 71 | "source": [ 72 | "X_train, y_train = load_house_data()\n", 73 | "X_features = ['size(sqft)','bedrooms','floors','age']" 74 | ] 75 | }, 76 | { 77 | "cell_type": "markdown", 78 | "metadata": {}, 79 | "source": [ 80 | "### Scale/normalize the training data" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": 3, 86 | "metadata": {}, 87 | "outputs": [ 88 | { 89 | "name": "stdout", 90 | "output_type": "stream", 91 | "text": [ 92 | "Peak to Peak range by column in Raw X:[2.41e+03 4.00e+00 1.00e+00 9.50e+01]\n", 93 | "Peak to Peak range by column in Normalized X:[5.85 6.14 2.06 3.69]\n" 94 | ] 95 | } 96 | ], 97 | "source": [ 98 | "scaler = StandardScaler()\n", 99 | "X_norm = scaler.fit_transform(X_train)\n", 100 | "print(f\"Peak to Peak range by column in Raw X:{np.ptp(X_train,axis=0)}\") \n", 101 | "print(f\"Peak to Peak range by column in Normalized X:{np.ptp(X_norm,axis=0)}\")" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": {}, 107 | "source": [ 108 | "### Create and fit the regression model" 109 | ] 110 | }, 111 | { 112 | "cell_type": "code", 113 | "execution_count": 4, 114 | "metadata": {}, 115 | "outputs": [ 116 | { 117 | "name": "stdout", 118 | "output_type": "stream", 119 | "text": [ 120 | "SGDRegressor(alpha=0.0001, average=False, early_stopping=False, epsilon=0.1,\n", 121 | " eta0=0.01, fit_intercept=True, l1_ratio=0.15,\n", 122 | " learning_rate='invscaling', loss='squared_loss', max_iter=1000,\n", 123 | " n_iter_no_change=5, penalty='l2', power_t=0.25, random_state=None,\n", 124 | " shuffle=True, tol=0.001, validation_fraction=0.1, verbose=0,\n", 125 | " warm_start=False)\n", 126 | "number of iterations completed: 128, number of weight updates: 12673.0\n" 127 | ] 128 | } 129 | ], 130 | "source": [ 131 | "sgdr = SGDRegressor(max_iter=1000)\n", 132 | "sgdr.fit(X_norm, y_train)\n", 133 | "print(sgdr)\n", 134 | "print(f\"number of iterations completed: {sgdr.n_iter_}, number of weight updates: {sgdr.t_}\")" 135 | ] 136 | }, 137 | { 138 | "cell_type": "markdown", 139 | "metadata": {}, 140 | "source": [ 141 | "### View parameters\n", 142 | "Note, the parameters are associated with the *normalized* input data. The fit parameters are very close to those found in the previous lab with this data." 143 | ] 144 | }, 145 | { 146 | "cell_type": "code", 147 | "execution_count": 5, 148 | "metadata": {}, 149 | "outputs": [ 150 | { 151 | "name": "stdout", 152 | "output_type": "stream", 153 | "text": [ 154 | "model parameters: w: [110.21 -21.07 -32.51 -38.02], b:[363.16]\n", 155 | "model parameters from previous lab: w: [110.56 -21.27 -32.71 -37.97], b: 363.16\n" 156 | ] 157 | } 158 | ], 159 | "source": [ 160 | "b_norm = sgdr.intercept_\n", 161 | "w_norm = sgdr.coef_\n", 162 | "print(f\"model parameters: w: {w_norm}, b:{b_norm}\")\n", 163 | "print( \"model parameters from previous lab: w: [110.56 -21.27 -32.71 -37.97], b: 363.16\")" 164 | ] 165 | }, 166 | { 167 | "cell_type": "markdown", 168 | "metadata": {}, 169 | "source": [ 170 | "### Make predictions\n", 171 | "Predict the targets of the training data. Use both the `predict` routine and compute using $w$ and $b$." 172 | ] 173 | }, 174 | { 175 | "cell_type": "code", 176 | "execution_count": 6, 177 | "metadata": {}, 178 | "outputs": [ 179 | { 180 | "name": "stdout", 181 | "output_type": "stream", 182 | "text": [ 183 | "prediction using np.dot() and sgdr.predict match: True\n", 184 | "Prediction on training set:\n", 185 | "[295.21 485.91 389.6 492.07]\n", 186 | "Target values \n", 187 | "[300. 509.8 394. 540. ]\n" 188 | ] 189 | } 190 | ], 191 | "source": [ 192 | "# make a prediction using sgdr.predict()\n", 193 | "y_pred_sgd = sgdr.predict(X_norm)\n", 194 | "# make a prediction using w,b. \n", 195 | "y_pred = np.dot(X_norm, w_norm) + b_norm \n", 196 | "print(f\"prediction using np.dot() and sgdr.predict match: {(y_pred == y_pred_sgd).all()}\")\n", 197 | "\n", 198 | "print(f\"Prediction on training set:\\n{y_pred[:4]}\" )\n", 199 | "print(f\"Target values \\n{y_train[:4]}\")" 200 | ] 201 | }, 202 | { 203 | "cell_type": "markdown", 204 | "metadata": {}, 205 | "source": [ 206 | "### Plot Results\n", 207 | "Let's plot the predictions versus the target values." 208 | ] 209 | }, 210 | { 211 | "cell_type": "code", 212 | "execution_count": 7, 213 | "metadata": {}, 214 | "outputs": [ 215 | { 216 | "data": { 217 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtgAAADpCAYAAAAeakAkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzdeVyU1f4H8M/MyL4Mm6IM2wiu5G6AaSgCppWZmmlJ4nJTK62u/tLrQppjpXb12s1uuZUGlrebJpmZAe4LuORSaC4wIIsLiqyCMMz5/TEy8sw8DzA4DDPwfb9evnDO88zMeWbOeZ4zzznne0SMMQZCCCGEEEKIUYibOwOEEEIIIYS0JNTAJoQQQgghxIiogU0IIYQQQogRUQObEEIIIYQQI6IGNiGEEEIIIUZEDWxCCCGEEEKMiBrYhBCLM3nyZERGRgo+biyRSIT4+PjHfh1TMNYxE8u1dOlSBAYGah9v2bIFbdq0afL3HTJkCP72t781+fs0VGOOW/ezI8TYmr4mEmJkkZGR8Pb2xpYtW5o7Kxzmmq/W4NNPP4VarW7w/kLf1Y0bN+Di4mLk3DUNQ4+ZtHzjx4/HiBEjmjsbhBBQA5u0YowxqFQqWFlZNXdWjKKyshLW1tbNnY0GM2Z+pVKpUV6nffv2RnkdUzDWMbcU5lj+TZ0nOzs72NnZmez9CCHCaIgIsSiTJ09GcnIytm7dCpFIBJFIhIMHDwIAFi1ahG7dusHe3h4+Pj6YOXMmioqKtM+t6UY8cOAA+vTpAxsbG+zbtw/l5eWYPn06pFIpXF1d8eabb2LBggV63Yfbt29H7969YWtrC39/f8yZMwdlZWX15qu2q1evQiQS4fjx45z01NRUiEQi/PXXXwCA0tJSvPPOO5DJZLC3t0efPn2wc+dO7f6ZmZkQiUTYtm0bnn32WTg4OGDhwoWoqqrCnDlz4O3tDRsbG3To0AETJkzgfH66wwri4+MhEom0j3NycjB27Fh4eHjAzs4OHTt2xCeffCL4nRw8eBAikQi7d+9GcHAwbG1tERQUhMTERL199uzZg0GDBsHW1hYbNmwAAHz22Wfo2rUrbG1t0alTJ3z44YdQqVTa5967dw/jx4+Hg4MDPD09sXjxYuguQMt3XP/973/Rr18/2Nrawt3dHSNGjMC9e/fq/K50h4jcuHEDEyZMgIuLC+zs7DBkyBCcPn1a77gSExMRFhYGe3t7dO/eHfv27RP8vAD+7umjR49CJBIhMzMTAFBcXIwpU6agffv2sLGxgY+PD+bMmSN4zDWPN2zYAD8/Pzg7O2PUqFHIz8/nvM/atWvh7e0Ne3t7PPPMM4iLi4NIJEJOTg5vXmuOUfefv79/nce4adMmdOvWTfv5h4WFcd7jzJkzGD58OJydneHo6Ijg4GCkpqZqt2/duhXdu3eHjY0NvL29sXjxYk65GDJkCKZNm4bY2Fh06NABMpkMAHDt2jWMHTsWLi4ucHV1xbBhw/DHH3/UmdeaIQ8KhQLt27eHm5sbJk+erK3fgOYH+T//+U907NgR1tbWCAgIwNq1azmv4+/vj8WLF+PNN9+Eu7s7Bg4cqP38fvnlFwwYMAB2dnbo168f0tLSkJaWhkGDBsHe3h7BwcG4ePGi9rXu3buH6Oho+Pr6ws7ODl26dMHq1av1yn5tukMl/P39eb+7mvKuUqmwdOlSyOVybb1dv3495zWzsrIwfPhw2NnZwdfXF5999lmdnyWARh8zAPzyyy/o168fbGxs0K5dO7z55pt630NsbCzatWsHR0dHTJgwAffu3dPLQ2JiIgYOHAg7OzvIZDJMmTIFd+/erTfvhBgNI8SCFBYWsqeffpq9/PLL7MaNG+zGjRvswYMHjDHGFAoFO3z4MFMqlSwpKYl16dKFTZo0Sfvcr7/+molEIta/f3+WnJzM0tPT2e3bt9ns2bNZu3btWEJCAvvrr7/YP/7xD+bs7MwCAgI4z3VxcWHffPMNS09PZ4cOHWI9evRg0dHR9eZLV2hoKJs+fTon7a233mLBwcGMMcbUajUbMmQIGzx4MDty5AhLT09n69evZ1ZWViwpKYkxxphSqWQAmEwmY3FxcSw9PZ1lZGSw1atXM5lMxg4cOMCysrLYyZMn2b/+9S/t+8TExLCIiAjOe8fFxbHap4KRI0eyiIgIdvbsWaZUKtn+/fvZt99+K/idHDhwgAFggYGBbPfu3ezixYts6tSpzNbWluXk5HD26dKlC0tISGAZGRksOzubLVmyhPn6+rKdO3eyjIwMtmfPHubj48MWL16sff0XX3yRBQQEsOTkZPbnn3+yiRMnMicnJ85x6B7XV199xdq0acOWLVvG0tLS2Pnz59natWtZfn5+nd8VABYXF6f9HoKDg1mvXr3YkSNH2IULF9jLL7/MXFxcWH5+Pue4evbsyfbu3cuuXLnCXnvtNSaVStm9e/cEP7MlS5ZwyhdjjB05coQBYEqlkjHG2OzZs1nPnj1ZSkoKy8rKYseOHWMbNmwQPOaYmBjm7OzMJkyYwP744w927Ngx5uvry6kDO3bsYBKJhK1du5ZduXKFff3116xDhw4MAMvOzubN64MHD7Sf040bN1haWhrz8vJikydPFjy+06dPM4lEwrZu3coyMzPZhQsX2MaNG7Xv8eeffzJ7e3s2YcIEdurUKXblyhX27bffsuPHjzPGGPv555+ZWCxmH330Ebt8+TLbvn07c3Fx4ZSLwYMHM0dHRzZjxgyWlpbGLly4wG7evMk8PT3ZzJkz2YULF9hff/3FZs2axdzc3Njt27cF8zt48GAmlUrZu+++yy5dusT27t3LpFIpe//997X7rFu3jtna2rL169ezK1eusC+++ILZ2NiwTZs2affx8/NjTk5ObMmSJezy5cssLS1NW0Z69+7NkpOTWVpaGgsNDWU9evRgTz/9NEtKSmIXL15kAwcO1J4DGGPsxo0bbMWKFezMmTMsIyODxcXFMQcHB/bVV19p99EtR19//TWTSCTax7dv3+Z8dzExMax9+/bsxo0bjDFNmenRowfbt28fy8jIYNu3b2dSqVR7TGq1mvXp04f179+fpaSksLNnz7LIyEjm5OTEpk2bJvh5NvaYz58/zyQSCXv33XfZxYsX2S+//MJ8fHy051nGGFu7di2zt7dnW7ZsYZcvX2YrV65kUqmUc9zJycnMzs6O/fvf/2ZXrlxhJ0+eZEOGDGFPP/00U6vVvJ8dIcZGDWxicSIiIlhMTEy9++3cuZNZW1uz6upqxpjm4gOAHT58WLtPaWkps7a25lwkGWMsJCSEc/L18/NjX3zxBWefQ4cOMQCsoKDAoHx98cUXzMXFhVVUVDDGGKusrGQeHh5s3bp1jDHNxcnGxoYVFhZynjdlyhQ2atQoxtijBvayZcs4+7z99tssPDxcexHR1ZAGds+ePdmSJUvqPY4aNRfT2p9hVVUV8/X1ZYsWLeLs880332j3KSsrY3Z2dmzv3r2c19u6dSuTSqWMMcauXr3KALDffvtNu/3BgwfMy8urzga2j48Pe+uttwTzLPRd1W5gJyUlMQAsLS1Nu72iooK1b9+effDBB5zj2rFjh3afGzduMADs119/FXz/hjSwX3jhhTrLE18D28PDQ1uuGGPs448/Zu3bt9c+fuqppziNFcYYmz9/fp0N7NoqKyvZkCFD2KBBgzjvo2vnzp3M2dmZFRUV8W6Pjo5mPXv21NZNXYMGDWLjxo3jpK1du5bZ2tpqfwwNHjyYderUifMaS5YsYSEhIZznqdVq1rFjR84PTV2DBw9mPXr04KTNmDGDhYaGah97e3uz9957j7PPu+++y+Ryufaxn58fGzp0KGefmjLy448/atO+//57BoD98MMP2rSdO3cyAKykpEQwn2+//TaLjIzkHG9dDezaNm7cyOzt7dnJkycZY4xlZGQwkUjELl26xNnvgw8+YL169WKMMZaYmMgAsMuXL2u33759m9na2jaogW3oMUdHR7Mnn3yS81q7du1iIpGIZWZmMsYYk8lkbOHChZx9xo4dyznuwYMHs/nz53P2ycrKYgDY2bNnGWPUwCZNj4aIkBZj586dCAsLg5eXFxwdHTFx4kRUVlbi5s2bnP2efPJJ7f+vXbuGyspKhIaGcvYZMGCA9v/5+fnIysrCnDlz4OjoqP1XM5no2rVrBuVz/PjxKC8vx08//QRA0yVaXFysHcpx6tQpVFZWQiaTcd4vPj4eV69e5bxWcHAw5/GUKVPwxx9/IDAwEDNnzsSOHTtQWVlpUP7effddfPTRRwgJCcH8+fNx+PDhBj2v9mfWpk0b3u7f2vlNS0tDeXk5xo4dyznOGTNmoKioCPn5+drnP/XUU9rnWVtbc75DXbdv30Z2djaGDRvWoHwLSUtLg7u7O7p3765Ns7GxQUhICNLS0jj79u7dW/v/9u3bQyKR4NatW4/1/m+++SZ++OEHPPHEE3jnnXewd+/eeic1duvWDTY2NtrHMpmMk4+LFy/WWdbr88YbbyA7Oxs//vij9n1GjBjB+f4AICoqCh07doRcLseECROwYcMG3LlzR/s6Z86cQUREBMRi/ktQWloawsLCOGmDBw9GRUUF0tPTtWn9+vXjvMapU6dw5swZTn6cnJyQmZmpV3d01f4OAe5nV1xcjJycHN48ZWZm4v79+9o03TpZo1evXtr/14z179mzp17a7du3AQBqtRorVqxA79694eHhAUdHR3z55ZfIysqq8zj47N+/H7NmzUJcXJy27pw+fRqMMfTv35/zeX300Ufaz+rixYvw8PBA586dta/Vtm1bdOnSpUHva+gxC33vjDFcvHgRxcXFyM3N5ZwPAGDQoEGcx6dOncLatWs5x1VTj+srB4QYC01yJC1Camoqxo0bhwULFuCTTz6Bq6srUlJSEBMTw2lgSiQS2Nra6j2/9hhkXTWNmk8//RTh4eF62729vQ3Kq6urK0aOHIlvvvkG48aNwzfffIPnnnsO7u7u2veTSqU4deqU3nN1J0w5ODhwHvfu3RtKpRKJiYk4cOAA3nnnHcTGxiIlJQXOzs4Qi8V6Yzirqqo4j6dMmYLhw4fj119/xYEDBzBixAiMHj3a4PB1uu+jm9+az/V///sf5wJew83Nrc7xpvWp6zt9nNdgjOml801kq6sx3JDv4ZlnnsH169exb98+HDx4ENHR0ejRoweSk5MhkUh4X1c3HyKRSO99Gvu5rFq1Cjt37sSJEyfg4eGhTd+0aRPKy8s5+zo6OuL06dM4duwYkpKS8OWXX2LevHlITk5Gv379GpQP3e01x1E7Xbf8q9VqREREYN26dXqvV9+kUL7PTvc7FMpTbbp5qlF7MnXN6/Cl1bzn6tWr8fHHH2PNmjXo27cvnJyc8K9//Qt79uyp8zh0XblyBS+99BIUCgXGjBmjTa95n+PHj8Pe3p7znJq88JV1Qxh6zLXTdNUuy/XlSa1WY/78+Xjttdf0tlnSRGZi2egONrE41tbWqK6u5qQdPXoUHh4eWL58OUJCQtC5c2fBSVu1BQYGwtraGidOnOCkp6SkaP/v6ekJHx8fXL58GYGBgXr/ahrsfPkSMmnSJPz666+4fPky9uzZg5iYGO22/v37o7CwEBUVFXrv5evrW+9rOzo6YvTo0fj3v/+N06dP49KlSzh06BAAoF27dsjLy+Ps//vvv+u9RocOHTBlyhR888032Lx5M7Zt24bi4uI637f2Z6ZSqXDq1Cl069ZNcP+goCDY2toiIyOD93OVSCQICgoCAM6k0MrKSt4fHzXatWsHb2/vOicaNuS7CgoKwp07dzh34R88eICTJ09q89VY7dq1w+3btzl54Pse3Nzc8Morr2D9+vXYs2cPDh06pNcrYIju3bvXWdaF7Nq1C++//z527typd/dSJpNxvrcaEokEYWFhWLZsGc6cOYMOHTrg22+/BaC585yUlCT4IyQoKEhbZmscPnxYO+lWSP/+/ZGWlqaXp8DAQLRt27be4xTi7OwMb29v3jzJ5XK9BqoxHD58GMOHD8e0adPQp08fBAYGGnz3taCgAM899xzGjh2L9957j7Ot5ofO9evX9T6rgIAAAJrvIT8/n/O+d+7cwZUrVx7z6Pjxfe+HDh2CSCRC9+7dIZVKIZPJcOzYMc4+uo9rygHfeaWml4WQpkZ3sInFkcvlOHDgANLT0yGVSiGVStGlSxfk5+dj8+bNCA8Px9GjR/Gf//yn3tdycHDAjBkzsHjxYnh6eqJz587YunUrLl26xLkgf/jhh5g2bRpcXFzw4osvwsrKCpcuXcLevXu1s+758iUUAnDEiBFwc3PDhAkT4OTkhGeffVa7bejQoYiMjMSYMWOwcuVK9OrVC/fu3cPx48dha2uL119/XfB4PvnkE3h5eaF3796wt7fHd999B4lEor1DHBkZiZUrV2LdunUYMWIE9u/fj++//57zGrNmzcKzzz6LLl26oKKiAjt37oSPjw+cnJzq/CxXrFiB9u3bQy6XY82aNbh16xbeeOMNwf0dHR2xcOFCLFy4EIBmWIFKpcIff/yBs2fPYuXKlQgMDMQLL7yAt956C+vXr4enpydWrFiBkpKSOvOyZMkSvPHGG/D09MRLL70EtVqNAwcOYMKECfDw8GjQdzV06FAEBwfj1Vdfxeeffw6pVAqFQoGKioo6j6shwsPDcf/+fcTGxmLatGn4/fff8fnnn3P2WbRoEfr164egoCCIxWJs27YNjo6ODfqRJWTu3LkYP348goODMWLECBw/fhzffPMNAOG7gmlpaYiOjsbSpUvRtWtX7ZAriUQi2GhNSEhARkYGwsLC0LZtW5w5cwbZ2dnabvp58+YhJCQEEydOxNy5c+Hq6orff/8d3t7eGDBgABYsWICRI0dixYoVGDNmDM6dO4elS5di7ty5dYa9mzVrFjZv3owXX3wRixcvho+PD3JycrB3714899xzekMLDLFgwQLMnTsXnTp1wpAhQ7B//3588cUXet+bsXTp0gVxcXE4cOAAZDIZvvnmG6SmpsLV1bXBrzFmzBi4uLhgyZIlnKFybm5uCAwMxNSpU/H6669j1apVGDBgAMrKynDmzBnk5+dj/vz5iIiIQK9evRAdHY3PPvsM1tbWmD9/fpMtZvPee++hb9++mDNnDqZPn47MzEzMnj0bEydO1Jb7uXPnIjY2Fl27dkVoaCh++uknJCUlcV5n2bJlGDZsGP7+978jJiYGTk5OuHr1Kv73v/9h3bp1FMqQmEbzDP0mpPHS09PZ008/zRwcHBgAduDAAcYYY4sXL2bt2rVj9vb2bMSIEezbb7/lTBoTmgB0//599vrrrzMnJycmlUrZG2+8wd555x32xBNPcPb78ccfWWhoKLOzs2NOTk6sV69e2sludeVLyLvvvssAsFmzZvHmaf78+czf359ZWVkxT09P9swzz7Dk5GTG2KNJjkeOHOE878svv2R9+/ZlTk5OzMHBgfXv35/t2rWLs8/y5cuZl5cXc3BwYBMmTGDr1q3jTHJ88803WadOnZitrS1zc3Njzz77LPvzzz8Fj6NmQlNCQgLr27cvs7a2Zt26deNM8qvZh28i3aZNm1ivXr2YjY0Nc3FxYcHBwew///mPdvudO3fYuHHjmL29PfPw8GD/+Mc/2KRJk+qc5MgYY/Hx8axnz57M2tpaexw1kT2EvivUmuTIGGN5eXls/PjxTCqVMltbWxYWFsZOnTpV73FJJBL29ddfC35mjDG2efNmJpfLma2tLRs+fDj77rvvOOV12bJlLCgoiDk4ODBnZ2cWFhbG+b75JjnWN4GVMcbWrFnDvLy8mK2tLRs2bBhbv349A8Du3LnDm8+aycG6//z8/ASP7dChQyw8PJx5eHgwGxsbFhgYyD7++GPO5NvU1FQWERHB7O3tmaOjIwsODmapqana7Vu2bGFdu3ZlVlZWzMvLiy1cuJBVVVVptw8ePJh3ol1mZiZ79dVXmYeHB7O2tma+vr5s4sSJLCMjQzC/fK+lUCg4x6hWq9mqVauYv78/a9OmDZPL5XoTJ/38/JhCoeCk8ZUR3QmtjDF24sQJBoBdvXqVMaaJTDRu3Djm5OTE3Nzc2JtvvskWL17MyVN9kxz5vrfa5V2lUrGVK1eyLl26MCsrK+bu7s7CwsLY999/r30NpVLJoqKimI2NDZPJZGzt2rWCn/3jHjNjjO3Zs0d7HvHw8GAzZ85kpaWl2u3V1dVswYIFzN3dndnb27OxY8eyNWvW6J3bDx8+zCIiIpijoyOzt7dnXbt2Ze+88462DNEkR9LURIw9xiBHQlqooUOHwtXVFTt27GjurJi9gwcPIjw8HNnZ2QaPRyfNb9myZfj0008pRjAhhBgRDREhrd4ff/yB33//HQMGDEBlZaW2W/aXX35p7qwRYlRVVVVYvXq1dnGiAwcO4JNPPsFbb73V3FkjhJAWhRrYpNUTiUT44osv8Pbbb0OtVqNr16748ccftWH4CGkpalbxW716NUpKSiCXy7Fw4UK9CXCEEEIeDw0RIYQQQgghxIgoTB8hhBBCCCFGRA1sQgghhBBCjIga2IQQQgghhBgRNbAJIYQQQggxImpgE0IIIYQQYkQtKkxfUVFRc2eBtCBSqdQk70PllhgblV1iiUxVbgEqu8S4+Mou3cEmhBBCCCHEiKiBTQghhBBCiBG1qCEitZmyq4m0HM3dbUjlljQWlV1iiZq73AJUdknj1Fd26Q42IYQQQgghRkQNbEIIIYQQQoyoxQ4RIUSIshiIPQnklgEyB0ARDMidmztXhNSNyi2xVFR2ibFYUllqVQ3swsJCVFVVNXc2WgQXFxdYWVk1dzYMpiwGonYD6cWP0lJuAYkjzbeSEkLlllgqKrvEWCytLLWaBvb9+/cBAG3btm3mnFg+tVqN27dvw93d3eIa2bEnuZUT0DyOPQnERzZPngipD5VbYqmo7BJjsbSy1GrGYJeWltJMYSMRi8Vo164d7t2719xZMVhuGX96nkA6IeaAyi2xVFR2ibFYWllqNQ1skUgEkUjU3NloMcRisUV+njIH/nQvgXRCzAGVW2KpqOwSY7G0stRqGtiEAJoJEQE6Y7UCnDXphJgrKrfEUlHZJcZiaWWp1YzBJgTQTIRIHKkZs5VXpvnla86zkAkBqNwSy0VllxiLpZUlamCTVkfubJ4TIgipC5VbYqmo7BJjsaSyRENEzIC/vz+SkpJa3XsTQgghhLREdAfbwlVXV0MikTR3NixLkRJIjQXKcgEHGRCiAKTy5s4VIXWjckssFZVdYiwWVJaogV2Ppl416LXXXsP169cxcuRISCQSvP/++zh16hSOHDmC8vJy9OrVC1988QWCgoIAAJMnT4adnR2ysrJw6NAhJCQkwM3NDdOmTcO1a9cwfPhwiMVidOrUCcuXLwcA/Pzzz1i8eDEyMzPRvXt3fPnll+jZsyfve8+bN894B2eOipRAQhRQnP4o7WYKMCrRbCspIVRuicWiskuMxcLKEg0RqUPNqkHbrgIH8zR/o3Zr0o0lLi4Ovr6+2L17N0pLSzFv3jyMGDECV69exe3bt9G3b19MnDiR85xvv/0WixYtQklJCYKDgzF69GhMnjwZBQUFeOWVV/Djjz9q9/39998xdepUrF+/Hnfv3sWMGTPwwgsv4MGDB7zv3eKlxnIrJ6B5nBrbPPkhpCGo3BJLRWWXGIuFlSVqYNehrlWDmtLUqVPh5OQEGxsbLF26FOfPn0dRUZF2+6hRozBw4ECIxWKcO3cOKpUKb7/9NqysrDBmzBgEBz+KWbNx40bMmDEDISEhkEgkiImJgY2NDVJSUpr2IMxVWa5Aep5p80GIIajcEktFZZcYi4WVJWpg16E5Vg2qrq7GP/7xDwQEBMDZ2Rn+/v4AgDt37mj38fHxeZSXvDzIZDLOoi+1t2dlZWH16tVwcXHR/svOzkZeXvMUSGUxEJ0EhCdo/hqzN6BBHGQC6V6mzQchhqBySyxVKyi7zX5day0srCxRA7sOplo1qHbj+Ntvv0VCQgKSkpJQVFSEzMxMAABjjHf/Dh06IDc3l7M9Oztb+38fHx8sWrQIhYWF2n/379/HK6+8ovdaTc0UQ27qFaIAnAO4ac4BmnRCzFR2kAJZYm65zRIHIDuIyi0xby297JrFda21sLDrNzWw62CqVYM8PT2RkZEBACgpKYGNjQ3c3d1x//59LFy4sM7nDhgwABKJBOvWrYNKpUJCQgJOnnw0huX111/Hl19+idTUVDDGUFZWhj179qCkpETvvZtacw254ZDKNRMiOk8EZOGav2Y6QYKQGgsuyTGYJSJeNBH7EY540UQMZolYcInKLTFvLb3smsV1rbWwsOs3NbDrULNq0MROQLiX5m/iSOOvGrRgwQIsX74cLi4uKCgogJ+fH2QyGbp3747Q0NA6n2ttbY2dO3di8+bNcHFxQXx8PJ5//nnY2NgAAPr374+NGzdi1qxZcHV1RWBgILZs2cL73v/85z+Ne2A6mmPIDS+pHBgWD4zer/lrppWTkBq5ZUCWWI7XxPGIkOzHa+J4ZInlpq87hBiopZdds7mutRYWdP2mMH31MMWqQaNGjcKoUaMEt0+aNEn7/9qN4xr9+/fHuXPntI9DQkIwcuRI7ePhw4dj+PDhjXpvYzLVkBtCWhqqO8RStfSy29KPjzQe3cFuAQ4dOoSbN29CpVJh69atuHDhgmCDujmZasgNIS0N1R1iqVp62W3px0caj+5gtwCXL1/Gyy+/jNLSUgQEBOCHH35Ahw4dmjtbemqG3MSe1HSfeTXBwj2EtERUd4ilaullt6UfH2k8EasdfsLC1Y4VLZVKOdvy8/PRtm1bU2epRWuJn2ldZaglvSdpeajsEkvUXGWIyi55XPWVIRoiQgghhBBCiBFRA5sQQgghhBAjogY2IYQQQgghRkQNbEIIIYQQQoyIoogQ0ypSAqmxQFku4CDTLHFqxoHiCTEbVHeIpWrpZbelHx9pFGpgt3D+/v7YtGkTIiMj8dFHHyEjIwObNm1qnswUKYGEKKA4/VHazRSzXuqUELNAdYdYqpZedlv68ZFGoyEircjChQsb1LiePHkyFi9ebPwMpMZyT0KA5nFqrPHfiynDzCAAACAASURBVJCWhOoOsVQtvey29OMjjUYNbAuiUqmaOwuPpyxXID3PtPkgxNJQ3SGWqqWX3ZZ+fKTRqIFdnyIl8Fs08GO45m+R0uhv4e/vj48//hjdu3eHq6srpkyZgoqKChw8eBDe3t5YuXIl2rdvjylTpkCtVmPFihUICAiAu7s7Xn75ZRQUFGhfKy4uDn5+fnB3d8eHH37IeZ+lS5ciOjpa+/jo0aN46qmn4OLiAh8fH2zZsgUbNmzAtm3bsGrVKjg6OmLkyJHGO1AHmUC6l/Heg5CWiOoOsVQtvey29OMjjWbyBvb27dvRrVs3ODg4ICAgAEeOHAEAJCcno2vXrrC3t0d4eDiysrK0z2GMYf78+XB3d4e7uzvmzZsHkyxAWTO26so2IPeg5m9CVJM0srdt24Z9+/YhPT0dV65cwfLlywEAN2/eREFBAbKysrBhwwb8+9//xq5du3Do0CHk5eXB1dUVb731FgDg4sWLeOONNxAXF4e8vDzcvXsXOTk5vO93/fp1jBgxArNnz0Z+fj7OnTuH3r17Y/r06Zg4cSLmzZuH0tJS7N6923gHGaIAnAO4ac4BmnRCiDCqO8RStfSy29KPjzSaSRvYiYmJmD9/Pr7++muUlJTg8OHD6NixI+7cuYMxY8ZAoVCgoKAA/fv3x/jx47XP27BhA3bt2oXz58/jwoUL+Pnnn7F+/fqmz7AJx1bNmjULPj4+cHNzw6JFi/Ddd98BAMRiMT744APY2NjAzs4O69evx4cffghvb2/Y2Nhg6dKl+OGHH6BSqfDDDz/g+eefR1hYGGxsbKBQKCAW83/F27ZtQ2RkJF555RVYWVnB3d0dvXv3Ns7BCN31l8o1Ez86TwRk4Zq/NBGEkPpJ5cDQrwAnf8DaRfN36FdUd4j5awllt66ebLquEQEmjSKyZMkSvP/++wgNDQUAyGSarpUNGzYgKCgI48aNA6AZyuDh4YG//voLXbt2xdatWzF37lx4e3sDAObOnYuNGzdi5syZTZthE46t8vHx0f7fz88PeXma92jbti1sbW2127KysjB69GhOw1kikeDWrVvIy8vjvI6DgwPc3d153y87OxsBAQG82x5LfTOqpXJgWLzx35eQlqxICeyfCpRkah5XFmoe04WcmDtLL7sNiRJC1zXCw2R3sKurq3H69Gnk5+cjMDAQ3t7emDVrFsrLy5GWloZevXpp960ZPpKWlgYAett79eql3dakTDi2Kjs7W/v/69evw8tL8x4ikYizn4+PD/bu3YvCwkLtv4qKCshkMnTo0IHzOvfv38fdu3d538/Hxwfp6em823Tf0yA0o5oQ46N6RSyVpZddS88/aTYma2DfunULVVVV+OGHH3DkyBGcO3cOZ8+exfLly1FaWgqpVMrZXyqVoqSkBAD0tkulUpSWljb9OGwTjq36/PPPkZOTg4KCAnz00UecITK1zZw5E4sWLdKOUc/Pz0dCQgIA4KWXXsLPP/+Mo0ePorKyEu+//z7UajXv60ycOBFJSUn4/vvvoVKpcPfuXZw7dw4A4OnpiYyMjMYdCM2oJsT4qF4RS2XpZdfS80+ajcka2HZ2dgCA2bNno0OHDvDw8MCcOXPwyy+/wNHREcXFxZz9i4uL4eTkBAB624uLi+Ho6Ph4d1obwoRjq1599VUMGzYMHTt2RMeOHQXjUL/zzjt44YUXMGzYMDg5OSE0NBSpqakAgKCgIHz++ed49dVX0aFDB7i6umqH1ejy9fXFL7/8gtWrV8PNzQ29e/fG+fPnAQDTpk3DxYsX4eLighdffNGwA6EZ1YQYH9UrYqksvexaev5JszHZGOyaxh5fozgoKAhbt27VPi4rK0N6ejqCgoK028+fP4/g4GAAwPnz57XbmpyJxlY9+eSTWLBgASdtyJAhelFAxGIx5syZgzlz5vC+TkxMDGJiYrSPFy1apP3/0qVLOfs+/fTT2sZ5bZ06ddLezTZYiEIzPq12lxrNqCbk8VC9IpbK0suupeefNBuTRhGZMmUKPvvsM9y+fRv37t3D2rVr8fzzz2P06NH4888/sWPHDlRUVGDZsmXo2bMnunbtCgCYNGkS1qxZg9zcXOTl5WH16tWYPHmyKbNOGopmVBNifFSviKWy9LJr6fknzcakUURiY2Nx584ddO7cGba2tnj55ZexaNEi2NraYseOHZg1axaio6MREhKC7du3a583Y8YMZGRkoEePHgCAv/3tb5gxY4Yps24yD6qB3DKgSg1YiQGZA2Ajae5cGYhmVBuFshiIPakpDzIHQBEMyJ2bO1ek2VhQvaKySzgsqOzy4sk/lXFSHxEzyYotplFUVKT9v+6kyfz8fLRt29bUWTLIg2rgSqHmbw0bCdDZxTwb2ZbwmRqqrjJkyvdUFgNRu4H0WlMTApyBxJF0Eif8qOwSS9Qc5fZx35fKOAHqL0O0VLoZyS3jNq6BR3e0SesSe5J78gY0j2NPNk9+CGkoKrukpaMyThrCpENEmhNjDIyxpo888hiq+CPqCabXZuqhJWq1WjBMInWdPT6hH1V59GOrWZhDmTaHPDQElV2iy1LKrhBlMfDuMSDlluaxSuCaTGWc1NZqGtiOjo4oKiqCi4tLc2dFkJVAf4JQeg2+oSVlVU03tEStVuP27du8q0TydZ2l3KKuM0PJHPjTvQTSSdMxhzJtDnloKGdr/nQngXTSsllS2eWjLAaGJADXS+vfl87PpLZW08C2t7dHZWUl8vPzmzsrgkRVwNkcoLTqUZqjFRDpDeQ/EH7esZuak4CudGdgYHvj55MxBnd3d1hZWeltq6vrLD7S+HlpqRTBmouQ7hg/RXDz5am1MocybQ55aCiRwKweoXTSsllS2eUTe7JhjWs6PxNdraaBDcCs714DQFsAw5kS15NjYV+Ri/u2MvhGKODjVXc4oE+PAwd5FpUK9wJe7NE0eRVC3cPGIXcGDgzRKQtDFPBxptBQpmYOZdoc8tBQRVWAn1qJ5YiFF8tFnkiGxVCguIrKbmtkSWWXj1D+/dRKrJbEoqOYzs+EX6tqYJu9IiV8kqPgUxPQvhJAckq9MTfNaTiBOeXFojWyLBDjM4cybQ55aKheVkpsZFEIxMOyy4BQpGCdVSIAKrutjSWVXT58+fdTK5HEohCoovMzEUZRRMxJaix3tShA8zg1ts6nKYI13VO1NVd3lTnlxaI1siwQ4zOHMm0OeWgoBYt91Lh+KBDpUDAqu62RJZVdPopgwNeRm7Yc+mWczs9EF93BNidluQLpPOM/apE/jL8Ze1LT7ealM0vblDO468sLaaBGlgVifHJn4KtwIGY/UPgAcLHRPDZlmbakeuVUyV92nSqp7LZG5lB/Gqvm2tnBDlAzoFwFtBEDPVW5QAXPE+j8TGqhBrY5cZAJpHvV+1S5M/+EkeaYwS2UF2KAxygLxLiUxcDUA0BmieZxYaXmsamjIFhMvaKyS2oxl/pjqDoXk0mRAVd4nkRlnNRCQ0TMSYgCcA7gpjkHaNIbiQLiW6gmKAukcagOGYjKLqnFUutPnfmmMk4agO5gmxOpXDNJIjVW09Xk4KWpsI2cNKEsBhJz+LdZygzuVsvIZYE0nqVHQTA5KrukFkuqP7WHU168x79PXhmojJMGoQa2uZHKgWHxhj+vSPmwsucCDjJkBykQdVCO2+X8u1vKDO5WrbFlgRiV1Io/7JyzFV1M60fBr1s7qf5yCQAAZ4H05sI3JKS2mnNAnzu5wG8yTYOazs+kDtTAbgmKlEBCFCfqhJUyBSpVIiDWbwRY0gxuQppb26qHIbl0ws6trKKwc7x4zke4SSHMWismMiy9ufANCamhDcuHdKAcmvHXVKZJPWgMdkvAE9KtfVU6lkM/ZJCnnflPLiHEnLx8lz/s3Pi7FJKLF4WYJLUUV/KnlwikNxehoSyedsAmOwrLRwxHDeyWQCCkmxfTDxkU6a3TuC5SAr9FAz+Ga/4WKZsok4RYJi/GX7868NQvAgoxSThkDpo7wHHqaCRXhyNOHQ0/tdLshikKLYgT6Q1EujayTNP1tVWjISItgUBYrBJrL6D60WPHNsD07rV2oK5cQurl5y4FeCYL+7lTNxAvCtNHavm4mxLVl6Pgzx4NsRokSoGkm3kNsVIEa0LY6oblUwQDSGlEmabra6tHd7BbAoGQQe3DFbCr9Q2XqoDoJM1kDgDUlUtIA4gEJuoJpbd6FMKM1OJyLvZR4/ohf5YOl3PmdZ2pWcxpYicg3Evzt2Y4ZapcgQxwy3SWOADZQXWUabq+tnp0B9vMZecqcT05FnYVuSi3lcE3QgEfmc6vX4GQQR8fk6NcrfN6ZcDfjwK7ngV15RKL0KA60ITS7xSjJ296CW96qyeVIy/0K7TZHwO76kKUS1ygCv0KXnTXrlVS3sjlrSfKG3lmV3/4FnNSFgORB+VwFyU+jCSUhzyRFxYzBQZdkuNjCJyf6Pra6lED24xl5yqhTojCQPXDX8GVQFZCCrJHJfI3snVCBp24zf+6KTXp1JVLzJxBdaCJXFHJeBsCV1VeZtdAMAfZuUqok6bCS50JAHBSFSIraSqyHUz3nRHzYen1J/akpve3VCzHa+BeY/veqeP8RNfXVo+GiJix68mx8FNzu5j81Om4nmxYF5PuBBOf6ocTLagrl5g5Y9WBx7HCSoFM+HLSMuGLFVZUT/iYw3dGzIdF158iJWKucydn1japqI6yTtfXVo/uYJsxuwr+LqZOxUnIzlXWezfoBVclxpa+jggcgBUejhVhQERVClCUSKtREbMnVAfsHpium7WPB2CXzQ2Qa4di9PYwWRYsijl8Z8R89PEArLPvc9Kscd8868/DBdsqCtJRVpgNR1U+ovAwniADxmAH9lUPw99Fa3HXWg4fdR1lna6vrR41sM1Yua0M4IkV2g63kJUQVXc3eZES/7kzGFbI1tvUQZUOHPk78PwuWi2QmDWhOlBuY7pu1lUl0XBFISfNE4VYVRIN4JjJ8mEpKiVS/nQxRV1pjdZWTocD7nDSvHAHayunA0hsnkzxqRX1wxaALc8u9qjAaPyESLYf14L34P55GVClv5+6zcOyTtfXVo2GiJgx3wgFssQBvNvq7XJNjYVVuX7jWiv7N4rJScweXx3IEgfAN8J03azOhccNSm/1RALRVYTSSYvmcOugQenNhi/qhwAnlKLPiecgD52OHPjobe/FztL1lTSugX3hwgXs2LED9+9run0ePHgAtVpdz7NInYqUKNkTjbTN4Uj8Khrv/qKEykkO8ahE3EY73qfU2eUqNIO5RnU5kBoLZbEmdF94gk4IP0LMgI9MDqvIr3C7jT9KRC643cYfVpFfmXSynNBJku5O8LNW8Z9ErKtLTJwTYh5UBqY3k/qumbpUpfA6EQO39l30Nlndv07XV2LYEJGCggKMGTMGhw8fhkgkwtWrV9GxY0e89dZbcHZ2xpo1a5oqny1bkRJVP0bBqTQdQQCCAMgzUjD5TiK2vCjHdecotCvepve0OrvJhWYw135+UR6idnMD66fcoqXUiRkpUsIrZSqgygSgiUiBlKmAJy3WYK4qJfwnj0qxk4lzQswBg4g3Zrwm3Yw04JqppyQT9vdv8m6i6ysx6CbM//3f/0EikSAzMxP29vba9Jdeegn79u0zeuZajdRYWJVyu6YCkY7XS2Lx6VElejuXoEJnRFi93eQhCkBsV+fb/l7uxan8gOZkEHvSoNwT0nTMYLGGfPDPxhJKb+2YQLNJKJ20bIViT4PSmw1f1I+H6hzcVF3Bm0zXV2JQA/u3337DqlWr4OvLDbnTuXNnXL9+3agZazWKlEA2/0SPsex/WJ0RAIecn2ALTSWugB1OO4yCuL44wFI58MKvwo1s5wCss+dvoOeVGXQEhDQdM1is4T3JZ3oXWPYwnehzquK/o+ekumXinBBzsMTtv6iChJNWBQmWuP23mXIkoCbqR+eJuNAmFBnwx3GEIl40EasxCwYNgqXrK4GBDeyCggK4urrqpZeUlEAsphGJBquZtVzOvyKMHSoh0bm026Ic/WWOUDnJ6x/b5R0GTEwDOk8E2oUCTv6AZ6jm8ahESFz4G+heDo95XIQYSamIPyJFqch0faxT2Sa9e6+ih+lEn0zFP1FMVnXNxDkh5qDAIwzh2I8M+KMALsiAP8KxHwUeYc2dNX0Po36sCjyBAIkSAyUnsIs9j7lYV39jyckfkIWjRD4R73okYn8pXV9bO4PGYPft2xe//vor3nzzTU761q1bERISYtSMtQoGzFqujW9s15EbQB93oKgKkDkAiuCH47zqCBOkCNaMCav9OgHOmnRCzMFfhQz9DUhvCgPYUYF0CtHHxxkFBqWTlm16dyDiWhgC2KOoGm1EwEfdmzFT9ah9bfwaf2vQ4KZyezluRuzXuzbXRtfX1sWgBvaSJUvw4osvIicnB9XV1YiLi0NaWhoSEhKwf//+pspjyyXQ/c2AOiv0yTIvpJdz066Xav7VaMhkCrmzZp/Yk5puK6/aDXNCzIC4kv9KJa4yXUQKxvg7hxmrNlkeLIlIoDNdKJ20bKvPAyqdMVYqpkkPM9NVw+XOwIEhSlxPjoVDccPGdPx61wtbjvE3rj3tgEhvur62NgaN64iKisLu3btx4sQJiMVirFy5Ejdv3sS+ffswcODApspjyyUwa7muxvU1BGButf7YLt3l0FWFygZNppA7A/GRwP5Rmr9U+Yk5KbflryOmXGjmDtwNSm/tynmX6BBOJy1byi3965OfWolUcx6SX6SET3IUBhZva1AjqRiOWK2ajhSBY+ruStfX1sjglRyHDh2KoUOHNkVeWp8QBXAzpUHDRNQAfkMEZoo2okLCHdvlp1YiiUUhEA9fhwGhSEFsYSIAzb7Kh7OXc8t0hpAQYsZ8IxTISkiBn/pRHTH1QjPX0BEy6E/c06QTXb+jD57GCYF00tr4VCuxnef69Er1o+uT2aln+CYDUA5r2D9cZtYZpdiCqRhxPxG3RfrHROOuWyeD7mCfOnUKqampeumpqak4ffq00TLVatTMWpa/IBgGqAoSZMAfYTiEEZIkZInl6OGuGcNWYzliH528HgpEOmbd14QyUxYDUbuBbVeBg3mav1G7Keg9MX8+Ms1iS8ecJ+J3m3Acc55YfwQdIwvCZYPSWzs3K/6zmVA6adk+ZPzXp+XMdKE2DVbPojPlkGgb1zUCkY4lPMdE465bL4Ma2LNnz0ZmZqZeenZ2NmbPnm2sPLUuUjnAhAPu/1c0AQESJY5JNDOuA5wBBwl3TJsX4z8Z9LXThDKLPak/LozicRJL4SOTY+CkePR9fT8GToo3aeMaABzB/0vUEbQyIZ9OkmyD0knL5lTFf31yrDJdqE2D1bPojD345194sUfH5GkHTOxEC8u0ZgY1sNPS0tC/v/7c/b59++LixYtGy1Rro7qp350KAGqRBINfVGBiJyDc61FlLarijmnzRybv8+2kmnGquQJzNCgeJyENQQukGEKs5l94QyidtGzZAgOpcmGmMxyBOhedqUue6NEx0bhrYlADWywWo7hY/27OvXv3oFbTDPHGKq0SSBe5wEcm15uE2MtKM+Y6mm3DUBxER2SiUnc4vXOA5iQBzZjr2gZWH0Z6tRw/5bgAW+VAzuEmOCpCWob7sBdIN/HAypzDmvq6wbzrbWm1RCDd4Ck/pAX4n810ncEUQCWA722mN0d2GqbWojNVsGrQU64hAIvxaG6IScddW8i5obUxqIH91FNPYfXq1Xrpq1evxoABA4yWqdbmT6tQ3vQL1gOhLIbegjIKnjFt1lCh0sEfkIVrF5KBVNOVrgjWDC0BNI3r/YhAR2TCkRUBJZlAQgRVSEIEVIocBdJNeAXNOayppyWZQKV511uh89mfVrRWQmu03l4Ba50064fp5oZzvT0lhzI0Hqm2I+p93nWRPyJFicgSa665Jh133YTnBr72B2k4g24pLF++HIMHD0afPn0QEREBkUiEpKQkXL16FQcPHmyiLFq4IqVmRnJZrmZcV4hC2/AFNAX2PclafIdz8Mej5eYz4Ysv3dYiRSdofcot4A9r/jFt1i5yYLR+PPLa8a4/uRoD62oVdwemApJjgBil3nMJaW7ZuZp4tHYVuSi3lcE3QmHScdh32sjQviqHJ90b7U2VieQYTT2tzUzr7da2a+GffRreeDQeNQde2Np2LQY1Y75I83AtOCKQzr+AU3OpCQage70Nb7sW3tnc67Outu3kGOQqR8cyoKeVEgoWC6dk7jW/ySJ5NdG5QejzoDHlDWdQA7tfv35ITU3FihUrsHfvXm1aXFwcnnjiiSbJoEWrWQq9drifmynau8vaAlwhxxDRQSxHLLxYHvJEXlgMBcqL5LitM2wxvRg4Zy0FX39BqcgZ/PfaHsW7RsY98M7PeFDYuGMkpAll5yqhTojCwJowfZVAVkIKsk0YSeS6qj34zm7XVZ686U3iwT2BdPOrtwUVgArcYSIqSFBAQ7BbJbW6mrerXK1WGdaF3sSEggF42skRjTj8gufgjFLe59pJvTTXV4FrfnZEIiYnA6+XxMKL5SJPJMPkPAW2vCh//MZqE50b6gqOEB/5WC/dahg8KC4oKAhxcXFNkZeWo+audXYiUH6bu604XbNtWDynAGeJ5XgN3CXNrR7wv/yDB/yzEzPulqFnfXmzcdV0I+mlu9T3TEJM7npy7KPG9UN+6nQcS46Fz6R4gWcZl4Tx1zeh9CZhQfV27J1Y+IMbMcQf2Rh7JxaAab4zYj4q0QZtoBJINx+6wQD81EosRyy883Lhi0zBxnXt+U688bOL03Er8V1sLk3jxgIvTsGnRxOx9tnHvFHQROcGCo7w+MzpB2TLUKRE1Y9RwJVt+o3rGmWarlOhAlxDIvDtdFdf4E33fvBH/fmL2AqIdE5rojaadELMjEs5/2IPUoH0ptAL/PWtFxpQ34zFguptR8b/3cgF0knLVgEb3vQHAunNpXYwgJrF26LZNgx5GEiAl50nZ76TUPzsjvdTeWOBP3/TCLHAm+jcoBscoQYtmtNw9Tawra2tkZ+fDwCwsrKCtbW14L/Whm8CQMnRWFiV1n0hqSxUAj+GY3mRZslYIYM8gTAH7hKzYQ5KWAl8a+KGRBPzDgNGJQNO/oC1i+bvqGRNOiFmxk2tv4IiALgLpDcFoZOkSe9OWFC97SDi/268BNJJyyYSCzSkxbamzUg9agcD4Fu8jU+l2A7YPxX4LVrTcy0QP1ss4l9kqQMzQizwJjo3KIL52x+0aE7D1dtDs3HjRjg7a0rdpk2bmjxDlkJoAsDO8tw6h2lUQQTrskygLBMDAZxAMgaoj2tnH9fwcQC+6q9E+18Gword0CQy4NWSH/DArS9wV//ueJsO/LP39XiHmd3EKEL45FR7ogPP3aOc6vboYKI8WLkGAPf065uVa0cT5eAhC6m3ZYy/4VTK7EycE2IOrLwGADkJPOkNvF6ZSO1gAP5XciG4vPJDakB7LQcAZCQATr6au8e1Jx06B6CN8xO8n4Fv+8eIBa4bQOHF/ZwACo9LzpRIUkfBij0a1jJenQIrZsZL3JuZem/CxMTEwMbGBiqVCm3btsXzzz+PmJgY3n8NdfXqVdja2iI6OlqblpycjK5du8Le3h7h4eHIysrSbmOMYf78+XB3d4e7uzvmzZsHxpp32V2hCQBXVPy/YG/CE7nwhJVOre2Am/jJZjpCPQF/JyC0rWZBmUMvAj6nXofVgxuc/cXsAezunkGVDbd5UWXnA8fwfz3+gRFiRnIF4nTkwtNkeXAtSTMovbXzB/+PADkyTJwTYg4c/cL12qrsYbq5qQkGUG7Lfx3PgD/OoDcYeBpPqlLg3kVu47qNIzD0KziG/wtVjtyFa6ocA+A0qJGhCmsmU17ZBuQe1PxNiNKkG0uqfm+8VenDOWSkQRrcy9mmTRuMGTMGJSWPvzzwW2+9hSeffFL7+M6dOxgzZgwUCgUKCgrQv39/jB8/Xrt9w4YN2LVrF86fP48LFy7g559/xvr16x87H49DaPz0SisFroFbka4hAKGiE3AF/6zenlVHcWIMoIwGTrxUa/WnG0JhjCph1SFYE+/6Ydxrq5cOaTb9Fg38GP6oy4oQCybUtSqU3iRUAsFfhdJbORu9ZUXqTictXOoivbVQRQ/TzdWe9vzX8aGi/XBFYcPXdlWVAhc3AFI5rEYncq/ZoxMbf8dZYDLlYzV+i5Q67QeBITJlZrzEvZkxaBJv9+7dkZmZCX9//0a/4fbt2+Hi4oKnnnoK165dAwDs3LkTQUFBGDduHABg6dKl8PDwwF9//YWuXbti69atmDt3Lry9vQEAc+fOxcaNGzFz5sxG5+NxCU0AcGknx/DsRCxl3JB7WWI5JNUCq12q+eLmAahjdczTucVYG7ALioiHjfF6QgISYol8rIsBnmg6PtaP/0O/oRj4F0sXSieEPMJU5fz1R1VhtvVnTG85opSJUED/Ou5aLRAWT0B5UR7sAM11eJhxouiUF+WCd8BVYxu/fO2HNgJBfx3MeIl7M2PQPJ01a9Zg/vz5OH78OCorDb8bUVxcjPfff19vNci0tDT06tVL+9jBwQEBAQFIS0vj3d6rVy/ttuaiCAbGWB9GVrU3KqqtUVZtiwPqKMirlUgXyfGaOB4Rkv14TRyPLLEcnnZAaZt2/C9mL5Qu3A3+l8oL265qxoEri2HQL9rDeYA8HnDZrPl7mH6QEjMV6MXfVRvoZbqTvNDPXOGfv03DUuptoZj/fCaUTlo2lUAzWj9wn/nYcFHzA9qD3UQojmMc+x6HWRjCqg/jvtjVoNf6vdy45yplMfDbPf7z4n9vejXu3MDXflCV6jeya4ckJPUyqIEdFRWF06dP4+mnn4adnZ3BUURiY2Mxbdo0+Pj4cNJLS0shlUo5aVKpVDscRXe7VCpFaWlps47Dtrl5GNvLI+CLXNigCvZ4gCEsCYqcpzCweiLw+wAAIABJREFU+jDi1NE4Vh2K9Go5jlUPQLw6Gu7h/wTTCafDRG2AqG95umeUONt3G6p0FmwAgFLYoCO7hjh1NFSFSsSehGB4IN1ftIfzgIifgMwSoKhS8zfiJ/O9WJPWrcLtSd7xmxVuT/Lt3iSu6nQV15feFCyp3v5g9Trvd7bD6vXmyA5pZpfR2aD0ZvXwOjzv2gBcZN0wHMmwxwPYoAq+yEEShqK0+7uo1On8F2qJXEMA1hl5SfjYk8AnldNRrLOs3DUEYL5a0bhzg1D7we0JzrAW6hE3jEFDRD777DNUV1fD0VFovUBh586dQ1JSEs6ePau3zdHREcXF3PGMxcXFcHJy4t1eXFwMR0dHiETN18HUZn8MrHh+g3vipmbFJ/YoKH1HZAL3U1B1IgWzreMw78ECuKAQhXDBe7ZbsVbkAx+e4R2zWSKA/fgWr8ITt6GGCNUQwREP8BRS8RRLRShSEFuYCDjz/6LV7c6J2Q+odM4GKqZJV0aDELPi8Dv/+E2H3xcBA94xSR4KITUovSlYUr197cE/eb+z6Af/BEB3v1qbHHjhCVwSSDcjtYZJCEUCs0I1HP/8J/ZjCMJwBG1QDREYrGotj1yJNvgDT+CSKAiLocAgF+M2SKsLldiCqZyFb4rhiMn4ShuNzOBzg0B4QbgEGG1YS2vUoAZ2QUEBYmJi8Ouvv0KtViM0NBTbtm0zaCz2wYMHkZmZCV9fXwCau9LV1dW4ePEiZs6cia1bHwVFLysrQ3p6OoKCggBoVo88f/48goM1ARjPnz+v3dZc7FV3BbcJrfhkVZqOQaKfESCpNfmwEpidGA0f3djZxemYLYnFBEk8/JADAIhTRyOabePsFoh0zLofC0QpNGOuazfSebpz7gmsDlkokE5Ic7Jm5QLpplt3ux/OG5TeFCyp3tqB/7sRSictW3dcNii92fANk+DRATmQPbwm87GGCpdEQXhNHI8AZxg9bvSs+/oxup1RipmiDTiGR7GvDTo3hDSs/UAM06AhIosWLUJqaio++OADfPLJJ7h165bBEwynT5+O9PR0nDt3DufOncPMmTPx3HPPYd++fRg9ejT+/PNP7NixAxUVFVi2bBl69uyJrl27AgAmTZqENWvWIDc3F3l5eVi9ejUmT55s8MEak0TUuBGYXjyB5e1Lr/HuO6L6J8SpHy1G48X4u3H62uVpum1GJdbbneMqEPPfxbwW1SIEAKAWGL9pyvHPEvBPQhZKbwqWVG+FusubN7AqaS5SgehZUvAs792chIZJ6GhIo8mL5cHfSRNXW+5cawPPUFBD9bXjz6du28Kgc0MD2w/EMA26g713715s3rwZI0eOBAAMHz4cPXv2RFVVFaysrBr0Rvb29rC3t9c+dnR0hK2tLdq2bQsA2LFjB2bNmoXo6GiEhIRg+/bt2n1nzJiBjIwM9OjRAwDwt7/9DTNmzGjYETYRkXMgUKR/B0uFuj/UPJH+hIe2uMW7rzNKEM22IRQpiFQnIk8k471K2Uk1r6kUyRErjkeuGJCJAYVIPxz81qGa8Vm1u5vbiDTphJibDHRGV57u5XR0RjcT5cEcGoyWVG8rIYYtz08gTTppbWxF1byVxVZkZtMchYZJNEKeyAtyJ57GtREifdlJZeBrMtRuWzTm3NCQ9gMxTIPuYOfm5qJfv37ax927d4e1tTVu3LhRx7PqtnTpUsTHPxrbExkZib/++gvl5eU4ePAgZ/iJSCTCqlWrUFBQgIKCAqxatapZx18DgK0n/+ixE5JIvYDyNXIkAVhcawyin1qzDKkbhIebAJphIKvEsfjSUYH7Djqv/bAbp2ZlyW1XgYN54EYYqSXMC0h+QbOojYu15m/yC5p0QsyNq09f3nQ3gfSmUAb+OSdC6U3BkurteSv+BUSE0knL9gD8t1KF0ptNiEJzPX1M16C5znvphvI1VuzqEIVeG+MaArBUpIBTm8adG5TFwORdSgy/HI3Y7HAMvxyNybuUeu0HYpgG3cGurq7Wu1MtkUhQXW26LlJzkx2kgPpaCvzUjypMBgJgHbkBVu2gqTSF6UD5TcC+PUrsAxB5XaGdhOCnViKJRemNpRLycvs8vDxaDhQlPlweNQ+wcgZEDNg/FXklMqhKFECtJdfTizUzjuMjua8V5mV+E6MI4VPZXwFlTgrk7FE9UYoC0Ka/6cYGahoC+vMqKmADZ/3dm0yYgxLKdrWWRnZQwBzvMX0nfR997xzkTPyqggTfSd9HSDPmizSP+8yat57cZ6atP/WqGSZRc321dkJplQilOSloL9DLDACZ8MFZ9IEUJdp42W1c5PpjrxsY6ash+bQanYiSo7G4fjMPN0Re+Lm9AomD5Nw75oD+cuohCt675Z8eVWJzca32CANCi1Pw6dFErH3W/M4xlqLBUUTGjRvHCcVXUVGBSZMmwc7uUbjz3377zbi5M2MLLslxlCViuYgbiH5QlhwhnaA38/aNJHCmdPyL/b3BjWsAj6KB1ASr1+luGggg6eFQkqxajew8gRUnCbEECy7JcRQ89eySHPHG69GtU7nAwIYKUw54sKCFpAYVbuA0rgFN9IWnCzcAtSZhkdbhHHpiOJJ40ntgeDPkp046i8E4AhiyWYkfyofAH9e16aWwxwX0RIYoQLsADaDpXXrOTzOxUa+xKzQEpTELt0jlcHouHkEAggBE8u1jwDnjuZv6EycDkY7nb8YCoCgijdWgBnZMTIxeWnR0674FmlsGZInleE2n8HWoNZ9D+fAOcm4ZcLHW4k9+aiWewT7e1y2EM8RQcyOR8M3m5eluCkQ6liOWkye9bipCLIhQPetowh+O+fCAL/TvPuWjLXxNlYm6upfNLIxWezX/nTpPtRkG7SZNTiIwW0Eo3dz08QCQzc3rHbjjVdG3nJtZgGZiIW/jGjB9pA4DzhlCARQ68ARlIA3XoAb2119/3dT5sDhCS6X/ee/RuOeo3ZphGrX5qZU4zJ6GvUDIqp9FI7EYCnxqFYthrnmaCYx83ToC3U21ZxI3RYggQkxJqJ6Z8oejp7iIN2yJp5g/OkKTMFb3sgmUWcvAd3orszbDAeOkyfUT/8Fbf/qJ/zR9ZhphTZtYOCGbk+aPbL2bWYBmAajBu4A+bYHiSs35S9vg1h2C4iBwbTcWA84Zvu1lAE9Aky7qNE20k6bMZwtm0EIz5BFFMJCgBNwrlViOWHixXOSJZFhcqUDsSU1B5GtcH2SDee+GAcAD2GKPpwK9HOT4msVjbZXwbN4SaxmceF7DxtkL4VJNA0TwlzQhFkIRDKTc4tYlU/9wLFHb86YXC6Q3CWN2LzexLzEd4fgO1rVaVZUQ40tMxzPNmC/SPBzFKt4GtqPEzKKICHAq+//27j0uqjL/A/hnGC7DdcALIIg4QIqSmmWoSSApuv1227J+lQUGqeVvd9st11IzJ1un7bY/S1ur9VLqT6y20tQutqLmtdAsr5iKMCCCoqLcERjm+/tjYGCYc5hB5j7f9+vFC+Y5M2eeMzznnO+c8zzPV3ga3SGUh/XajPZzf2tXkZI6oKTDHbYtauCb37aOfbLlTB3dOGYEJqnQfCUXXp3ycXg2XgbObrBrd7SOPQEinSyu4QD7JimCgLQgNd660mlgQGtmxUtexg3xVSgxsNM34Y5OBEzCa5MURle+c8sN59NUVwNZFSp8iFyDflPNAbEYd78Ku2yXYI4xq1IE6dq+8pBuPIE9vjhGoVCwfIBIuVU4USKI6Y1vGwTXAOANLWY0vg3ug+1+rmt9ESZU3iITLHc4DcIDHIfjJO6go7oHref+idocADC66PbbbxT45rfA9O+7PrdbVHeOGa0DJ3FQCZTsMN5mO3VHa5sdzWafmYVxgN0DcxqFBwY8U6/EBxHGDTGGxAc1agGs9pyD2kPGV747zwaiPATsrVNgoiSndUfWDf46HKrCUr6Nw1yMIsh4Jhxb8kZzt8qtwta3l3tgDH4ULE9Ero1rwhzBBW0owgTu2urKHd8Nn3DIaoqMyr1geAU+DgXYS3dDC4/2C2ltgXdTDjJ3KVBUY7gOsZm+LKK7x4y2AZ5fpgKlAl8q7NAdTWlGPOTIOMA2g9gtitt9SwGBeSJv9y0zurUdrVVjeBeplT0A/KZuJZbVCV/h6TgbSGmdbn0G35KhQkyz451sGespe98ilIrkjRQrt5pOMxw4KrkXgCbj8mBv4zLm+sJFukSKlTuan5tiMc7ML4dC3T/jUIB3aDamN24WfE3Hc7vFj3U3c8xwoO5oYrFOWZ1zxDocYJvQ5S0KkYxKvvIIg1vbLZVqLLt6DwI0DV2+VyTKzBrUNcJLjVVk3DVluVcOHHFeXMZulmPcInSO2Q4cRXPoGHhf2Gpc3nc0OMZ2P31EEqn1wTUb1+TmLPdTIaw6t3vT6nYyGduR4KnGgSbj83Pbud3ex7q24L6lWoV3vHIR3izStcTMubUtwdljHbMyObqzrm5RYLQK8Ov0rc4vQt8QFUFA9p1q/F9tGkI1RSbfq6AlAk8P1Q3i6qjzoC4VCXdNebG5mxmhGHNwykNA+PW9KGhR4FpLMApaFAi/vle3/9mIWM5YiY0Db3U1kLEDSN2i++2oWdZebZ6DZkgNypohxavNc+xUI2ZPnedEby93wEGOF/YC6xTAymDd7wt7IQ3WdcfMlqRjF1JxCaHdXq0fGrAxWNnlub3LWOMmmXvM6JgJ+tMrCozR5OBTj3Qc9krFgaB0lExoHeBYpUbzl2m6gY+lu4GzG3SPqwSmILEAsVhHRc4R6/AVbBO6vEVRUwJt/SWDbyna+kvwqCnRf6Or2a9EYK3pb77nEIv5pELSKdODugKbhG+tFZaVob7aOTr/M2aOXlf3Yg0m6E/GIajC95iAOVd3wp0GzNn76lZ33HZFONHMbVc40Qxr53D3hS7shXbLBHhQa+DfVAXtlgl4K20nksuTMa1a19VivTYDGbSh26sP05Z1eW63dHeI7hwzOgf3xR4KPIZs3eCwOiB2t+51fQTiGa/aAl2c81vLd18Ti3UCmxxvelIh7hVg38Stja5uUTRsz4Rvp36YHtDqyqfrvtGdv1SKBBPVOo9ITJToMjDG1JkxqEukj1SBNgLvOUnnf+YkbHg7UMjCa48bXenyggYLrz0O4IJN6tAET8gErrbpym3DmQb7DNAKX1CIEilnrq0FHvAQGK+ghUen+xz21bA9E75kuJ97kAYh+zOR899qLNuvxu8uKTGg5RxaWgIg1dQKrqcWfghAvfEC/4guz+2W7g7RnWNGqYnEXW2ve1Eknjl/qcxknHNTHKg/+M1wnwD7JlMNq0iJQJFbFM31FfAVeE1jfaW+vEwS2WXDK0IUxkv26DNCmZVAY7QKpQW5iGxpr9c56FK22jLDHXNxDpCeO1h7uVvl1rAXKZiEnSLltiF2AixzwP29v8hUpFFdTFHKXFcDfOEF44aqK3ccTfXXRc/nClJj6eXxQH17unR4+uFG8HDsrQxDnYYgRw3KJBH4yu9pZEumG84pbcaUml3FGjeTrrw7xwyxsV+dXycWz1yURFgnwHai6UmFuE8f7K7ShnZB7BbFxYsFkEF40GKlJFj/9zfhKpxDrOHbIgA/YAyyJekGwbXZCTTkCvwjur1PWLYkXX8FnFOjM4u5yX3GkrQiPaDFyq3hZf9VKEE/g7IS9MPL/qtsVgdHyGhpLl+R46KvSPZa5tqqENytcnu5LgkRLK+UBAP7nwNqzxsu0NRDFhiKWx7fjI3xW/Bq1C58NzgbbzyUrJtTelA6EJmq+23GRQlLd4cQO2YEeRn3y1YlGo/96izCXzieOYdYfB1upYC3barBbn6WjsJ9rmDfbKphkVsUnjcuwVvgtnELJHjedx2+aH38bJICWVdz8FRN+3zVbRmfACDAExgTAsQGd29KnmeTFEi7nm3XDHfMxTlAeu4qBEMG46vVunLbeC5Jgbu3HzCYc34hVHgjyXYHeUfIaGkuf29PwWn6/L3d53TD2p3DEEQJTF+nK3ccz/uuw8f1EwzO603w1J3PLz0s/KJLB0W6fdh/ejyhY0aUP3CkAjjfoXdLW7/stv7hBZXAyetAbYfwpv1YYxzPrApUYa01j4VOMj2pELc54omlFq/xjhAs1xutAkr3AnXttzeLEIVLCEMMioyefhQj8JfftA/kUQQBax9QQHkoG2V1um+PIyRATFPPstI5QoY75tpuep+xIFmwAqg0DrBlwTE2qgHwdXHHR2RQPvUW29TBmfb3o9KxGIstAuVjMNYO9WH2NbKPBrhqXH5bH+HZRezlL79JRtqmnViDTASjEpUIxpNYB9VvkqH5VjhY0pAZQZS541gs3B1C6JhR0wxsLTJ8Xsd+2W1fFNqm7Ot4rAF0ZU3+unTv4TLdhcG1DnoccgRuE2ArJSo80ym1+DnEYrlEhaUCz99bBmTuAgIbgG+aJJ2+aUtQjnDB9ynyTMDtAYZl1spEZ+8Md8y1dXefsQZ5aBxQeVCgPFbg2dZRX6HGDoHBR3MrbDsXq7Ps769q/orN+NpgJpFmSPGq5q/4xo71YvYR3CtSMMAO6eVYA9WSIwDVg8mYsEuNykYg2AdYd4+u/LB0LEaJfGkcJbCu9vhBja1NaRhIZoxj6ZR5scY7AkqJCsd2K2466UznY0aq8SYAMO6X3fl1QjOSSCXAxxxcd8lt+mAfazacy7Kt3/JxgeyHe8uACVuBohpgbpMSUWTY92ogzgMgwb5Ic1pUeO6ANbeEMdvozj5jNaNVaA4w3M+aA2w7yOXxa8JzsT5+zTnmYrW1xzXC0/Q9rllppxoxu3KAfdhcyRGAOgO4PkP3O7n1O8CSwHdQ1OkyWxGisCTwHaN1dI4fDIJroOtxLK3dIdQTdmFkZTaWFSuwu0w3P3XaV8LzWHdnfvybHcshNiPJPVsddz5+R+A2AXakv25ux2ke2Zgg3YVpHtmCgwLV1cBvvgZGa3TJLR6lTwXXF4bLgsFHsYcC20u40THnZ+4+Y01qiQKPaj5CIQbiOoJRiIF4VPMR1BLbBfm9WoT7ovdqcY65WG2tHwl/Xv2IPy93pJYokEGG+3AG2XYf7qkAL+AIRuISQnEJYfgS92O8ZA/q/Y23IXOX6fjB1DgW5SFAU6nGem0GdrakYr02A5pKtVHSmY4JYswJxGubAFmnuRHNGcshNiNJUY34+zE3CrCFRsl2blhtjfX2pr3YhQmIQZFoFqrhOA4AmOaRjemSDwEAH9F0rNdmIKzZeEdgzNmYs89Y2+u71HjrxnTEoAghqEQMivDWjel4fZd1MocJ8UCjSDnPiiGEPy/W0bL9avy9znAf/nvddCzbb7t9uEeq1Fh8MQ1TsBXhuIxwlGMYTgIAJALZcgbXmY4fTA1cbKnUdUvLoA24B7uRQRuwg9LQUtn+malbryCbk/2xLbbZUgzcaK2STAr8fqB5yao6XvmO1poO/JmO2/TBNmeQUNttkO3IFJwhpKMA1OMMDcbVll4IRiX8204qrf0zlZW27Z/JmKU5wsC6SZeEu2dMvnRzc8PejDtxpFvl7o4/L9bRb0X24d/ZcB/ukYNK9NMY1/9VKPFRs3H9V2m6jh+aA2LhZaJ7zDP1wp/ZM/W6z6wtYC6qEX595z7VbbFNtFaNd2g2xuJHoAW4UD4GCloKU7GKKhEoKVPjuernMBnb4df2ZZnjnS65TYANmB4k1HYbJATXzVqfD5oRiXKj8o47AmPOzN4D68K0wt0NQrW2624gNn8zz+ssjD8v1pEj7MM90VBVKpiAJoLKBLvLhUuvQyi+boYU/5ZMxSoPFdZKFF2Go7f7lgIC3S5u99V9ZkJ9og3q1qlebWnYd1MKBnZI+BReuxX48igwZXeXc0srSI0d2jR4dQr6AY53uuI2XUTMEemva4SeJq5em6NtR2CM3bw6ifDl8jqJrSYKZIz1xDVP4fmdr3k61iwiYn5pEK7/ZY8Iwe5yXr7CCWtKEIVpHtnYW6cw2aXCVy78nr5y3WfWVWpzoW58kf7AO/ScQXCtV3veZPKwmv1Kw8yUndzhJb7MnXGA3cHrQ9TYRWkIFEjr2l1tO4I1lZSqceD/MvDLylQc+L8MlJQ6SZ82xswUhPpulbsyZ9nfxXJs2i73JnMkKwJUKMIAg7IiDMCKAMebRUTIcj/h7IWf9VYJd5ebsA6QGHYOaIInnsA6/WOhdOUGRqt0c2B31GFObLHZQAYGCvepfn2IGvdiu+jbNVSJXxBUVwMnz4skHWvlceNSl8udWU+Ou27VRcSUqDwlIHALpNs8A6w+BVFJqRraLWkYp22tbxNQvCUXJffnICqS+0Ix13CL9rhgeZz2hM3q0ABf+Auk/9aV24Yz7e8E4WBarJy5tgAvoGOCJh1qLXd80mAFJpbnGGVyTeotst/1Twbu3wnszERtbSUuUzCewDockLYnoDM5E1OnObHhH2GQoEYss6vYgEVdbCPeReuXhgiME1mmPAQ81NL1wJsLzcGwXeqvnmlLolNaB5Pzi/f0uMtXsDsSSw3dHZ4BwO++6bI/kyWc36lEtNbwy0C0tgDnd/LcvMx1SEm4u5anSLk1zMPrAuGBrtxWnGl/r4G3SLmPjWvCHEHGdaVR14SBKEHGdcdru0JUiYBnsOF0pZ7Biq5nU+qfDGSqcWXadUwKURsE12bPxNSWInzKLt3vDjFF2wD09FuAqX3V2O+fgRPeqVDkZugyR3bWRWxThAFY7id+QbClUo2RJgYohzefE35fB9OdaQ2Bnh93OcDuyF+435MppQjFLqTiQFA68Nhx3c5lZb43hHcY30bu+81cxw2J0PAi4IZEZrM6TJLsMrryKmkttxVn2t+9RE4rXnz92i0FNgm33cBmx2u7QjoGs6kRut/mTG3X09eaXDepkX3jAXxyZSjGVW+Ab/lu4OwGYEuacbArEtucR3+Ml+yGNFj8guAz9cZfkDrzo1qT/bgdgVjCHLE+8T097nIXkVbqamCZRoX50n0Ibzlv+gUdnEECJkh3IVUO7JJbqYKdNMgigSaBch/nGDjCmDkqpFGI0Bgf5K5Ko2Crlp4M4dSsd4uUW4Mz7e9eaO5WOXNtztR2xfRkNiWrzMRUpdYF0tUCXVrbMkVO6jCrx2gVLqlzEd7c/vxziMVESQ4qvLu+Gp/gbeadfZHkOd3pkmFtYoNDxfrE97Tt8hVs6BpAymZgWbECP7aM7PbrB0KNaK3aphnuBkxQodjDcBBEsUcsBkxwjoEjjJmjQhYrWH5NpNwafKlWsNyPej4Y2lzOtL9LjDrUdF3OXJsztV2ncVApHFy36RzsyhUovicHn3qk4weMQSEG4jL64g2JEjvGq7sMePOazLyzL5A8p7tdMqytbaa4jolyuordetp2+Qo2gNn7gZLWc6UcVaLP00KCFp8weDUajpiNQRH2SNLgMcR2k61HRSpQcn8ODuxUwrexDA0+ERgwQeVwA54Y64nNsqdxV+2/DRI3NMETm2VPY7yN6nADPvARyE6oK7cNZ9rf6+ANucCAqjr4wEY3+JgDcaa26zRMjRfrFOyqq4H0QwpoSIUdSEMMihCDItylzQV+zAVCc0THjS33UyGsOtcg8U0TPA2OycUesfBIUCGq02u76pJhj/wKrw9RQ3s2rb1fNQF3e+SKxm49bbscYAP48XL732WSSOMBz608QPjFKxGjvI8DNUUGy6K1BUCeEoi03WTrUZEKRD3Bk7sz15VUudIoK5o3NLi7ciUA6491AHRzccvJ+JJLnSTQpgGjs+zvniJ9rflk476cpe06ja7Gi3WYzq9NW6C7HsYZIgW7lHQgNIvKv+hp/I9kZfusKqRC0q8KZHeqVne7ZFhbVJ4SEBi02FXs1pO26zbHvJJSNc7vVML3RikaZJGi30IWQoUxyDVuhK08mmuAPgONAmwAon2QGGM3J1wkC1yYDbPAXZX2R4TmgnC5zWrhPDQipxUNnGReNsYcTOf4RTHmaURcyjXsJiKVAQMmAUlLja5GtwW6ESRy5buL2EWVCKSVKzCt2jDIPNDpAkeMQNAsNl+3LbvTGhC78m+l2M0tAmxTcxmOCQO2FukWFXsokNXyEb7DZAQI3OZs8ImA6OS3An2QGGM3T+ITBIEpqCHxsV0mxxq/WKA6V7icGamVBEFONQLltr3iz5grEIxfduSibOJHiChaKThPdmdtga7oHXqh2KVKDRxUQlFXiqP+cpxuJkibqlFMkXhOo0Kxh+F7CQXNYvN1mzVNoTWIXfm3UuzmFgH2+Z3K9sbZKlpbgAM7lYh6IhtLxwFXy9X4Q50SMXQOw3BcMLi+gChd5/YAAJ2/PQrclmGM9YxUpLuBWLk1xA3/HWj/BoN3pNZyZqxaEopIgStl1ZJQ3NxEqIy5L9H4JXclIszsutAW6C6sFLhDLxS7dJqlJADAqNZFIwGM9MhFijZHH2SLBc2KIOD78bqr7343SlEvi8SA8SpEBdmp//1olU1jN7cIsE3NZaggNXa3pMCLup7rMaT/SPi3dStpzbJ041oBqisv4XJ9H1RtUfLgDcYsKFBTLlgeIFJuDWHHXjQqk7SV3zbVZvVwFjHavG6VM9dnbhdNZswSc+B3DHQbG/riMrUgSB4OWe9Y4SvfJmYpidYWYEOQEkp5NiK6mnqvSo2onWmIqm6/+t745UYcDpiEsLSltm8DJjJkWppbBNgm5zLc9TS8GroOrgHAv+NtT7kCJQkqaLekIVpbhFAUAdUHHTZ9MWPOKFRr3PcZAMJEyq2hpaECUsHya4Ll7s5L6GALngfbXfU03bS7s8g84p0DXQCovAQEhQq/Z1UphFN8tRsXWIZd95t4kkCg7oMbGFW7FcVb8uzTBtoyZNqAW8yDLTSXYQFisS5IhZJSNVBqZka2sr3AmU/1D50pfTFjzsi7pV6kXKBjtpU0a4QDw2aNcCDp7sRmuyaeB9st8XmyZywyj7jQFemWG4B6q2Dmx198XJFeAAAgAElEQVQaTHfmqvGOQMYOIHULkLFDZG7rLqYTtHsbOPMp8K8A4D1P3e8OsZ2luEWAHRWpgMf9OdgdkI7dklRkS9IxQZKDVRcVOLBFCUBr3oqoBch5TP+PcKb0xYw5I4nI1VCJwLzU1tIkkq69UaTc3WlFruuLlTPXxufJnmmLXw4EpeMXn1QcCEqHh5lXftXVuuD3l6Iu5s1um6avg+V+KpyD+CDu5oBY/K5CZTqBTFfTCcKObeDMp7pYTlOni+s0dQaxnaW4RReRNpWNQCg1IBm7cQh3QtsihRQtos/XQuQbyPczgcFTXSIFLGOOzFdoCpEuyq2hReIleFlWK/G2WR2cidgxtatjLXNdfJ7suZuZi1ldDWRtVuOpGiUi6FTXT26dpq6tr/yfaktxArfiBBIgRw2qEASAMMi3BglREXhBo0KxGliPDERQKcokkVhYqYLykMIwgYzQoMIO7NYGvp8pXj7YcuNq3CLAbusD9oBWvNN+Z7Xwgw8a4CF0Zm3RzTAyYIIKxVtyDW5/cQpYxizHQ+Tukli5NRR5D0fIjR0C5cMQYrNaMOac+DxpH8v2q/FhdZpoTo+OKi7mIf/DB9D/xhGMo/P68nOIxUSJbraQ2CAg5z4AQUD5F2rsoA7rJmAMcqGs7JQRsXVQYe33syG9sN3gwohd20CL8SxxXZbfJLfoIiLUB6wrGmkA3u6/DU3wE1zeLJEB6NmtG8aYc2hsEe47LFbOGGvH50n7+O0lgayNAFoEumr11l7GmIYt6N8huAaAOBRgiVSJ9Ft0wXXbTCHP1BuvOw4FeKZeoE+1XIGABzbj6pQ8h2kDbTGcueU3yy2uYIv1ARNSiIF40n8X1t6jwNPrVyObHjOa/zaLVmND62NOAcuY9dTDB4ECc9Lrym0jpvmESPlJG9WAMefG50nbE8va+BPuxDlJLCbSDoTD9HSnCmmZYbcPALf7lgICgxpv9xXvU+1IbSCLViMbXcd2luAWV7AbZOanNyiCAnvrFFAeAv4tnYqH8Qlq4I9mSFEDfzyMT/BvKc99y5gtXEF4t8qtQXxWDCakReS0IlbOGLO8AeHCcc85SSymeWTjFIaYtR6hftK+cuF1+8qdo1+9rWI7mx3xGhsbMWPGDERHRyMwMBAjR47Etm3b9Mt37tyJ+Ph4+Pn5ITU1FcXFxfplRIR58+ahd+/e6N27N+bOnQsi809vQtPciCmT6BpIWR0gkwIbpVMRJK2Ft1SDIGktNkqnQsaD4Rmziac81qGp0422JnjiKY91NqvDzx5jBct/8Rhjszo4kwMY3a1yxpjlBSap0BxgGPeUSGOxELp+z2US0xceRftJj1bpMiB25ETZrG0V29kswNZoNIiKisKePXtQVVUFlUqFRx55BEVFRbh69SoefPBBqFQqXLt2DaNGjcKjjz6qf+3KlSuxefNmHDt2DMePH8fXX3+NFStWmPW+6mrgxV8VeK5XDjZ7peO45xhc9YhEk2cINJ02/xzaG1+EP7B6vPA6xcoZY5b1VFoy7sFOFGIgriEYhRiIe7ATT6Ul26wOX0a+gyJEGZQVIQqbIt+xWR2cyZLeG1CGvgZlZeiLJb0tefOVMdYluQJeU3KAQelAZCowKB1LotvTmy+E8VR8JYhCrt/vTfeTbsuI2GHduD+nRxkR26YU7HJebQuxVWwnoe5cCraw4cOHY9GiRaioqMDatWvxww8/AADq6urQp08fHDlyBPHx8bjrrruQlZWFp59+GgDw4YcfYtWqVcjNzTVYX1VVlf5vuVwOdbVubsaCDv+otpGwiiDd7CKFOUpQXRkuIAILoTIYLasIAj7NB2buBm606L71rB4PTL3F2p8Ms5fObchV39NZqKuBxM+Bqx2m+erjDRx6WCQ1r5XqkLFJjT/UKRFBZSiTROADfxWyH1TYrA7mcJS2u7cMyPxSDRXaPy8lVFg3RYFk57iDzGzIXsc/dzzudo6JorVqvCFRIk5ahkbfCLulsDcVq1mDJWI7U23IboMcy8vLcfbsWSQkJOCDDz7AiBEj9Mv8/f0RGxuLvLw8xMfHIy8vz2D5iBEjkJeXZ/I9lIcM/2GA7rHyEJA9sbXTfVY21NXA6kNATB2Q5A+oEtv/qVNv4YCaMXtRHjIMrgHd47Z92BYUQUD2gwooD2WjrE53dys70XYBvrNZeQoo8lBgGrKNyjnAZsx+FK1Bq/IQWo9lCoxOzLb7scxUrGYNtojt7BJgNzc3Iz09HZmZmYiPj0dtbS369jW8pSiXy1FTUwMAqK2tNfh2IJfLUVtbCyKCRCKBmNI64fKyTuWKINudrBlj5jN3H7Y2PkaYz1H+Z4wxY454LHPVY4bNh3VrtVpMmzYN3t7eWL58OQAgICAA1dWGX1+qq6sRGBgouLy6uhoBAQFdBtcAEOmv+x2tVWO9NgM7W1KxXpuB4V5qC24RY8xaIv2N999orRoR/vauGRPD/zPGWHe46jHDplewiQgzZsxAeXk5vv32W3h5eQEAEhISsG5d+6wAdXV1KCgoQEJCgn75sWPHkJiYCAA4duyYfllXVIlAc8lerKr7LYJQ21oJ4NHLuUBVzzrkM8as7/UhakjOjG9PgEDAeMk+0JDdMMgYxhwG/88YY93hqscMm17B/sMf/oBff/0VX331FXx9ffXlU6ZMwcmTJ7Fx40bcuHEDixcvxvDhwxEfHw8AeOKJJ/D222+jtLQUZWVlWLJkCbKysky+n4LU+LixQ3Ddyqu2ADgokHGIMeZQoo4+Z5RdrD+dR9TR5+xUI2YK/88YY93hqscMm13BLi4uxooVK+Dj44Pw8PYkEStWrEB6ejo2btyIZ555BhkZGRg9ejQ+/fRT/XNmzZqFwsJCDBs2DAAwc+ZMzJo1y/SbHlRCqqkVXlYnnnHISJVaF5DXlQL+kbq5HvnqN2PWdylXpPygbevBzMf/M9YZn0NZV2xxzLBDG7RZgB0dHd1lcpiJEyfi9OnTgsskEgneeustvPXWW9170zrxFOk13hHmpVquUgNb0oDqgvayS7k9nvORMWaapqVF8CCladHYbwok1iUNCZ9YxMqZi+NzKDPB6scMO7VB185d6y+cqagaAVBKzMw4dFBp+E8BdI+5iwljVlepkXWrnNnfUalw5sujUs586Zb4HMpMsPoxw05t0LUD7NEqlEoNMxVVIwD/hW9wvNnMby1iV8G708WEMXZTyqVRguWXRMqZ/S0JFM58uSSQM1+6JT6HMhOsfsywUxt07QBbrsA/onOQLUnHLqQiW5KO4ZLjOCBNNn/6F5Gr4PDnjAmMWVulb6xgeZVIObM/abAC4yV7DI674yV7IA3m7gBuic+hzASrHzPs1AZdvkvcs0kKpF3PNkrBqUo07/UlCSpoz+UiWtt+e6HYIxYeCSrwNTTGrMtzrArn/pOLOLTvf+cQC8+xZnbxYjanSgTSyhWYVt2eybE7x1zmWvgcykyx9jHDXm1Q+sorr7xixfXbVGNjo/5vmUzXRzPEB7hvIHD1BtBHBowLB9bcY36a4z8dCsHiq/ehj+QqrqIPDkjG4QmsQT4p8GCMFTaC2ZVQG3LF93QW84+E4NUKw/0vS7IGlzx5/+vMUdpuT4+5zLWYOofa6/jHx13HYe1jhrXiOFNtyOWvYAM9Sw1aWgcUeygwDdkG5TFOnsKTMWfA+59zcsR0zMw+eB9m5rDmMcNebdC1+2BbQKRIX21nT+HJmDPg/Y8x58b7MLM3e7VBDrBNUCXq+gJ1xP0JGbMN3v8Yc268DzN7s1cbdIsuIj2hCAJy7gOUh4CyOt03HlUi9ydkzBZ4/2PMufE+zOzNXm1QQl2lV3QyVVVV+r/lcrkda8KclT3aELdbZgncdpkzslcb4rbLespUG+IuIowxxhhjjFmQy3YR6fjNgjFnwe2WOStuu8xZcdtl1sBXsBljjDHGGLMgDrAZY4wxxhizIJca5MgYY4wxxpi98RVsxhhjjDHGLIgDbMYYY4wxxizI5QPs5cuXY9SoUfDx8UFWVpbBsp07dyI+Ph5+fn5ITU1FcXGxfhkRYd68eejduzd69+6NuXPnomNvmqKiIqSmpsLPzw/x8fHYsWOHrTYJANDY2IgZM2YgOjoagYGBGDlyJLZt2+YS2wYAGRkZ6NevH4KCgjBo0CCsXr1av8zZt81c165dw5QpU+Dv74/o6Gh8/PHH9q6SzZlq5+4oPz8fMpkMGRkZdq1HV8fWzt555x2Eh4dDLpdj+vTpaGxstE0lbcTcz2Lt2rWQSqUICAjQ/+zevdtm9bSm7u6rrt4m7GX8+PGQyWT69jV48GD9sq7Ona7GWrFft5CL27hxI3355Zf0P//zP5SZmakvv3LlCgUFBdFnn31GDQ0N9Pzzz9Po0aP1y//1r3/RoEGDqKSkhC5cuEBDhgyhDz74QL98zJgxNHv2bKqvr6cvvviC5HI5Xb582WbbVVtbS4sWLSK1Wk0tLS301VdfUUBAAKnVaqffNiKikydP0o0bN4iI6Ndff6WwsDA6fPiwS2ybuaZOnUqPPPII1dTU0L59+ygoKIhOnjxp72rZVFft3F2lpaVRUlISpaen27UeYsfWzr777jsKDQ2lkydP0rVr1yglJYXmzZtnu4ragLmfxZo1a2jcuHG2q5gNdWdfdYc2YS8pKSm0atUqo3JT505XY63YrztcPsBu89JLLxl8yCtWrKCxY8fqH9fW1pJMJqNff/2ViIjGjh1LK1as0C9fvXq1/p9w5swZ8vb2purqav3ypKSkm/4nWMqwYcPoiy++cLltO336NIWHh9O///1vl9s2MbW1teTl5UVnzpzRl2VkZPBJiNrbuTv65JNP6OGHH6ZFixbZPcBu0/nY2tljjz1GL774ov7xjh07KCwszAY1sz1Tn4UrB9hCxPZVd2oTtiYWYJs6d7oqS8Z+3eXyXUTE5OXlYcSIEfrH/v7+iI2NRV5enuDyESNGGCyLiYlBYGCg4HJ7KC8vx9mzZ5GQkOAy2/bHP/5R35WjX79++K//+i+X2TZTzp49C6lUikGDBunLHLWuttSxnbub6upqvPzyy1iyZIm9q9ItQvtkeXk5Kioq7Fgr+zly5Aj69OmDQYMGQaVSQaPR2LtKVtHVvsptwrpefPFF9OnTB+PGjdN3QTJ17nQXPYkhusttA+za2lqj3PFyuRw1NTWCy+VyOWpra0FEJl9ra83NzUhPT0dmZibi4+NdZtvef/991NTUYN++fXjwwQfh4+PjMttmijPV1VY6t3N3o1QqMWPGDERFRdm7Kt0itE8CcMu2nJycjJMnT+Ly5cvYuHEjPvnkE/zjH/+wd7UsztS+ym3Cet58800UFhaitLQUTz/9NO677z4UFBTwOaVVT2KI7nLbADsgIADV1dUGZdXV1fqrm52XV1dXIyAgABKJxORrbUmr1WLatGnw9vbG8uXLAbjOtgGAVCpFUlISLly4gA8++MCltq0rzlRXWxBq5+7k6NGj2LFjB2bPnm3vqnSb0D4JwC3bckxMDBQKBTw8PDBs2DC8/PLL+OKLL+xdLYsyZ1/lNmE9o0ePRmBgIHx8fJCZmYlx48bh22+/5XNKq57EEN3ltgF2QkICjh07pn9cV1eHgoIC/e2szsuPHTtmsKywsNDgm1/H5bZCRJgxYwbKy8uxceNGeHl5CdbdGbetM41Go98GV9s2IYMGDYJGo0F+fr6+zFHram1i7dyd7N69G0VFRRgwYADCw8Pxv//7v9i4cSNuv/12e1fNJKF9MiwsDL1797ZjrRyDRCK5+RkKHJC5+yq3Cdtpa2Omzp3uoicxRLfdVM9tJ9Lc3EwNDQ00f/58ysjIoIaGBmpubqbLly9TUFAQffHFF9TQ0EBz58416Mj+wQcfUHx8PF24cIFKS0tp6NChBoPhRo8eTXPmzKGGhgbatGmTXWajmDVrFo0ePZpqamoMyp1928rLy+mTTz6hmpoa0mg09N1335Gfnx9t3rzZ6betOx599FGaOnUq1dbW0v79+91yFhEi8XbuTurq6ujixYv6nzlz5tBDDz1k17YrdmztbNu2bRQWFkZ5eXl07do1Sk1NdbnBuuZ+Ft9++y1dunSJiHSzIyUkJNArr7xi6+pajbn7qju0CXu4fv06fffdd/r2l52dTX5+fnT69GmT505XY63YrztcPsBetGgRATD4WbRoERER5eTk0ODBg0kmk1FKSorBdEJarZZeeOEFCgkJoZCQEHrhhRdIq9Xql6vVakpJSSGZTEaDBg2inJwcm25XUVERASAfHx/y9/fX/2RnZzv9tl2+fJmSk5NJLpdTYGAg3XrrrbRy5Ur9cmfetu6oqKig+++/n/z8/CgqKoo2bNhg7yrZnKl27q4cYRYRsWNrcXEx+fv7U3Fxsf65S5YsodDQUAoMDKSsrCz9FJyuwtzPYs6cORQaGkp+fn6kUChIqVRSU1OTnWtvGV3tq+7YJuzh8uXLNGrUKAoICCC5XE6jR4+m7du365d3de50NdaK/bpDQuRC96cYY4wxxhizM7ftg80YY4wxxpg1cIDNGGOMMcaYBXGAzRhjjDHGmAVxgM0YY4wxxpgFcYDNGGOMMcaYBXGAzRhjjDHGmAVxgO1gJBIJsrOzLbpOrVaLkSNHWjwl74kTJ5CYmAiZTIaBAweKPm/WrFl4/vnnLfrezHbGjx+PmTNnWny9WVlZmDhxosXXy5ilLFiwAGFhYZBIJBg4cCDi4uLsXSXGmJPgANvBXLx4Ef/93/9t0XWuWbMGRISHHnrIouudO3cugoKCcPr0afz000/Izs6GRCIxet7LL7+MDz74AIWFhRZ9f8YYs5aDBw/i9ddfx8qVK3Hx4kU8+uij9q4SY8yJcIDtYMLDwyGTySy6znfeeQdPP/20YPDbE/n5+UhJScHAgQPRt29f0edFRkZiwoQJeP/99y36/sy1NTU12bsKzI3l5+fDw8MD999/P8LDw+Hr62uT9+V2z5hr4ADbDvbv349x48YhMDAQgYGBGDFiBP7zn/8AMOwi8sorr0AikRj9ZGVl6deVk5ODcePGwdfXF5GRkXjyySdRUVGhX3706FHk5eXhgQceMKjD6tWrMWTIEMhkMvTu3RvJycm4cOGCfvlnn32GuLg4yGQy3HXXXdi6dSskEgn279+PoqIiSCQSFBQU4OWXX4ZEIsH48eMxbdo0/TZ0rueUKVMs3vWF2Y5Wq8X8+fPRp08fBAUFYebMmWhoaNAv/+c//4n4+HjIZDLccsst+Pvf/w6NRqNffv36dTz66KPw9/dHWFgYFi5ciM5JZMePH48ZM2ZAqVSiX79+iIyMBADk5uYiOTkZvr6+CAkJweOPP47Lly8bvHbdunUYOnQofHx80L9/fyxcuNDg/dvWvXDhQoSGhiI4OBgvvfQStFotFi9ejLCwMPTt2xcvvfSSwXq3bNmCkSNHws/PD8HBwUhMTMSRI0cs9rkyx5SVlYVp06ZBq9Xqj2dCTLW75uZmzJ8/H5GRkfD29sbQoUPx8ccfG6xDIpHg3XffxeOPPw65XI709HQAwGuvvYaYmBj4+Pigb9++mDx5ssE+x9jNyMnJwfjx49GrVy/I5XKkpKTg0KFD+uVqtRqTJk2CTCbDgAED8N577xl1E9RoNHjllVegUCggk8mQkJCAFStW2GNzHFsP072zbtJoNBQSEkKzZ8+ms2fP0tmzZ2nTpk20d+9eIiICQOvXryciopqaGrp48aL+Z+vWreTp6Ulr1qwhIqKdO3eSr68vvfvuu3T27Fk6dOgQjR8/nu6++27SarVERLR06VKKjIw0qMPhw4dJKpXSunXrqKioiI4fP06rVq2ikpISIiL65ZdfSCKR0Pz58+n06dO0ceNGGjhwIAGgffv2kUajoYsXL1L//v1p3rx5dPHiRaqqqqLly5cTAH19Kysr9e+Zl5dHAOjUqVPW/oiZhaWkpFBgYCDNnDmTTp06RVu3bqW+ffvSn//8ZyIiWrRoEQ0YMIA2bdpEhYWF9M0331BUVBQtXLhQv44HHniAYmNjaefOnXTy5ElKT0+nwMBAmjBhgsH7BAQE0KxZsygvL4+OHz9OFy9epMDAQHrsscfo+PHjtG/fPho2bBglJSXpX/f111+Th4cHvfbaa3TmzBn69NNPKTg42OD9U1JSKCgoiObOnUtnzpyhDz/8kADQvffeSy+88AKdOXOG1q5dSwDo22+/JSKiixcvkpeXF7355ptUWFhIp06dog0bNtDx48et/ZEzO6usrKSlS5eSVCrVH88WLVpEsbGx+ueY0+6ef/556tWrF3322Wd05swZ+vvf/04SiYR27Nihfw4A6tWrF7377rt07tw5OnPmDG3cuJECAwNp69atVFxcTEeOHKF33nmH6uvrbfo5MNezadMmfXs8efIkzZgxg0JCQujq1auk1WppxIgRlJiYSAcPHqQjR47QvffeS0FBQTRjxgz9OjIzM2nYsGH0n//8hwoLC+nTTz8luVxOq1evtuOWOR4OsG3s2rVrBIC+//57weUdA+yOzp8/T+Hh4fTCCy/oy1JSUmjevHkGzysuLiYAdOTIESIievbZZykxMdHgOZs2baKgoCCqqqoSrEN6ejqNHTvWoOyf//ynPsBuEx0dTSqVSv94/fr1JPadraqqigDQ119/LbicOa6UlBSKjo4mjUajL1uxYgV5e3tTbW0t+fr60rZt2wxes27dOpLL5URElJ+fTwBo+/bt+uWNjY0UERFhFGDfcsst1NLSoi9buHAhRUZGUmNjo77s6NGjBID27NlDRERJSUn08MMPG7z/0qVLSSaT6V+XkpJCI0aMMHjO0KFD6dZbbzUoGz58OM2ZM4eIdF80AZBarTbvg2IuZc2aNSSVSvWPOwfYptpdXV0deXt703vvvWfwnAceeIBSU1P1jwHQ9OnTDZ7z9ttv0y233EJNTU2W3CTGjLS0tFBwcDBlZ2fT9u3bCQDl5+frl1dUVJCvr68+wC4sLCSJREK//vqrwXr+9re/GR1j3R13EbGxkJAQzJw5E5MnT8a9996LN954A2fOnOnyNbW1tbjvvvswduxYvPHGG/ryn376CUuXLkVAQID+Z+jQoQB0/QcBoKGhwahPd1paGmJiYqBQKDB16lSsXLkSV69e1S8/deoUxo0bZ/CapKSkHm13Wx34FqdzSkxMhFQq1T8eN24cmpqacPjwYTQ0NOChhx4yaIezZs1CVVUVrly5glOnTgEA7rrrLv3rvb29ceeddxq9zx133AEPj/bDUl5eHsaMGQNvb2992YgRIyCXy5GXl6d/TnJyssF6UlJScOPGDRQUFBi8rqPw8HAMHz7cqKyt+8nw4cMxefJk3HrrrZgyZQqWLVuGkpIS8z4w5vJMtbtz586hqalJ8DltbbdNYmKiweNHHnkEzc3NiI6ORlZWFtavX4+amhrrbAhzK2q1GtOmTUNcXByCgoIQFBSEqqoqFBcX49SpU+jTp4/BbDm9evXC4MGD9Y8PHz4MIsKoUaMMjvmvvfaaPu5gOhxg28GqVavw888/Iy0tDXv27MGtt94q2n9Jq9Xi8ccfh5eXF7Kzsw2CD61Wi3nz5uHo0aMGP/n5+bj33nsBAH379sW1a9cM1hkQEIDDhw/jyy+/xKBBg/Cvf/0LcXFx+PnnnwEARGTxAZFtdehqMCRzHtSp//Tnn39u0AZPnDiB/Px89OrVy+i5XfH39zcqE2uLHcs7P6ftPTuWe3l5Gb1eqEyr1QIApFIptm3bhl27duHOO+/Exo0bMWjQIHz99ddmbw9zbea0O6HndC7r3O4jIyNx+vRpfPTRRwgNDYVKpcLgwYP5Cx7rsd/97nc4f/483nvvPeTm5uLo0aMIDQ3VD641de5vOz7+8MMPBsf8kydP4vjx41avvzPhANtObr31Vvz1r3/Ftm3bMGPGDKxcuVLwec8//zyOHj2Kr776Cn5+fgbLRo0ahby8PMTFxRn9BAQEAABuv/125OfnG41Ml0qlSE5OxuLFi/Hzzz+jX79++sE3CQkJOHDggMHzOz8W0naVsaWlxWjZiRMnIJVKMXLkSJPrYY7np59+Mvi//vjjj/D29sZtt90GmUyGwsJCwXYolUqRkJAAQHdAbtPU1ISffvrJ5PsmJCTgxx9/NGi/x44dQ1VVlX69CQkJ2LNnj8Hr9u7dC19fX8TExPRouyUSCRITE7FgwQLs3bsXKSkpWLNmTY/WyVyDqXYXFxcHHx8fwee0td2u+Pj44De/+Q3eeustnDhxAvX19di8ebNFt4G5l4qKCpw6dQrz58/H5MmTMXToUMhkMv1du6FDh+LKlSs4d+6c/jXXr1/H2bNn9Y/vuOMOAMD58+eNjvexsbG23SAH52nvCribc+fOYdWqVbjvvvsQFRWFsrIy7Nu3D7fffrvRc9euXYv3338fW7duBQBcunQJAODr6wu5XI7Fixdj0qRJmD17NjIzMxEYGIj8/Hx8/vnnWL58OXx9fZGamgqJRIKDBw/i7rvvBqCbGaGwsBDJycno27cvfv75Z5SUlOi7l8yePRt33nknXnrpJWRmZiIvLw9LliwxuW0KhQIAsHXrViQlJcHX11cf6O/evRtJSUkICgrq+YfIbK6iogJ/+tOf8Oyzz6KwsBBKpRJPPfUU5HI5FixYgAULFgDQdT/SaDQ4ceIEjhw5gjfffBNxcXH4/e9/jz/96U9YsWIFwsLC8MYbb5h1y/uZZ57BsmXLkJWVhQULFqCyshJ//OMfkZSUpG/PL774Iu677z688cYbePDBB3H06FG88sormDNnjkHXku764YcfsHPnTkyaNAn9+vVDfn4+jh8/jhkzZtz0OpnrMNXuvL298Ze//AVKpRJ9+/bFbbfdhs8//xxbtmxBTk5Ol+v+8MMPodVqkZiYiODgYOzcuRM1NTX6YzRjNyMkJAR9+/bFqlWrEBsbi/f3qOUAAAMoSURBVIqKCsydO1c/BeXEiRMxYsQIPPHEE1i2bBm8vb3x0ksvwdPTU39lOy4uDtOnT8dTTz2Ft956C2PHjkVdXR1+/vlnXLlyBfPmzbPnJjoWO/b/dktlZWU0ZcoUioyMJG9vb+rXrx/NnDlTP+MGOgxyzMzMJABGP5mZmfr17d27lyZMmEABAQHk5+dH8fHx9Oyzz1Jzc7P+OVlZWfTUU0/pH+/Zs4dSU1OpT58+5OPjQ3FxcfT666/rZx4hIvrkk08oJiaGvL29KTExkTZv3mxykCORblBlaGgoSSQSfT21Wi0NHDiQPv74Y4t9jsx2UlJS6Mknn9TPiBAQEEBPPvkk1dXV6Z+zevVqGjFiBPn4+FBwcDAlJibS+++/r19+9epVevjhh8nPz4/69OlD8+fPpyeeeMJokGPHkeptfvzxR7r77rtJJpORXC6nxx57jMrLyw2es3btWoqPjycvLy+KiIigBQsWGOwDQuueMGGCwb5ERDR58mRKT08nIqKTJ0/SvffeS2FhYeTt7U0DBgyg559/3mDAJXNdpgY5Eplud01NTTRv3jyKiIggLy8vGjJkCG3YsMFgHRAY2L5x40YaO3YsBQcHk6+vLyUkJPAMDcwidu/eTcOHDycfHx8aNGgQffHFFxQbG0uLFi0iIt0gxokTJ5KPjw/179+fli9fTnfeeSc988wz+nVoNBp68803afDgweTl5UW9e/em5ORk+uyzz+y0VY5JQtSNDpLMKRUUFOi7k0RERNzUOoqKiqBQKLBv375uD3j87LPPoFKpcPToUYOBcowxxhhzXDU1Nejfvz9effVV/PnPf7Z3dZwKdxFxA7GxsVixYgXUavVNB9g90djYiDVr1nBwzRhjjDmwrVu3wtPTE0OGDMHly5fxt7/9DRKJBI888oi9q+Z0OMB2E/bcOdoyPDLGGGPMcdXX12Px4sUoKiqCv78/7rjjDuzfvx9hYWH2rprT4S4ijDHGGGOMWRBP08cYY4wxxpgFcYDNGGOMMcaYBXGAzRhjjDHGmAVxgM0YY4wxxpgFcYDNGGOMMcaYBXGAzRhjjDHGmAX9P7YnHxrxBf2EAAAAAElFTkSuQmCC\n", 218 | "text/plain": [ 219 | "
" 220 | ] 221 | }, 222 | "metadata": {}, 223 | "output_type": "display_data" 224 | } 225 | ], 226 | "source": [ 227 | "# plot predictions and targets vs original features \n", 228 | "fig,ax=plt.subplots(1,4,figsize=(12,3),sharey=True)\n", 229 | "for i in range(len(ax)):\n", 230 | " ax[i].scatter(X_train[:,i],y_train, label = 'target')\n", 231 | " ax[i].set_xlabel(X_features[i])\n", 232 | " ax[i].scatter(X_train[:,i],y_pred,color=dlc[\"dlorange\"], label = 'predict')\n", 233 | "ax[0].set_ylabel(\"Price\"); ax[0].legend();\n", 234 | "fig.suptitle(\"target versus prediction using z-score normalized model\")\n", 235 | "plt.show()" 236 | ] 237 | }, 238 | { 239 | "cell_type": "markdown", 240 | "metadata": {}, 241 | "source": [ 242 | "## Congratulations!\n", 243 | "In this lab you:\n", 244 | "- utilized an open-source machine learning toolkit, scikit-learn\n", 245 | "- implemented linear regression using gradient descent and feature normalization from that toolkit" 246 | ] 247 | }, 248 | { 249 | "cell_type": "code", 250 | "execution_count": null, 251 | "metadata": {}, 252 | "outputs": [], 253 | "source": [] 254 | } 255 | ], 256 | "metadata": { 257 | "kernelspec": { 258 | "display_name": "Python 3", 259 | "language": "python", 260 | "name": "python3" 261 | }, 262 | "language_info": { 263 | "codemirror_mode": { 264 | "name": "ipython", 265 | "version": 3 266 | }, 267 | "file_extension": ".py", 268 | "mimetype": "text/x-python", 269 | "name": "python", 270 | "nbconvert_exporter": "python", 271 | "pygments_lexer": "ipython3", 272 | "version": "3.7.6" 273 | } 274 | }, 275 | "nbformat": 4, 276 | "nbformat_minor": 5 277 | } 278 | --------------------------------------------------------------------------------