\n",
9 | "\n",
10 | "In this part of the exercise, you will implement a neural network to recognize handwritten digits using the same training set as before. The neural network will be able to represent complex models that form non-linear hypotheses. For this week, you will be using parameters from a neural network that we have already trained. Your goal is to implement the feedforward propagation algorithm to use our weights for prediction. In next week’s exercise, you will write the backpropagation algorithm for learning the neural network parameters.
\n",
11 | "\n",
12 | "The file ex3data1 contains a training set. \n",
13 | "The structure of the dataset described blow: \n",
14 | "1. X array = 400 columns describe the values of pixels of 20*20 images in flatten format for 5000 samples\n",
15 | "2. y array = Value of image (number between 0-9)\n",
16 | "\n",
17 | "\n",
18 | "
\n",
19 | "\n",
20 | "Our assignment has these sections:\n",
21 | "1. Visualizing the Data\n",
22 | " 1. Converting .mat to .csv\n",
23 | " 2. Loading Dataset and Trained Neural Network Weights\n",
24 | " 3. Ploting Data\n",
25 | "2. Model Representation\n",
26 | "3. Feedforward Propagation and Prediction\n",
27 | "\n",
28 | "\n",
29 | "In each section full description provided."
30 | ]
31 | },
32 | {
33 | "cell_type": "markdown",
34 | "metadata": {},
35 | "source": [
36 | "## 1. Visualizing the Dataset\n",
37 | "Before starting on any task, it is often useful to understand the data by visualizing it. "
38 | ]
39 | },
40 | {
41 | "cell_type": "markdown",
42 | "metadata": {},
43 | "source": [
44 | "### 1.A Converting .mat to .csv\n",
45 | "In this specific assignment, the instructor added a .mat file as training set and weights of trained neural network. But we have to convert it to .csv to use in python. \n",
46 | "After all we now ready to import our new csv files to pandas dataframes and do preprocessing on it and make it ready for next steps."
47 | ]
48 | },
49 | {
50 | "cell_type": "code",
51 | "execution_count": 2,
52 | "metadata": {},
53 | "outputs": [],
54 | "source": [
55 | "# import libraries\n",
56 | "import scipy.io\n",
57 | "import numpy as np\n",
58 | "\n",
59 | "data = scipy.io.loadmat(\"ex3data1\")\n",
60 | "weights = scipy.io.loadmat('ex3weights')"
61 | ]
62 | },
63 | {
64 | "cell_type": "markdown",
65 | "metadata": {},
66 | "source": [
67 | "Now we extract X and y variables from the .mat file and save them into .csv file for further usage. After running the below code you should see X.csv and y.csv files in your directory."
68 | ]
69 | },
70 | {
71 | "cell_type": "code",
72 | "execution_count": 3,
73 | "metadata": {},
74 | "outputs": [],
75 | "source": [
76 | "for i in data:\n",
77 | " if '__' not in i and 'readme' not in i:\n",
78 | " np.savetxt((i+\".csv\"),data[i],delimiter=',')\n",
79 | " \n",
80 | "for i in weights:\n",
81 | " if '__' not in i and 'readme' not in i:\n",
82 | " np.savetxt((i+\".csv\"),weights[i],delimiter=',')"
83 | ]
84 | },
85 | {
86 | "cell_type": "markdown",
87 | "metadata": {},
88 | "source": [
89 | "### 1.B Loading Dataset and Trained Neural Network Weights\n",
90 | "First we import .csv files into pandas dataframes then save them into numpy arrays.
\n",
91 | "There are 5000 training examples in ex3data1.mat, where each training example is a 20 pixel by 20 pixel grayscale image of the digit. Each pixel is represented by a floating point number indicating the grayscale intensity at that location. The 20 by 20 grid of pixels is \"flatten\" into a 400-dimensional vector. Each of these training examples becomes a single row in our data matrix X. This gives us a 5000 by 400 matrix X where every row is a training example for a handwritten digit image.
\n",
92 | "The second part of the training set is a 5000-dimensional vector y that contains labels for the training set.
\n",
93 | "Notice: In dataset, the digit zero mapped to the value ten. Therefore, a \"0\" digit is labeled as \"10\", while the digits \"1\" to \"9\" are labeled as \"1\" to \"9\" in their natural order. \n",
94 | "But this make thing harder so we bring it back to natural order for 0!"
95 | ]
96 | },
97 | {
98 | "cell_type": "code",
99 | "execution_count": 4,
100 | "metadata": {},
101 | "outputs": [],
102 | "source": [
103 | "# import library\n",
104 | "import pandas as pd\n",
105 | "\n",
106 | "# saving .csv files to pandas dataframes\n",
107 | "x_df = pd.read_csv('X.csv',names= np.arange(0,400))\n",
108 | "y_df = pd.read_csv('y.csv',names=['label'])"
109 | ]
110 | },
111 | {
112 | "cell_type": "code",
113 | "execution_count": 5,
114 | "metadata": {},
115 | "outputs": [],
116 | "source": [
117 | "# saving .csv files to pandas dataframes\n",
118 | "Theta1_df = pd.read_csv('Theta1.csv',names = np.arange(0,401))\n",
119 | "Theta2_df = pd.read_csv('Theta2.csv',names = np.arange(0,26))"
120 | ]
121 | },
122 | {
123 | "cell_type": "code",
124 | "execution_count": 6,
125 | "metadata": {},
126 | "outputs": [
127 | {
128 | "name": "stdout",
129 | "output_type": "stream",
130 | "text": [
131 | "#5000 Number of training samples, #400 features per sample\n"
132 | ]
133 | }
134 | ],
135 | "source": [
136 | "# saving x_df and y_df into numpy arrays\n",
137 | "x = x_df.iloc[:,:].values\n",
138 | "y = y_df.iloc[:,:].values\n",
139 | "\n",
140 | "m, n = x.shape\n",
141 | "\n",
142 | "# bring back 0 to 0 !!!\n",
143 | "y = y.reshape(m,)\n",
144 | "y[y==10] = 0\n",
145 | "y = y.reshape(m,1)\n",
146 | "\n",
147 | "print('#{} Number of training samples, #{} features per sample'.format(m,n))"
148 | ]
149 | },
150 | {
151 | "cell_type": "code",
152 | "execution_count": 7,
153 | "metadata": {},
154 | "outputs": [],
155 | "source": [
156 | "# saving Theta1_df and Theta2_df into numpy arrays\n",
157 | "theta1 = Theta1_df.iloc[:,:].values\n",
158 | "theta2 = Theta2_df.iloc[:,:].values"
159 | ]
160 | },
161 | {
162 | "cell_type": "markdown",
163 | "metadata": {},
164 | "source": [
165 | "### 1.C Plotting Data\n",
166 | "You will begin by visualizing a subset of the training set. In first part, the code randomly selects selects 100 rows from X and passes those rows to the display_data function. This function maps each row to a 20 pixel by 20 pixel grayscale image and displays the images together. \n",
167 | "After plotting, you should see an image like this:"
168 | ]
169 | },
170 | {
171 | "cell_type": "code",
172 | "execution_count": 8,
173 | "metadata": {},
174 | "outputs": [
175 | {
176 | "data": {
177 | "text/plain": [
178 | "
\n",
232 | "You have been provided with a set of network parameters (Θ(1); Θ(2)) already trained by instructor.
\n",
233 | "Theta1 and Theta2 The parameters have dimensions that are sized for a neural network with 25 units in the second layer and 10 output units (corresponding to the 10 digit classes)."
234 | ]
235 | },
236 | {
237 | "cell_type": "code",
238 | "execution_count": 9,
239 | "metadata": {},
240 | "outputs": [
241 | {
242 | "name": "stdout",
243 | "output_type": "stream",
244 | "text": [
245 | "theta1 shape = (25, 401), theta2 shape = (10, 26)\n"
246 | ]
247 | }
248 | ],
249 | "source": [
250 | "print('theta1 shape = {}, theta2 shape = {}'.format(theta1.shape,theta2.shape))"
251 | ]
252 | },
253 | {
254 | "cell_type": "markdown",
255 | "metadata": {},
256 | "source": [
257 | "It seems our weights are transposed, so we transpose them to have them in a way our neural network is."
258 | ]
259 | },
260 | {
261 | "cell_type": "code",
262 | "execution_count": 10,
263 | "metadata": {},
264 | "outputs": [
265 | {
266 | "name": "stdout",
267 | "output_type": "stream",
268 | "text": [
269 | "theta1 shape = (401, 25), theta2 shape = (26, 10)\n"
270 | ]
271 | }
272 | ],
273 | "source": [
274 | "theta1 = theta1.transpose()\n",
275 | "theta2 = theta2.transpose()\n",
276 | "print('theta1 shape = {}, theta2 shape = {}'.format(theta1.shape,theta2.shape))"
277 | ]
278 | },
279 | {
280 | "cell_type": "markdown",
281 | "metadata": {},
282 | "source": [
283 | "# 3. Feedforward Propagation and Prediction\n",
284 | "Now you will implement feedforward propagation for the neural network. \n",
285 | "You should implement the feedforward computation that computes hθ(x(i)) for every example i and returns the associated predictions. Similar to the one-vs-all classification strategy, the prediction from the neural network will be the label that has the largest output hθ(x)k."
286 | ]
287 | },
288 | {
289 | "cell_type": "markdown",
290 | "metadata": {},
291 | "source": [
292 | "Implementation Note: The matrix X contains the examples in rows. When you complete the code, you will need to add the column of 1’s to the matrix. The matrices Theta1 and Theta2 contain the parameters for each unit in rows. Specifically, the first row of Theta1 corresponds to the first hidden unit in the second layer. \n",
293 | "You must get a(l) as a column vector.
"
237 | ],
238 | "text/plain": [
239 | " y\n",
240 | "0 2.134311\n",
241 | "1 1.173257\n",
242 | "2 34.359109"
243 | ]
244 | },
245 | "execution_count": 5,
246 | "metadata": {},
247 | "output_type": "execute_result"
248 | }
249 | ],
250 | "source": [
251 | "y_df.head(3)"
252 | ]
253 | },
254 | {
255 | "cell_type": "markdown",
256 | "metadata": {},
257 | "source": [
258 | "Now we convert all **pandas dataframes** to **numpy arrays** for calculations."
259 | ]
260 | },
261 | {
262 | "cell_type": "code",
263 | "execution_count": 6,
264 | "metadata": {},
265 | "outputs": [
266 | {
267 | "name": "stdout",
268 | "output_type": "stream",
269 | "text": [
270 | "#12 Number of training samples, #1 features per sample\n",
271 | "#21 Number of validation samples, #1 features per sample\n",
272 | "#21 Number of test samples, #1 features per sample\n"
273 | ]
274 | }
275 | ],
276 | "source": [
277 | "# saving x, y, xval, yval, xtest and ytest into numpy arrays\n",
278 | "x = x_df.iloc[:,:].values\n",
279 | "xval = xval_df.iloc[:,:].values\n",
280 | "xtest = xtest_df.iloc[:,:].values\n",
281 | "\n",
282 | "y = y_df.iloc[:,:].values\n",
283 | "yval = yval_df.iloc[:,:].values\n",
284 | "ytest = ytest_df.iloc[:,:].values\n",
285 | "\n",
286 | "# number of examples and number of features\n",
287 | "m, n = x.shape\n",
288 | "\n",
289 | "m_val = xval.shape[0]\n",
290 | "m_test = xtest.shape[0]\n",
291 | "\n",
292 | "print('#{} Number of training samples, #{} features per sample'.format(m,n))\n",
293 | "print('#{} Number of validation samples, #{} features per sample'.format(m_val,n))\n",
294 | "print('#{} Number of test samples, #{} features per sample'.format(m_test,n))"
295 | ]
296 | },
297 | {
298 | "cell_type": "code",
299 | "execution_count": 7,
300 | "metadata": {},
301 | "outputs": [],
302 | "source": [
303 | "# define some hypter parameters\n",
304 | "\n",
305 | "# define theta as zero\n",
306 | "theta = None\n",
307 | "\n",
308 | "# define hyperparameter λ\n",
309 | "lambda_ = None\n",
310 | "\n",
311 | "# reshape (-1,1) because we just have one feature in y column\n",
312 | "y = y.reshape(-1,1)"
313 | ]
314 | },
315 | {
316 | "cell_type": "markdown",
317 | "metadata": {},
318 | "source": [
319 | "### 1.C Ploting Dataset\n",
320 | "We will begin by visualizing the dataset containing historical records on **the change in the water level**, `x`, and **the amount of water flowing out of the dam**, `y`.
\n",
321 | "This dataset is divided into three parts: \n",
322 | " • A **training set** that your model will learn on: `x`, `y` \n",
323 | " • A **cross validation set** for determining the regularization parameter: `xval`, `yval` \n",
324 | " • A **test set** for evaluating performance. These are **\"unseen\" examples** which your model did not see during training: `xtest`, `ytest` "
325 | ]
326 | },
327 | {
328 | "cell_type": "code",
329 | "execution_count": 9,
330 | "metadata": {},
331 | "outputs": [
332 | {
333 | "data": {
334 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmcXFWd9/HPFwiiNNoi0EZgCLsig0gahjEu3bih44aPqIw6uIxxYYY2LAPqDDQjLjw6xDAjahTGoGDDICgiOMYMLeDDlrAlMSIKCEgEERpoQCTk9/xxbk0qTVfV7aTureqq7/v1qlfX3X910qlfn3PuPUcRgZmZda9NWh2AmZm1lhOBmVmXcyIwM+tyTgRmZl3OicDMrMs5EZiZdTknAutqkjaVNC7pL5q5r9l04kRg00r2RVx5rZX0eNXye6Z6voh4KiJ6IuLOZu47VZKeK+lbkn4v6WFJt0g6Juex35E03OyYrHts1uoAzKYiInoq7yXdAfx9RPy01v6SNouINWXEtpFOAzYFXgg8DOwJvKilEVnXcI3AOoqkkyWdK+m7kh4B3ivpryVdLWlM0mpJp0make2/maSQNCtb/k62/VJJj0i6StLOU9032/4GSb+S9JCkf5f0c0nvrxH6/sA5ETEWEWsjYlVEXFB1rr0k/VTSA5J+Ken/ZOs/DrwL+FRWK7qwuSVq3cCJwDrRIcA5wHOAc4E1wBCwDTAHOBj4SJ3j/xb4F2Br4E7gM1PdV9J2wHnAsdl1bwcOqHOeq4HPS3q/pN2rN0jaClgMnAVsB7wHWChpz4g4PfuMn8uarQ6pcw2zSTkRWCe6MiJ+mP1l/XhEXBcR10TEmoi4DVgIvKrO8edHxNKIeBI4G9h3A/Z9E3BjRPwg2zYfuL/OeT5O+kI/Elgl6VZJr8u2vQX4VUSclX2GZcD3gXfULwazfJwIrBPdVb0g6YWSflTpiAX+lfRXei2/r3r/GNBTa8c6+76gOo5IozveXeskEfFYRJwcEfsBzwMuAL4n6TnATsCcrGlrTNIYqTloZp24zHJzIrBONHFI3a8DK4DdIuLZwAmACo5hNbBDZUGSgO3zHBgRDwGfJyWVWaSEsiQieqtePRHxD5VDmhq5dR0nAusGWwEPAY9KehH1+wea5WJgP0lvlrQZqY9i21o7SzpRUr+kzSVtQWoiegC4FbgIeLGkv5U0I3sdIGnP7PB7gV2K/TjWyZwIrBscDRwOPEKqHZxb9AUj4l5S882pwB+BXYEbgCfqHLYo2/ceYAD4m6zJ6CHg9cB7STWN35NqDM/Ijvsm8BJJD0o6v/mfxjqdPDGNWfEkbUr6gn9HRFzR6njMqrlGYFYQSQdLeo6kZ5BuMV0DXNvisMyexonArDgvB24j3TZ6MPC2iKjXNGTWEm4aMjPrcq4RmJl1uWkx6Nw222wTs2bNanUYuT366KNsueWWrQ6jLblsanPZ1Oayqa1e2Sxbtuz+iKh523LFtEgEs2bNYunSpa0OI7fR0VEGBgZaHUZbctnU5rKpzWVTW72ykfTbPOdw05CZWZdzIjAz63JOBGZmXc6JwMysyzkRmJm1m4nPdxX8vJcTgZlZOxkehnnz1n35R6Tl4eHCLulEYGbWLiJgbAwWLFiXDObNS8tjY4XVDKbFcwRmZl1Bgvnz0/sFC9ILYGgorVcx8ym5RmBm1k6qk0FFgUkAnAjMzNpLpTmoWnWfQQGcCMzM2kV1n8DQEKxdm35W9xkUwH0EZmbtQoLe3vX7BCrNRL29hTUPORGYmbWT4eH0l3/lS7+SDNxHYGbWRSZ+6ReYBMCJwMys6xWWCCRtIelaSTdJWinppGz9tyTdLunG7LVvUTGYmVljRfYRPAEcFBHjkmYAV0q6NNt2bEScX+C1zcwsp8ISQUQEMJ4tzshexY6cZGZmU6Yo8CEFSZsCy4DdgK9ExHGSvgX8NanGsAQ4PiKemOTYucBcgL6+vtkjIyOFxdls4+Pj9PT0tDqMtuSyqc1lU5vLprZ6ZTM4OLgsIvobnaPQRPC/F5F6gQuBfwT+CPwe2BxYCPwmIv613vH9/f3hOYs7g8umNpdNbR1dNsPDaUC5yi2ilYfKentzjTjaYM7iXImglLuGImIMGAUOjojVkTwB/CdwQBkxmJm1nRaNNjpRYX0EkrYFnoyIMUnPBF4DnCJpZkSsliTgbcCKomIwM2trLRptdKIiawQzgcsk3QxcByyOiIuBsyUtB5YD2wAnFxiDmVl7a8FooxMVedfQzcBLJ1l/UFHXNDObdmqNNtohNQIzM6unRaONTuRB58zMWqVFo41O5ERgZtZKLRhtdKKGiUDSJsBLgBcAjwMrI+LeogMzM+saJY82OlHNRCBpV+A40m2ftwJ/ALYA9pD0GPB1YFFErC0jUDMzK0a9GsHJwFeBj8SEx48lbQf8LfA+YFFx4ZmZWdFqJoKIOKzOtvuALxcSkZmZlarh7aOSlko6QtJzywjIzMzKlec5gneTOoqvkzQi6fXZ8BBmZtYBGiaCiPh1RHwa2AM4BzgTuFPSSZK2LjpAMzMrVq4niyXtA/wb8EXge8A7gIeB/ykuNDMzK0Oe5wiWAWPAGaw/icw1kuYUGZyZmRUvz5PFh0bEbZNtiIi3NzkeMzMrWc2mIUnvlbRJrSQgaVdJLy8uNDMzK0O9GsHzgBuypqFlrHuyeDfgVcD9wPGFR2hmZoWq90DZAkn/ARwEzAH2IY01tAp4X0TcWU6IZmZWpLp9BBHxFLA4e5mZWQfyxDRmZl3OicDMrMsVlggkbSHpWkk3SVop6aRs/c6SrpF0q6RzJW1eVAxmZtZYngfKeoG/A2ZV7x8RRzY49AngoIgYlzQDuFLSpcBRwPyIGJH0NeBDpOGuzcysBfLUCC4hJYHlpNtIK6+6IhnPFmdkryDdhXR+tn4R8LaphWxmZs2kCXPOPH0H6fqI2G+DTi5tSkoauwFfIY1VdHVE7JZt3xG4NCL2nuTYucBcgL6+vtkjIyMbEkJLjI+P09PT0+ow2pLLpjaXTW0um9rqlc3g4OCyiOhvdI48Q0x8W9KHgYtJzT0ARMQDjQ7Mbj/dN2teuhB40WS71Th2IbAQoL+/PwYGBnKE2h5GR0eZTvGWyWVTm8umNpdNbc0omzyJ4M+kv+Q/zbov7QB2yXuRiBiTNAocCPRK2iwi1gA7APdMKWIzM2uqPH0ERwG7RcSsiNg5ezVMApK2zWoCSHom8BrSU8mXkYaxBjgc+MGGhW5mZs2Qp0awEnhsA849E1iU9RNsApwXERdL+gUwIulk4AbS8NZmZtYieRLBU8CNki5j/T6CurePRsTNwEsnWX8bcMAU4zQzs4LkSQTfz15mZtaBGiaCiFhURiBmZtYaeZ4s3h34PLAXaT4CAPJ0GJuZWfvLc9fQf5KGgFgDDAJnAd8uMigzMytPnkTwzIhYQnoK+bcRMUwaJsLMzDpAns7iP0naBLhV0j8AvwO2KzYsMzMrS54awSeAZwFHArOB95EeBDMzsw6Q566h67K348AHig3HzMzKVjMRSPohNQaEA4iItxQSkZmZlapejeBL2c+3A88HvpMtHwbcUWBMZmZWopqJICJ+BiDpMxHxyqpNP5R0eeGRmZlZKfJ0Fm8r6X8fHpO0M7BtcSGZmVmZ8tw+Og8YlXRbtjyLbOYwMzOb/vLcNfTjbJiJF2arfhkRT9Q7xszMpo88NQKyL/6bCo7FzMxaIE8fgZmZdTAnAjOzLtcwESh5r6QTsuW/kOQZxszMOkSeGsHpwF+THiQDeAT4SmERmZlZqfIkgr+KiCOAPwFExIPA5o0OkrSjpMskrZK0UtJQtn5Y0u8k3Zi93rhRn8DMzDZKnruGnpS0Kdm4Q5K2BdbmOG4NcHREXC9pK2CZpMXZtvkR8aU6x5qZWUny1AhOAy4EtpP0WeBK4HONDoqI1RFxffb+EWAVsP1GxGpmZgVQRM0BRtftJL0QeDUgYElErJrSRaRZwOXA3sBRwPuBh4GlpFrDg5McM5fsCea+vr7ZIyMjU7lkS42Pj9PT09PqMNqSy6Y2l01tLpva6pXN4ODgsojob3SOvIlgU6CPqqakiLgzT5CSeoCfAZ+NiAsk9QH3k5qaPgPMjIgP1jtHf39/LF26NM/l2sLo6CgDAwOtDqMtuWxqc9nU5rKprV7ZSMqVCBr2EUj6R+BE4F7gKVKtIIB9chw7A/gecHZEXAAQEfdWbf8GcHGj85iZWXHydBYPAXtGxB+ncmJJAs4AVkXEqVXrZ0bE6mzxEGDFVM5rZmbNlScR3AU8tAHnnkOa33i5pBuzdZ8CDpO0L6lWcQfwkQ04t5mZNUm9qSqPyt7eRhqG+kfA/446Wv1X/mQi4kpSM9JEl2xAnGZmVpB6NYKtsp93Zq/NWfcgWeMeZjMzmxbqTVV5EoCkQyPiv6q3STq06MDMzKwceR4o+2TOdWZmrTHxNvgct8XbOvX6CN4AvBHYXtJpVZueTRo+wsys9YaHYWwM5s8HKSWBefOgtzdts4bq1QjuIT35+ydgWdXrIuD1xYdmZtZAREoCCxakL/9KEliwIK13zSCXen0ENwE3STonIp4sMSYzs3ykVBOA9OW/YEF6PzS0roZgDTXsI3ASMLO2Vp0MKpwEpsRTVZrZ9FZpDqpWaSayXGomAknfzn4OlReOmdkUVPcJDA3B2rXpZ3WfgTVU74Gy2ZJ2Aj4o6SwmPCUcEQ8UGpmZWSNSujuouk+g0kzU2+vmoZzqJYKvAT8GdiHdLVRdopGtNzNrreHh9Jd/5Uu/kgycBHKr2TQUEadFxIuAMyNil4jYuerlJGBm7WPil76TwJQ0HH00Ij4m6SXAK7JVl0fEzcWGZWZmZWl415CkI4Gzge2y19nZZDVmZtYB8sxH8PfAX0XEowCSTgGuAv69yMDMzKwceZ4jEGmKyorKdJVmZtYB8tQI/hO4RtKF2fLbSFNQmplZB8jTWXyqpFHg5aSawAci4oaiAzMzs3LkqREQEdcD10/lxJJ2BM4Cng+sBRZGxAJJWwPnArNIcxa/MyIenMq5zcyseYoca2gNcHT2LMKBwBGS9gKOB5ZExO7AkmzZzMxapLBEEBGrs5oEEfEIsArYHngrsCjbbRGpz8HMzFokz3MEp+RZ1+Acs4CXAtcAfRGxGlKyID2bYGZmLaJoMDqfpOsjYr8J626OiH1yXUDqAX4GfDYiLpA0FhG9VdsfjIjnTnLcXGAuQF9f3+yRkZE8l2sL4+Pj9PT0tDqMtuSyqc1lU5vLprZ6ZTM4OLgsIvobnaPenMUfAz4O7CKpekiJrYCf5wlQ0gzge8DZEXFBtvpeSTMjYrWkmcB9kx0bEQuBhQD9/f0xMDCQ55JtYXR0lOkUb5lcNrW5bGpz2dTWjLKpd9fQOcClwOdZv0P3kTxDUEsS6XmDVRFxatWmi4DDgS9kP38w1aDNzKx56s1Z/BDwkKTjJmzqkdQTEXc2OPcc4H3Ackk3Zus+RUoA50n6EHAncOiGhW5mZs2Q5zmCH5HmHxCwBbAzcAvw4noHRcSV1B6K4tVTiNHMzAqU58niv6xelrQf8JHCIjIzs1JN+TmC7NmA/QuIxczMWqBhjUDSUVWLmwD7AX8oLCIzMytVnj6CrareryH1GXyvmHDMzKxsefoITgKQtFVajPHCozIzs9LkGWJib0k3ACuAlZKWSdq7+NDMzKwMeTqLFwJHRcROEbETcHS2zszMOkCeRLBlRFxWWYiIUWDLwiIyM7NS5eksvk3SvwDfzpbfC9xeXEhmZlamPDWCDwLbAhdkr22ADxQZlJmZlSfPXUMPAkeWEIuZmbVAkVNVmpnZNOBEYGbW5ZwIzMy6XJ6xhk6bZPVDwNKI8KQyZmbTXJ4awRbAvsCt2WsfYGvgQ5K+XGBsZmZWgjzPEewGHBQRawAkfRX4CfBaYHmBsZmZWQny1Ai2Z/0nibcEXhARTwFPFBKVmZmVJk+N4P8CN0oaJU09+Urgc5K2BH5aYGxmZlaChjWCiDgDeBnw/ez18oj4ZkQ8GhHH1jpO0pmS7pO0omrdsKTfSboxe72xGR/CzMw2XN7bRzchzUr2ALCbpFfmOOZbwMGTrJ8fEftmr0tyXt/MpqOI+svWFvLcPnoK8C5gJbA2Wx3A5fWOi4jLJc3ayPjMbLoaHoaxMZg/H6SUBObNg97etM3ahqJBhpZ0C7BPREy5YzhLBBdHxN7Z8jDwfuBhYClwdDaW0WTHzgXmAvT19c0eGRmZ6uVbZnx8nJ6enlaH0ZZcNrV1XNncdRfcdx9stx3suOPTl6eg48qmieqVzeDg4LKI6G94koio+wIuBXoa7Vfj2FnAiqrlPmBTUlPTZ4Ez85xn9uzZMZ1cdtllrQ6hbblsauu4slm7NmJoKCLVBdJraCitn6KOK5smqlc2pAd/G37H5rlr6DHSXUNLqLpdNCKmPCJpRNxbeS/pG8DFUz2HmU0TUmoWWrBg3bpKM5G1lTyJ4KLstdEkzYyI1dniIaR5kM2sE1X6BKrNm+dk0IbyzEewaENOLOm7wACwjaS7gROBAUn7kjqb7wA+siHnNrM2V0kCCxbA0FD68q8sg5NBm6mZCCSdFxHvlLSc9MW9nojYp96JI+KwSVafMfUQzWzakdLdQZUkUGkmgrTeSaCt1KsRDGU/31RGIGbWYYaHU82g8qVfSQZOAm2nZiKoast/NXBFRNxaTkhm1jEmfuk7CbSlPJ3Fs4D3StoJWAZcQUoMNxYZmJmZlSPPWEMnRMRBwN7AlcCxpIRgZmYdIM8QE/8MzAF6gBuAY0i1AjMz6wB5mobeDqwBfgT8DLg6Iv5UaFRmZlaaPE1D+5E6jK8lm5VM0pVFB2ZmZuXI0zS0N/AK4FVAP3AXbhoyM+sYeZqGTiENOX0acF1EPFlsSGZmVqY8Q0z8jaTNgT2APSXd4mRgZtY58jQNvQo4izQ2kIAdJR0eEXUnpjEzs+khT9PQqcDrIuIWAEl7AN8FZhcZmJmZlSPPnMUzKkkAICJ+BcwoLiQzMytTnhrBUklnAN/Olt+Dnyw2M+sYeRLBx4AjgCNJfQSXA6cXGZSZmZUnz11DT5D6CU4tPhwzMytbvYlpJp2QpqLRxDRmZjY91KsRHAo8XlYgZmbWGvUSwTkRsZ+kb0fE+0qLyMzMSlUvEWwu6XDgZZLePnFjRFxQ78SSziRNc3lfROydrdsaOJc02c0dwDsj4sENC93MzJqh3nMEHwUOBHqBN0945ZnH+FvAwRPWHQ8siYjdgSXZspmZtVC9OYuvBK6UtDQizpjqiSPickmzJqx+KzCQvV8EjALHTfXcZmbWPIqoeWPQxp88JYKLq5qGxiKit2r7gxHx3BrHzgXmAvT19c0eGRkpLM5mGx8fp6enp9VhtCWXTW0um9pcNrXVK5vBwcFlEdHf6Bx5HihriYhYCCwE6O/vj4GBgdYGNAWjo6NMp3jL5LKpzWVTm8umtmaUTd2xhpTsuFFXWN+9kmZm554J3NfEc5uZ2QaomwgitRt9v4nXuwg4PHt/OPCDJp7bzMw2QJ7RR6+WtP9UTyzpu8BVpMls7pb0IeALwGsl3Uqa//gLUz2vmZk1V54+gkHgo5LuAB4lDTwXjYaYiIjDamx69ZQiNDOzQuVJBG8oPAozM2uZhk1DEfFbYEfgoOz9Y3mOMzOz6aHhF7qkE0kPfX0yWzUD+E6RQZmZWXny/GV/CPAWUv8AEXEPsFWRQZlZk018cLTAB0lt+smTCP6c3UYaAJK2LDYkM2uq4WGYN2/dl39EWh4ebmVU1kbyJILzJH0d6JX0YeCnwDeLDcvMmiICxsZgwYJ1yWDevLQ8NuaagQH5pqr8kqTXAg8DewInRMTiwiMzs40nwfz56f2CBekFMDSU1kuti83aRp7O4lMiYnFEHBsRx0TEYkmnlBGcmTVBdTKocBKwKnmahl47yTo/W2A2XVSag6pV9xlY16uZCCR9LJvAfk9JN1e9bgduLi9EM9tg1X0CQ0Owdm36Wd1nYF2v7pzFwKXA51l/JrFHIuKBQqMys+aQoLd3/T6BSjNRb6+bhwyoP0PZQ8BDwGEAkrYDtgB6JPVExJ3lhGhmG2V4OP3lX/nSryQDJwHL5OksfnM2WujtwM9Ik85fWnBcZtZME7/0nQSsSp7O4pNJk9j/KiJ2Jo0e+vNCozIzs9LkSQRPRsQfgU0kbRIRlwH7FhyXmZmVJM8w1GOSeoDLgbMl3QesKTYsMzMrS54awVuBx4F5wI+B3wBvLjIoMzMrT80agaRPkPoCboiIp7LVi0qJyszMSlOvaWgHYAHwQkk3A/+PlBiu2tjnCLJpLx8BngLWRET/xpzPzMw2XL3nCI4BkLQ50A+8DPgg8A1JYxGx10ZeezAi7t/Ic5iZ2UbK01n8TODZwHOy1z3A8iKDMjOz8tTrI1gIvJjUhHMNqWno1Ih4sAnXDeAnkgL4ekQsbMI5zcxsAyhqDDol6cfANsAKUhK4ClgRtQ6YykWlF0TEPdmwFYuBf4yIyyfsMxeYC9DX1zd7ZGRkYy9bmvHxcXp6elodRlty2dTmsqnNZVNbvbIZHBxclqcPtmYiAJAkUq3gZdlrb+ABUofxiRsS9CTXGAbGI+JLtfbp7++PpUuXNuNypRgdHWVgYKDVYbQll01tLpvaXDa11SsbSbkSQd3nCCJZAVxCGl/o58CuwNCUo10X2JaStqq8B15HqnWYmVkL1OsjOJJUC5gDPEl26yhwJhvXWdwHXJgqG2wGnBMRP96I8zVP9QiNky2blcW/i1aiencNzQLOB+ZFxOpmXTAibgNe0qzzNc3wcJrMuzI8b2VCj97etM2sLP5dtJLVbBqKiKMi4vxmJoG2FZH+41XP2lSZ1WlszLM4WXn8u2gtkOc5gs5XPWvTggXpBevP6mRWBv8uWgvkGXSuO1T/B6zwfzxrBf8uWsk6NxFMrEI3qlJXquDVPLm3tYJ/F61knZkIhofX/49T+Y9Vq6Otuh12aAjWrk0/q9tpyzbVRGadoR1/F63jdV4fQXVnG6QqdfV/rMluw5PSHRnV7bCVqnlvb/lVct810r3a7XfRukLnJYIN7WwbHl4/SVTOU/Z/vA1JZNZZ2uV30bpGZzYNbWhn22Q1hbJVYq80B2yyybok0I1fBmU0kbVjM1w7/C5a1+jMRDDdO9t810gy1b6edr2GWZvrvETQCZ1t0z2RNUMZD1b54S0zoFP7CKZzZ9vERFbdRwDdUzMo48EqP7xlBnRiIoDp3dk23RNZM1U+e+ULGpr/71jGNczaXGcmApjenW3NTmTTdSTLWk1kzfyiLuMaZm2u8/oIOkWzEtl07Qwto6+nE/qTzJqgc2sENr2fSSijiczNcGaAE0Fnm+6doWX09Uzn/iSzJnHTUKeb7s8klNHXM537k8yawImg0/mZBDNrwImgk7kz1MxyaEkfgaSDgQXApsA3I+ILrYij47kz1MxyKD0RSNoU+ArwWuBu4DpJF0XEL8qOpSu4M9TMGmhF09ABwK8j4raI+DMwAry1BXF0D3eGmlkdrWga2h64q2r5buCvJu4kaS4wF6Cvr4/R0dFSgmuG8fHxaRVvmVw2tblsanPZ1NaMsmlFIpjsz9Gn9VpGxEJgIUB/f38MDAwUHFbzjI6OMp3iLZPLpjaXTW0um9qaUTataBq6G9ixankH4J4WxGFmZrQmEVwH7C5pZ0mbA+8GLmpBHGZmBihacC+5pDcCXybdPnpmRHy2wf5/AH5bRmxNsg1wf6uDaFMum9pcNrW5bGqrVzY7RcS2jU7QkkTQ6SQtjYj+VsfRjlw2tblsanPZ1NaMsvGTxWZmXc6JwMysyzkRFGNhqwNoYy6b2lw2tblsatvosnEfgZlZl3ONwMysyzkRmJl1OSeCAkg6RlJI2iZblqTTJP1a0s2S9mt1jGWT9EVJv8w+/4WSequ2fTIrm1skvb6VcbaKpIOzz/9rSce3Op5WkrSjpMskrZK0UtJQtn5rSYsl3Zr9fG6rY20VSZtKukHSxdnyzpKuycrm3Oxh3dycCJpM0o6kIbbvrFr9BmD37DUX+GoLQmu1xcDeEbEP8CvgkwCS9iI9Xf5i4GDg9Gyo8q5RNTT7G4C9gMOyculWa4CjI+JFwIHAEVl5HA8siYjdgSXZcrcaAlZVLZ8CzM/K5kHgQ1M5mRNB880H/on1B9J7K3BWJFcDvZJmtiS6FomIn0TEmmzxatIYU5DKZiQinoiI24Ffk4Yq7yYemr1KRKyOiOuz94+QvvC2J5XJomy3RcDbWhNha0naAfgb4JvZsoCDgPOzXaZcNk4ETSTpLcDvIuKmCZsmG3p7+9ICaz8fBC7N3rtsXAY1SZoFvBS4BuiLiNWQkgWwXesia6kvk/7YXJstPw8Yq/pDa8q/Py2ZqnI6k/RT4PmTbPo08CngdZMdNsm6jrtvt17ZRMQPsn0+Tar6n105bJL9O65sGnAZTEJSD/A94BMR8bA8oRKS3gTcFxHLJA1UVk+y65R+f5wIpigiXjPZekl/CewM3JT9wu4AXC/pALpk6O1aZVMh6XDgTcCrY90DLF1RNg24DCaQNIOUBM6OiAuy1fdKmhkRq7Om1ftaF2HLzAHekg3cuQXwbFINoVfSZlmtYMq/P24aapKIWB4R20XErIiYRfrPvV9E/J40zPbfZXcPHQg8VKnidgtJBwPHAW+JiMeqNl0EvFvSMyTtTOpQv7YVMbaQh2avkrV5nwGsiohTqzZdBByevT8c+EHZsbVaRHwyInbIvmPeDfxPRLwHuAx4R7bblMvGNYJyXAK8kdQR+hjwgdaG0xL/ATwDWJzVmK6OiI9GxEpJ5wG/IDUZHRERT7UwztJFxBpJ/wD8N+uGZl/Z4rBaaQ7wPmC5pBuzdZ8CvgCcJ+lDpLvyDm1RfO3oOGBE0snADaREmpuHmDAz63JuGjIz63JOBGZmXc6JwMysyzkRmJl1OScCM7Mu50RgTSHp+ZJGJP1G0i8kXSJpD0kDlRESW03Sv0qq+9Bbk67TK+njTTjPqKSmTthe75ySzpe0S51jN5d0uSTfdt5hnAhso2UPAF0IjEbErhGxF+m+777WRra+iDghIn5awqVfu66GAAAEa0lEQVR6gSklguxhw5b9f5T0YmDTiLit1j7ZgHhLgHeVFpiVwonAmmEQeDIivlZZERE3RsQV2WJP9tfmLyWdnSUOJJ0g6TpJKyQtrFo/KukUSddK+pWkV2TrnyXpvGxOg3Oz8df7s22vk3SVpOsl/Vc2Ts16JH1L0juy93dIOinbf7mkF06y/yWS9sne3yDphOz9ZyT9vaQeSUuqzlEZMfQLwK6SbpT0xeyYY7PPerOkk7J1s5TG3D8duJ71h5mYGMvTPp+kN2QP41X2GZD0w7zlMcF7yJ5GlbST0rj220jaRNIVkipjaH0/29c6iBOBNcPewLI6218KfII01v4upCdHAf4jIvaPiL2BZ5LGIarYLCIOyI47MVv3ceDBbE6DzwCzAZQmAPpn4DURsR+wFDgqR9z3Z/t/FThmku2XA6+Q9GzSU8+VuF8OXAH8CTgkO8cg8G9ZMjse+E1E7BsRx2ZforuThpveF5gt6ZXZufYkDVH+0oj47WRB1vl8i4EDJW2Z7fou4NwNLI85ZP+GWRynAF8DjgZ+ERE/yfZbAezf4Fw2zbitz8pwbUTcDZANGTALuBIYlPRPwLOArYGVwA+zYyoDjS3L9of0BbwAICJWSLo5W38gKcn8PKtUbA5clSOu6mu8fZLtVwBHArcDPwJeK+lZwKyIuEVpYLTPZV/qa0lD/07WHPa67HVDttxDSgx3Ar/N5qioZ9LPlw1N8WPgzZLOJ41R/0/Aqybbv8E1ZgJ/qCxExDclHQp8lJS8KuufkvRnSVtlcwVYB3AisGZYyboBrybzRNX7p4DNJG0BnA70R8RdkoZJoylOPOYp1v2e1hqHWMDiiDhsinFPdo1q1wH9wG2kv763AT7MutrPe4BtgdkR8aSkOyZ8hur4Ph8RX19vZRpr/9Eccdb7fOcCRwAPANdFxCNZrWSq5fF4dexZwqtMHtQDVH/pP4NUG7IO4aYha4b/AZ4h6cOVFZL2l/SqOsdUvnTuz9qv6yWSiiuBd2bn3wv4y2z91cAcSbtl254laY8pfoanyTpH78queTWphnBM9hPgOaSx4Z+UNAjslK1/BNiq6lT/DXyw0k4vaXtJU5lUpd7nGwX2IyWoc3PsX8sqYLeq5VNIc0acAHyjslLS84A/RMSTU4jf2pwTgW20bG6BQ0hNJ7+RtBIYps6Y6BExRvqCWU7qgLwux6VOB7bNmoSOA24mDen9B+D9wHezbVcDT+v83UBXAPdmQ2dfQforuZIIzgb6JS0l1Q5+CRARfyQ1y6yQ9MWsff0c4CpJy0lTCm5FTvU+XzZS68Wk+Y4vbrR/HT8CBgCyBL4/cEpEnA38WVJlxNxB0mi61kE8+qhNG0qTvM+IiD9J2pV0K+Me2V/uthEkPZM0pv2cesOAS7oA+GRE3FJacFY49xHYdPIs4LKsk1bAx5wEmiMiHpd0IqnD+87J9lGaNOf7TgKdxzUCM7Mu5z4CM7Mu50RgZtblnAjMzLqcE4GZWZdzIjAz63L/Hw66S+5MENHgAAAAAElFTkSuQmCC\n",
335 | "text/plain": [
336 | "
"
337 | ]
338 | },
339 | "metadata": {},
340 | "output_type": "display_data"
341 | }
342 | ],
343 | "source": [
344 | "# import libraries\n",
345 | "import matplotlib.pyplot as plt\n",
346 | "%matplotlib inline\n",
347 | "\n",
348 | "plt.scatter(x, y, color='red', marker='x')\n",
349 | "plt.title('Training Set')\n",
350 | "plt.xlabel('Change in water level (x)')\n",
351 | "plt.ylabel('Water flowing out of the dam (y)')\n",
352 | "plt.grid()\n",
353 | "plt.show()"
354 | ]
355 | },
356 | {
357 | "cell_type": "markdown",
358 | "metadata": {},
359 | "source": [
360 | "# 2. Adding Polynomial Features\n",
361 | "\n",
362 | "For use polynomial regression, our hypothesis has the form:\n",
363 | "Notice that by defining `x1 = (waterLevel)`, `x2 = (waterLevel)`2, ... , `xp = (waterLevel)`p, we obtain a linear regression model where the features are the various powers of the original value (waterLevel).
\n",
364 | "Now, you will **add more features using the higher powers** of the existing feature `x` in the dataset. Your task write a function so that the function **maps the original training set `x` of size `m × 1` into its higher powers**. Specifically, when a training set `x` of size `m × 1` is passed into the function, the **function should return a `m × p` matrix `x_poly`**, where column `1` holds the original values of `x`, column `2` holds the values of `x^2`, column `3` holds the values of `x^3`, and so on. \n",
365 | "Note that **you don’t have to account for the zero-eth power in this function**."
366 | ]
367 | },
368 | {
369 | "cell_type": "markdown",
370 | "metadata": {},
371 | "source": [
372 | "## 2.A Adding Features\n",
373 | "`poly_features(x, p)` takes a data matrix `x` (size `m x 1`) and **maps each example into its polynomial features** where \n",
374 | "`x_poly[i, :] = [x(i) x(i)^2 x(i)^3 ... x(i)^p]`"
375 | ]
376 | },
377 | {
378 | "cell_type": "code",
379 | "execution_count": 12,
380 | "metadata": {},
381 | "outputs": [],
382 | "source": [
383 | "x_poly = None # the output of polu_features\n",
384 | "p = 8 # order of polynomial features\n",
385 | "\n",
386 | "from sklearn.preprocessing import PolynomialFeatures # import libraries\n",
387 | "def poly_features(x,p): \n",
388 | " polynomial_features = PolynomialFeatures(degree=8, include_bias=False)\n",
389 | " x_poly = polynomial_features.fit_transform(x)\n",
390 | " return x_poly\n",
391 | " "
392 | ]
393 | },
394 | {
395 | "cell_type": "code",
396 | "execution_count": 13,
397 | "metadata": {},
398 | "outputs": [
399 | {
400 | "data": {
401 | "text/html": [
402 | "
"
668 | ],
669 | "text/plain": [
670 | " 0 1 2 3 4 \\\n",
671 | "0 -33.318004 1110.089390 -36985.962723 1.232298e+06 -4.105772e+07 \n",
672 | "1 -37.912164 1437.332181 -54492.373421 2.065924e+06 -7.832364e+07 \n",
673 | "2 -51.206938 2622.150494 -134272.297639 6.875673e+06 -3.520822e+08 \n",
674 | "\n",
675 | " 5 6 7 \n",
676 | "0 1.367961e+09 -4.557774e+10 1.518559e+12 \n",
677 | "1 2.969419e+09 -1.125771e+11 4.268041e+12 \n",
678 | "2 1.802905e+10 -9.232124e+11 4.727488e+13 "
679 | ]
680 | },
681 | "execution_count": 15,
682 | "metadata": {},
683 | "output_type": "execute_result"
684 | }
685 | ],
686 | "source": [
687 | "xtest_poly = poly_features(xtest,p)\n",
688 | "xtest_poly_df = pd.DataFrame(xtest_poly,columns=None)\n",
689 | "xtest_poly_df.head(3)"
690 | ]
691 | },
692 | {
693 | "cell_type": "markdown",
694 | "metadata": {},
695 | "source": [
696 | "## 2.B Normalize Features\n",
697 | "It turns out that if we run the training directly on the projected data, will not work well as the **features** would be **badly scaled** (e.g., an example with **x = 40 will now have a feature x8 = 408 = 6.5 × 1012**). \n",
698 | "Therefore, you will need to use **feature normalization**.\n",
699 | "`feature_normalize(x)` returns a **normalized version** of `x` where the **mean value of each feature is 0** and the **standard deviation is 1**. This is often a **good preprocessing step** to do when working with learning algorithms."
700 | ]
701 | },
702 | {
703 | "cell_type": "code",
704 | "execution_count": 16,
705 | "metadata": {},
706 | "outputs": [],
707 | "source": [
708 | "def feature_normalize(x, xtest, xval):\n",
709 | " sigma = x.std()\n",
710 | " mean = x.mean()\n",
711 | " x_norm = (x-mean)/sigma\n",
712 | " xtest_norm = (xtest-mean)/sigma\n",
713 | " xval_norm = (xval-mean)/sigma\n",
714 | " return (x_norm, xtest_norm, xval_norm)"
715 | ]
716 | },
717 | {
718 | "cell_type": "code",
719 | "execution_count": 17,
720 | "metadata": {},
721 | "outputs": [],
722 | "source": [
723 | "x_poly_norm, xtest_poly_norm, xval_poly_norm = feature_normalize(x_poly,xtest_poly, xval_poly)"
724 | ]
725 | },
726 | {
727 | "cell_type": "code",
728 | "execution_count": 18,
729 | "metadata": {},
730 | "outputs": [
731 | {
732 | "name": "stdout",
733 | "output_type": "stream",
734 | "text": [
735 | "x_poly_norm : mean= -1.850371707708594e-17, std=0.9999999999999999.\n",
736 | "xval_poly_norm : mean= 0.044848630228004685, std=1.2247731967735336.\n",
737 | "xtest_poly_norm : mean= 0.1606768875137592, std=2.5245490041714636.\n"
738 | ]
739 | }
740 | ],
741 | "source": [
742 | "print('x_poly_norm : mean= {}, std={}.'.format(x_poly_norm.mean(),x_poly_norm.std()))\n",
743 | "print('xval_poly_norm : mean= {}, std={}.'.format(xval_poly_norm.mean(),xval_poly_norm.std()))\n",
744 | "print('xtest_poly_norm : mean= {}, std={}.'.format(xtest_poly_norm.mean(),xtest_poly_norm.std()))"
745 | ]
746 | },
747 | {
748 | "cell_type": "markdown",
749 | "metadata": {},
750 | "source": [
751 | "Look at this [link](https://stackoverflow.com/questions/40405803/mean-of-data-scaled-with-sklearn-standardscaler-is-not-zero) if you have any question why after using **scaling**, we still do not have **mean = 0** and **std = 1**. \n",
752 | "Actually the values are zero."
753 | ]
754 | },
755 | {
756 | "cell_type": "code",
757 | "execution_count": 20,
758 | "metadata": {},
759 | "outputs": [],
760 | "source": [
761 | "# add 1's to the features of x as bias\n",
762 | "x_poly_norm = np.append(np.ones(shape=(m,1)),x_poly_norm,axis = 1)\n",
763 | "xval_poly_norm = np.append(np.ones(shape=(xval_poly_norm.shape[0],1)),xval_poly_norm,axis = 1)\n",
764 | "xtest_poly_norm = np.append(np.ones(shape=(xtest_poly_norm.shape[0],1)),xtest_poly_norm,axis = 1)"
765 | ]
766 | },
767 | {
768 | "cell_type": "markdown",
769 | "metadata": {},
770 | "source": [
771 | "# 3. Learning Polynomial Regression\n",
772 | "We will proceed to **train polynomial regression using your linear regression cost function**.
\n",
773 | "Keep in mind that even though **we have polynomial terms** in our feature vector, we are **still solving a linear regression** optimization problem. The **polynomial terms have simply turned into features** that we can use for linear regression. We are using the **same cost function and gradient** that you wrote for the earlier part of this exercise. "
774 | ]
775 | },
776 | {
777 | "cell_type": "markdown",
778 | "metadata": {},
779 | "source": [
780 | "## 3.A Regularized Linear Regression Cost Function\n",
781 | "Recall that regularized linear regression has the following cost function:
\n",
782 | "where `lambda` is a regularization parameter which controls the degree of regularization (thus, **help preventing overfitting**). The **regularization** term puts a **penalty** on the overal **cost J**.
\n",
783 | "As the magnitudes of the model parameters **θj**\n",
784 | "increase, the penalty increases as well. Note that you should not regularize the **θ0** term.
\n",
785 | "For this part of the exercise, you will be using a **polynomial of degree 8**. \n",
786 | "After learning the parameters **θ**, you should draw two plots generated for polynomial regression with **λ = 0**. "
787 | ]
788 | },
789 | {
790 | "cell_type": "markdown",
791 | "metadata": {},
792 | "source": [
793 | "### Implementation\n",
794 | "`linear_reg_cost(x, y, theta, lambda_)` computes the cost of using `theta` as the parameter for linear regression to fit the data points in `x` and `y`. Returns the cost in `j` as cost."
795 | ]
796 | },
797 | {
798 | "cell_type": "code",
799 | "execution_count": 22,
800 | "metadata": {},
801 | "outputs": [],
802 | "source": [
803 | "def hypothesis(x,theta):\n",
804 | " return np.dot(x,theta)"
805 | ]
806 | },
807 | {
808 | "cell_type": "code",
809 | "execution_count": 23,
810 | "metadata": {},
811 | "outputs": [],
812 | "source": [
813 | "def linear_reg_cost(theta_flatten, x_flatten, y, lambda_, num_of_samples, num_of_features):\n",
814 | " x = x_flatten.reshape(num_of_samples, num_of_features)\n",
815 | " theta = theta_flatten.reshape(n,1)\n",
816 | " loss = hypothesis(x,theta)-y\n",
817 | " regularizer = lambda_*np.sum(theta[1:,:]**2)/(2*m)\n",
818 | " j = np.sum(loss ** 2)/(2*m) \n",
819 | " return j"
820 | ]
821 | },
822 | {
823 | "cell_type": "markdown",
824 | "metadata": {},
825 | "source": [
826 | "## 3.B Regularized Linear Regression Gradient\n",
827 | "Correspondingly, the **partial derivative of regularized linear regression’s cost for θj** is defined as:"
828 | ]
829 | },
830 | {
831 | "cell_type": "markdown",
832 | "metadata": {},
833 | "source": [
834 | "### Implementation\n",
835 | "`linear_reg_grad(x, y, theta, lambda_)` computes the gradient of cost of using `theta` as the parameter for linear regression to fit the data points in `x` and `y`. Returns the gradient in `grad`."
836 | ]
837 | },
838 | {
839 | "cell_type": "code",
840 | "execution_count": 24,
841 | "metadata": {},
842 | "outputs": [],
843 | "source": [
844 | "def linear_reg_grad(theta_flatten, x_flatten, y, lambda_, num_of_samples, num_of_features):\n",
845 | " x = x_flatten.reshape(num_of_samples, num_of_features)\n",
846 | " m,n = x.shape\n",
847 | " theta = theta_flatten.reshape(n,1)\n",
848 | " new_theta = np.zeros(shape=(theta.shape))\n",
849 | " loss = hypothesis(x,theta)-y\n",
850 | " gradient = np.dot(x.T,loss)\n",
851 | " new_theta[0:,:] = gradient/m\n",
852 | " new_theta[1:,:] = gradient[1:,:]/m + lambda_*(theta[1:,]/m)\n",
853 | " return new_theta.flatten()"
854 | ]
855 | },
856 | {
857 | "cell_type": "markdown",
858 | "metadata": {},
859 | "source": [
860 | "## 3.C Fitting Linear Regression\n",
861 | "Once your cost function and gradient are working correctly, the next part is to **compute the optimal values** of **θ**. \n",
862 | "This training function uses `fmin_cg` to optimize the cost function. See official doc "
863 | ]
864 | },
865 | {
866 | "cell_type": "markdown",
867 | "metadata": {},
868 | "source": [
869 | "### Implementation\n",
870 | "Once you have implemented the cost and gradient correctly, the `fmin_cg` function will use your cost function to train regularized linear regression and update theta each time."
871 | ]
872 | },
873 | {
874 | "cell_type": "code",
875 | "execution_count": 28,
876 | "metadata": {},
877 | "outputs": [
878 | {
879 | "name": "stdout",
880 | "output_type": "stream",
881 | "text": [
882 | "Optimization terminated successfully.\n",
883 | " Current function value: 15.795185\n",
884 | " Iterations: 198\n",
885 | " Function evaluations: 454\n",
886 | " Gradient evaluations: 454\n"
887 | ]
888 | }
889 | ],
890 | "source": [
891 | "m,n = x_poly_norm.shape\n",
892 | "lambda_ = 0\n",
893 | "theta = np.ones(n)\n",
894 | "\n",
895 | "from scipy.optimize import fmin_cg\n",
896 | "new_theta = fmin_cg(f=linear_reg_cost, x0=theta, fprime=linear_reg_grad, args=(x_poly_norm.flatten(), y, lambda_, m,n))"
897 | ]
898 | },
899 | {
900 | "cell_type": "code",
901 | "execution_count": 31,
902 | "metadata": {},
903 | "outputs": [
904 | {
905 | "data": {
906 | "text/plain": [
907 | "array([ 286.25746025, -46.25445386, -46.25037961, -45.57463053,\n",
908 | " -44.4634646 , 229.73026807, 1062.20194521, 561.40028703,\n",
909 | " 10.76226527])"
910 | ]
911 | },
912 | "execution_count": 31,
913 | "metadata": {},
914 | "output_type": "execute_result"
915 | }
916 | ],
917 | "source": [
918 | "new_theta"
919 | ]
920 | },
921 | {
922 | "cell_type": "markdown",
923 | "metadata": {},
924 | "source": [
925 | "## 3.D Visualization of Fitted Model\n",
926 | "Finally, you should also **plot the best fit line**. The best fit line tells us that the model is a good fit to the data because the **data has a non-linear pattern**. While **visualizing the best fit** as shown is **one possible way to debug** your learning algorithm, it is not always easy to visualize the data and model.
\n",
927 | "In the next section, you will implement a function to **generate learning curves** that can help you debug your learning algorithm even if it is **not easy to visualize** the data."
928 | ]
929 | },
930 | {
931 | "cell_type": "code",
932 | "execution_count": 43,
933 | "metadata": {},
934 | "outputs": [
935 | {
936 | "data": {
937 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd8VFXawPHfSUghhYQSQgm9JLTQAopYgohiQbGtDcXuskXEVxd1QVBQwQKCnbVhxV0bGBAEJFIElSaGNCCEkBBCejLpmXneP+4EQ0kyCZnMTHK+H/Mhc9s8c8H7zD3n3PMoEUHTNE1rudwcHYCmaZrmWDoRaJqmtXA6EWiaprVwOhFomqa1cDoRaJqmtXA6EWiaprVwOhFoLZpSyl0pZVJKdW/MbTXNlehEoLkU64W46seilCqp9vqO+h5PRMwi4iciKY25bX0ppdoqpT5USh1XShUopRKUUo/ZuO8nSqm5jR2T1nK0cnQAmlYfIuJX9btSKhm4X0Q21LS9UqqViFQ2RWznaCngDoQBBUAoMMChEWkthr4j0JoVpdR8pdQXSqnPlVKFwBSl1Bil1A6lVJ5SKl0ptVQp5WHdvpVSSpRSPa2vP7Gu/14pVaiU2q6U6lXfba3rr1RKJSql8pVSrymltiml7q4h9FHAZyKSJyIWEYkTka+rHWugUmqDUipHKRWvlLrRuvxvwC3AU9a7om8a94xqLYFOBFpzdD3wGRAAfAFUAtOBDsBYYCLwUC373w7MBtoBKcC8+m6rlOoI/Bd43Pq+h4HRtRxnB/CCUupupVS/6iuUUv7AeuAjoCNwB7BMKRUqIm9aP+Pz1mar62t5D007K50ItOZoq4h8Z/1mXSIiv4nILyJSKSJJwDLgklr2/1JEdopIBfApMKwB214D7BWRldZ1i4GsWo7zN4wL+sNAnFLqgFLqcuu6a4FEEfnI+hl2Ad8CN9V+GjTNNjoRaM3R0eovlFJhSqnVVR2xwLMY39Jrcrza78WAX00b1rJtl+pxiDG7Y2pNBxGRYhGZLyIjgPbA18BXSqkAoAcw1tq0laeUysNoDupcS1yaZjOdCLTm6PQpdd8BYoC+ItIGeBpQdo4hHQipeqGUUkBXW3YUkXzgBYyk0hMjoWwUkcBqP34i8o+qXRo1cq3F0YlAawn8gXygSCk1gNr7BxpLFDBCKTVJKdUKo48iqKaNlVJzlFIRSilPpZQ3RhNRDnAAWAUMUkrdrpTysP6MVkqFWnfPAHrb9+NozZlOBFpL8H/AVKAQ4+7gC3u/oYhkYDTfLAKygT7AHqCslt2WW7c9BkQCV1ubjPKBK4ApGHcaxzHuGLys+70LDFVK5Sqlvmz8T6M1d0oXptE0+1NKuWNc4G8SkS2OjkfTqtN3BJpmJ0qpiUqpAKWUF8YQ00rgVweHpWln0IlA0+znQiAJY9joRGCyiNTWNKRpDqGbhjRN01o4fUegaZrWwrnEpHMdOnSQnj17OjoMmxUVFeHr6+voMJySPjc10+emZvrc1Ky2c7Nr164sEalx2HIVl0gEPXv2ZOfOnY4Ow2bR0dFERkY6OgynpM9NzfS5qZk+NzWr7dwopY7YcgzdNKRpmtbC2T0RWKs67VFKRVlf91JK/WKdVOsLpZSnvWPQNE3TatYUdwTTgbhqrxcCi0WkH5AL3NcEMWiapmk1sGsfgVIqBLgaeA541Drx1qUYc7iD8Uj9XOCt+h67oqKC1NRUSktLGynaxhMQEEBcXFzdGzZj3t7ehISE4OHh4ehQNE2rg707i18F/oUx6RcY0+vmVSsdmEoNMzIqpR4EHgQIDg4mOjr6lPV+fn4EBwfTtWtXjPziPMxmM+7u7o4Ow2FEhPz8fH7//XdMJtMp60wm0xl/l5pBn5ua6XNTs8Y4N3ZLBEqpa4ATIrJLKRVZtfgsm571iTYRWYZRQISIiAg5vVc8Li6OkJAQp0sCAIWFhfj7+9e9YTPm7++PyWQiIiLilOV69EfN9LmpmT43NWuMc2PPO4KxwLVKqasAb6ANxh1CYLWC4iEYE3E1iDMmAc2g/240zXXYrbNYRJ4UkRAR6QncCvwoIncAm/izxN5UYKW9YtA0TXNVJwpL+Wh7MqUVZru/lyOeI5iJ0XF8EKPP4D0HxHDOkpOTGTx48DkfJzo6mp9//vms68rKyrjssssYNmwYX3zxBffffz+xsbEAPP/88+f83raIjIys82E+W7bRNM02lWYLH247zPiXf+LplftJOF5o9/dskieLRSQaiLb+ngSMbor3dQXR0dH4+flxwQUXnLFuz549VFRUsHfvXgBuueWWk+uef/55nnrqqSaLU9M0+9udksusb2KITS8g0McYcdcpwNvu76ufLD4HlZWVTJ06lfDwcG666SaKi4sB4wJ+ySWXMHLkSK644grS09MBWLp0KQMHDiQ8PJxbb72V5ORk3n77bRYvXsywYcPYsuXPeiUnTpxgypQp7N27l2HDhnHo0KGT37yfeOIJSkpKGDZsGHfccccZcfn5+TFz5kxGjhzJZZddxq+//kpkZCS9e/dm1apVAJSWlnLPPfcwZMgQhg8fzqZNmwAoKSnh1ltvJTw8nFtuuYWSkpKTx/3hhx8YM2YMI0aM4Oabbz5jRJCmaQ2TU1TOzC/3ccObP5NTVM4bt4/glohueLq7EeTnVfcBzpFLzDVUl2e+20/ssYJGPebALm2YM2lQrdskJCTw3nvvMXbsWO69917efPNNpk+fzuOPP05UVBRBQUF88cUX/Pvf/+b9999nwYIFHD58GC8vL/Ly8ggMDOSvf/0rfn5+PPbYY6ccu2PHjrz77ru8/PLLREVFnbJuwYIFvP766yfvFE5XVFREZGQkCxcu5Prrr2fWrFmsX7+e2NhYpk6dyrXXXssbb7wBwB9//EF8fDyXX345iYmJvPXWW/j4+LBv3z727dvHiBEjAMjKymL+/Pls2LABX19fFi5cyKJFi3j66acbeoo1rcWzWIQVvx3lxXXxFJZW8uDFvXl4fD/8vFqxJiadLoHeuLnZf+BFs0gEjtKtWzfGjh0LwJQpU1i6dCkTJ04kLi6OCRMmAMYzBZ07dwYgPDycO+64g8mTJzN58mS7xeXp6cnEiRMBGDJkCF5eXnh4eDBkyBCSk5MB2Lp1K//85z8BCAsLo0ePHiQmJrJ582Yefvjhk/GGh4cDsGPHDmJjY09+3vLycsaMGWO3z6BpzV1MWj6zvo1h79E8Rvdsx7zJgwnt9Oew89TcEkLa+jRJLM0iEdT1zd1eTh8iqZRCRAgLC+PXX8+sSLh69Wo2b97MqlWrmDdvHvv377dLXB4eHidjc3Nzw8vL6+TvlZXGs3y1FSQ629BPEWHChAl8/vnndohY01qO/JIKXvkhgU92HKGdryeL/jKU64ef+WBsWm4J48M6NklMuo/gHKSkpLB9+3YAPv/8cy688EJCQ0PJyso6ubyiooL9+/djsVg4evQo48aN48UXXyQvLw+TyYS/vz+FhfUfFeDh4UFFRUWDY7/44ov59NNPAUhMTCQlJYXQ0NBTlsfExLBv3z4Azj//fLZt28bBgwcBKC4uJjExscHvr2ktjYjw1a5Uxr8SzSc7jnDn+T3Y+H+R3DDizAdjSyvMZJnKCGnbukli04ngHAwYMIDly5cTHh5OTk4O06ZNw9PTk48//piZM2cydOhQhg0bxs8//4zZbGbKlCknO2dnzJhBYGAgkyZN4ptvvjmjs7guDz744Mmmpob429/+htlsZsiQIdxyyy18+OGHeHl5MW3aNEwmE+Hh4bz44ouMHm0M8AoKCuLDDz/ktttuIzw8nPPPP5/4+PgGvbemtTQJxwu5ZdkO/u9/v9O1rQ+r/nEhz1w3mIDWZ5+LKy3PGKQR0q5pEgEi4vQ/I0eOlNPFxsaescxZFBQUODoEp3C2v6NNmzY1fSAuQp+bmrnquSksrZD5Uful95OrZegz6+SzX46I2Wypc7/o+AzpMTNKfknKNhZYat6ntnMD7BQbrrHNoo9A0zTNmYgIa/44zryoWI4XlHJLRDdmXhlGO18byq/MnUtaoS94DDSahkRgxgwIDIS5c+0Sr24a0jRNa0RJmSbuev9X/v7Zbtr5evLVtAtYeFO4bUlABPLySN0ZQyuxEOzvZSSBJUsgL89Ybwf6jkDTNK0RlJSbeWPTQZZtTsKrlRtzJw1kyvk9aOVej+/bSsHixaT+6z0655/AvZV1Ovvp02HxYmO9HehEoGmado42xGYw97v9pOaWMHlYF566egAd/Rs4NYRSpPUeSMjm6D+X2TEJgG4a0jRNa7CjOcXcv/w37v9oJ6093Pn8gfN59dbhDU8CACKkJqfTteDEn8tmzLBbsxDoOwJN07R6K6s0s+ynJF7fdBB3N8WTV4Zx74W98KhPM9DZiFA241FOeF1KSFhviLL82UcAdrsz0HcEDZSXl8ebb77ZoH2vuuoq8vLyGvzefn5+ta4/l9g0Tavd5sRMJr66hVfWJ3JpWEc2PHoJD13S59yTAIBSpLfpiCg3ut52/ck+A6ZPN0YN2al5qOUkgtNvq87xNqu2i63ZXHshiTVr1hAYGHhO718bnQg0rfGl55fw9093c9f7vyIiLL93NG9NGUmXwMZ96CvtrgcA/pxnqCoZ2GnoKLSURDB37qltbFXjcs/hxD7xxBMcOnSIYcOG8fjjjxMdHc24ceO4/fbbOf/88wGYPHkyI0eOZNCgQSxbtuzkvj179iQrK4vk5GQGDBjAAw88wKBBg7j88stPmfa5yuHDhxkzZgyjRo1i9uzZJ5ebTCbGjx/PiBEjGDJkCCtXrjxrbDVtp2la3SrMFpZtPsT4V35iQ1wGj07oz9pHLuaS/kF2eb/UXGM6+1Oml7B36Vdbnjpz9M85PVlssYhMny4Cxp9ne90Ahw8flkGDBp18vWnTJvHx8ZGkpKSTTxZnZxtPBRYXF8ugQYMkKytLRER69OghmZmZcvjwYXF3d5c9e/aIiMjNN98sH3/88RnvNWnSJFm+fLmIiLz++uvi6+srIiIVFRWSn58vIiKZmZnSp08fsVgsZ8RW03b2pp8srh99bmrmqHOz41CWTFgULT1mRsk9H/wqR7KK7P6eL6+Ll15PREl5pdmm7ZvkyWKllBswFOgClAD7RSTDhv28gc2AF0an9JciMkcp9SFwCZBv3fRuETn7xPqNoeq2CowOl6pOFzuMyx09ejS9evU6OYnc0qVL+eabbwA4evQoBw4coH379qfs06tXL4YNGwbAyJEjT04TXd22bdv46quvALjzzjuZOXMmYCTxp556is2bN+Pm5kZaWhoZGWf+1dS0XadOnRrts2tac5JZWMYLa+L4ek8aXQNbs+zOkUwYGHzWmXkbW1puCZ0DWjdOn4ONakwESqk+GPWFLwMOAJmAN9BfKVUMvAMsFxFLDYcoAy4VEZNSygPYqpT63rrucRH5srE+RJ2qkkFVEgC79L77+vqe/D06OpoNGzawfft2fHx8iIyMpLS09Ix9qqaIBnB3dz9r0xCcfWroTz/9lMzMTHbt2oWHhwc9e/Y863vYup2mtXRmi/DJjiO8/EMCpRVm/j6uD/8Y14/Wnu5NFkNqbgldG7nfoS61pZz5wCdAHxG5QkSmiMhNIhIOXAsEAHfWtLP1zqSqlqGH9cd+A2FrU9UnUN05jsuta/ro/Px82rZti4+PD/Hx8ezYsaPB7zV27FhWrFgBcHKK6Kr36NixIx4eHmzatIkjR46cNbaattM07U97UnK57o2tzFm1n6Ehgax95GIevyKsaZJAtWtRWl5Jk00/XaXGOwIRua2WdSeAV+s6uFLKHdgF9AXeEJFflFLTgOeUUk8DG4EnRKTsLPs+CDwIEBwcTHR09CnrAwICbJvHXwSvJ57A8623KJ82jbIFC4zXS5ZQXl5O2YIFDboz8PT0ZPTo0QwcOJAJEyZwxRVXUFlZSWFhIWazmbFjx/L6668zePBg+vXrx6hRoyguLqawsBARwWQyYTKZsFgsJz9HWVkZZWVlZ3yu5557jvvuu49FixZx3XXXAVBYWMh1113HX/7yl5OdwP3798dkMtG+fftTYpsxY8ZZt2tIHYT6KC0tPePvzWQynbFMM+hzUzN7nhtTufC/xHI2p1YS4KWYNtSL0Z2KObp/J0ft8o6nSU+Hykro1o1Ki3Asr4TK1INEfx4H1uqGtWmUc1NXJwKwE/g70NaWTocajhEIbAIGA50BhdF3sBx4uq79z3ka6jlzTu0YruownjPH9mPUg56G2qA7i+tHn5ua2ePcmM0W+fyXIzLsmXXS+8nVMu+7/VJYWtHo71Or0wavpGSZpMfMKFkRPsHmwSxNNQ31rcA9wG9KqZ3AB8AP1jexNdnkKaWigYki8rJ1cZlS6gPgsZr3bCRz5xq3XlXf/Kv6DJqg40fTNOez/5hRL3hPSh6jerZl3uTBhHVq0/SBnDaYJfXrH+H2FwgZfyG8MrvJrlF1dkuLyEER+TfQH/gMeB9IUUo9o5RqV9N+SqkgpVSg9ffWGJ3O8UqpztZlCpgMxJz7x7DB6SdUJwFNa3EKSiuYu2o/k17bSkp2Ma/cPJT/PjTGMUmgSrVkkBpg1Cju+uSjTXqNsmmuIaVUOMZdwVXAV8CnwIXAj8CwGnbrDCy39hO4Af8VkSil1I9KqSCM5qG9wF8bGryINMlwLq3+6nHDqGl2JyJ8uzeN51bHk11UxpTzevDY5aEE+Jy9VGQTB3dyMEtaQEeUWOj83GxYvKjJkoEtzxHsAvKA9zi1Y/cXpdTYmvYTkX3A8LMsv7SBsZ7C29ub7Oxs2rdvr5OBkxERsrOz8fY+hxkYNa2RJGYUMvvbGH45nMPQkADevzuC8BD7TfFSL1VJYMkSmD6d1LF3E7zzEF5LXgWkyZqwbbkjuFlEks62QkRuaOR4bBYSEkJqaiqZmZmOCqFGpaWlLf4i6O3tTUhIiKPD0FqworJKlm48wHtbD+Pr1Yrnrh/MraO64+7mRF8clTImk7M+4Jr2n1/o2rOz3SeZO11tD5RNAT6rKQlYHzjrLCJb7RVcbTw8POjVq5cj3rpO0dHRDB9+xs2QpmlNQERYG3OcZ6NiSc8v5S8RIcycGEZ7P6+6d3aEaoNZUvOKGdG9bZMPZqntjqA9sMfaNLSLP58s7osxRUQW8ITdI9Q0TbPR4awi5qzaz+bETMI6+fP67cMZ2aPGMS3OQynMFiE9r5Su4a2bfDBLbQ+ULVFKvQ5cCowFwjHmGooD7hSRlKYJUdM0rXalFWbe3HSQt39KwrOVG09fM5C7xtSzXrCDZRSUUmmRP6efbkK19hGIiBlYb/3RNE1zOhvjjHrBR3NKuG5YF/591QA6tnG9PrrUXGOesaaeXgJ0qUpN01zU0Zxino2KZX1sBn07+vHZA+dxQZ8Ojg6rwdLyjDoEXXUi0DRNq11ZpZl3txzmtR8PoFDMnBjGfRf2wrOV6zQDnU1qjnFH0NQzj4JOBJqmuZCtB7J4emUMSVlFTBzUidmTBjrkwmkPaXkldPDzwtuj6aa8rmLLA2WBwF1Az+rbi8jD9gtL0zTtT7mlFv7x2W6i9qXTo70PH94zisjQjo4Oq1Gl5jb99NNVbLkjWAPsAP4AaipCo2ma1ugqzBaW/5zMy1tKsKgyHrmsH3+9pI9DvjXbU3TCCf5Iy+eifo7p47AlEXiLyKN2j0TTNK2aXw/nMPvbGBIyCgkPcue1uy+iR3vfund0IQdPFDJ/dRzRCZn0aO/DQxf3cUgctiSCj5VSDwBRGOUnARCRHLtFpWlai5VlKuOFNfF8tTuVroGteefOkXieiGtWSSC3qJxXNyTyyS8p+Hi68++rBnDXBT3wauWYOx1bEkE58BLwb/4sNSlAb3sFpWlay2O2CJ/9coSX1iVQUmFmWmQf/nlpX3w8WxGdGe/o8BpFeaWFj3ccYcmGRExlldxxXg8euayfw6e/sCURPAr0FZEsewejaVrLtPdoHrO/jeGPtHwu6NOeZ68bTN+Ofo4Oq9GICBvjTvDcmjgOZxVxUb8OzLp6IKGd/B0dGmBbItgPFNs7EE3TWp684nIWrk1gxW8pBPl5sfS24UwK79ysppaPSy9g/upYth3MpneQLx/cPYrI0CCn+oy2JAIzsFcptYlT+wj08FFN0xrEYhG+3JXKgrXx5JdUcO/YXjxyWT/8vZ2gUEwjyTKV8coPiXzxWwr+3h7MnTSQO87vgYcTzn9kSyL41vqjaZp2zmKPFTB7ZQy7juQS0cOoFzygswNLRTayskozH25L5vUfD1JSYWbqBT2ZPr4fgT6ejg6tRnUmAhFZ3pADK6W8gc2Al/V9vhSROUqpXsAKoB2wG2Mm0/KGvIemaa6joLSCxesTWf5zMoE+nrx0Uzg3jgjBzZkKxZyDqjoIL3wfT0pOMePDOvLU1QPoE+T8fR22PFncD3gBGIhRjwAAEalr1FAZcKmImJRSHsBWpdT3GJ3Pi0VkhVLqbeA+4K2GfgBN05ybiLDq92PMXx1HlqmM20d35/ErQp36G3J9xaTl82xULL8eziE02J+P7xvNRf2CHB2WzWxpGvoAmAMsBsZhFLGvM4WLUb3cZH3pYf0RjPoGt1uXLwfmohOBpjVLB08UMvvb/WxPyiY8JIB374pgaDcnqRfcCE4UlPLSugS+3J1KWx9P5k8ezK2jurlUHQQAZVyva9lAqV0iMlIp9YeIDLEu2yIiF9V5cKXcMaqb9QXewHgeYYeI9LWu7wZ8LyKDz7Lvg8CDAMHBwSNXrFhRv0/mQCaTCT8/578ddAR9bmrWnM5NWaWw8lAF65Ir8HKHm/p7EtmtFW4NHCnjbOem3CysTa5gdVIFlRaY0MODSX088PVo+mau2s7NuHHjdolIRF3HsOWOoFQp5QYcUEr9A0gDbJrtyVrYZph14rpvgAFn26yGfZcBywAiIiIkMjLSlrd0CtHR0bhSvE1Jn5uaNYdzIyKs23+cZ7+L5Vh+BTeNDOGJK8PocI4PTDnLuRERvtuXzsLv40nLq+CKQcE8eeUAenZw3FPPjXFubEkEjwA+wMPAPIymnan1eRMRyVNKRQPnA4FKqVYiUgmEAMfqFbGmaU4p2Vov+CdrveAltw1nVE8XqBdso71H85gXFcuuI7kM7NyGl28eypg+7R0dVqOwZdTQb9ZfTRj9AzZRSgUBFdYk0Bq4DFgIbAJuwhg5NBVYWd+gNU1zHqUVZt6KPsRbPx3C092N2dcMZKqL1QuuTXp+CS+uTeCbPWl08PNi4Y1DuGlkN9ybyWgnqCURKKW+o4ZmGwARubaOY3cGllv7CdyA/4pIlFIqFlihlJoP7AHeq3/YmqY5g03xJ5izaj8pOcVMGtqFWVcPINgF6wWfTXF5Je/8lMQ7mw9hEfhbZB/+Nq4vfl7Nr55XbZ/oZeufNwCdgE+sr28Dkus6sIjsA4afZXkSMLpeUWqa5lRSc4t59rtYfojNoE+QL5/efx5j+7puveDqLBbh271pvLg2geMFpVwd3pknJobRrZ2Po0OzmxoTgYj8BKCUmiciF1db9Z1SarPdI9M0zemUV1p4d2sSSzca9YL/NTGU+y/s7fL1gqvsTM5hXlQsv6fmEx4SwGu3N69+jprYco8TpJTqbf0mj/XJYNd5UkLTtEbx88EsZq+M4VBmEZcPDObpSQMJads8viUfzSlmwdp4Vu9LJ7iNF6/cPJTrh3dtNk8918WWRDADiFZKJVlf98Q6vl/TtObvREEp81fHser3Y3Rv58MHd49iXFjzqBdsKqvkreiD/GfLYdwUTB/fj4cu6Y2PZ/PrB6iNLaOG1lqnmQizLooXkbLa9tE0zfVVmi0s336ExesTKTdbmD6+H9Mim0e9YLNF+GpXKi/9kEBmYRmTh3XhXxPD6BLomOLxjmZT2rNe+H+3cyyapjmJnck5zPo2hvjjhVzSP4hnrh3k0IemGtP2Q9nMi4olNr2AEd0DWXbnSIZ3b+vosByqZd3/aJpWq2xTGS98H8+Xu1LpEuDN21NGcMWgTk5VRKWhjmQX8fyaONbtz6BrYOtmWQSnoXQi0DQNs0X4/NcUXlqXQFFZJX+9pA8Pj+/bLNrKC0oreP3Hg3y4LZlW7orHLu/P/Rf1bhZNXI3FlmmoFXAH0FtEnlVKdQc6icivdo9O0zS725eax6xvY9iXms/5vdsx77rB9At2jlq656LSbGHFb0dZvD6RnOJybhoRwmNXhDabB94aky3p/k3AgjHH0LNAIfAVMMqOcWmaZmf5xRW8uC6ez35NoYOfF0tuHca1Q7s0i6aSLQcymR8VR0JGIaN7tWP5NQMZ3DXA0WE5LVsSwXkiMkIptQdARHKVUs2nooSmtTAWi/DV7lQWfB9PbnE5d1/QkxkT+tOmGdQLPpRp4vnVcWyMP0G3dq15644RTBzcPPo47MmWRFBhnS9I4ORkcha7RqVpml3EpRcw+9sYdh7JZUT3QD66bzSDurj+N+W84nKWbDzAx9uP4O3hzhNXhnH3BT11P4CNbEkESzFqCXRUSj2HMXPoLLtGpWlaoyosrWDx+gMs355MQGsPXrwxnJtGun694AqzhU93HOHVjQcoKKngllHdeXRCf4L8z63+QUtjywNlnyqldgHjMUpUThaROLtHpmnaOasqpDI/KpZMUxm3je7Ov5pBvWARITohk/mrYzmUWcTYvu2ZdfVABnRu4+jQXJKtY8MOAAVV2yuluotIit2i0jTtnB08YeLplTH8fCibwV3bsOyuCIY1g3rBiRmFzF8dx+bETHp18OU/d0Vw2YCOuh/gHNgyfPSfGMXrMwAzxl2BAOH2DU3TtIYoLq/ktR8P8u6WJFp7uDPvukHcfl4Ply+kklNUzuL1iXz2awq+nu7MunoAd43p2WxmPnUkW+4IpgOhIpJt72A0TWs4EeGH2Aye/S6WtLwSbhwRwpNXnXu9YEcrr7Sw9nAF/4zeRHG5mTvO684jl/Wnna9rN285E1sSwVEg396BaJrWcCnZxcxZFcOmhExCg/3570NjGN3LtefRFxHWx2bw/Jo4krPLuaR/ELOuHtAsHnZzNrWVqnzU+msSxjTUq4GTs46KyKLaDqx48r+pAAAgAElEQVSU6gZ8hFHdzAIsE5ElSqm5wANApnXTp0RkTYM/gaa1YKUVZt75KYk3og/i4aaYdfUApl7QEw8Xrxcce6yAeVGxbE/Kpm9HPx4d6cXDN+vChvZS2x1BVdpNsf54Wn+gllrG1VQC/yciu5VS/sAupdR667rFIvJyLftqmlaH6ASjXvCR7GKuCe/MrKsH0inAtadPyCwsY9H6BFb8dpSA1h48e90gbhvdnW1bdFFEe6qtVOUzAEqpm0Xkf9XXKaVuruvAIpIOpFt/L1RKxQFdzy1cTdOO5ZXw7HexrN1/nN5Bvnxy33lc2M+16wWXVpj5YFsyb2w6SGmFmXvH9uLhS/sR4OP6Tzu7AiVS+5d7pdRuERlR17I6jtET2AwMBh4F7sYYjroT464h9yz7PIi1ElpwcPDIFStW2Pp2DmcymfDz83N0GE5Jn5ua1XVuKi3CuuQKVh6qAIFJfTyY2MsDDxceDSQi7Mww80VCOVklwrAgd24N86ST76lNW/rfTc1qOzfjxo3bJSIRdR2jxkSglLoSuAr4C/BFtVVtgIEiYlODnVLKD/gJeE5EvlZKBQNZGM1L84DOInJvbceIiIiQnTt32vJ2TiE6OprIyEhHh+GU9LmpWW3nZvuhbGavjOHgCRMTBgbz9DUD6dbOtesF/5Gaz7yoWH5NziGskz+zrxnI2L5nv7PR/25qVtu5UUrZlAhq6yM4hvGN/VpgV7XlhRh1jOuklPLAmKn0UxH5GkBEMqqt/w8QZcuxNK0lOlFQynNr4li59xjd2rXmvakRjB8Q7OiwzklGQSkvrk3g6z2ptPPx5Pnrh3DLqG4u/5yDK6utj+B34Hel1GciUlHfA1vrGLwHxFUfYaSU6mztPwC4Hoip77E1rbmrNFv4yFovuKzSwsOX9uVv4/q69CRqJeVm/rMlibeiD2G2CA9e3Ju/j+vbLGY9dXW2zDVU7yRgNRa4E/hDKbXXuuwp4Dal1DCMpqFk4KEGHl/TmqVdR3KZ9W0McekFXGytF9zLhesFiwirfj/Gwu/jOZZfypWDO/HklQPo3t61m7aaE7vVoRORrRjTUZxOPzOgaWdRWC7868vf+e/OVDoHeDeLufR3p+QyLyqWPSl5DOrShkW3DOP83u0dHZZ2mtoeKPtYRO5USk0XkSVNGZSmtSQWi/D5byk8v6WYMnMJD13cm4fH98PXy3XrBR/LK2Hh2nhW7j1GkL8XL94Uzo0jQnQ/gJOq7V/aSKVUD+BepdRHnPbtXkRy7BqZprUAf6TmM2tlDL8fzSO0rRuv3X0h/V14CoWiskre+ekQy7YkIQL/GNeXaZF9XDqptQS1/e28DawFemOMGqqeCMS6XNO0BsgvruDlHxL45JcjtPf14tVbhhGQl+iyScBiEb7ek8ZL6+LJKChj0tAuzJwYSkjbJuoHEIHqTWinv9ZqVduooaXAUqXUWyIyrQlj0rRmS0T4encaz6+JI7e4nKljjHrBAa09iI4+4OjwGuS35Bye/S6WP9LyGdotkDfvGMHIHk044d3cuZCXB4sXGxd/EZgxAwIDjXVanWwZNTRNKTUUuMi6aLOI7LNvWJrW/MQfL+Dpb/fza3IOw7sHsvze0Qzu6rr1go/mFLPg+3hW/5FOpzbevHrLMK4d2qVpy1+KGElgibUbc/FiIwksWQLTp+s7AxvZUpjmYYypHr62LvpUKbVMRF6za2Sa1kyYyip5dX0iH/ycTBvvViy8cQg3j+zmsvWCC0sreDP6EO9tPYy7UjxyWT8evLg3Pp4O6AdQyrj4g3Hxr0oI06f/eYeg1cmWv7n7gfNEpAhAKbUQ2A7oRKBptRARVv+RzryoWDIKyrhtdDf+dUUYbV20oIrZIvxv51Fe/iGBLFM5NwzvyuMTQ+kc0NqxgVUlgyXVBjfqJFAvtiQChVGiskpVuUpN02pwKNPEnJX72Xowi0Fd2vD2lJEM797W0WE12M+HspgXFUdcegEje7TlvamjGOos9Y+r+gSqmzFDJ4N6sCURfAD8opT6xvp6MsbUEZqmnaak3Mzrmw6wbHMS3h7uPHvdIO5w4XrBh7OKeH5NHOtjM+ga2JrXbx/O1UM6O89DblVJoKpPoHofAehkYCNbOosXKaWigQsx7gTuEZE99g5M01zN+tgM5q7aT1peCTcM78qTVw0gyN816wXnl1Tw2sYDLN+ejKe7G49fEcp9F/ZyvrmOlDJGB1XvE6jqMwgM1EnARjb17ojIbmC3nWPRNJd0NKeYuav2szH+BP2D/fjiwfM5z0WnUag0W/j8t6MsXp9IbnE5N48M4bHLQ+nYxokrn82de+rooKpkoJOAzfTjfprWQGWV1nrBmw7Syk3x76sGcPdY160X/FNiJs+tjiUxw8R5vdox+5qBrjO89fSLvk4C9aITgaY1wE+JmcxZGUNydjFXD+nMrGsGOH70TAMdPGHiudWxbErIpHs7H96eMpIrBgU7Tz+AZne2PEewUERm1rVM01qC9PwS5kXFsuaP4/Tq4MtH947m4v5Bjg6rQXKLylmy8QAf7ziCj4c7T10VxtQLeuLVysn6ATS7s+WOYAJw+kX/yrMs07Rmq8Js4f2th1my8QBmi/DY5f154OLeLnnRrDBb+Hj7EZZsPEBhaQW3je7OjAn96eDnmh3b2rmrbRrqacDfgN5KqepTSvgD2+wdmKY5ix1J2cz+NoYDJ0xcNqAjcyYNcsl6wSLCj/EneG5NHEmZRVzYtwOzrhlAWKc2jg5Nc7Da7gg+A74HXgCeqLa8UE9BrbUEJwpLeWFNPN/sSSOkbWvevSuCywa6Zr3ghOOFzF8dy5YDWfTu4Mt7UyO4NKyj7gfQgNpnH80H8pVSpzcB+Sml/EQkpbYDK6W6AR8BnQALsExEliil2gFfAD0xSlX+RURyG/4RNK1xVZotfLLjCK/8YNQL/uelfflbZF9ae7peM1C2qYxF6xP5/NcU/L09ePqagdw5pofLjmzS7MOWPoLVGPUHFOAN9AISgEF17FcJ/J+I7FZK+QO7lFLrgbuBjSKyQCn1BMbdhu5v0JzC7pRcZn0TQ2x6ARf168Az1w6id5Cfo8Oqt7JKM8t/Tua1jQcprjBz15ieTB/fz2XnOdLsy5Yni4dUf62UGoENBedFJB1It/5eqJSKA7oC1wGR1s2WA9HoRKA5WE5ROS+ujWfFb0fp1MabN24fwVVDXK9esIiwbn8GL3wfx5HsYsaFBvHvqwfQt6NrFrzRmoYSkfrvpNRuERlRj+17ApuBwUCKiARWW5crImfMxqWUehBj+muCg4NHrlixot5xOorJZMLPz/W+RTYFZzs3FhE2p1byv8RySirh8h4eXNfXg9atmj4BnOu5OVJg5vP4cuJzLHTxU9wW6smQoObxqJCz/btxJrWdm3Hjxu0SkYi6jmHLcwSPVnvpBowAMm0NUinlB3wFPCIiBbZ+wxKRZcAygIiICImMjLT1LR0uOjoaV4q3KTnTuYlJy2fWtzHsPVrM6F7tmHfdYEI7Oe6bc0PPzYnCUl5el8D/dqUS2NqDeZMHctuobrRqRv0AzvTvxtk0xrmx5etC9f8zKjH6DL6y5eBKKQ/rtp+KSFVhmwylVGcRSVdKdQZO1CdgTTtX+SUVLPohgY93HKGdryeL/jKU64d3dblmoNIKM+9tPcybmw5SbrZw39he/HN8PwJaezg6NM3F2NJH8AyAtcNXRMRky4GV8X/Ve0CciCyqtmoVMBVYYP1zZX2D1rSGEBG+2ZPG82viySkq487ze/Do5aEud+EUEaL2pbPg+3jS8kq4fGAwT141gF4dfB0dmuaibGkaGgx8DLSzvs4CpopITB27jgXuBP5QSu21LnsKIwH8Vyl1H5AC3NzA2DXNZgnHC5m9MoZfD+cwtFsgH94zynUmVKvm96N5zIuKZeeRXMI6+fPZ/edxQd8Ojg5Lc3G2NA0tAx4VkU0ASqlI67ILattJRLZScyWz8fWIUdMarKiskiUbD/D+1sP4ebfihRuGcEuE69ULPp5fyotr4/l6Txod/DxZcMMQbo7o5rIFbzTnYksi8K1KAgAiEq2U0vegmlMTEdb8cZx5UbEcLyjl1lHd+NfEMNq52Dj6knIz72w+xDs/JWG2CNMi+/C3yD74e7tWc5bm3GxJBElKqdkYzUMAU4DD9gtJ085NUqaJOav2s+VAFgM7t+GNO0Ywsodr1Qu2WISVv6fx4toE0vNLuXpIZ564Mswl5zjSnJ8tieBe4BmgatTPZuAeu0WkaQ1UUm7mzeiDvPNTEl6t3Jg7aSBTzu/hcsModx3J5dmoWH4/mseQrgEsuXU4o3u1c3RYWjNmy6ihXODhJohF0xpsQ2wGc7/bT2puCdcP78qTV4XR0d+JyyueRVaJhX9+vofvfj9GR38vXr55KDcM7+py/Rma62kejx1qLdbRnGKe+S6WDXEZ9Ovox+cPnM+YPq5VL7iorJK3og/xzpYS3NzKePjSvjx0SR98vfT/nlrT0P/SNJdUVmnmP5uTeH3TQdyU4skrw7j3wl4uNaumxSJ8uTuVl9YlkFlYxvmd3Xll6iV0DXTNkpea69KJQHM5Ww5kMmflfpKyirhycCdmXzOQLi528fwlKZt5q2OJSStgWLdA3rlzJAVJv+skoDmELQ+ULT3L4nxgp4jop4K1JnM8v5R5q2NZvS+dnu19WH7vaC5xsXrBKdnFvPB9HN/HHKdLgDdLbh3GtUO7oJQiOsnR0WktlS13BN5AGPA/6+sbgf3AfUqpcSLyiL2C0zQwaux+uC2ZVzckUmkRHp3Qnwcv7o23h+sUiiksreD1TQf5YGsy7m6KRyf054GLertksRut+bElEfQFLhWRSgCl1FvADxhF7f+wY2yaxi9J2Ty9cj8JGYVcGtaRuZMG0b2964ylN1uEL347yis/JJBdVM6NI0J4/IpQOgW41ogmrXmzJRF0BXwxmoOw/t5FRMxKqTK7Raa1aJmFZbywJo6v96TRNbA1y+4cyYSBwS41Q+i2g1nMi4ol/ngho3q25YN7RhEeElj3jprWxGxJBC8Ce5VS0RhzB10MPG+dZmKDHWPTWiCzRfj0lyO8tC6B0gozfx/Xh3+M6+dSTShJmSaeXxPHhrgThLRtzZt3jODKwa5X7UxrOWx5oOw9pdQaYDRGInhKRI5ZVz9uz+C0lmVPSi6zV8YQk1bAhX078Mx1g+jjQvWC84srWLLxAB9tT8bbw51/TQzl3rG9XKovQ2uZbB0+6oZRlawV0Fcp1VdENtsvLK0lyS0q58V1Caz4LYWO/l68dttwrgnv7DLfoCvMFj77JYXFGxLJL6ng1lHdeHRCKEH+Xo4OTdNsYsvw0YXALRgjhSzWxYIx55CmNZjFIvxv11EWfB9PQWkl943txSMT+uPnQk/URiecYP7qOA6eMDGmd3tmXTOAQV1cr86B1rLZ8n/cZCBURHTHsNZo9h/LZ/a3MexOyWNUz7bMmzyYsE5tHB2WzQ5kFDJ/dRw/JWbSs72PS3Zma1oVm6ahBjwAnQi0c1ZcIcxdtZ+PtifT1seTV24eyg0jXKdecE5ROa9uSOTTX1Lw8XRn1tUDuGtMTzxbuc7UFpp2OlsSQTHGqKGNVEsGIlLrjKRKqfeBa4ATIjLYumwu8ABGfwMYHc9rGhC35mJEhJV7jzFnawkF5clMOa8Hj10eSoCPaxRYKa+08NH2ZJZuPICprJI7zuvBI5f1o72f7geolQhUT/Knv9acgi2JYJX1p74+BF4HPjpt+WIRebkBx9Nc1IEMo17wjqQcege48ckDYxkS4hrt6CLCxrgTPLcmjsNZRVzUrwOzrxlI/2B/R4fm/ObOhbw8WLzYuPiLwIwZEBhorNOchi3DR5c35MAislkp1bMh+2rNQ1FZJUt/PMB7Ww7j69WK568fQqfiQy6TBOLSC5i/OpZtB7PpE+TLB3ePIjI0yGWasRxKxEgCS5YYrxcvNpLAkiUwfbq+M3AySkTOvkKp/4rIX5RSf2CMEjqFiITXeXAjEUSd1jR0N1AA7AT+z1r45mz7Pgg8CBAcHDxyxYoVdX8aJ2EymfDzc53x741NRNiZYebz+HJySoWLurbi5lBP2ngqlzg3BWXC1wfK+Sm1Eh8PmNzXk3HdWtHKzgViXOHc1NvRo3DixJ+vO3aEbt3qfZhmeW4aSW3nZty4cbtEJKKuY9SWCDqLSLpSqsfZ1ovIkToPfmYiCAayMBLLPKCziNxb13EiIiJk586ddW3mNKKjo4mMjHR0GA5xOKuIOav2szkxkwGd2zB/8iBG9vizzKIzn5uySjMfbEvm9R8PUlph5s4xPZg+vh+BPk1T8N6Zz02DiYBbtY50i6VBdwLN8tw0ktrOjVLKpkRQY9OQiKRbfx0PbBGRAw0J8rRjZlQL8D9A1LkeU3MOpRVm3ow+xNvRh/Bs5cacSQO500XqBYsIa2OO88L38aTkFDM+rCNPXT3ApZ5qdkpVfQLVzZjxZ5+B5jRs6SzuCUyx3hnsArZgJIa99X2zqrsM68vrgZj6HkNzPj/GZzBn1X6O5pRw3bAu/PuqAXRs4xqza8ak5fNsVCy/Hs4hNNifj+8bzUX9XKvGgVOqSgJVfQLV+whAJwMnY0tn8dMASqnWGEM/HwdeBWqdQEUp9TkQCXRQSqUCc4BIpdQwjKahZOChc4hdc7DUXKNe8PrYDPp29OOzB87jgj4dHB2WTU4UlPLSugS+3J1KWx9P5k8ezK2jurnEHYxLUMoYHVSVBJQy/gRjuU4CTsWWKSZmAWMBP2AP8BjGXUGtROS2syx+r74Bas6nrNLMu1sO89qPB1AoZk4M474Le7nEQ1WlFWbe3ZLEm9GHqDBbePCi3vz90r608XaN5xlcyty5p44OqkoGOgk4HVuahm4AKoHVwE/ADhEptWtUmtPadjCL2StjSMosYuKgTsyeNNAl6uyKCN/tS2fh9/Gk5ZVwxaBgnrxyAD07+Do6tObt9Iu+TgJOyZamoRFKKX/gQoyqZP9RSmWIyIV2j05zGhkFpcyLiiVqXzo92vvw4T2jiAzt6OiwbLInJZd5UbHsTsljYOc2vHzzUMb0ae/osDTNadjSNDQYuAi4BIgAjmJD05DWPFSYLSz/OZnF6xOpsAiPXNaPv17SxyXm2D+WV8KLa+P5du8xOvh5sfDGIdw0shvudn4eQNNcjS1NQwsxppxeCvwmIhX2DUlzFr8l5zD72xjijxcyLjSIudcOokd7529KKS6v5O2fkli2+RAWgb+P68O0yL4uNb21pjUlW5qGrlZKeQL9gVClVIJOBs1blqmMF9bE89XuVLoGtuadO0dyuQtMsWyxCN/uTWPh2ngyCsq4OrwzT0wMo1s71yl2r2mOYEvT0CUYE8clY5Sq7KaUmqorlDU/Zovw2a8pvLQ2npIKM9Mi+/DPS/vi4+n836R3JufwbFQs+1LzCQ8J4PXbRzCqZ7u6d9Q0zaamoUXA5SKSAKCU6g98Doy0Z2Ba0/r9aB6zV8awLzWfC/q059nrBtO3o/M/WXs0p5gFa+NZvS+d4DZeLPrLUCYP64qb7gfQNJvZkgg8qpIAgIgkKqX0oOtmIq/YqBf8+a8pBPl5sfS24UxygXrBprJK3tx0kHe3HsZNwfTx/Xjokt4ucfeiac7Glv9rdiql3gM+tr6+A2OqCc2FWSzCl7tTWfB9PPklFdw7thePXNYPfyd/sMpsEb7cdZSX1iWSZSrj+uFdefyKULq4wLMMmuasbEkE04C/Aw9j9BFsBt60Z1CafcUeK2D2yhh2HckloodRL3hAZ+evF7z9UDbzomKJTS9gRPdA3p0awbBugY4OS9Ncni2jhsow+gkW2T8czZ4KSytYtD6Rj7YfIbC1By/dFM6NI0Kcvj39SHYRz6+JY93+DLoGtnaZ5itNcxU1JoKaCtJUsaUwjeYcRIRVvx9j/uo4skxl3HFedx6/PMzp6wUXlFbw+o8H+WDbYTzc3Xjs8v7cf1Fvl3iYTdNcSW13BDcDJU0ViGYfB08UMvvb/WxPyiY8JIB374pgqJM3p1SaLaz47SiL1ieSW1zOTSNCePyKUJeZ2lrTXE1tieAz6zxDH4vInU0WkdYoissrWbrxIO9tTaK1hzvzJw/mttHdnX56hS0HMpkXFUtihonRvdrx9DUDGdzVNWoca5qrqi0ReCqlpgIXKKVuOH2liHxtv7C0hhIR1u3PYF5ULGl5Jdw0MoQnrgyjg5+Xo0Or1cETJp5fE8eP8Sfo1q41b90xgomDO+l+AE1rArUlgr9iDBUNBCadtk4AnQiczJFso15wdEImYZ38+fKvY4hw8qdr84rLeXXDAT7ZcQRvD3eevDKMu8f2xKuV7gfQtKZSW83ircBWpdROEdEFZZxYaYWZt6IP8dZPh/B0d2P2NQOZOsa56wVXmC18uuMIizccoLC0gltGdefRCf0J8nfuOxdNa45sGT7aoCSglHofuAY4ISKDrcvaAV9g1EFOBv4iIrkNOb5m2JRwgrmr9nMku5hJQ7sw6+oBBDtxp6qI8GN8Bs+tjuNQZhFj+7Zn1tUDXeI5Bk1rruz5PP6HwOsYE9ZVeQLYKCILlFJPWF/PtGMMzVZaXgnPfrefdfsz6BPky6f3n8fYvs5dLzgxo5BXdpYRk72TXh18efeuCMYP6Kj7ATTNwWpNBMr4PzRERI7W98Aislkp1fO0xddhFLQHWA5EoxNBvZRXWnh3axKvbTwIwL8mhnL/hb2dul5wtqmMxRsS+eyXFLzdYfY1A7nz/B5OHbOmtSRKpMZnxowNlNolIg2aadSaCKKqNQ3liUhgtfW5ItK2hn0fBB4ECA4OHrlixYqGhOAQJpMJP7/Gn7kzLtvMR7FlpBcJIzq6c/sATzq0dt6LaaVFWH+kklWHyikzw7hurZjQuYJObZ1/VlNHsNe/m+ZAn5ua1XZuxo0bt0tEIuo6hi1NQzuUUqNE5Lf6BnguRGQZsAwgIiJCIiMjm/Ltz0l0dDSNGe+JglLmr45j1e/H6N7Ohw9uHsS4MOetFywi/BCbwQtr4kjOLicyNIh/XzWAfsH+jX5umhN9bmqmz03NGuPc2JIIxgF/VUolA0UYE89JA6eYyFBKdRaRdKVUZ+BEA47RYlSaLSzffoTF6xMpN1uYPr4f0yKdu15w7LEC5kXFsj0pm74d/fjgnlGMc5Ei95rWUtmSCK5sxPdbBUwFFlj/XNmIx25WdibnMMtaL/iS/kE8c+0genZw3nrBJwpLWfRDIl/sPEpAaw+evW4Qt43ujocTD2HVNM1gy/DRI0qpC4F+IvKBUioIqLOxTin1OUbHcAelVCowByMB/FcpdR+QgjGfkVZNtqmMBd/H879dqXQJ8ObtKSO4YpDzPmFbWmHm/W2HeePHg5RVWrh3bC8evrSf009op2nan2ypWTwHiABCgQ8AD+ATYGxt+4nIbTWsGl/PGFsEs0X4/NcUXlqXQFFZJX+9pA8Pj3feesEiwpo/jvPC93Gk5pZw2YBgnroqjN5BukNP01yNLVeZ64HhwG4AETmmlPK3a1QtzL7UPGZ9a9QLHtO7PfMmD6JvR+c9xftS85gXFctvybmEdfJ3iWcYtOavpNyMqaxSP53eALYkgnIREaWUACilnLeh2sXkF1fw0g/xfPpLCh38vFhy6zCuHdrFaZuBjueX8uK6eL7enUZ7X0+ev34It4zq5vQzmmquRUQoLjeTU1ROXnEFOcXl/HysksPbDpNbVE6udVlecTk5RRXWP8spq7TQu4MvPz4W6eiP4HJsSQT/VUq9AwQqpR4A7gXetW9YzZuI8NXuNF5YE0ducTl3X9CTGRP608ZJ6wWXlJv5z5Yk3oo+hNkiPHRJb/4+rq/TxqudhQhU/4Jx+mu7va1gKqskt8i4eOcWl5+8mBt/Vi2rOOX3crPlzIPti0UpCGjtQTsfT9r6etI10JvBXdrQ1teTtj6edA5w3ulVnJktncUvK6UmAAUY/QRPi8h6u0fWTMWlF/D0yhh+S85lRPdAPrpvNIO6OOd8+xaLUdls4dp40vNLuXJwJ568cgDd2/s4OjStPubOhbw8WLzYuPiLwIwZEBhorLORiFBQWkluUflZv5GfcXG3vq60nP2hVTcFbX08CfTxoJ2vJ93a+TA0JJBAX+uF3nqxb+vjwYGYvVwx7kICWnvoO1A7sKWzeKGIzATWn2WZZqPC0gpe3XCAD39OJqC1By/eGM5NI523XvDulFye/S6WvUfzGNSlDYtvGcb5vds7OiytvkSMJLBkifF68WKYMQPLkqXkT/8/ck8UkltScfIb++kX9xzrRb+qmaY2fl6tCPTxINDHg+A23oR2akNbHw+C/LzoFOBNO19PAn08aefrSTsfT/y9W9n879+U7EY7X89zPRtaDWxpGprAmfMBXXmWZdpZiAjf7UtnflQsmaYybhvdnX9dEUqgj3P+o07LK2Hh9/Gs+v0YQf5eDilwn19SQWpuMWm5JaSe/ClmzrWD6BrY+pyPLyJkmco5nFXE4SwTSZlFJGUVcTy/FLNFThbqrpp+pWoWFkEQ4ZT1Yqyg6o+Ty6z7Ve1zynGqjmtdVlxeSUFppbFy7eoz4q26AFYdu/pxanv/skoLZouA9wSYOcFY+OSaU18v2mz7iauDqawSU1klqbmnVrj1cFd8+/exTnvnq9VevH4a8Degt1JqX7VV/sA2ewfWHBzKNPH0yhi2HcxmSNcAlt0VwTAnrRdcVFbJ2z8dYtnmJAD+Ma4v0yL74OvV+MNXqy701S/y1X8vrLooWvl4uhPStjV5xeX1SgRFZZUczjIu8oczrRd96++FZX++h6e7Gz3a+9C1bWtauSmMh+eNVhTFn03pCmUsq/ba+p91e3Xa9sbFOdtUzrG8Eo7ll1BacZa271ooBVcN6XTq+1d7vyqpuSUkZhSSklNs87F9PN2tzS8etLPVi7kAAB4ASURBVHJzO5lISivMZJuM5p0aWnXOysNd0cHPi/Z+nrT3Nf7s4OdFlwBv+uhhxU6t1prFwPfACxjTRVcpFJEcu0bl4soqhRfXxvOfLUa94HmTB3O7k9YLtliEr3an8tK6BE4UljFpaBdmTgwlpG3D+gFEhIKSSo6evLifepE/klVEydofTtnH19OdkLY+hLRtzeiebU/+3rVta0La+tDWx6PGkVQVZgtHc4qNC771m/3hLBOHs4rIKCg7uZ1S0CWgNb2DfLl+RFd6dfCld5AfvTv40iWwdYP/bkrKzaTllRgXeutPWl7pyQt/el7p2Ts+q2nlpugX7E8Ht2IujwhlUNcABnRqQ2vPM6cSySkqJ/54AQnHC0k4Xkj88UIOZBRSVG4+uU33dj706uBLBz8vOvh7EuTrif/Kr8jb+gtZvoFk+wSSNSCc7A69yTaVk3C8kArz2a/4gT4e/9/emYfHVZ6H/vfOjDTSbNpGlmRJtmRjy9gyYGMMhM0OiUMChaZJGnJpk7RpaW9yC4GspAmhkFvCJYXAk5CGJDxNc0khyQXCvgRQApTFrLa8YltetFrraF/nvX+cM6MZLSPJljRavt/z6JmZc75zznc+zbzvd97tI8ebSo7PTe4oIW9tj+wPpLnmbMSbITGJVigLASHgMwAisgRIA3wi4lPVo7PTxflDpNjat17uobn3IJ/YWMQNH5u76wW/UdXCLY/vZmdNiNOLM/nJX23kzOWJl7ZUVXtGP1rIV7f2UNPaEzfbBkvQF2dbwr0otYdz1q+iMDM9KvAzEwj6yDUb2ns52GgJeGt2bwn9oy3dlvnDJsuTQmnQywWrci1hH/RSmuulJMc75RpN4bDS1NVHbVsvNa0RId8TFfK1bb20dPVP6ZwpTmFNfoDywgDlhRmUL82gLN9PWorTKh52bglgZWxX1oTYW9/B3rp29tS38/qhlnEdrwBOh5DtTaW1q3/0k0HKOti6DrfLQXCgi2DdMfJQ1m7eQI7PTdCevUcEfdBnOWpNiZDFwWScxX8G3AEsxSoStxzYA6yb2a7NL442d3PTY7t4Ye9xinzCb/7hXDaXzs31go+1dHPrU3t4cmc9+YE0fvhpK3/B4RBUlbbukYK+m5q2YXt95whB73O7LCGf5eGcFTn2+2FBn5E+LOgrKirYcsGKMfvV3jswLOQbbTOO/dcdM+N1uxyUBr2cWuDnY+vzKQ36WJHrpTTHS9YUHIo9/UO2QLeFfOvEs3lvqpPCrHSWZqZTvjSDvsEwzV1WSGRLVz9NnX30DVrHpLocnFoQoHxpgPWFGZQXZrA6zx9dh6Gnf4jjHb08XVnPKweaeGFXN81j+AgS4RDLhxCZpUdm6LGz9eCDvyLY0ULOrTfjcbss09J114E3BJ/6/JSuZ1iYTMYA/D3gHOAPqrpBRLZiPyUYrJnbT/94iHsqDuByCN++9FRKBo7MSSXQ0TvAj188yH0vV9E/FGZNvp+PlhfwztFWHnuvNir0Y80MAH63i6JsD8XZsYLeEvLFWR4C6ZM3CQyElfcbOoaFfHR230lT5/Ds2iFQlGWZODaXZlsz+6CP0lwvBYG0CZ3XsbP5YZONJezHm807BPICaSzNTOe0okwuKU+jMDOdpRnp5AXS6BkY4khzF7tq26msCfFoVW1UQaWlOCjL83N6cQYFGekUZKSR5UmlrWeA5s4+3jnaxgPbj/HusbZJjVOElblezlyexRJ/WlTQB21TTNBnReFMaNa6+Wuj8wYioaQGA5NTBAOq2iwiDhFxqOqLInLbjPdsHlBhrxd8uLmby04r4NuXriU/I42KirlnNatt6+HyH71CU+ew3XyvbWMOpLkoyvKwLMfDB07JiQr5iMDPSJ9a4lg4rNS190YdtAcbh2f2x1q60WeHI1WCPjcrgl4uXpNHaa6X0qCXlbleirM9uF3jm3JGzeZHCPyJZvOnFWVaQj4zjaUZ1rb8jDRSnA4GhsLsb+igsibEa4da+OH+9yc0AaU6HeyoCfFedWhKY+VPc3HeyiDnnZKDo/kQf7HtojF9AyfNSKFvlIAhhskogjYR8QF/Au4XkePA4ATHLGhq23q4+bHdPL2rnhW5Xv7vF87m/FVzu9ZOWoqTrWW5+NNS4oR8oW26ORHauvs5ZDtpIw7aQ41dHG7uiouO8aQ6KQ16Oa0ogw1ZA2zdtI7SoJeSoHfMa4fDSnNXPzVtHXHC3XpvCfzmEYJZBPL8aSzNTBs1m1+amU5hZjqBdBcDQxo14TR39VPX1sP9rx/lj/sbCfUkjpOPJZDmIsubSlffIE2d/cPhn2OwvjCDUwv8lOUHWJPvpyzfP6bfqKLi8MwoAYNhAiajCK4AeoHrgKuADODmmezUXKV/MMx9r1Rx9/PvE1blax8p4+8uKE04c50rZHtTuf1Tp0/5uN6BIQ43WyackUK/NSbByOkQlmV7WBH0cv4pwZjZvY8lfnecj+Cc8nxq23rYWR2ipq07bjZv2ed76R+Mn817Up32DD6d8sIMirKs2XxBRjo+twuXUwh1D9Dc1U9zZx9Nnf3sb+jg1YPNNHf209TVR01rT9R+PxGpLgdbVueycXlWVGg3dfbR1NFHY2cfhxq7eP94R1ThOQRKcryU2YLeEvgBlmV75mS0mMEQS6I8gi9j5Qu8o6oRo/EvZ6VXc5BXDzbznd9XcuB4Jx9em8eNl62lOHthlFoYCiu1bT3DUTlNXdHZfW2oJ5rABJAXcFMa9HJJeQErbWFfGrRMOSlORzRZKyLUH3uvNm42f7ixi46nn467fuxsvrwwg4tW55KW4sSd4sTtcuB2OegfCtPS2U9zVz/Vrd28d6yN5q4+mjvHL2EwEekpTraU5bKlLJdzVuSwLNtDd/8Q+xuGQzP/uK+RfQ0dcaahXL+bNfl+/urs5bbQD7AqzzenV44zGBKR6ImgCLgLWGMnlP03lmJ4dTHlERzv6OVfn9jDI+/WUpydzn2f38QH1+Qlu1tTRtUyt0QctIfsyJyqpi6ONHfH2dP9bhcrcr2cVZJFabCY0lwrDLMk6MXlkDgTzY7qEE9X1seEVY6ezaelOPCmunC7HHhcQmG2zxbwTtwpDpwOob3Hms3vb+gcFZUUex4rxNFNQUYa6wszyPGlkp7ipKGjl7q2XupCvdSFeuKeVnK8qZQXZtiRO1bYZn4gjcPNXey14/G/98Qe9tXHJ2R5Up2szvOzbW1ezEw/YEodGBYcifIIvgogIqlYC9N8AKvy6M9EpE1V185OF5PD4FCYX712hDue3U/fYJhrPngKX9x6ypyf9XX3D0Zn89GZfVMXVY2dcXbsFKewPMeazX9wzRJW2LH2gfQU+gbD1EXt8r08uaOO2pAVcTPSNj8RLofQPximd2D4uIbuDiAS+ui2Qx1TOT0rM5qNGhyRnZrjS8WT6qKjdyAatVNZE+KZXfUcauqKPrUs8bvZsCzLjtEPsL4oA0GiSVjP7mrg7ucPcKCxM6qwHAKlQS/rCzP45JlFlOX7OTU/QFFW+pytBWUwTCeT8RGkAwEs30AGUAvsPJmLishhoAMYAgZVddPJnG+6eetIK995pJLdde1caK8XXDqH1gseGApT3doTrZMTK/jr23vj2i7NSKM018sVZxSyNDOdtBRH1KfR0G7N6itrQzy7u4Gatp5Rs/mp4ne74uLZc6JC3XpffWAPHzp/Mzk+N5npKQkFbah7gF21IZ7cWUelLfyrmrqi+wsy0li3NIPLTy+kvDBAadBLa/eAnXXbzi9ermJfQ0dcsbS8gJuy/ADnrwpSlmfN8k9ZYsw6hsVNIh/BvVhJYx3A61imoTtUtXWarr1VVZum6VzTQktXP7c9tZcH3zxGQUYaP7lqI5eUJ2e9YFWlsaNvdFROUxdHm7vj7OIZ6VY2bVm+n9OKMkhLcZLqcpDidNDa1U9tqIenKuvi4vQni8shMbPySAx7bOJSfCLTRAK1onU/q/JGr77W2tVPZW2InfZMv7KmPc5MU5iZTnlhgE9sLGRNfgCP20lTZz/76tvZWdPGb986FlfszJvqpMzOk4hE6pTl+aeUcGYwLBYSPREsA9zA+0ANUA1MLRtmnhAOKw9sP8b/eWYvnb2D/MOFK7jm4lUzUnBtJB29A3HO2UhyVVVj16jErlSXgzSXgyxvatSJ6nY56RkYYk9d+6QjYjLSUyzhPsL0EpusFNk/lWSxydLU2Rc17ey0hX5N27AQX5btYX1hBlduLibodeNyCg3tfeyrb+fxHXXc/fyBqE/D6RBWBL2cUZzJlWcVR0M0CzONWcdgmCyiOn7EhVgSYB2Wf+ADQDnQguUw/u4JX1SkCmjFKs74U1W9d4w2VwNXA+Tl5Z35wAMPnOjlEnI4NMR/7u7nUChMWZaDz651U+g/ufoqnZ2d+HzD1RYHw8rxbqWhO0xdV5j6LqWhK0x9txLqGx5/AYLpQp7XQb5HePHYIOPUAovD5YCMVCGQKvjd1mv0zy0EUsEf2Z8qdoXN2aGtN8zh9jBH2q3XqrZB2vqHr5/nEUoCDpZ4HLgclr2+tVep7gxT3RGmO8ZvnJ0mFPkcFPntP59Q4HOQskAE/sjvjWEYMzbjk2hstm7d+tZkTO8JFUG0kUgRcB6WMrgMyFHVE66nLCJLVbXWLmT3HPBPqjpuYfRNmzbpm2++eaKXG5NQzwD/9uw+fvXaEXK8br596alcccaJrxccDiv17b1UNXXxzH+/gyurMFr2+FhLd1w53xxvql390i6bYL9flu2JM61c81/v0NjRNzxrj52tR2zvPjfeVGfSqz6qWve/szoUtefvrAnR2GFlMovAiqCXbGcvJYUFuJwOXA6hpq2HffUdcU8EfrdrVDx+WZ6fDM/CXhqzoqKCLVu2JLsbcxIzNuOTaGxEZFKKIJGP4BoswX8eMIAdOgrcx0k6i1W11n49LiIPA5uxMpdnHFXlobdruPWpPbR09fO5c0u4ftvk1wsOdQ9Ypps4U04Xh5u66BkYNuWkpxylNOilvDCDy09fGi17XJrjnbRAu/szG07oHmcaVaWmrSdqy99ZE2JXbSjqg3AIrLRLPK/J95PidOAQONbSw9sNQ2yvrwYs/8PKXB9nLs/if5y9LJp9uzQjLemKLekkaY1hw+IkkRG8BPgdcJ2q1k3XBUXECzhUtcN+v41ZylTeV9/Bdx6p5I3DLWxYlsl//M1mygtjVk2yf2y9A0Mcae6mqrGTQzFZtVVNXXGJRU6HUJyVTmnQywdW5kTLHtcf2MHHP7J1QQgzVeVYS49ly68NRW37kTh9p0PI87vxul1kelJxOQSHCMdaunn/eGf0PIWZ6VaEjqeXbWevoyzfz4qgL1qJ0xDDNK0xbDBMlkR5BNfP0DXzgIdtIekCfq2qTyc+5OTo7Bvkrj/s575XDhNIc3HbJ9bziY1F1IV6+dP+RmtW/8QLHOp3UhUspqYtPps2128VRvvIujw7k9Yqe1yc5RlTkFVUO+alEgiHlSMt3TGRO9ZfbP6BQ8DldOBzu0hxWvdYGxoOWQ2kuViTH+DPNxRGTTur8/3RJ66Kigq2nFE4uzc2nxhnjWHuuguuvdY8GRhmhJkPixmBqh4Cpl705sSuxRM767jl8d00tPchAqvz/Pzi5Sq+8/tdcTHzPs2h9HgVGxngk9supPSph1jx659TcuUV+G/9wYL78Q2FlaqmrpjInRC7a9tHLSozkrBaNZdUleJs/3Bopi308wPGrHNSiFjCHyzhH1EI115rSkcbZoxZVwSzyYPbj/HNh4bdGS6H0NTZR2nQx5ayJXErWOV6U5Hrr4cf3AU/sA+49lq4c/4rgcGhMAcbh4X+rtoQu2rb4xZ7SURRVnqMwLfCM0uDXrN61UwRUQYRJQBGCRhmlIWrCFS5YHUuN1+xjuIsD6U5HoqyPbgSCa8F8OMbGArzfkNnnD1/d137pBZNz/SkUJYXE6mT72d1ng//JB3phmki4hOI5brr5uX30TA/WJiKwHa2Fd55J589t2Ryzra59uObRNRI/+DwAiqWM7edPXXtE5aJSHU5WLXEFxeeuSbfH1cu2pAkIt/DiE8g1kcARhkYZoSFpwhOxNk21358Y0SN9F73FfYFCqi89C+jgn9ffQcDE2ScLcv2xAh8q3pmSc4ET0aG5CFiTVhifQIRn0FmplEChhlh4SmCE3G2zaUfnyq9be3s/u3TVDqLqDz3w+x8Yw/vp1zEYL8LHq4c87Bsb2q0iFpE6K/O889KmQzDNHPTTfETlsj30SgBwwyxMKXEiTjbkvTj6+4fZHdte7TmTmVNiAOeDzH01xdbDd6sBsdwkTa3y8HqEQK/LN9Prm+BmnVmI7FqLiZvmTWGDbPIwlQEJ2rvn+EfX0fvQFTo77JfDzZ2xuUsxF56eXMtZY2HKWs6zJof305ZQYCSHO/iWfpwNhKrTPKWwbAAFcEcsfeHeqxa+pY9v51dNSEOxdTSjyXoS7XLJNuLm+f5WHXHLXh+dudwo/tyF5d5YDYSq0zylsEALERFkAR7f1t3f7TmTmVNiO0Huzn+9LOj2qWnOFmd54uLxy/L90cXRwfmjCJLOrORWGWStwwGYCEqAphRe39zZ99wdc1qq/5O7IIoYJWTXmEvFLPGjsdfk++nONszsVlnLjmuk81sJFaZ5C2DYYEqApgWe//xjt74Cps1obi6OmDVIbogZtnDNfkB6va9zbaLt5x436dbkc1FZ+hkmI3cjrmWP2IwJIGFqwhOkMNNXXzviT3sqG7juF1LH8CT6mR1np8LV+fG1NYJkD3G0ofNB6bJbJHo82SZr87Q2TCRGTOcwQAYRTCKxs4+Gtp7Oas0mzUxs/yirHm49OF8dobOhonMmOEMBsAoglGcVZLNY/90frK7MT3Md2fobOR2mOQtgwFTZ2ChE6sMIswnQTcbiVUmecuwyDGKYKEznjN0EmtVGwyGxYFRBAuZkc7QcNh6vesuowwMBkOUpPgIROQS4C7ACfxcVb+fjH4seIwz1GAwTIJZVwQi4gR+DHwYqAa2i8ijqrp7tvuyKDDOUIPBMAHJMA1tBg6o6iFV7QceAK5IQj8WD8YZajAYEpAM01AhcCzmczVw9shGInI1cDVAXl4eFRUVs9K56aCzs3Ne9Xc2MWMzPmZsxseMzfhMx9gkQxGMNR0d5bVU1XuBewE2bdqkW7ZsmeFuTR8VFRXMp/7OJmZsxseMzfiYsRmf6RibZJiGqoHimM9FQG0S+mEwGAwGkqMItgOrRKRURFKBK4FHk9APg8FgMACiSYglF5GPAT/ECh+9T1X/9wTtG4Ejs9G3aSIINCW7E3MUMzbjY8ZmfMzYjE+isVmuqrkTnSApimChIyJvquqmZPdjLmLGZnzM2IyPGZvxmY6xMZnFBoPBsMgxisBgMBgWOUYRzAz3JrsDcxgzNuNjxmZ8zNiMz0mPjfERGAwGwyLHPBEYDAbDIscoAoPBYFjkGEUwA4jIV0VERSRofxYRuVtEDojIDhHZmOw+zjYicruI7LXv/2ERyYzZd4M9NvtE5CPJ7GeyEJFL7Ps/ICLfTHZ/komIFIvIiyKyR0R2ici19vZsEXlORN63X7OS3ddkISJOEXlHRB63P5eKyOv22DxoJ+tOGqMIphkRKcYqsX00ZvNHgVX239XAT5LQtWTzHFCuqqcB+4EbAERkLVZ2+TrgEuAeu1T5oiGmNPtHgbXAZ+xxWawMAl9R1VOBc4Av2ePxTeB5VV0FPG9/XqxcC+yJ+XwbcKc9Nq3AF6ZyMqMIpp87ga8TX0jvCuA/1eI1IFNECpLSuyShqs+q6qD98TWsGlNgjc0DqtqnqlXAAaxS5YsJU5o9BlWtU9W37fcdWAKvEGtMfmk3+yXw58npYXIRkSLgUuDn9mcBPgj8zm4y5bEximAaEZHLgRpVfW/ErrFKbxfOWsfmHn8LPGW/N2NjxmBcRKQE2AC8DuSpah1YygJYkryeJZUfYk02w/bnHKAtZqI15e9PUpaqnM+IyB+A/DF2/TPwLWDbWIeNsW3Bxe0mGhtV/b3d5p+xHv3vjxw2RvsFNzYTYMZgDETEB/w/4Muq2i5mQSVE5DLguKq+JSJbIpvHaDql749RBFNEVT801nYRWQ+UAu/ZX9gi4G0R2cwiKb093thEEJHPAZcBF+twAsuiGJsJMGMwAhFJwVIC96vqQ/bmBhEpUNU627R6PHk9TBrnAZfbhTvTgADWE0KmiLjsp4Ipf3+MaWiaUNWdqrpEVUtUtQTrx71RVeuxymx/1o4eOgcIRR5xFwsicgnwDeByVe2O2fUocKWIuEWkFMuh/kYy+phETGn2GGyb9y+APap6R8yuR4HP2e8/B/x+tvuWbFT1BlUtsmXMlcALqnoV8CLwSbvZlMfGPBHMDk8CH8NyhHYDf5Pc7iSFHwFu4Dn7iek1Vf1HVd0lIr8BdmOZjL6kqkNJ7Oeso6qDIvK/gGcYLs2+K8ndSibnAX8N7BSRd+1t3wK+D/xGRL6AFZX3qST1by7yDeABEfke8A6WIp00psSEwWAwLHKMachgMBgWOUYRGAwGwyLHKAKDwWBY5BhFYDAYDIscowgMBoNhkWMUgWFaEJF8EXlARA6KyG4ReVJEVovIlkiFxGQjIjeLSMKkt2m6TqaIfHEazlMhItO6YHuic4rI70RkRYJjU0XkTyJiws4XGEYRGE4aOwHoYaBCVVeq6lqsuO+85PYsHlW9UVX/MAuXygSmpAjsZMOk/R5FZB3gVNVD47WxC+I9D3x61jpmmBWMIjBMB1uBAVX998gGVX1XVV+yP/rs2eZeEbnfVhyIyI0isl1EKkXk3pjtFSJym4i8ISL7ReQCe7tHRH5jr2nwoF1/fZO9b5uIvCoib4vIb+06NXGIyH+IyCft94dF5F/s9jtFZM0Y7Z8UkdPs9++IyI32+1tE5O9ExCciz8ecI1Ix9PvAShF5V0Rut4/5mn2vO0TkX+xtJWLV3L8HeJv4MhMj+zLq/kTko3YyXqTNFhF5bLLjMYKrsLNRRWS5WHXtgyLiEJGXRCRSQ+sRu61hAWEUgWE6KAfeSrB/A/BlrFr7K7AyRwF+pKpnqWo5kI5VhyiCS1U328d91972RaDVXtPgFuBMALEWAPo28CFV3Qi8CVw/iX432e1/Anx1jP1/Ai4QkQBW1nOk3+cDLwG9wMftc2wF/s1WZt8EDqrqGar6NVuIrsIqN30GcKaIXGifqwyrRPkGVT0yVicT3N9zwDki4rWbfhp48ATH4zzs/6Hdj9uAfwe+AuxW1WftdpXAWROcyzDPMLY+w2zwhqpWA9glA0qAl4GtIvJ1wANkA7uAx+xjIoXG3rLbgyWA7wJQ1UoR2WFvPwdLybxiP1SkAq9Ool+x1/iLMfa/BFwDVAFPAB8WEQ9Qoqr7xCqM9q+2UA9jlf4dyxy2zf57x/7sw1IMR4Ej9hoViRjz/uzSFE8DfyYiv8OqUf914KKx2k9wjQKgMfJBVX8uIp8C/hFLeUW2D4lIv4j47bUCDAsAowgM08EuhgtejUVfzPshwCUiacA9wCZVPSYiN2FVUxx5zBDD39Px6hAL8JyqfmaK/R7rGrFsBzYBh7Bm30Hg7xl++rkKyAXOVNUBETk84h5i+3erqv40bqNVa79rEv1MdH8PAl8CWoDtqtphP5VMdTx6YvtuK7zI4kE+IFbou7GehgwLBGMaMkwHLwBuEfn7yAYROUtELkpwTEToNNn260SKJMLLwF/a518LrLe3vwacJyKn2Ps8IrJ6ivcwCts5esy+5mtYTwhftV8BMrBqww+IyFZgub29A/DHnOoZ4G8jdnoRKRSRqSyqkuj+KoCNWArqwUm0H489wCkxn2/DWjPiRuBnkY0ikgM0qurAFPpvmOMYRWA4aey1BT6OZTo5KCK7gJtIUBNdVduwBMxOLAfk9klc6h4g1zYJfQPYgVXSuxH4PPBf9r7XgFHO3xPkJaDBLp39EtYsOaII7gc2icibWE8HewFUtRnLLFMpIrfb9vVfA6+KyE6sJQX9TJJE92dXan0ca73jxydqn4AngC0AtgI/C7hNVe8H+kUkUjF3K1Y1XcMCwlQfNcwbxFrkPUVVe0VkJVYo42p75m44CUQkHaum/XmJyoCLyEPADaq6b9Y6Z5hxjI/AMJ/wAC/aTloB/qdRAtODqvaIyHexHN5Hx2oj1qI5jxglsPAwTwQGg8GwyDE+AoPBYFjkGEVgMBgMixyjCAwGg2GRYxSBwWAwLHKMIjAYDIZFzv8HOGYdXqtJZfAAAAAASUVORK5CYII=\n",
938 | "text/plain": [
939 | "
"
940 | ]
941 | },
942 | "metadata": {},
943 | "output_type": "display_data"
944 | }
945 | ],
946 | "source": [
947 | "# import libraries\n",
948 | "import matplotlib.pyplot as plt\n",
949 | "%matplotlib inline\n",
950 | "\n",
951 | "plt.scatter(x, y, color='red', marker='x', label= 'train data')\n",
952 | "plt.plot(x.flatten(),np.dot(x_poly_norm,new_theta.reshape(n,1).flatten()), label = 'best fit model')\n",
953 | "#plt.axis([-40,50,-75,75])\n",
954 | "plt.title('Training Set')\n",
955 | "plt.xlabel('Change in water level (x)')\n",
956 | "plt.ylabel('Water flowing out of the dam (y)')\n",
957 | "plt.grid()\n",
958 | "plt.legend()\n",
959 | "plt.show()"
960 | ]
961 | }
962 | ],
963 | "metadata": {
964 | "kernelspec": {
965 | "display_name": "Python 3",
966 | "language": "python",
967 | "name": "python3"
968 | },
969 | "language_info": {
970 | "codemirror_mode": {
971 | "name": "ipython",
972 | "version": 3
973 | },
974 | "file_extension": ".py",
975 | "mimetype": "text/x-python",
976 | "name": "python",
977 | "nbconvert_exporter": "python",
978 | "pygments_lexer": "ipython3",
979 | "version": "3.6.6"
980 | }
981 | },
982 | "nbformat": 4,
983 | "nbformat_minor": 2
984 | }
985 |
--------------------------------------------------------------------------------
/Week 6 - Regularized Linear Regression and Bias v.s. Variance/Regularized Linear Regression.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Regularized Linear Regression\n",
8 | "\n",
9 | "In the first half of the exercise, you will implement regularized linear regression to predict the amount of water flowing out of a dam using the change of water level in a reservoir. In the next half, you will go through some **diagnostics of debugging learning algorithms** and examine the **effects of bias v.s. variance.**
\n",
10 | "In the following parts, you will implement linear regression and use that to fit a **straight line** to the data and plot learning curves. Following that, you will implement **polynomial regression** to find a **better fit** to the data.\n",
11 | "\n",
12 | "The file ex5data1 contains a data set which includes train set, test set, validation set. \n",
13 | "The structure of the dataset described blow: \n",
14 | "1. x = **Water level (a single feature)**\n",
15 | "2. y = **Value of flowed water**\n",
16 | "3. xval = **x validation set**\n",
17 | "4. yval = **y validation set**\n",
18 | "5. xtest = **x test set**\n",
19 | "6. ytest = **y test set**\n",
20 | "\n",
21 | "\n",
22 | "
\n",
23 | "\n",
24 | "Our assignment has these sections:\n",
25 | "1. Visualizing the Data\n",
26 | " 1. Converting .mat to .csv\n",
27 | " 2. Loading Dataset\n",
28 | " 3. Visualizing The Dataset\n",
29 | "2. Regularized Linear Regression Cost Function\n",
30 | " 1. Implementation\n",
31 | " 2. Test Cost Function\n",
32 | "3. Regularized Linear Regression Gradient\n",
33 | " 1. Implementation\n",
34 | " 2. Test Gradient Function\n",
35 | "4. Fitting Linear Regression\n",
36 | " 1. Implementation\n",
37 | " 2. Visualziation of Fitted Model\n",
38 | "5. Bias-variance\n",
39 | " 1. Learning Curves\n",
40 | " 2. Ploting Learning Curve\n",
41 | " \n",
42 | "\n",
43 | "\n",
44 | "In each section full description provided."
45 | ]
46 | },
47 | {
48 | "cell_type": "markdown",
49 | "metadata": {},
50 | "source": [
51 | "## 1. Visualizing the Dataset\n",
52 | "Before starting on any task, it is often useful to understand the data by visualizing it. "
53 | ]
54 | },
55 | {
56 | "cell_type": "markdown",
57 | "metadata": {},
58 | "source": [
59 | "### 1.A Converting .mat to .csv\n",
60 | "In this specific assignment, the instructor added a .mat file as training set and weights of trained neural network. But we have to convert it to .csv to use in python. \n",
61 | "After all we now ready to import our new csv files to pandas dataframes and do preprocessing on it and make it ready for next steps."
62 | ]
63 | },
64 | {
65 | "cell_type": "code",
66 | "execution_count": 1,
67 | "metadata": {},
68 | "outputs": [],
69 | "source": [
70 | "# import libraries\n",
71 | "import scipy.io\n",
72 | "import numpy as np\n",
73 | "\n",
74 | "data = scipy.io.loadmat(\"ex5data1\")"
75 | ]
76 | },
77 | {
78 | "cell_type": "markdown",
79 | "metadata": {},
80 | "source": [
81 | "Now we **extract** `x`, `y`, `xval`, `yval`, `xtest` and `ytest` variables from the .mat file and save them into .csv file for further usage. After running the below code you should see:\n",
82 | "1. X.csv\n",
83 | "2. y.csv \n",
84 | "3. Xtest.csv\n",
85 | "4. ytest.csv\n",
86 | "5. Xval.csv\n",
87 | "6. yval.csv\n",
88 | "\n",
89 | "files in your directory."
90 | ]
91 | },
92 | {
93 | "cell_type": "code",
94 | "execution_count": 2,
95 | "metadata": {},
96 | "outputs": [],
97 | "source": [
98 | "for i in data:\n",
99 | " if '__' not in i and 'readme' not in i:\n",
100 | " np.savetxt((i+\".csv\"),data[i],delimiter=',')"
101 | ]
102 | },
103 | {
104 | "cell_type": "markdown",
105 | "metadata": {},
106 | "source": [
107 | "### 1.B Loading Dataset\n",
108 | "First we import .csv files into pandas dataframes then save them into numpy arrays.
"
236 | ],
237 | "text/plain": [
238 | " y\n",
239 | "0 2.134311\n",
240 | "1 1.173257\n",
241 | "2 34.359109"
242 | ]
243 | },
244 | "execution_count": 5,
245 | "metadata": {},
246 | "output_type": "execute_result"
247 | }
248 | ],
249 | "source": [
250 | "y_df.head(3)"
251 | ]
252 | },
253 | {
254 | "cell_type": "markdown",
255 | "metadata": {},
256 | "source": [
257 | "Now we convert all **pandas dataframes** to **numpy arrays** for calculations."
258 | ]
259 | },
260 | {
261 | "cell_type": "code",
262 | "execution_count": 6,
263 | "metadata": {},
264 | "outputs": [
265 | {
266 | "name": "stdout",
267 | "output_type": "stream",
268 | "text": [
269 | "#12 Number of training samples, #1 features per sample\n",
270 | "#21 Number of validation samples, #1 features per sample\n",
271 | "#21 Number of test samples, #1 features per sample\n"
272 | ]
273 | }
274 | ],
275 | "source": [
276 | "# saving x, y, xval, yval, xtest and ytest into numpy arrays\n",
277 | "x = x_df.iloc[:,:].values\n",
278 | "xval = xval_df.iloc[:,:].values\n",
279 | "xtest = xtest_df.iloc[:,:].values\n",
280 | "\n",
281 | "y = y_df.iloc[:,:].values\n",
282 | "yval = yval_df.iloc[:,:].values\n",
283 | "ytest = ytest_df.iloc[:,:].values\n",
284 | "\n",
285 | "# number of examples and number of features\n",
286 | "m, n = x.shape\n",
287 | "\n",
288 | "# add 1's to the features of x as bias\n",
289 | "x = np.append(np.ones(shape=(m,1)),x,axis = 1)\n",
290 | "xval = np.append(np.ones(shape=(xval.shape[0],1)),xval,axis = 1)\n",
291 | "xtest = np.append(np.ones(shape=(xtest.shape[0],1)),xtest,axis = 1)\n",
292 | "\n",
293 | "\n",
294 | "\n",
295 | "m_val = xval.shape[0]\n",
296 | "m_test = xtest.shape[0]\n",
297 | "\n",
298 | "print('#{} Number of training samples, #{} features per sample'.format(m,n))\n",
299 | "print('#{} Number of validation samples, #{} features per sample'.format(m_val,n))\n",
300 | "print('#{} Number of test samples, #{} features per sample'.format(m_test,n))"
301 | ]
302 | },
303 | {
304 | "cell_type": "code",
305 | "execution_count": 7,
306 | "metadata": {},
307 | "outputs": [],
308 | "source": [
309 | "# define some hypter parameters\n",
310 | "\n",
311 | "# define theta as zero\n",
312 | "theta = None\n",
313 | "\n",
314 | "# define hyperparameter λ\n",
315 | "lambda_ = None\n",
316 | "\n",
317 | "# reshape (-1,1) because we just have one feature in y column\n",
318 | "y = y.reshape(-1,1)"
319 | ]
320 | },
321 | {
322 | "cell_type": "markdown",
323 | "metadata": {},
324 | "source": [
325 | "### 1.C Ploting Dataset\n",
326 | "We will begin by visualizing the dataset containing historical records on **the change in the water level**, `x`, and **the amount of water flowing out of the dam**, `y`.
\n",
327 | "This dataset is divided into three parts: \n",
328 | " • A **training set** that your model will learn on: `x`, `y` \n",
329 | " • A **cross validation set** for determining the regularization parameter: `xval`, `yval` \n",
330 | " • A **test set** for evaluating performance. These are **\"unseen\" examples** which your model did not see during training: `xtest`, `ytest` "
331 | ]
332 | },
333 | {
334 | "cell_type": "code",
335 | "execution_count": 8,
336 | "metadata": {},
337 | "outputs": [
338 | {
339 | "data": {
340 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmcXFWd9/HPFwiiNNoi0EZgCLsig0gahjEu3bih44aPqIw6uIxxYYY2LAPqDDQjLjw6xDAjahTGoGDDICgiOMYMLeDDlrAlMSIKCEgEERpoQCTk9/xxbk0qTVfV7aTureqq7/v1qlfX3X910qlfn3PuPUcRgZmZda9NWh2AmZm1lhOBmVmXcyIwM+tyTgRmZl3OicDMrMs5EZiZdTknAutqkjaVNC7pL5q5r9l04kRg00r2RVx5rZX0eNXye6Z6voh4KiJ6IuLOZu47VZKeK+lbkn4v6WFJt0g6Juex35E03OyYrHts1uoAzKYiInoq7yXdAfx9RPy01v6SNouINWXEtpFOAzYFXgg8DOwJvKilEVnXcI3AOoqkkyWdK+m7kh4B3ivpryVdLWlM0mpJp0make2/maSQNCtb/k62/VJJj0i6StLOU9032/4GSb+S9JCkf5f0c0nvrxH6/sA5ETEWEWsjYlVEXFB1rr0k/VTSA5J+Ken/ZOs/DrwL+FRWK7qwuSVq3cCJwDrRIcA5wHOAc4E1wBCwDTAHOBj4SJ3j/xb4F2Br4E7gM1PdV9J2wHnAsdl1bwcOqHOeq4HPS3q/pN2rN0jaClgMnAVsB7wHWChpz4g4PfuMn8uarQ6pcw2zSTkRWCe6MiJ+mP1l/XhEXBcR10TEmoi4DVgIvKrO8edHxNKIeBI4G9h3A/Z9E3BjRPwg2zYfuL/OeT5O+kI/Elgl6VZJr8u2vQX4VUSclX2GZcD3gXfULwazfJwIrBPdVb0g6YWSflTpiAX+lfRXei2/r3r/GNBTa8c6+76gOo5IozveXeskEfFYRJwcEfsBzwMuAL4n6TnATsCcrGlrTNIYqTloZp24zHJzIrBONHFI3a8DK4DdIuLZwAmACo5hNbBDZUGSgO3zHBgRDwGfJyWVWaSEsiQieqtePRHxD5VDmhq5dR0nAusGWwEPAY9KehH1+wea5WJgP0lvlrQZqY9i21o7SzpRUr+kzSVtQWoiegC4FbgIeLGkv5U0I3sdIGnP7PB7gV2K/TjWyZwIrBscDRwOPEKqHZxb9AUj4l5S882pwB+BXYEbgCfqHLYo2/ceYAD4m6zJ6CHg9cB7STWN35NqDM/Ijvsm8BJJD0o6v/mfxjqdPDGNWfEkbUr6gn9HRFzR6njMqrlGYFYQSQdLeo6kZ5BuMV0DXNvisMyexonArDgvB24j3TZ6MPC2iKjXNGTWEm4aMjPrcq4RmJl1uWkx6Nw222wTs2bNanUYuT366KNsueWWrQ6jLblsanPZ1Oayqa1e2Sxbtuz+iKh523LFtEgEs2bNYunSpa0OI7fR0VEGBgZaHUZbctnU5rKpzWVTW72ykfTbPOdw05CZWZdzIjAz63JOBGZmXc6JwMysyzkRmJm1m4nPdxX8vJcTgZlZOxkehnnz1n35R6Tl4eHCLulEYGbWLiJgbAwWLFiXDObNS8tjY4XVDKbFcwRmZl1Bgvnz0/sFC9ILYGgorVcx8ym5RmBm1k6qk0FFgUkAnAjMzNpLpTmoWnWfQQGcCMzM2kV1n8DQEKxdm35W9xkUwH0EZmbtQoLe3vX7BCrNRL29hTUPORGYmbWT4eH0l3/lS7+SDNxHYGbWRSZ+6ReYBMCJwMys6xWWCCRtIelaSTdJWinppGz9tyTdLunG7LVvUTGYmVljRfYRPAEcFBHjkmYAV0q6NNt2bEScX+C1zcwsp8ISQUQEMJ4tzshexY6cZGZmU6Yo8CEFSZsCy4DdgK9ExHGSvgX8NanGsAQ4PiKemOTYucBcgL6+vtkjIyOFxdls4+Pj9PT0tDqMtuSyqc1lU5vLprZ6ZTM4OLgsIvobnaPQRPC/F5F6gQuBfwT+CPwe2BxYCPwmIv613vH9/f3hOYs7g8umNpdNbR1dNsPDaUC5yi2ilYfKentzjTjaYM7iXImglLuGImIMGAUOjojVkTwB/CdwQBkxmJm1nRaNNjpRYX0EkrYFnoyIMUnPBF4DnCJpZkSsliTgbcCKomIwM2trLRptdKIiawQzgcsk3QxcByyOiIuBsyUtB5YD2wAnFxiDmVl7a8FooxMVedfQzcBLJ1l/UFHXNDObdmqNNtohNQIzM6unRaONTuRB58zMWqVFo41O5ERgZtZKLRhtdKKGiUDSJsBLgBcAjwMrI+LeogMzM+saJY82OlHNRCBpV+A40m2ftwJ/ALYA9pD0GPB1YFFErC0jUDMzK0a9GsHJwFeBj8SEx48lbQf8LfA+YFFx4ZmZWdFqJoKIOKzOtvuALxcSkZmZlarh7aOSlko6QtJzywjIzMzKlec5gneTOoqvkzQi6fXZ8BBmZtYBGiaCiPh1RHwa2AM4BzgTuFPSSZK2LjpAMzMrVq4niyXtA/wb8EXge8A7gIeB/ykuNDMzK0Oe5wiWAWPAGaw/icw1kuYUGZyZmRUvz5PFh0bEbZNtiIi3NzkeMzMrWc2mIUnvlbRJrSQgaVdJLy8uNDMzK0O9GsHzgBuypqFlrHuyeDfgVcD9wPGFR2hmZoWq90DZAkn/ARwEzAH2IY01tAp4X0TcWU6IZmZWpLp9BBHxFLA4e5mZWQfyxDRmZl3OicDMrMsVlggkbSHpWkk3SVop6aRs/c6SrpF0q6RzJW1eVAxmZtZYngfKeoG/A2ZV7x8RRzY49AngoIgYlzQDuFLSpcBRwPyIGJH0NeBDpOGuzcysBfLUCC4hJYHlpNtIK6+6IhnPFmdkryDdhXR+tn4R8LaphWxmZs2kCXPOPH0H6fqI2G+DTi5tSkoauwFfIY1VdHVE7JZt3xG4NCL2nuTYucBcgL6+vtkjIyMbEkJLjI+P09PT0+ow2pLLpjaXTW0um9rqlc3g4OCyiOhvdI48Q0x8W9KHgYtJzT0ARMQDjQ7Mbj/dN2teuhB40WS71Th2IbAQoL+/PwYGBnKE2h5GR0eZTvGWyWVTm8umNpdNbc0omzyJ4M+kv+Q/zbov7QB2yXuRiBiTNAocCPRK2iwi1gA7APdMKWIzM2uqPH0ERwG7RcSsiNg5ezVMApK2zWoCSHom8BrSU8mXkYaxBjgc+MGGhW5mZs2Qp0awEnhsA849E1iU9RNsApwXERdL+gUwIulk4AbS8NZmZtYieRLBU8CNki5j/T6CurePRsTNwEsnWX8bcMAU4zQzs4LkSQTfz15mZtaBGiaCiFhURiBmZtYaeZ4s3h34PLAXaT4CAPJ0GJuZWfvLc9fQf5KGgFgDDAJnAd8uMigzMytPnkTwzIhYQnoK+bcRMUwaJsLMzDpAns7iP0naBLhV0j8AvwO2KzYsMzMrS54awSeAZwFHArOB95EeBDMzsw6Q566h67K348AHig3HzMzKVjMRSPohNQaEA4iItxQSkZmZlapejeBL2c+3A88HvpMtHwbcUWBMZmZWopqJICJ+BiDpMxHxyqpNP5R0eeGRmZlZKfJ0Fm8r6X8fHpO0M7BtcSGZmVmZ8tw+Og8YlXRbtjyLbOYwMzOb/vLcNfTjbJiJF2arfhkRT9Q7xszMpo88NQKyL/6bCo7FzMxaIE8fgZmZdTAnAjOzLtcwESh5r6QTsuW/kOQZxszMOkSeGsHpwF+THiQDeAT4SmERmZlZqfIkgr+KiCOAPwFExIPA5o0OkrSjpMskrZK0UtJQtn5Y0u8k3Zi93rhRn8DMzDZKnruGnpS0Kdm4Q5K2BdbmOG4NcHREXC9pK2CZpMXZtvkR8aU6x5qZWUny1AhOAy4EtpP0WeBK4HONDoqI1RFxffb+EWAVsP1GxGpmZgVQRM0BRtftJL0QeDUgYElErJrSRaRZwOXA3sBRwPuBh4GlpFrDg5McM5fsCea+vr7ZIyMjU7lkS42Pj9PT09PqMNqSy6Y2l01tLpva6pXN4ODgsojob3SOvIlgU6CPqqakiLgzT5CSeoCfAZ+NiAsk9QH3k5qaPgPMjIgP1jtHf39/LF26NM/l2sLo6CgDAwOtDqMtuWxqc9nU5rKprV7ZSMqVCBr2EUj6R+BE4F7gKVKtIIB9chw7A/gecHZEXAAQEfdWbf8GcHGj85iZWXHydBYPAXtGxB+ncmJJAs4AVkXEqVXrZ0bE6mzxEGDFVM5rZmbNlScR3AU8tAHnnkOa33i5pBuzdZ8CDpO0L6lWcQfwkQ04t5mZNUm9qSqPyt7eRhqG+kfA/446Wv1X/mQi4kpSM9JEl2xAnGZmVpB6NYKtsp93Zq/NWfcgWeMeZjMzmxbqTVV5EoCkQyPiv6q3STq06MDMzKwceR4o+2TOdWZmrTHxNvgct8XbOvX6CN4AvBHYXtJpVZueTRo+wsys9YaHYWwM5s8HKSWBefOgtzdts4bq1QjuIT35+ydgWdXrIuD1xYdmZtZAREoCCxakL/9KEliwIK13zSCXen0ENwE3STonIp4sMSYzs3ykVBOA9OW/YEF6PzS0roZgDTXsI3ASMLO2Vp0MKpwEpsRTVZrZ9FZpDqpWaSayXGomAknfzn4OlReOmdkUVPcJDA3B2rXpZ3WfgTVU74Gy2ZJ2Aj4o6SwmPCUcEQ8UGpmZWSNSujuouk+g0kzU2+vmoZzqJYKvAT8GdiHdLVRdopGtNzNrreHh9Jd/5Uu/kgycBHKr2TQUEadFxIuAMyNil4jYuerlJGBm7WPil76TwJQ0HH00Ij4m6SXAK7JVl0fEzcWGZWZmZWl415CkI4Gzge2y19nZZDVmZtYB8sxH8PfAX0XEowCSTgGuAv69yMDMzKwceZ4jEGmKyorKdJVmZtYB8tQI/hO4RtKF2fLbSFNQmplZB8jTWXyqpFHg5aSawAci4oaiAzMzs3LkqREQEdcD10/lxJJ2BM4Cng+sBRZGxAJJWwPnArNIcxa/MyIenMq5zcyseYoca2gNcHT2LMKBwBGS9gKOB5ZExO7AkmzZzMxapLBEEBGrs5oEEfEIsArYHngrsCjbbRGpz8HMzFokz3MEp+RZ1+Acs4CXAtcAfRGxGlKyID2bYGZmLaJoMDqfpOsjYr8J626OiH1yXUDqAX4GfDYiLpA0FhG9VdsfjIjnTnLcXGAuQF9f3+yRkZE8l2sL4+Pj9PT0tDqMtuSyqc1lU5vLprZ6ZTM4OLgsIvobnaPenMUfAz4O7CKpekiJrYCf5wlQ0gzge8DZEXFBtvpeSTMjYrWkmcB9kx0bEQuBhQD9/f0xMDCQ55JtYXR0lOkUb5lcNrW5bGpz2dTWjLKpd9fQOcClwOdZv0P3kTxDUEsS6XmDVRFxatWmi4DDgS9kP38w1aDNzKx56s1Z/BDwkKTjJmzqkdQTEXc2OPcc4H3Ackk3Zus+RUoA50n6EHAncOiGhW5mZs2Q5zmCH5HmHxCwBbAzcAvw4noHRcSV1B6K4tVTiNHMzAqU58niv6xelrQf8JHCIjIzs1JN+TmC7NmA/QuIxczMWqBhjUDSUVWLmwD7AX8oLCIzMytVnj6CrareryH1GXyvmHDMzKxsefoITgKQtFVajPHCozIzs9LkGWJib0k3ACuAlZKWSdq7+NDMzKwMeTqLFwJHRcROEbETcHS2zszMOkCeRLBlRFxWWYiIUWDLwiIyM7NS5eksvk3SvwDfzpbfC9xeXEhmZlamPDWCDwLbAhdkr22ADxQZlJmZlSfPXUMPAkeWEIuZmbVAkVNVmpnZNOBEYGbW5ZwIzMy6XJ6xhk6bZPVDwNKI8KQyZmbTXJ4awRbAvsCt2WsfYGvgQ5K+XGBsZmZWgjzPEewGHBQRawAkfRX4CfBaYHmBsZmZWQny1Ai2Z/0nibcEXhARTwFPFBKVmZmVJk+N4P8CN0oaJU09+Urgc5K2BH5aYGxmZlaChjWCiDgDeBnw/ez18oj4ZkQ8GhHH1jpO0pmS7pO0omrdsKTfSboxe72xGR/CzMw2XN7bRzchzUr2ALCbpFfmOOZbwMGTrJ8fEftmr0tyXt/MpqOI+svWFvLcPnoK8C5gJbA2Wx3A5fWOi4jLJc3ayPjMbLoaHoaxMZg/H6SUBObNg97etM3ahqJBhpZ0C7BPREy5YzhLBBdHxN7Z8jDwfuBhYClwdDaW0WTHzgXmAvT19c0eGRmZ6uVbZnx8nJ6enlaH0ZZcNrV1XNncdRfcdx9stx3suOPTl6eg48qmieqVzeDg4LKI6G94koio+wIuBXoa7Vfj2FnAiqrlPmBTUlPTZ4Ez85xn9uzZMZ1cdtllrQ6hbblsauu4slm7NmJoKCLVBdJraCitn6KOK5smqlc2pAd/G37H5rlr6DHSXUNLqLpdNCKmPCJpRNxbeS/pG8DFUz2HmU0TUmoWWrBg3bpKM5G1lTyJ4KLstdEkzYyI1dniIaR5kM2sE1X6BKrNm+dk0IbyzEewaENOLOm7wACwjaS7gROBAUn7kjqb7wA+siHnNrM2V0kCCxbA0FD68q8sg5NBm6mZCCSdFxHvlLSc9MW9nojYp96JI+KwSVafMfUQzWzakdLdQZUkUGkmgrTeSaCt1KsRDGU/31RGIGbWYYaHU82g8qVfSQZOAm2nZiKoast/NXBFRNxaTkhm1jEmfuk7CbSlPJ3Fs4D3StoJWAZcQUoMNxYZmJmZlSPPWEMnRMRBwN7AlcCxpIRgZmYdIM8QE/8MzAF6gBuAY0i1AjMz6wB5mobeDqwBfgT8DLg6Iv5UaFRmZlaaPE1D+5E6jK8lm5VM0pVFB2ZmZuXI0zS0N/AK4FVAP3AXbhoyM+sYeZqGTiENOX0acF1EPFlsSGZmVqY8Q0z8jaTNgT2APSXd4mRgZtY58jQNvQo4izQ2kIAdJR0eEXUnpjEzs+khT9PQqcDrIuIWAEl7AN8FZhcZmJmZlSPPnMUzKkkAICJ+BcwoLiQzMytTnhrBUklnAN/Olt+Dnyw2M+sYeRLBx4AjgCNJfQSXA6cXGZSZmZUnz11DT5D6CU4tPhwzMytbvYlpJp2QpqLRxDRmZjY91KsRHAo8XlYgZmbWGvUSwTkRsZ+kb0fE+0qLyMzMSlUvEWwu6XDgZZLePnFjRFxQ78SSziRNc3lfROydrdsaOJc02c0dwDsj4sENC93MzJqh3nMEHwUOBHqBN0945ZnH+FvAwRPWHQ8siYjdgSXZspmZtVC9OYuvBK6UtDQizpjqiSPickmzJqx+KzCQvV8EjALHTfXcZmbWPIqoeWPQxp88JYKLq5qGxiKit2r7gxHx3BrHzgXmAvT19c0eGRkpLM5mGx8fp6enp9VhtCWXTW0um9pcNrXVK5vBwcFlEdHf6Bx5HihriYhYCCwE6O/vj4GBgdYGNAWjo6NMp3jL5LKpzWVTm8umtmaUTd2xhpTsuFFXWN+9kmZm554J3NfEc5uZ2QaomwgitRt9v4nXuwg4PHt/OPCDJp7bzMw2QJ7RR6+WtP9UTyzpu8BVpMls7pb0IeALwGsl3Uqa//gLUz2vmZk1V54+gkHgo5LuAB4lDTwXjYaYiIjDamx69ZQiNDOzQuVJBG8oPAozM2uZhk1DEfFbYEfgoOz9Y3mOMzOz6aHhF7qkE0kPfX0yWzUD+E6RQZmZWXny/GV/CPAWUv8AEXEPsFWRQZlZk018cLTAB0lt+smTCP6c3UYaAJK2LDYkM2uq4WGYN2/dl39EWh4ebmVU1kbyJILzJH0d6JX0YeCnwDeLDcvMmiICxsZgwYJ1yWDevLQ8NuaagQH5pqr8kqTXAg8DewInRMTiwiMzs40nwfz56f2CBekFMDSU1kuti83aRp7O4lMiYnFEHBsRx0TEYkmnlBGcmTVBdTKocBKwKnmahl47yTo/W2A2XVSag6pV9xlY16uZCCR9LJvAfk9JN1e9bgduLi9EM9tg1X0CQ0Owdm36Wd1nYF2v7pzFwKXA51l/JrFHIuKBQqMys+aQoLd3/T6BSjNRb6+bhwyoP0PZQ8BDwGEAkrYDtgB6JPVExJ3lhGhmG2V4OP3lX/nSryQDJwHL5OksfnM2WujtwM9Ik85fWnBcZtZME7/0nQSsSp7O4pNJk9j/KiJ2Jo0e+vNCozIzs9LkSQRPRsQfgU0kbRIRlwH7FhyXmZmVJM8w1GOSeoDLgbMl3QesKTYsMzMrS54awVuBx4F5wI+B3wBvLjIoMzMrT80agaRPkPoCboiIp7LVi0qJyszMSlOvaWgHYAHwQkk3A/+PlBiu2tjnCLJpLx8BngLWRET/xpzPzMw2XL3nCI4BkLQ50A+8DPgg8A1JYxGx10ZeezAi7t/Ic5iZ2UbK01n8TODZwHOy1z3A8iKDMjOz8tTrI1gIvJjUhHMNqWno1Ih4sAnXDeAnkgL4ekQsbMI5zcxsAyhqDDol6cfANsAKUhK4ClgRtQ6YykWlF0TEPdmwFYuBf4yIyyfsMxeYC9DX1zd7ZGRkYy9bmvHxcXp6elodRlty2dTmsqnNZVNbvbIZHBxclqcPtmYiAJAkUq3gZdlrb+ABUofxiRsS9CTXGAbGI+JLtfbp7++PpUuXNuNypRgdHWVgYKDVYbQll01tLpvaXDa11SsbSbkSQd3nCCJZAVxCGl/o58CuwNCUo10X2JaStqq8B15HqnWYmVkL1OsjOJJUC5gDPEl26yhwJhvXWdwHXJgqG2wGnBMRP96I8zVP9QiNky2blcW/i1aiencNzQLOB+ZFxOpmXTAibgNe0qzzNc3wcJrMuzI8b2VCj97etM2sLP5dtJLVbBqKiKMi4vxmJoG2FZH+41XP2lSZ1WlszLM4WXn8u2gtkOc5gs5XPWvTggXpBevP6mRWBv8uWgvkGXSuO1T/B6zwfzxrBf8uWsk6NxFMrEI3qlJXquDVPLm3tYJ/F61knZkIhofX/49T+Y9Vq6Otuh12aAjWrk0/q9tpyzbVRGadoR1/F63jdV4fQXVnG6QqdfV/rMluw5PSHRnV7bCVqnlvb/lVct810r3a7XfRukLnJYIN7WwbHl4/SVTOU/Z/vA1JZNZZ2uV30bpGZzYNbWhn22Q1hbJVYq80B2yyybok0I1fBmU0kbVjM1w7/C5a1+jMRDDdO9t810gy1b6edr2GWZvrvETQCZ1t0z2RNUMZD1b54S0zoFP7CKZzZ9vERFbdRwDdUzMo48EqP7xlBnRiIoDp3dk23RNZM1U+e+ULGpr/71jGNczaXGcmApjenW3NTmTTdSTLWk1kzfyiLuMaZm2u8/oIOkWzEtl07Qwto6+nE/qTzJqgc2sENr2fSSijiczNcGaAE0Fnm+6doWX09Uzn/iSzJnHTUKeb7s8klNHXM537k8yawImg0/mZBDNrwImgk7kz1MxyaEkfgaSDgQXApsA3I+ILrYij47kz1MxyKD0RSNoU+ArwWuBu4DpJF0XEL8qOpSu4M9TMGmhF09ABwK8j4raI+DMwAry1BXF0D3eGmlkdrWga2h64q2r5buCvJu4kaS4wF6Cvr4/R0dFSgmuG8fHxaRVvmVw2tblsanPZ1NaMsmlFIpjsz9Gn9VpGxEJgIUB/f38MDAwUHFbzjI6OMp3iLZPLpjaXTW0um9qaUTataBq6G9ixankH4J4WxGFmZrQmEVwH7C5pZ0mbA+8GLmpBHGZmBihacC+5pDcCXybdPnpmRHy2wf5/AH5bRmxNsg1wf6uDaFMum9pcNrW5bGqrVzY7RcS2jU7QkkTQ6SQtjYj+VsfRjlw2tblsanPZ1NaMsvGTxWZmXc6JwMysyzkRFGNhqwNoYy6b2lw2tblsatvosnEfgZlZl3ONwMysyzkRmJl1OSeCAkg6RlJI2iZblqTTJP1a0s2S9mt1jGWT9EVJv8w+/4WSequ2fTIrm1skvb6VcbaKpIOzz/9rSce3Op5WkrSjpMskrZK0UtJQtn5rSYsl3Zr9fG6rY20VSZtKukHSxdnyzpKuycrm3Oxh3dycCJpM0o6kIbbvrFr9BmD37DUX+GoLQmu1xcDeEbEP8CvgkwCS9iI9Xf5i4GDg9Gyo8q5RNTT7G4C9gMOyculWa4CjI+JFwIHAEVl5HA8siYjdgSXZcrcaAlZVLZ8CzM/K5kHgQ1M5mRNB880H/on1B9J7K3BWJFcDvZJmtiS6FomIn0TEmmzxatIYU5DKZiQinoiI24Ffk4Yq7yYemr1KRKyOiOuz94+QvvC2J5XJomy3RcDbWhNha0naAfgb4JvZsoCDgPOzXaZcNk4ETSTpLcDvIuKmCZsmG3p7+9ICaz8fBC7N3rtsXAY1SZoFvBS4BuiLiNWQkgWwXesia6kvk/7YXJstPw8Yq/pDa8q/Py2ZqnI6k/RT4PmTbPo08CngdZMdNsm6jrtvt17ZRMQPsn0+Tar6n105bJL9O65sGnAZTEJSD/A94BMR8bA8oRKS3gTcFxHLJA1UVk+y65R+f5wIpigiXjPZekl/CewM3JT9wu4AXC/pALpk6O1aZVMh6XDgTcCrY90DLF1RNg24DCaQNIOUBM6OiAuy1fdKmhkRq7Om1ftaF2HLzAHekg3cuQXwbFINoVfSZlmtYMq/P24aapKIWB4R20XErIiYRfrPvV9E/J40zPbfZXcPHQg8VKnidgtJBwPHAW+JiMeqNl0EvFvSMyTtTOpQv7YVMbaQh2avkrV5nwGsiohTqzZdBByevT8c+EHZsbVaRHwyInbIvmPeDfxPRLwHuAx4R7bblMvGNYJyXAK8kdQR+hjwgdaG0xL/ATwDWJzVmK6OiI9GxEpJ5wG/IDUZHRERT7UwztJFxBpJ/wD8N+uGZl/Z4rBaaQ7wPmC5pBuzdZ8CvgCcJ+lDpLvyDm1RfO3oOGBE0snADaREmpuHmDAz63JuGjIz63JOBGZmXc6JwMysyzkRmJl1OScCM7Mu50RgTSHp+ZJGJP1G0i8kXSJpD0kDlRESW03Sv0qq+9Bbk67TK+njTTjPqKSmTthe75ySzpe0S51jN5d0uSTfdt5hnAhso2UPAF0IjEbErhGxF+m+777WRra+iDghIn5awqVfu66GAAAEa0lEQVR6gSklguxhw5b9f5T0YmDTiLit1j7ZgHhLgHeVFpiVwonAmmEQeDIivlZZERE3RsQV2WJP9tfmLyWdnSUOJJ0g6TpJKyQtrFo/KukUSddK+pWkV2TrnyXpvGxOg3Oz8df7s22vk3SVpOsl/Vc2Ts16JH1L0juy93dIOinbf7mkF06y/yWS9sne3yDphOz9ZyT9vaQeSUuqzlEZMfQLwK6SbpT0xeyYY7PPerOkk7J1s5TG3D8duJ71h5mYGMvTPp+kN2QP41X2GZD0w7zlMcF7yJ5GlbST0rj220jaRNIVkipjaH0/29c6iBOBNcPewLI6218KfII01v4upCdHAf4jIvaPiL2BZ5LGIarYLCIOyI47MVv3ceDBbE6DzwCzAZQmAPpn4DURsR+wFDgqR9z3Z/t/FThmku2XA6+Q9GzSU8+VuF8OXAH8CTgkO8cg8G9ZMjse+E1E7BsRx2ZforuThpveF5gt6ZXZufYkDVH+0oj47WRB1vl8i4EDJW2Z7fou4NwNLI85ZP+GWRynAF8DjgZ+ERE/yfZbAezf4Fw2zbitz8pwbUTcDZANGTALuBIYlPRPwLOArYGVwA+zYyoDjS3L9of0BbwAICJWSLo5W38gKcn8PKtUbA5clSOu6mu8fZLtVwBHArcDPwJeK+lZwKyIuEVpYLTPZV/qa0lD/07WHPa67HVDttxDSgx3Ar/N5qioZ9LPlw1N8WPgzZLOJ41R/0/Aqybbv8E1ZgJ/qCxExDclHQp8lJS8KuufkvRnSVtlcwVYB3AisGZYyboBrybzRNX7p4DNJG0BnA70R8RdkoZJoylOPOYp1v2e1hqHWMDiiDhsinFPdo1q1wH9wG2kv763AT7MutrPe4BtgdkR8aSkOyZ8hur4Ph8RX19vZRpr/9Eccdb7fOcCRwAPANdFxCNZrWSq5fF4dexZwqtMHtQDVH/pP4NUG7IO4aYha4b/AZ4h6cOVFZL2l/SqOsdUvnTuz9qv6yWSiiuBd2bn3wv4y2z91cAcSbtl254laY8pfoanyTpH78queTWphnBM9hPgOaSx4Z+UNAjslK1/BNiq6lT/DXyw0k4vaXtJU5lUpd7nGwX2IyWoc3PsX8sqYLeq5VNIc0acAHyjslLS84A/RMSTU4jf2pwTgW20bG6BQ0hNJ7+RtBIYps6Y6BExRvqCWU7qgLwux6VOB7bNmoSOA24mDen9B+D9wHezbVcDT+v83UBXAPdmQ2dfQforuZIIzgb6JS0l1Q5+CRARfyQ1y6yQ9MWsff0c4CpJy0lTCm5FTvU+XzZS68Wk+Y4vbrR/HT8CBgCyBL4/cEpEnA38WVJlxNxB0mi61kE8+qhNG0qTvM+IiD9J2pV0K+Me2V/uthEkPZM0pv2cesOAS7oA+GRE3FJacFY49xHYdPIs4LKsk1bAx5wEmiMiHpd0IqnD+87J9lGaNOf7TgKdxzUCM7Mu5z4CM7Mu50RgZtblnAjMzLqcE4GZWZdzIjAz63L/Hw66S+5MENHgAAAAAElFTkSuQmCC\n",
341 | "text/plain": [
342 | ""
343 | ]
344 | },
345 | "metadata": {},
346 | "output_type": "display_data"
347 | }
348 | ],
349 | "source": [
350 | "# import libraries\n",
351 | "import matplotlib.pyplot as plt\n",
352 | "%matplotlib inline\n",
353 | "\n",
354 | "plt.scatter(x[:,1], y, color='red', marker='x')\n",
355 | "plt.title('Training Set')\n",
356 | "plt.xlabel('Change in water level (x)')\n",
357 | "plt.ylabel('Water flowing out of the dam (y)')\n",
358 | "plt.grid()\n",
359 | "plt.show()"
360 | ]
361 | },
362 | {
363 | "cell_type": "markdown",
364 | "metadata": {},
365 | "source": [
366 | "# 2 Regularized Linear Regression Cost Function\n",
367 | "Recall that regularized linear regression has the following cost function:
\n",
368 | "where `lambda` is a regularization parameter which controls the degree of regularization (thus, **help preventing overfitting**). The **regularization** term puts a **penalty** on the overal **cost J**.
\n",
369 | "As the magnitudes of the model parameters **θj**\n",
370 | "increase, the penalty increases as well. Note that you should not regularize the **θ0** term.
\n",
371 | "\n",
372 | "**Your task is to write a function to calculate the regularized linear regression cost function. If possible, try to vectorize your code and avoid writing loops.**"
373 | ]
374 | },
375 | {
376 | "cell_type": "markdown",
377 | "metadata": {},
378 | "source": [
379 | "# 2.A Implementation\n",
380 | "`linear_reg_cost(x, y, theta, lambda_)` computes the cost of using `theta` as the parameter for linear regression to fit the data points in `x` and `y`. Returns the cost in `j` as cost."
381 | ]
382 | },
383 | {
384 | "cell_type": "code",
385 | "execution_count": 9,
386 | "metadata": {},
387 | "outputs": [],
388 | "source": [
389 | "def hypothesis(x,theta):\n",
390 | " return np.dot(x,theta)"
391 | ]
392 | },
393 | {
394 | "cell_type": "code",
395 | "execution_count": 10,
396 | "metadata": {},
397 | "outputs": [],
398 | "source": [
399 | "def linear_reg_cost(theta_flatten, x_flatten, y, lambda_, num_of_samples, num_of_features):\n",
400 | " x = x_flatten.reshape(num_of_samples, num_of_features)\n",
401 | " theta = theta_flatten.reshape(n,1)\n",
402 | " loss = hypothesis(x,theta)-y\n",
403 | " regularizer = lambda_*np.sum(theta[1:,:]**2)/(2*m)\n",
404 | " j = np.sum(loss ** 2)/(2*m) \n",
405 | " return j"
406 | ]
407 | },
408 | {
409 | "cell_type": "markdown",
410 | "metadata": {},
411 | "source": [
412 | "## 2.B Test Cost Function\n",
413 | "Cost at `theta` = [1,1] should be about **303.993192**"
414 | ]
415 | },
416 | {
417 | "cell_type": "code",
418 | "execution_count": 11,
419 | "metadata": {},
420 | "outputs": [
421 | {
422 | "data": {
423 | "text/plain": [
424 | "303.95152555359761"
425 | ]
426 | },
427 | "execution_count": 11,
428 | "metadata": {},
429 | "output_type": "execute_result"
430 | }
431 | ],
432 | "source": [
433 | "m,n = x.shape\n",
434 | "theta = np.ones(n)\n",
435 | "lambda_= 1\n",
436 | "linear_reg_cost(theta.flatten(), x.flatten(),y,lambda_,m,n)"
437 | ]
438 | },
439 | {
440 | "cell_type": "markdown",
441 | "metadata": {},
442 | "source": [
443 | "compute_gradient(theta, x, y,lambda): computes the gradient of the cost using theta as the parameter."
444 | ]
445 | },
446 | {
447 | "cell_type": "markdown",
448 | "metadata": {},
449 | "source": [
450 | "# 3. Regularized Linear Regression Gradient\n",
451 | "Correspondingly, the **partial derivative of regularized linear regression’s cost for θj** is defined as:"
452 | ]
453 | },
454 | {
455 | "cell_type": "markdown",
456 | "metadata": {},
457 | "source": [
458 | "## 3.A Implementation"
459 | ]
460 | },
461 | {
462 | "cell_type": "markdown",
463 | "metadata": {},
464 | "source": [
465 | "`linear_reg_grad(x, y, theta, lambda_)` computes the gradient of cost of using `theta` as the parameter for linear regression to fit the data points in `x` and `y`. Returns the gradient in `grad`."
466 | ]
467 | },
468 | {
469 | "cell_type": "code",
470 | "execution_count": 12,
471 | "metadata": {},
472 | "outputs": [],
473 | "source": [
474 | "def linear_reg_grad(theta_flatten, x_flatten, y, lambda_, num_of_samples, num_of_features):\n",
475 | " x = x_flatten.reshape(num_of_samples, num_of_features)\n",
476 | " m,n = x.shape\n",
477 | " theta = theta_flatten.reshape(n,1)\n",
478 | " new_theta = np.zeros(shape=(theta.shape))\n",
479 | " loss = hypothesis(x,theta)-y\n",
480 | " gradient = np.dot(x.T,loss)\n",
481 | " new_theta[0:,:] = gradient/m\n",
482 | " new_theta[1:,:] = gradient[1:,:]/m + lambda_*(theta[1:,]/m)\n",
483 | " return new_theta.flatten()"
484 | ]
485 | },
486 | {
487 | "cell_type": "markdown",
488 | "metadata": {},
489 | "source": [
490 | "## 3.B Test Gradient Function\n",
491 | "Run your gradient function using `theta = [1; 1]`, You should expect to see a `gradient = [-15.30; 598.250]`."
492 | ]
493 | },
494 | {
495 | "cell_type": "code",
496 | "execution_count": 13,
497 | "metadata": {},
498 | "outputs": [],
499 | "source": [
500 | "theta = np.ones(n)\n",
501 | "lambda_ = 0\n",
502 | "grad = linear_reg_grad(theta,x.flatten(),y,lambda_,m,n)"
503 | ]
504 | },
505 | {
506 | "cell_type": "code",
507 | "execution_count": 14,
508 | "metadata": {},
509 | "outputs": [
510 | {
511 | "data": {
512 | "text/plain": [
513 | "array([ -15.30301567, 598.16741084])"
514 | ]
515 | },
516 | "execution_count": 14,
517 | "metadata": {},
518 | "output_type": "execute_result"
519 | }
520 | ],
521 | "source": [
522 | "grad"
523 | ]
524 | },
525 | {
526 | "cell_type": "markdown",
527 | "metadata": {},
528 | "source": [
529 | "# 4. Fitting Linear Regression\n",
530 | "Once your cost function and gradient are working correctly, the next part is to **compute the optimal values** of **θ**. \n",
531 | "This training function uses `fmin_cg` to optimize the cost function. See official doc \n",
532 | "In this part, we set regularization parameter **λ** to **zero**. Because our current implementation of linear regression is trying to fit a 2-dimensional **θ**, regularization will not be incredibly helpful for a **θ** of such **low dimension**. \n",
533 | "In the later parts of the exercise, you will be using **polynomial regression with regularization**."
534 | ]
535 | },
536 | {
537 | "cell_type": "markdown",
538 | "metadata": {},
539 | "source": [
540 | "## 4.A Implementation\n",
541 | "Once you have implemented the cost and gradient correctly, the `fmin_cg` function will use your cost function to train regularized linear regression and update theta each time."
542 | ]
543 | },
544 | {
545 | "cell_type": "code",
546 | "execution_count": 15,
547 | "metadata": {},
548 | "outputs": [
549 | {
550 | "name": "stdout",
551 | "output_type": "stream",
552 | "text": [
553 | "Optimization terminated successfully.\n",
554 | " Current function value: 22.373906\n",
555 | " Iterations: 18\n",
556 | " Function evaluations: 28\n",
557 | " Gradient evaluations: 28\n"
558 | ]
559 | }
560 | ],
561 | "source": [
562 | "theta = np.ones(n)\n",
563 | "\n",
564 | "from scipy.optimize import fmin_cg\n",
565 | "new_theta = fmin_cg(f=linear_reg_cost, x0=theta, fprime=linear_reg_grad, args=(x.flatten(), y, lambda_, m,n))"
566 | ]
567 | },
568 | {
569 | "cell_type": "code",
570 | "execution_count": 16,
571 | "metadata": {},
572 | "outputs": [
573 | {
574 | "data": {
575 | "text/plain": [
576 | "array([ 13.08790734, 0.36777925])"
577 | ]
578 | },
579 | "execution_count": 16,
580 | "metadata": {},
581 | "output_type": "execute_result"
582 | }
583 | ],
584 | "source": [
585 | "new_theta"
586 | ]
587 | },
588 | {
589 | "cell_type": "markdown",
590 | "metadata": {},
591 | "source": [
592 | "## 4.B Visualization of Fitted Model\n",
593 | "Finally, you should also **plot the best fit line**. The best fit line tells us that the model is not a good fit to the data because the **data has a non-linear pattern**. While **visualizing the best fit** as shown is **one possible way to debug** your learning algorithm, it is not always easy to visualize the data and model.
\n",
594 | "In the next section, you will implement a function to **generate learning curves** that can help you debug your learning algorithm even if it is **not easy to visualize** the data."
595 | ]
596 | },
597 | {
598 | "cell_type": "code",
599 | "execution_count": 17,
600 | "metadata": {},
601 | "outputs": [
602 | {
603 | "data": {
604 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl4VNX5wPHvC4QgBIkCIooSFmWPCJGCoCZ1w7VqpS7gD1fq0opYLbhj3UBbKdSVulGKgnVFwQWUFFRQQZBdUIiIUgQ0yA5J3t8f906YZJbcLDN3JvN+nidPcs/c5Z1DmDf3nHPPEVXFGGNM6qrjdwDGGGP8ZYnAGGNSnCUCY4xJcZYIjDEmxVkiMMaYFGeJwBhjUpwlApPSRKSuiGwXkSNrcl9jkoklApNU3A/iwFeJiOwK2h5Y2fOparGqZqjquprct7JE5CAReUFE/iciv4jIVyJyi8dj/y0iI2s6JpM66vkdgDGVoaoZgZ9FpAC4WlVnRtpfROqpalE8YqumcUBdoCPwC9AB6ORrRCZl2B2BqVVE5H4RmSIiL4nINmCQiPQRkXkiUigiG0RknIikufvXExEVkSx3+9/u6++IyDYRmSsibSq7r/v6GSKySkS2isg/RORjEbk8QujHAS+qaqGqlqjqClV9LehcnUVkpoj8JCIrReS3bvn1wEXA7e5d0es1W6MmFVgiMLXR+cCLQBNgClAEDAWaAX2B/sDvoxx/KXAXcDCwDrivsvuKyCHAy8Ct7nXXAr2inGce8JCIXC4iRwW/ICKNgRnAv4BDgIHAeBHpoKpPuO/xQbfZ6vwo1zAmLEsEpjb6SFXfcv+y3qWqn6vqp6papKprgPHASVGOf0VV56vqPmAS0L0K+54NLFLVN93XxgCbo5znepwP9BuBFSKyWkROc187F1ilqv9y38MC4A3gwujVYIw3lghMbfRd8IaIdBSRaYGOWOAvOH+lR/K/oJ93AhmRdoyy72HBcagzu+P6SCdR1Z2qer+q9gCaAq8Br4pIE6A10Ndt2ioUkUKc5qCWUeIyxjNLBKY2Kj+l7tPAUqC9qh4I3A1IjGPYALQKbIiIAId7OVBVtwIP4SSVLJyE8oGqZgZ9ZajqHwKH1GjkJuVYIjCpoDGwFdghIp2I3j9QU94GeojIOSJSD6ePonmknUXkHhHJEZH6ItIAp4noJ2A1MBXoIiKXikia+9VLRDq4h28E2sb27ZjazBKBSQV/AgYD23DuDqbE+oKquhGn+eZRYAvQDlgI7Ily2AR33x+AXOAst8loK3A6MAjnTuN/OHcM6e5xzwDHiMjPIvJKzb8bU9uJLUxjTOyJSF2cD/gLVXWO3/EYE8zuCIyJERHpLyJNRCQdZ4hpEfCZz2EZE8ISgTGx0w9YgzNstD9wnqpGaxoyxhfWNGSMMSnO7giMMSbFJcWkc82aNdOsrCy/w/Bsx44dNGrUyO8wEpLVTWRWN5FZ3UQWrW4WLFiwWVUjDlsOSIpEkJWVxfz58/0Ow7P8/Hxyc3P9DiMhWd1EZnUTmdVNZNHqRkS+9XIOaxoyxpgUZ4nAGGNSnCUCY4xJcUnRRxDOvn37WL9+Pbt37/Y7lBBNmjRhxYoVfofhqwYNGtCqVSvS0tL8DsUYU4GkTQTr16+ncePGZGVl4UzsmDi2bdtG48aN/Q7DN6rKli1bWL9+PW3atKn4AGNMWaoQ/LlWfruGJW3T0O7du2natGnCJQEDIkLTpk0T8m7NmIQ3ciQMG+Z8+IPzfdgwpzxGkjYRAJYEEpj92xhTBapQWAhjx+5PBsOGOduFhfuTQw1L2qYhY4ypdURgzBjn57FjnS+AoUOd8hj9gZXUdwR+KigooGvXrtU+T35+Pp988knY1/bs2cMpp5xC9+7dmTJlCldffTXLly8H4MEHH6z2tb3Izc2t8GE+L/sYYzwKTgYBMUwCYInAd9ESwcKFC9m3bx+LFi3ioosu4plnnqFz585A/BKBMSbOAs1BwYL7DGLAEkE1FBUVMXjwYLKzs7nwwgvZuXMn4HyAn3TSSfTs2ZPTTz+dDRs2ADBu3Dg6d+5MdnY2F198MQUFBTz11FOMGTOG7t27M2fO/vVKfvzxRwYNGsSiRYvo3r0733zzTelf3iNGjGDXrl10796dgQMHhsSVkZHB8OHD6dmzJ6eccgqfffYZubm5tG3blqlTpwJOZ/sVV1xBt27dOPbYY5k1axYAu3bt4uKLLyY7O5uLLrqIXbt2lZ73/fffp0+fPvTo0YMBAwawffv2mNWtMSkpuE9g6FAoKXG+B/cZxECt6CO4961lLP/hlxo9Z+fDDuSec7pE3eerr77i2WefpW/fvlx55ZU88cQTDB06lFtvvZW3336b5s2bM2XKFO644w6ee+45Ro0axdq1a0lPT6ewsJDMzEyuvfZaMjIyuOWWW8qc+5BDDuGZZ57hr3/9K2+//XaZ10aNGsVjjz3GokWLwsa1Y8cOcnNzGT16NOeffz533nknM2bMYPny5QwePJhzzz2Xxx9/HIAlS5awcuVKTjvtNFatWsWTTz5Jw4YNWbx4MYsXL6ZHjx4AbN68mfvvv5+ZM2fSqFEjRo8ezaOPPsrdd99d1So2xpQnApmZZfsEAs1EmZkxax6qFYnAL0cccQR9+/YFYNCgQYwbN47+/fuzYsUKTj31VACKi4tp2bIlANnZ2QwcOJDzzjuP8847L2Zx1a9fn/79+wPQrVs30tPTSUtLo1u3bhQUFADw0Ucf8cc//hGAjh070rp1a1atWsXs2bO58cYbS+PNzs4GYN68eSxfvrz0/e7du5c+ffrE7D0Yk7JGjiz73EAgGcSwj6BWJIKK/nKPlfJDJEUEVaVjx4589lnoioTTpk1j9uzZTJ06lfvuu49ly5bFJK60tLTS2OrUqUN6enrpz0VFRYDz0Fck4YZ+qiqnnnoqL730UgwiNsaUUf7/YIyHY1sfQTWsW7eOuXPnAvDSSy/Rr18/OnTowObNm0vL9+3bx7JlyygpKeG7774jLy+Phx9+mMLCQrZv307jxo3Ztm1bpa+dlpbGvn37qhz7iSeeyKRJkwBYtWoV69ato0OHDmXKly5dyuLFiwHo3bs3H3/8MV9//TUAO3fuZNWqVVW+vjEmcVgiqIZOnToxYcIEsrOz+emnn7juuuuoX78+EydOZPjw4RxzzDF0796dTz75hOLiYgYNGlTaOTts2DAyMzM555xzeP3110M6iysyZMiQ0qamqrj++uspLi6mW7duXHTRRbzwwgukp6dz3XXXsX37drKzs3n44Yfp1asXAM2bN+eFF17gkksuITs7m969e7Ny5coqXdsYk1iSYs3inJwcLT9OfcWKFXTq1MmniKJL9bmGAsL9G9kCI5FZ3URW6+umGnMLVbAwzQJVzanoHHZHYIwxfvJhbqHyLBEYY4xffJpbqLxaMWrIGGOSkk9zC5VndwTGGOMnH+YWKs8SgTHG+MmHuYXKs0RgjDF+8WluofIsEVRRYWEhTzzxRJWOPfPMMyksLKzytTMyMqK+Xp3YjDFxFGluoaFDYzq3UHmpkwjKZ9ZqZtpoH7bFxcVRj50+fTqZmZnVun40lgiMSSIjR5btEwgkAxs+WsNiME53xIgRfPPNN3Tv3p1bb72V/Px88vLyuPTSS+nduzcA5513Hj179qRLly6MHz++9NisrCw2b95MQUEBnTp14pprrqFLly6cdtppZaZ9Dli7di19+vThuOOO46677iot3759OyeffDI9evSgW7duvPnmm2Fji7SfMSZBxHluoRCqmvBfPXv21PKWL18eUhZWSYnq0KGq4HwPt10Fa9eu1S5dupRuz5o1Sxs2bKhr1qzRX375RVVVt2zZoqqqO3fu1C5duujmzZtVVbV169a6adMmXbt2rdatW1cXLlyoqqoDBgzQiRMnhlzrnHPO0QkTJqiq6mOPPaaNGjVSVdV9+/bp1q1bVVV106ZN2q5dOy0pKQmJLdJ+sRbu32jWrFkxv26ysrqJzOomsmh1A8xXD5+xFT5HICJ1gGOAw4BdwDJV3Rjj/FRz4jhOt1evXrRp06Z0Erlx48bx+uuvA/Ddd9+xevVqmjZtWuaYNm3a0L17dwB69uxZOk10sI8//phXX30VgMsuu4zhw4cDThK//fbbmT17NnXq1OH7779n48bQf5pI+x166KE19t6NMckrYiIQkXbAcOAUYDWwCWgAHC0iO4GngQmqWhKPQKslkAwCSQBiMk63UaNGpT/n5+czc+ZM5s6dS8OGDcnNzWX37t0hxwSmiAaoW7du2KYhCD819KRJk9i0aRMLFiwgLS2NrKyssNfwup8xJjVF6yO4H/g30E5VT1fVQap6oapmA+cCTYDL4hFktcVgnG5F00dv3bqVgw46iIYNG7Jy5UrmzZtX5Wv17duXyZMnA5ROER24xiGHHEJaWhqzZs3i22+/DRtbpP2MMQaiJAJVvURVZ7vtTOVf+1FV/66qE2IbXg2I0Tjdpk2b0rdvX7p27cqtt94a8nr//v0pKioiOzubu+66q7QDuSrGjh3L448/znHHHcfWrVtLywcOHMj8+fPJyclh0qRJdOzYMWxskfYzxhjwMNeQiMwHngdeVNWfYx9SDYvhGqAvvvhime3gqWDT09N55513wh4X6Ado1qwZS5cuLS0vv25xQJs2bUoXugFnVFDg+ODyaLFF2s8YY7xMOncxcAXweVBSeD/cnULC8mENUGOMSRYVPkegql+r6h3A0cCLwHPAOhG5V0QOjnWANcbvcbrGGJOgPD1QJiLZwN+AR4BXgQuBX4APYxdaxZLppiTV2L+NMcnDSx/BAqAQeBYYoap73Jc+FZG+sQwumgYNGrBlyxaaNm0admil8Y+qsmXLFho0aOB3KMYYD7z0EQxQ1TXhXlDVC2o4Hs9atWrF+vXr2bRpk18hRLR79+6U/xBs0KABrVq18jsMY4wH0R4oG4QzUihsEnAfOGupqh/FKrho0tLSaNOmjR+XrlB+fj7HHnus32EYY4wn0e4ImgIL3aahBex/srg9cBKwGRgR6WARaQDMBtLd67yiqveISBtgMnAw8AVwmarurYH3YowxpgqiPVA2FugBvAQ0B052t7/H+fD+raqujnLuPcCvVfUYoDvQX0R6A6OBMap6FPAzcFWNvBNjjDFVErWPQFWLgRnuV6W4zxlsdzfT3C8Ffg1c6pZPAEYCT1b2/MYYY2qGxHKYn4jUxWlWag88jjP8dJ6qtndfPwJ4R1W7hjl2CDAEoEWLFj0Dc+0kg+3bt1e4iliqsrqJzOomMqubyKLVTV5e3gJVzanoHF5GDVWZe0fRXUQygdeBTuF2i3DseGA8QE5OjgZP35Do8vPzSaZ448nqJjKrm8isbiKribqJywplqloI5AO9gUwRCSSgVsAP8YjBGGNMeF4eKMsE/g/ICt5fVW+s4LjmwD5VLRSRA3DWNRgNzMJ5MnkyMBiwdRONMcZHXpqGpgPzgCVAZRahaQlMcPsJ6gAvq+rbIrIcmCwi9wMLcZ5YNsYY4xMviaCBqt5c2ROr6mIg5Kkq9wG1XpU9nzHGmNjw0kcwUUSuEZGWInJw4CvmkRljjIkLL3cEe3GGfd7B/hE+CrSNVVDGGGPix0siuBlor6qbYx2MMcaY+PPSNLQM2BnrQIwxxvjDyx1BMbBIRGbhzB8EVDx81BhjTHLwkgjecL+MMcbUQhUmAlWdEI9AjDHG+MPLk8VHAQ8BnXHWIwBAVW3UkDHG1AJeOoufx5kmugjIA/4FTIxlUMYYk8rmF/xE1ohptL1tGnuKimN+PS99BAeo6gciIqr6LTBSROYA98Q4NmOMSSnz1mzh4vHzSrfr1alD/bqxnxvUSyLYLSJ1gNUi8gecFcoOiW1YxhiTOuas3sRlz35Wpuzl3/ehV5v4TOLgJRHcBDQEbgTuw1lhbHAsgzLGmFQwa+WPXPHC52XKXrv+eHoceVBc4/AyaigQ5XbgitiGY4wxtd/7y/7HkIkLypS99Yd+dGvVxJd4IiYCEXmLCKuHAajquTGJyBhjaqlpizdww4tflCmbfuMJdD7sQJ8ickS7I/ir+/0C4FDg3+72JUBBDGMyxpha5Y2F33PTlEVlyt4fdiJHt2jsU0RlRUwEqvpfABG5T1VPDHrpLRGZHfPIjDEmyf1n/nfc+sriMmUf/Okk2jUPv9i8X7x0FjcXkbbugjKISBugeWzDMsaY5DXp02+54/WlZcr+e2surZs28imi6LwkgmFAvoiscbezgCExi8gYY5LUCx+vZeRby0u369YR8m/J5YiDG/oYVcW8jBp6151moqNbtFJV90Q7xhhjUsn42d/w4PSVpdsN69flgz+dRMsmB/gYlXde7ghwP/i/jHEsxhiTVB6f9TWPvPdV6XaTA9KYMexEDjmwQZSjEo+nRGCMMcahqoyZuZpxH6wuLWveOJ13hp5As4x0HyOrOksExhjjgaoy+t2veOq/35SWHZ55AG/9sR8HN6rvY2TV52UaagEGAm1V9S8iciRwqKp+VsGhxhiT9FSV+95ewXMfry0ta9usEa9f35cmDdN8jKzmeLkjeAIowZlj6C/ANuBV4LgYxmWMMb4qKVHunrqUf89bV1rW8dDGvHxtHw5sUDsSQICXRPArVe0hIgsBVPVnEUnu+yBjjImgpEQZ8dpiXp6/vrQsu1UTXrqmN43Sa2drupd3tU9E6uLOOyQizXHuEIwxJjGogkjkbQ+KS5Q/vbyINxb9UFqW0/ogJl71Kw6oX7emIk1IXhLBOOB14BAReQC4ELgzplEZY4xXI0dCYSGMGeN8+KvCsGGQmem8VoGi4hKGTl7EtCUbSsv6tG3K81ccR4O02p0AArw8UDZJRBYAJwMCnKeqK2IemTHGVETVSQJjxzrbY8Y4SWDsWBg6NOqdwd6iEq6ftICZK34sLcvt0JynL+tJer3USAABXhu8VgO/BPYXkSNVdV30Q4wxJsZEnA9/cD78Awlh6ND9dwjl7Ckq5pp/LWD2qk2lZad3acFjl/YgLQ7LQiYiL8NH/4izPvFGoBjnrkCB7NiGZowxHgSSQSAJQNgksHtfMYOf+4xP1/5UWnZ2dkv+flF36qVoAgjwckcwFOigqltiHYwxxlRaoE8g2LBhpclg594iBj3zKV+sKyx9+YIeh/PIhcdQt07lOpRrKy+J4Dtga6wDMcaYSgskgUCfQFAfwXbqcnHb81j6wy+lu1/S6wgeOK8bdSwBlBFtqcqb3R/X4ExDPQ0onXVUVR+NcWzGGBOdiDM6KKhP4JeHHubC+n1YVScD3CQwuE9rRp7bBankkNJUEe2OILCG2jr3q777BVHWMjbGmLgaORJU+XHbHvo9PIu9RSVQx1kB7JoT2nD7mZ0sAVQg2lKV9wKIyABV/U/wayIyINaBGWOMF98X7qLvqA/LlN2Q145bTutgCcAjL30EtwH/8VBmjDFxs27LTk58ZFaZsj5tm/LSkN4+RZS8ovURnAGcCRwuIuOCXjoQKIp1YMYYE86aTdv59d/+W6Ysr0Nznr+il08RJb9odwQ/APOBc4EFQeXbcNYxNsaYuFm1cRunjZldpuysbi15fGAPnyKqPaL1EXwJfCkiL6rqvsqeWESOAP4FHIozSd14VR0rIgcDU4AsoAD4nar+XIXYjTEpYNkPW7n83R3w7v4k8Nserfjb747xMaraxctcQ5VOAq4i4E+q+oWINAYWiMgM4HLgA1UdJSIjgBHA8CpewxhTS335XSG/efzjMmUDf3UkD5zfzaeIaq+YTa6tqhuADe7P20RkBXA48Bsg191tApCPJQJjjGt+wU9c+NTcMmWnt67H09ed7lNEtV/ECTZEZKL7fWh1LyIiWcCxwKdACzdJBJLFIdU9vzEm+X3yzWayRkwrkwRuyGtHwaizuKRTci4KnyxENfyzYSKyHDgDmIrzF3yZAbmq+lOYw8KdJwP4L/CAqr4mIoWqmhn0+s+qelCY44YAQwBatGjRc/LkyZ7eUCLYvn07GRkZfoeRkKxuIkvVulmyqYi/LdhTpuz89mn8pv3+hRBTtW68iFY3eXl5C1Q1p6JzRGsaegp4F2iLM2ooOBGoWx6ViKThrG88SVVfc4s3ikhLVd0gIi2BH8Mdq6rjgfEAOTk5mpubW9HlEkZ+fj7JFG88Wd1Elmp1M3P5Rq7+1/wyZbed0ZHfn9QuZN9Uq5vKqIm6iTZqaBwwTkSeVNXrKnticR7pexZYUW5eoqnAYGCU+/3Nyp7bGJO8pi3ewA0vflGm7N5zuzD4+Cx/AjKeRg1dJyLHACe4RbNVdbGHc/cFLgOWiMgit+x2nATwsohchTOHkU1XYUwKeGPh99w0ZVGZsocu6MYlvY70KSIT4GVhmhtx2uoDTTuTRGS8qv4j2nGq+hHl+hWCnFypKI0xSWvK5+sY/uqSMmWP/u4YLujRyqeITHleho9eDfxKVXcAiMhoYC4QNREYY1Lb1RM+L7MeMMBjlx7L2dmH+RSRicRLIhCcJSoDAstVGmNMiIvHz2XemrKDCp++rCendznUp4hMRbwkgueBT0XkdXf7PJxOYGOMKXX2P+aw9PtfypT9uX8Hrs9t71NExisvncWPikg+0A/nTuAKVV0Y68CMMckh95FZFGzZWaZs5DmdubxvG58iMpXlaYoJVf0C+KLCHY0xKaPnfTPYsmNvmbJRF3TjYhsFlHRiNteQMaZ2OvrOd5zlIIP8/aLunHfs4T5FZKrLEoExxpOsEdNCyp4a1JP+Xa0TONl5eY5gtKoOr6jMGFM7hUsAz19xHHkdbL7I2sLLHcGphE4TfUaYMmNMLRIuAbx4za84vl0zH6IxsRRtzeLrgOuBtiISPKVEY+Dj8EcZY5JduATw6nXH07N1yCTBppaIdkfwIvAO8BDOKmIB27xOQW2MSQ6qSpvbpoeUv/3HfnQ9vIkPEZl4ijb76FZgq4iUbwLKEJEMVV0X29CMMbEWKQG8d9OJdDi0sQ8RGT946SOYhrP+gAANgDbAV0CXGMZljImhkhKl7e2hCWDWLbm0adbIh4iMn7w8WVxmpWgR6QH8PmYRGWNipqi4hPZ3vBNSPufPeRxxcEMfIjKJoNLPEajqFyJyXCyCMcbExt6iEo6+MzQBfHr7ybQ4sIEPEZlE4uU5gpuDNusAPYBNMYvIGFNjdu8rpuNd74aUz7/zFJpl2ILwxuHljiC4x6gIp8/g1diEY4ypCTv2FNHlnvdCyr+8+zSaNEyLTxCqIBJ52yQML30E9wKISGNnU7fHPCpjTJX8snsf2SPfDylfeu/pZKTHcUaZkSOhsBDGjHE+/FVh2DDIzHReMwnFS9NQV2AicLC7vRkYrKpLYxybMcajn3fs5dj7ZoSUr/hLfw6oXze+wag6SWDsWGd7zBgnCYwdC0OH2p1BAvLyJ8J44GZVnQUgIrlu2fExjMsY48GP23bT64EPQsq/ur8/6fXinAACRJwPf3A+/AMJYejQ/XcIJqF4SQSNAkkAQFXzRcQGGhvjow1bd9HnoQ9Dylc/cAZpdev4EFE5gWQQSAJgSSCBeUkEa0TkLpzmIYBBwNrYhWSMiWTdlp2c+MiskPJvHjyTunUS6EM20CcQbNgwSwYJyksiuBK4F3jN3Z4NXBGziIwxIb7+cTunPPrfkPI1D55JnURKALA/CQT6BIL7CMCSQQLyMmroZ+DGOMRijClnxYZfOGPsnJDytQ+diSTqh6mIMzoouE8g0GeQmWlJIAHZCmXGJKBZK3/kihc+DylP6AQQbOTIsqODAskgGWJPQZYIjEkg05ds4PpJX4SUF4w6y4doqqn8h74lgYRlicCYBPDKgvXc8u4OoGwSSMoEYJKOlwfKxoUp3grMV9U3az4kY1LHxLkF3PXmspBySwAmnrzcETQAOgL/cbd/CywDrhKRPFW9KVbBGVNbPZn/DaPfXRlSbgnA+MFLImgP/FpViwBE5EngfZxF7ZfEMDZjap2/vf8V//jw65DyglFnkZ+fH/+AjMFbIjgcaITTHIT782GqWiwie2IWmTG1yMipy3jhk4KQcrsDMInASyJ4GFgkIvk4y1WeCDzoTjMxM4axGZP0bvnPl7yyYH1IuSUAk0i8PFD2rIhMB3rhJILbVfUH9+VbYxmcMcnq2okLeHfZ/0LKLQGYROR1+GgdnFXJ6gHtRaS9qs6OXVjGJKdLxs9j7potIeWWAEwi8zJ8dDRwEc5IoRK3WHHmHDLGAGf/Yw5Lv/+lTFlGej2W3nu6TxEZ452XO4LzgA6qah3DxpTTb/SHrP95V5mywzMP4OMRv/YpImMqz9M01EAaYInAGFe3ke+xbXdRmbLOLQ9k+tATfIrImKrzkgh24owa+oCgZKCqNiOpSTlZI6aFlPVuezCTh/TxIRpjaoaXRDDV/TImZYVLAKd1bsH4/8vxIRpjapaX4aMTqnJiEXkOOBv4UVW7umUHA1OALKAA+J273oExCSlcAvhtj1b87XfH+BCNMbERMRGIyMuq+jsRWYIzSqgMVc2u4NwvAI8B/woqGwF8oKqjRGSEuz280lEbE2PhEsDgPq259zddfYjGmNiKdkcw1P1+dlVOrKqzRSSrXPFvgFz35wlAPpYITAIJlwBuyGvHrad39CEaY+JDVEP+2C+7g8iVwBxVXV3pkzuJ4O2gpqFCVc0Mev1nVT0owrFDgCEALVq06Dl58uTKXt4327dvJyMjw+8wElKi1s3l7+4IKbvw6DTObls/bjEkat0kAqubyKLVTV5e3gJVrbAjy0tncRYwSERaAwuAOTiJYVElYq00VR0PjAfIycnR3NzcWF6uRuXn55NM8cZTotVNuDuAe8/twuDjs+IeS6LVTSKxuomsJurGS2fx3QAicgBwDc78Qn8H6lbhehtFpKWqbhCRlsCPVTiHMdUWLgE8cmE2A3KO8CEaY/zlZYqJO4G+QAawELgF566gKqYCg4FR7ndb4czEVbgE8PilPTgru6UP0RiTGLw0DV0AFAHTgP8C81R1d0UHichLOB3DzURkPXAPTgJ4WUSuAtYBA6oYtzGeqSptbpseUv7c5Tn8umMLHyIyJrF4aRrqISKNgX44q5L9U0Q2qmqNDBGuAAAT+klEQVS/Co67JMJLJ1c+TGMqL1ICePGaX3F8u2Y+RGRMYvLSNNQVOAE4CcgBvqPqTUPGxFxJidL29tAE8Nr1x9PjyLCD1IxJaV6ahkbjTDk9DvhcVffFNiRjqqaouIT2d7wTUj7txn50OayJDxEZkxy8NA2dJSL1gaOBDiLylSUDk0j2FBXT4c53Q8pn3nwS7Q+xsecAqIJI5G2T0rw0DZ2EM01EAc5SlUeIyGBbocz4bdfeYjrdHZoA5vw5jyMObuhDRAlq5EgoLIQxY5wPf1UYNgwyM53XTMrz0jT0KHCaqn4FICJHAy8BPWMZmDGRbNu9j24j3w8pn3fbyRzapIEPESUwVScJjB3rbI8Z4ySBsWNh6FC7MzCAt0SQFkgCAKq6SkTSYhiTMWH9vGMvx943I6R8wZ2n0DQj3YeIkoCI8+EPzod/ICEMHbr/DsGkPC+JYL6IPAtMdLcH4kw1YUxc/G/rbno/9EFI+Zd3n0aThvY3SYUCySCQBMCSgCnDSyK4DrgBuBGnj2A28EQsgzIGYN2WnZz4yKyQ8mX3nk6jdC+/ugbY3ycQbNgwSwamlJdRQ3tw+gkejX04xsDqjds4dUzoWIQVf+nPAfWrMsVVCgskgUCfQHAfAVgyMED0hWnCLkgT4GFhGmMqZen3Wzn7Hx+FlK+6/wzq16vjQ0S1gIgzOii4TyDQZ5CZaUnAANHvCAYAu+IViEld8wt+4sKn5oaUf/3AGdSrawmg2kaOLDs6KJAMLAkYV7RE8KI7z9BEVb0sbhGZlPHR6s0MevbTkPI1D55JnTr2IVWjyn/oWxIwQaIlgvoiMhg4XkQuKP+iqr4Wu7BMbfb+sv8xZGLowLO1D52J2AeUMXEXLRFcizNUNBM4p9xrClgiMJXy5qLvGfruDsqPPi4YdZY/ARljgCiJQFU/Aj4Skfmq+mwcYzK1zIufruP215eElFsCMCYxeBk+aknAVMk/Z6/hgekrQsotARiTWOypHFPjHp2xinEfrA4pLxh1Fvn5+fEPyBgTVdREIE7PXStV/S5O8Zgkdu9by3j+44KQcrsDMCaxRU0Eqqoi8gY206iJ4pb/fMkrC9aHlFsCMCY5eGkamicix6nq5zGPxiSV30+cz3vLNoaUWwIwJrl4SQR5wLUiUgDswJl4TmvlFBO2ipMnF4+fy7w1P4WUWwKoQfa7aOLISyI4I+ZRJAJbxalCZ4ydw4oNv4SUWwKoYfa7aOLMy/DRb0WkH3CUqj4vIs2B2rUQrK3iFFWfhz5gw9bdIeWWAGLAfheND7ysWXwPkAN0AJ4H0oB/A31jG1oc2SpOYXW86x127ysJKbcEEEP2u2h84KVp6HzgWOALAFX9QUQaxzQqP9gqTqWyRkwLKWuQVoeV96VGK6Hv7HfRxJmXOX73qqrirk0gIo1iG1INUY2+HW7/cKs4VXRcLZI1YlpIEjisSQMKRp1lSSCe7HfRxJmXRPCyiDwNZIrINcBM4JnYhlVNI0eW/Y8T+I8VqaOt/CpOJSXO97Fj/fsPWNlEVg3hEkDHQxtTMOosPrnt5Jhd14SRiL+Lptbz0ln8VxE5FfgFp5/gblWdEfPIqqoqnW2JtopTnEaNhGsC6tO2KS8N6V1j1zCVlGi/iyYleOksHq2qw4EZYcoST1U72xJlFac4jBoJlwBO79KCpy/LqdZ5YyIe4+kTbcx+ovwumpThpbP4VKD8h/4ZYcoSR1U72xJhFacYjhoJlwAG9GzFIwOOqfI5Yyoed0aJOmY/EX4XTcqI2EcgIte5C9h3EJHFQV9rgcXxC7EKkr2zLTgZBFQjCYTrA7iybxsKRp2VuEkg+M4o8G8XuDMqLKyZf8t4XMOYJBB1zWLgHeAhYERQ+TZVDZ1fIFGU72wLblqB5LjFjpTIKhl7uDuAG08+iptPPbq6EcZePMbT25h9Y4DoK5RtBbYClwCIyCFAAyBDRDJUdV18QqykZO9sq4FEFi4B/OOSYznnmMP2XyPR6wHiM57exuwb46mz+BzgUeAw4EegNbAC6BLb0KohmTvbqpHIwiWAZ9JWccq2byH7TKcgUdrAvaihOyPfr2FMgvPSWXw/0BuYqarHikge7l1CQkvmzrZKJrJwCeDfV/2Kfu2bwrAZyTlvTTya+GpDM6IxNcBLItinqltEpI6I1FHVWSIyOuaRpboKEpmq0ua26SGHvXJtH3KyDt5fkKxt4PFo4kv2ZkRjaoiXRFAoIhnAbGCSiPwIFMU2LBNJSYnS9vbQBPDWH/rRrVWT0AOSuQ08Hk18ydyMaEwN8ZIIfgPsBoYBA4EmwF9iGZQJVVyitAuTAN676UQ6HBplDsBkbwOPRxNfMjcjGlMDIiYCEbkJ+BhYqKrFbvGEuERlSu0rLuGoO94JKZ91Sy5tmlUw/5+1gRtjPIh2R9AKGAt0FJHFwCc4iWFudZ8jEJH+7rnrAs+o6qjqnK82ipQAPhqeR6uDGno7ibWBG2M8iPYcwS0AIlIfZ2Ga44ErgX+KSKGqdq7KBUWkLvA4ztQV64HPRWSqqi6vyvlqmz1FxXS4892Q8k9vP5kWBzao/AmtDdwYUwEvfQQHAAfi9A00AX4AllTjmr2Ar1V1DYCITMbph0jpRLBrbzGd7g5NAAvuPIWmGenVO7m1gRtjohCNMJ+KiIzHeWhsG/ApMA+Yp6o/V+uCIhcC/VX1anf7MuBXqvqHcvsNAYYAtGjRoufkyZOrc9m42r59OxkZ3pZ13lWkXDdzZ0j506c0JL1e7fvArkzdpBqrm8isbiKLVjd5eXkLVLXCaYWj3REcCaQDq4HvcZpxCqsQZ3nhPt1CspGqjgfGA+Tk5Ghubm4NXDo+8vPzqSjerbv2ccy974eUr7yvPw3S6sYoMv95qZtUZXUTmdVNZDVRN9H6CPqLiODcFRwP/AnoKiI/4XQY31PFa64HjgjaboXT3JQSftqxlx73ha7rs+r+M6hfz8uCccYYU7Oi9hG4axUvFZFCnAnotgJn47TzVzURfA4cJSJtcO40LgYureK5ksambXs47oGZIeVfP3AG9epaAjDG+CfacwQ34twJ9AX24Q4dBZ6jGp3FqlokIn8A3sMZPvqcqi6r6vkS3Yatu+jz0Ich5d88eCZ169S+PgBjTPKJdkeQBbwCDFPVDTV5UVWdDoQ+JluLfPfTTk54eFZI+ZoHz6SOJQBjTAKJ1kdwczwDqS3Wbt7B5e/ugHfLJoG1D52J2LBNY0wC8vIcgfFg1cZtnDZmdki5JQBjTKKzRFBNS7/fytn/+KhMWf26sOqBs3yKyBhjKscSQRUtXPcz5z/xSZmyQxqn89kdp5Cfn+9PUMYYUwWWCCrp0zVbuGj8vDJlbZo1YtYtuf4EZIwx1WSJwKM5qzdx2bOflSnrdngT3vpjP58iMsaYmmGJoAIfrNjIVRPmlynr1eZgXv59H58iMsaYmmWJIILpSzZw/aQvypTldmjOC1f08ikiY4yJDUsE5fx31SYGP1e2Ceis7JY8fmkPnyIyxpjYskTgWrJ+K+c8VnYY6ICerXhkwDE+RWSMMfGR8ongi3U/c0G5YaAPXdCNS3od6VNExhgTXymbCD4v+IkBT80tUzbxql6ccFRznyIyxhh/pFwi+OTrzVz6zKdlyl66pjd92jX1KSJjjPFXyiSC2as28X/lOoFfubYPOVkH+xSRMcYkhlqfCMI9B/DGDX3pfkSmTxEZY0xiqdWJ4PFZX/PIe1+Vbr/9x350PbyJjxEZY0ziqdWJoPsRmbRt1ojHB/agU8sD/Q7HGGMSUq1OBH3bN+NDmwzOGGOislXTjTEmxVkiMMaYFGeJwBhjUpwlAmOMSXGWCIwxJsVZIjDGmBRnicAYY1KcJQJjjElxoqp+x1AhEdkEfOt3HJXQDNjsdxAJyuomMqubyKxuIotWN61VtcK59ZMiESQbEZmvqjl+x5GIrG4is7qJzOomspqoG2saMsaYFGeJwBhjUpwlgtgY73cACczqJjKrm8isbiKrdt1YH4ExxqQ4uyMwxpgUZ4nAGGNSnCWCGiYit4iIikgzd1tEZJyIfC0ii0Wkh98xxpuIPCIiK933/7qIZAa9dptbN1+JyOl+xuknEenv1sHXIjLC73j8JCJHiMgsEVkhIstEZKhbfrCIzBCR1e73g/yO1Q8iUldEForI2+52GxH51K2XKSJSv7LntERQg0TkCOBUYF1Q8RnAUe7XEOBJH0Lz2wygq6pmA6uA2wBEpDNwMdAF6A88ISJ1fYvSJ+57fhznd6UzcIlbN6mqCPiTqnYCegM3uPUxAvhAVY8CPnC3U9FQYEXQ9mhgjFsvPwNXVfaElghq1hjgz0BwD/xvgH+pYx6QKSItfYnOJ6r6vqoWuZvzgFbuz78BJqvqHlVdC3wN9PIjRp/1Ar5W1TWquheYjFM3KUlVN6jqF+7P23A+9A7HqZMJ7m4TgPP8idA/ItIKOAt4xt0W4NfAK+4uVaoXSwQ1RETOBb5X1S/LvXQ48F3Q9nq3LFVdCbzj/mx147B6iEBEsoBjgU+BFqq6AZxkARziX2S++TvOH5sl7nZToDDoD60q/e7U6sXra5qIzAQODfPSHcDtwGnhDgtTVuvG7EarG1V9093nDpzb/kmBw8LsX+vqxgOrhzBEJAN4FbhJVX9x/vhNXSJyNvCjqi4QkdxAcZhdK/27Y4mgElT1lHDlItINaAN86f6ytgK+EJFeOBn6iKDdWwE/xDjUuItUNwEiMhg4GzhZ9z+8khJ144HVQzkikoaTBCap6mtu8UYRaamqG9zm1R/9i9AXfYFzReRMoAFwIM4dQqaI1HPvCqr0u2NNQzVAVZeo6iGqmqWqWTj/sXuo6v+AqcD/uaOHegNbA7e3qUJE+gPDgXNVdWfQS1OBi0UkXUTa4HSof+ZHjD77HDjKHf1RH6cDfarPMfnGbfd+Flihqo8GvTQVGOz+PBh4M96x+UlVb1PVVu5nzMXAh6o6EJgFXOjuVqV6sTuC2JsOnInTEboTuMLfcHzxGJAOzHDvmOap6rWqukxEXgaW4zQZ3aCqxT7G6QtVLRKRPwDvAXWB51R1mc9h+akvcBmwREQWuWW3A6OAl0XkKpyReQN8ii/RDAcmi8j9wEKcJFopNsWEMcakOGsaMsaYFGeJwBhjUpwlAmOMSXGWCIwxJsVZIjDGmBRnicDUCBE5VEQmi8g3IrJcRKaLyNEikhuYJdFvIvIXEYn64FsNXSdTRK6vgfPki0iNLtge7Zwi8oqItI1ybH0RmS0iNuy8lrFEYKrNfQDodSBfVdupameccd8t/I2sLFW9W1VnxuFSmUClEoH7wKFv/x9FpAtQV1XXRNrHnRDvA+CiuAVm4sISgakJecA+VX0qUKCqi1R1jruZ4f61uVJEJrmJAxG5W0Q+F5GlIjI+qDxfREaLyGciskpETnDLG4rIy+66BlPcOdhz3NdOE5G5IvKFiPzHnaemDBF5QUQudH8uEJF73f2XiEjHMPtPF5Fs9+eFInK3+/N9InK1iGSIyAdB5wjMGDoKaCcii0TkEfeYW933ulhE7nXLssSZc/8J4AvKTjNRPpaQ9yciZ7gP5AX2yRWRt7zWRzkDcZ9IFZHW4sxt30xE6ojIHBEJzKP1hruvqUUsEZia0BVYEOX1Y4GbcObab4vz5CjAY6p6nKp2BQ7AmYsooJ6q9nKPu8ctux742V3X4D6gJ4A4iwDdCZyiqj2A+cDNHuLe7O7/JHBLmNdnAyeIyIE4Tz4H4u4HzAF2A+e758gD/uYmsxHAN6raXVVvdT9Ej8KZbro70FNETnTP1QFnmvJjVfXbcEFGeX8zgN4i0sjd9SJgShXroy/uv6Ebx2jgKeBPwHJVfd/dbylwXAXnMknG2vpMPHymqusB3CkDsoCPgDwR+TPQEDgYWAa85R4TmGhsgbs/OB/AYwFUdamILHbLe+MkmY/dm4r6wFwPcQVf44Iwr88BbgTWAtOAU0WkIZClql+JMzHag+6HegnO9L/hmsNOc78WutsZOIlhHfCtu05FNGHfnzs1xbvAOSLyCs489X8GTgq3fwXXaAlsCmyo6jMiMgC4Fid5BcqLRWSviDR21wowtYAlAlMTlrF/0qtw9gT9XAzUE5EGwBNAjqp+JyIjcWZULH9MMft/TyPNQyzADFW9pJJxh7tGsM+BHGANzl/fzYBr2H/3MxBoDvRU1X0iUlDuPQTH95CqPl2m0Jlrf4eHOKO9vynADcBPwOequs29K6lsfewKjt1NeIEFhDKA4A/9dJy7IVNLWNOQqQkfAukick2gQESOE5GTohwT+NDZ7LZfR0skAR8Bv3PP3xno5pbPA/qKSHv3tYYicnQl30MIt3P0O/ea83DuEG5xvwM0wZkffp+I5AGt3fJtQOOgU70HXBlopxeRw0WkMouqRHt/+UAPnAQ1xcP+kawA2gdtj8ZZN+Ju4J+BQhFpCmxS1X2ViN8kOEsEptrc9QXOx2k6+UZElgEjiTIvuqoW4nzALMHpgPzcw6WeAJq7TULDgcU403pvAi4HXnJfmweEdP5W0Rxgozt99hycv5IDiWASkCMi83HuDlYCqOoWnGaZpSLyiNu+/iIwV0SW4Cwr2BiPor0/d7bWt3HWO367ov2jmAbkArgJ/DhgtKpOAvaKSGDW3DycGXVNLWKzj5qkIc4i72mqultE2uEMZTza/cvdVIOIHIAzr33faFOBi8hrwG2q+lXcgjMxZ30EJpk0BGa5nbQCXGdJoGao6i4RuQenw3tduH3EWTTnDUsCtY/dERhjTIqzPgJjjElxlgiMMSbFWSIwxpgUZ4nAGGNSnCUCY4xJcf8PQo0oD77AaxcAAAAASUVORK5CYII=\n",
605 | "text/plain": [
606 | ""
607 | ]
608 | },
609 | "metadata": {},
610 | "output_type": "display_data"
611 | }
612 | ],
613 | "source": [
614 | "# import libraries\n",
615 | "import matplotlib.pyplot as plt\n",
616 | "%matplotlib inline\n",
617 | "\n",
618 | "plt.scatter(x[:,1], y, color='red', marker='x', label= 'train data')\n",
619 | "plt.plot(x[:,1],np.dot(x,new_theta.reshape(n,1)), label = 'best fit model')\n",
620 | "plt.title('Training Set')\n",
621 | "plt.xlabel('Change in water level (x)')\n",
622 | "plt.ylabel('Water flowing out of the dam (y)')\n",
623 | "plt.grid()\n",
624 | "plt.legend()\n",
625 | "plt.show()"
626 | ]
627 | },
628 | {
629 | "cell_type": "markdown",
630 | "metadata": {},
631 | "source": [
632 | "# 5. Bias-Variance\n",
633 | "An important concept in machine learning is the **bias-variance tradeoff**. Models with **high bias** are not complex enough for the data and tend to **underfit**, while models with **high variance** **overfit** to the training data. \n",
634 | "In this part of the exercise, you will plot training and test errors on a learning curve to diagnose bias-variance problems."
635 | ]
636 | },
637 | {
638 | "cell_type": "markdown",
639 | "metadata": {},
640 | "source": [
641 | "## 5.1 Learning Curves\n",
642 | "You will now implement code to generate the learning curves that will be useful in debugging learning algorithms. Recall that a learning curve **plots training and cross validation error as a function of training set size.** \n",
643 | "To plot the learning curve, we need a training and cross validation set error for **different training set** sizes. To obtain different training set sizes, you should use **different subsets** of the original training set `x`. Specifically, for a training set size of `i`, **you should use the first `i` examples **(i.e., `X[1:i,:]` and `y[1:i]`).
\n",
644 | "You can use the `fmin_cg` function to find the `θ` parameters. Note that the `lambda` is passed as a parameter to the `learning_curve` function. After learning the `θ` parameters, you should **compute the error on the training and cross validation sets.** Recall that the **training error** for a dataset is defined as: \n",
645 | "\n",
646 | "In particular, note that the **training error does not include the regularization** term. One way to compute the training error is to **use your existing cost function and set `λ` to `0`** only when using it to compute the training error and cross validation error. \n",
647 | "When you are **computing the training set error**, make sure you compute it on the **training subset (i.e., `x[1:n,:]` and `y[1:n]`)** (instead of the entire training set). However, for the **cross validation error**, you should compute it over the **entire cross validation set**. You should store the computed errors in the vectors `error_train` and `error_val`."
648 | ]
649 | },
650 | {
651 | "cell_type": "markdown",
652 | "metadata": {},
653 | "source": [
654 | "`learning_curve(x, y, xval, yval, lambda_)` **returns the train and cross validation set errors for a learning curve**.
In particular, it returns two vectors of the same length - `error_train` and `error_val`. Then, `error_train[i]` contains the training error for `i` examples (and similarly for `error_val[i]`)."
655 | ]
656 | },
657 | {
658 | "cell_type": "code",
659 | "execution_count": 18,
660 | "metadata": {},
661 | "outputs": [],
662 | "source": [
663 | "error_train = np.zeros(m)\n",
664 | "error_val = np.zeros(m)\n",
665 | "\n",
666 | "lambda_ = 1"
667 | ]
668 | },
669 | {
670 | "cell_type": "code",
671 | "execution_count": 38,
672 | "metadata": {},
673 | "outputs": [],
674 | "source": [
675 | "def train_val_error(x, y, xval, yval, lambda_):\n",
676 | " m,n = x.shape\n",
677 | " theta = np.ones(n) # initialize theta for each time of training with subset of training set.\n",
678 | " new_theta = fmin_cg(f=linear_reg_cost, x0=theta, fprime=linear_reg_grad, args=(x.flatten(), y, lambda_, m,n))\n",
679 | " t_error = linear_reg_cost(new_theta.flatten(), x.flatten(), y, 0, x.shape[0], x.shape[1])\n",
680 | " v_error = linear_reg_cost(new_theta.flatten(), xval.flatten(), yval, 0, xval.shape[0], xval.shape[1])\n",
681 | " return (t_error, v_error)"
682 | ]
683 | },
684 | {
685 | "cell_type": "code",
686 | "execution_count": 39,
687 | "metadata": {},
688 | "outputs": [],
689 | "source": [
690 | "def learning_curve(x, y, xval, yval, lambda_):\n",
691 | " m,n = x.shape\n",
692 | " et_array = np.zeros(m)\n",
693 | " ev_array = np.zeros(m)\n",
694 | " for i in range(1,m):\n",
695 | " et_array[i] , ev_array[i] = train_val_error(x[0:i,:],y[0:i],xval,yval,lambda_)\n",
696 | " return (et_array,ev_array)"
697 | ]
698 | },
699 | {
700 | "cell_type": "code",
701 | "execution_count": 56,
702 | "metadata": {},
703 | "outputs": [
704 | {
705 | "name": "stdout",
706 | "output_type": "stream",
707 | "text": [
708 | "Warning: Desired error not necessarily achieved due to precision loss.\n",
709 | " Current function value: 0.000001\n",
710 | " Iterations: 7\n",
711 | " Function evaluations: 145\n",
712 | " Gradient evaluations: 133\n",
713 | "Warning: Desired error not necessarily achieved due to precision loss.\n",
714 | " Current function value: 0.000001\n",
715 | " Iterations: 11\n",
716 | " Function evaluations: 130\n",
717 | " Gradient evaluations: 118\n",
718 | "Warning: Desired error not necessarily achieved due to precision loss.\n",
719 | " Current function value: 0.821654\n",
720 | " Iterations: 12\n",
721 | " Function evaluations: 133\n",
722 | " Gradient evaluations: 118\n",
723 | "Warning: Desired error not necessarily achieved due to precision loss.\n",
724 | " Current function value: 0.947566\n",
725 | " Iterations: 12\n",
726 | " Function evaluations: 89\n",
727 | " Gradient evaluations: 76\n",
728 | "Warning: Desired error not necessarily achieved due to precision loss.\n",
729 | " Current function value: 5.480855\n",
730 | " Iterations: 10\n",
731 | " Function evaluations: 135\n",
732 | " Gradient evaluations: 123\n",
733 | "Warning: Desired error not necessarily achieved due to precision loss.\n",
734 | " Current function value: 9.721983\n",
735 | " Iterations: 15\n",
736 | " Function evaluations: 130\n",
737 | " Gradient evaluations: 118\n",
738 | "Warning: Desired error not necessarily achieved due to precision loss.\n",
739 | " Current function value: 11.724155\n",
740 | " Iterations: 13\n",
741 | " Function evaluations: 132\n",
742 | " Gradient evaluations: 118\n",
743 | "Warning: Desired error not necessarily achieved due to precision loss.\n",
744 | " Current function value: 12.115397\n",
745 | " Iterations: 8\n",
746 | " Function evaluations: 78\n",
747 | " Gradient evaluations: 67\n",
748 | "Warning: Desired error not necessarily achieved due to precision loss.\n",
749 | " Current function value: 16.957055\n",
750 | " Iterations: 14\n",
751 | " Function evaluations: 109\n",
752 | " Gradient evaluations: 94\n",
753 | "Warning: Desired error not necessarily achieved due to precision loss.\n",
754 | " Current function value: 19.384552\n",
755 | " Iterations: 9\n",
756 | " Function evaluations: 118\n",
757 | " Gradient evaluations: 106\n",
758 | "Warning: Desired error not necessarily achieved due to precision loss.\n",
759 | " Current function value: 22.290813\n",
760 | " Iterations: 22\n",
761 | " Function evaluations: 141\n",
762 | " Gradient evaluations: 128\n"
763 | ]
764 | }
765 | ],
766 | "source": [
767 | "error_train, error_val = learning_curve(x,y,xval,yval,lambda_,)"
768 | ]
769 | },
770 | {
771 | "cell_type": "markdown",
772 | "metadata": {},
773 | "source": [
774 | "## 5.2 Ploting Learning Curve\n",
775 | "You must observe that both the **train error and cross validation error are high** when the number of **training examples** is **increased**. This reflects a **high bias** problem in the model. (the **linear regression** model is too simple and is unable to fit our dataset well so it is **underfitting**)"
776 | ]
777 | },
778 | {
779 | "cell_type": "code",
780 | "execution_count": 74,
781 | "metadata": {},
782 | "outputs": [
783 | {
784 | "data": {
785 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl4VeW1+PHvykAGMockDBECiGFKmBFFGRwQcKJ1VlTqwLV6W3+9lWpbb/W2tWrtvbZWW4tW0RZR6myLVlAD2oKAiIAMgggYDUOAhIQMJDnr98feiSchIzlDcrI+z3Oes+e93kM46+y93722qCrGGGNMc8KCHYAxxpiOz5KFMcaYFlmyMMYY0yJLFsYYY1pkycIYY0yLLFkYY4xpkSULYxohIuEiUioifX25rDGdlSULExLcL+val0dEyr3Gr2nr9lS1RlXjVHWPL5dtKxFJFpEFIrJXRI6IyDYRuaOV6/5VRO71dUyma4oIdgDG+IKqxtUOi8gu4CZVXdbU8iISoarVgYitnR4BwoHBwBEgGxgS1IhMl2RHFqZLEJFfisgLIrJIREqA2SJymoisEpEiESkQkUdEJNJdPkJEVESy3PG/uvPfFJESEVkpIv3buqw7f4aIfCYixSLyexH5l4jMaSL0ccBzqlqkqh5V3aKqL3tta6iILBORQyKyVUQucaffClwB/MQ9unrFt5+o6WosWZiu5FvAc0Ai8AJQDdwO9AAmAtOB/2hm/auB/wZSgD3AL9q6rIikA4uBee5+vwDGN7OdVcD9IjJHRAZ5zxCReGAp8CyQDlwDzBeRbFX9g9vGX7mnyL7VzD6MaZElC9OVfKCqb7i/0MtVdY2qfqiq1aq6E5gPTG5m/RdVda2qVgELgZEnsOwFwHpVfc2d9zBQ2Mx2bsX50v8+sEVEtovINHfeRcBnqvqs24aPgFeBS5v/GIxpO0sWpiv50ntERAaLyD9qLx4DP8f5td+UvV7DZUBcUws2s2xv7zjUqeSZ39RGVLVMVX+pqqOBVOBl4CURSQT6ARPd02hFIlKEc+qpVzNxGXNCLFmYrqRhieU/AZuAk1U1AfgZIH6OoQDIrB0REQH6tGZFVS0G7sdJPFk4SecdVU3yesWp6n/WruLTyE2XZsnCdGXxQDFwVESG0Pz1Cl/5OzBaRC4UkQicayZpTS0sIveIyFgR6SYi0Tinow4B24HXgWEicrWIRLqv8SKS7a6+Dxjg3+aYrsKShenKfghcD5TgHGW84O8dquo+nFNF/wccBAYCHwOVzaz2jLvs18AU4Hz39FQxcB4wG+eIZS/OkUeUu96TwAgROSwiL/q+NaYrEXv4kTHBIyLhOEngUlV9P9jxGNMUO7IwJsBEZLqIJIpIFE732mpgdZDDMqZZfksWIvKUiOwXkU0Npn/PLVnwqYj82mv6j0VkhzvvPH/FZUwHcAawE6fL7HRglqo2dxrKmKDz22koEZkElALPqupwd9pU4Kc451wrRSRdVfeLyFBgEc7NSb2BZcApqlrjl+CMMca0id+OLFR1BU6vDW/fBR6o/RWlqvvd6RcDz6tqpap+Aeyg+btajTHGBFCgCwmeApwpIvcBFcAdqroGp5/5Kq/l8mmi77mIzAXmAkRHR4/p2zd0q0J7PB7CwjrGZaX88nxqtIZ+sf18ts2O1D5/COX2hXLbIPTb99lnnxWqapNdthsT6GQRASQDE3AKpC0WkQE0fiNUo+fHVHU+TlkGsrOzddu2bX4KNfjy8vKYMmVKsMMAYN7b8/j96t/z6U8+JSLMN382Hal9/hDK7QvltkHot09Edrd1nUCnznzgZXWsBjw45RXygZO8lsvE6U5oOojcjFwqayrZfnB7sEMxxgRBoJPFq8BZACJyCtANp0fI68CVIhLllnIehHUl7FByM3IB2LBvQ5AjMcYEgz+7zi4CVgLZIpIvIjcCTwED3O60zwPXu0cZn+KUbd4MvAXcZj2hOpbBPQYTERZhycKYLspv1yxU9aomZs1uYvn7gPv8FY9pn6iIKAb3GMyG/ZYsgqWqqor8/HwqKiqCHQqJiYls2bIl2GH4Tai0Lzo6mszMTCIjI9u9LXusqmm1nPQc/vXlv4IdRpeVn59PfHw8WVlZOMVqg6ekpIT4+PigxuBPodA+VeXgwYPk5+fTv3//lldoQej2DTM+l5uRy57iPRRXFAc7lC6poqKC1NTUoCcK0zmICKmpqT47ErVkYVqt9iL3xv0bgxxJ12WJwrSFL/9eLFmYVrMeUcZ0XZYsTKv1ie9DcnSyJYsuqqioiD/84Q8ntO7MmTMpKirycUQmkCxZmFYTEXIzci1ZdFHNJYuamuZ7ui9ZsoSkpCSfxlNdXd3seFNaitU0zpKFaZOc9Bw27t+IRz3BDsUE2F133cXnn3/OyJEjufvuu8nLy2Pq1KlcffXV5OTkADBr1izGjBnDsGHDmD9/ft26WVlZFBYWsmvXLoYMGcLNN9/MsGHDmDZtGuXl5cft68CBA1xyySWMGzeOcePG8a9/Ob3w7r33XubOncu0adO47rrrWLBgAZdddhkXXngh06ZNQ1WZN28ew4cPJycnhxdecB5+2Fispm2s66xpk9yMXEqPlbK7aDf9k9vfHc+cmP9541M2f33Ep9sc2juBey4c1uT8Bx54gE2bNrF+/XpKSkr46KOPWL16NZs2barrmvnUU0+RkpJCeXk548aN45JLLiE1NbXedrZv386iRYt44oknuPzyy3nppZeYPbv+7Ve33347P/jBDzjjjDPYs2cP5513Xt19Dx999BEffPABMTExLFiwgJUrV7JhwwZSUlJ46aWXWL9+PZ988gmFhYWMGzeOSZMmARwXq2kbSxamTbwvcluyMOPHj6/35fvII4/wyiuvAPDll1+yffv245JF//79GTlyJABjxoxh165dx2132bJlbN68uW78yJEjlJSUAHDRRRcRExNTN+/cc88lJSUFgA8++ICrrrqK8PBwMjIymDx5MmvWrCEhIeG4WE3bWLIwbTIsfRiCsGHfBi4efHGww+mymjsCCKTu3bvXDefl5bFs2TJWrlxJbGwsU6ZMabSPf1RUVN1weHh4o6ehPB4PK1eurJcUGttnw/HmHubWcD3TNnbNwrRJXLc4BqYMtLIfXVB8fHzdr/vGFBcXk5ycTGxsLFu3bmXVqlVNLtuSadOm8eijj9aNr1+/vlXrTZo0iRdeeIGamhoOHDjAihUrGD/enqPmC5YsTJtZj6iuKTU1lYkTJzJ8+HDuvvvu4+ZPnz6d6upqcnNz+e///m8mTJhwwvt65JFHWLt2Lbm5uQwdOpTHH3+8Vet961vfIjc3lxEjRnDWWWfx61//mp49e55wHOYbfnsGdyDYw4+C4968e/n58p9T+pNSYiNjT3g7HbV9vuLr9m3ZsoUhQ4b4bHvtEQq1k5oTSu1r7O9GRD5S1bFt2Y4dWZg2y83IRVE2H9jc8sLGmJBgycK0mZX9MKbrsWRh2mxA8gBiI2MtWRjThViyMG0WJmHkpOdYsjCmC7FkYU5IbY+oztxBwhjTev58BvdTIrLffd52w3l3iIiKSA93XETkERHZISIbRGS0v+IyvpGTnsPB8oMUlBYEOxRjTAD488hiATC94UQROQk4F9jjNXkGMMh9zQX+6Me4jA/UPQhpnz0IyXQsU6ZMYe3atUDTpdHvvfdefvOb3zS7nVdffbVeyZGf/exnLFu2zLfBdiJ+SxaqugI41Mish4EfAd7nLy4GnlXHKiBJRHr5KzbTfjkZTuVOu25hvLW2THigtKc0esNk8fOf/5xzzjnHV6E1q2EZdVXF42ldpWd/lWAP6DULEbkI+EpVP2kwqw/wpdd4vjvNdFApMSlkJmRa2Y8u5tlnnyU3N5fTTz+da6+9FoA5c+bwX//1X0ydOpU777yTQ4cOMWvWLHJzc5kwYQIbNjh/I8uXL2fkyJGMHDmSUaNGUVJSQkFBAZMmTWLkyJEMHz6c999/v97+3nzzTS6//PK68by8PC688EIAvvvd7zJ27FiGDRvGPffc02i8taXRAe677z6ys7M555xz8L6Z94knnmDcuHGMGDGCSy65hLKyMj788ENef/115s2bx8iRI/n888+ZM2cOL774IgDvvPMOo0aNIicnhxtuuIHKysq6/d1zzz2MHj2anJwctm7delxMNTU1zJs3j3HjxpGbm8uf/vSnurZ5l1GvLed+6623Mnr0aL788ksWLVpETk4Ow4cP584776zbZlxcHD/72c849dRTWblyZRv+RVsvYIUERSQW+CkwrbHZjUxr9MqpiMzFOVVFWloaeXl5vgqxwyktLe3Q7esT0Yd/f/7vE46xo7evvXzdvsTExLraTHe+dycbD/j2FGBOWg4PTn2wyflbtmzhF7/4BUuXLiUpKYni4mJKSkqoqqpi8+bNvPLKK4SHh3PHHXcwdOhQ/vKXv7B8+XJmz57Nv/71Lx544AEeeughJkyYQGlpKdXV1Tz99NNMmTKFefPmUVNTQ1lZWb36UxMmTGDu3Lns3buX7t2789e//pWLLrqIkpIS7rrrLlJSUqipqeHCCy9k+vTpDB8+nJqaGo4ePUpJSQmqSmlpKZs3b+a5555jxYoVVFdXc+aZZzJ8+HBKSko499xzufLKKwHn6OGxxx7j5ptvZsaMGUyfPp1Zs2YBUFVVRXl5OQcOHOD666/n9ddfZ9CgQcydO5eHH36Y2267DVUlLi6O5cuX88QTT3D//ffXq3EF8PTTTxMdHc27775LZWUl06ZN4/TTT6esrIzVq1ezatUqsrKy2L17N9u2bePRRx/lwQcfpKCggB/96EesWLGCpKQkZs2axaJFi7jgggs4evQoAwcOrDtN5v0ZVlRU+OTvMJBVZwcC/YFP3IeIZwLrRGQ8zpHESV7LZgJfN7YRVZ0PzAen3IeViwieqdVT+d+V/8vpZ55Ot/BubV6/o7evvfxR7qO2BEW3bt0IDw/32bZrt9lciYsPP/yQyy+/nKysLEpKSujXrx8AkZGRXHXVVXWne1avXs1LL71EfHw8F1xwAd/97nfxeDxMnjyZu+++m2uuuYZvf/vbJCcnc8YZZ3DDDTcQFhbGrFmz6kqXe5sxYwZ5eXlceumlvP322zz88MPEx8ezcOFC5s+fT3V1NQUFBezevZvTTjuN8PBwunfvTnx8PCJCXFwc69at45JLLiEjIwNwHtIUFRVFfHw869at49prr6WoqIjS0lLOO+88wsPDiYyMJCYmpu4zqR3/+uuvGTBgAKNHO/1wbrrpJh577DHuuusuRISrr76a+Ph4Jk6cyJIlS477TFesWMGGDRt44403AKcAY0FBAbGxsYwfP77u4UxxcXH069ePs88+G4B3332XqVOn1pVZv+6661izZk1dSfbZs2c3+jcRHR3NqFGjWvlX0LSAJQtV3Qik146LyC5grKoWisjrwH+KyPPAqUCxqlo3mw4uJyOHKk8V2wq31V3DMIHx2+m/Dfg+VRX3h95xWioTLiLcddddnH/++SxZsoQJEyawbNkyJk2axIoVK/jHP/7Btddey7x587juuuvqrXvFFVfw2GOPkZKSwrhx44iPj+eLL77gN7/5DWvWrCE5OZk5c+Y0Wg69YQyNmTNnDq+++iojRoxgwYIFLf4Kb6m7eG0J9vDw8Eav4agqv//97znvvPPqTc/Lyzvh8uvR0dE+//HQkD+7zi4CVgLZIpIvIjc2s/gSYCewA3gCuNVfcRnfqesRtd96RHUFZ599NosXL+bgwYMAHDrUWP8Vp0z4woULAecLsEePHiQkJPD555+Tk5PDnXfeydixY9m6dSu7d+8mPT2dm2++mRtvvJF169Ydt70pU6awbt06nnjiCa644grAeRhS9+7dSUxMZN++fbz55pvNxj5p0iReeeUVysvLKSkpqftVD84pm169elFVVVUXNzRdkn3w4MHs2rWLHTt2APCXv/yFyZMnN7t/b+eddx5//OMfqaqqAuCzzz7j6NGjLa536qmnsnz5cgoLC6mpqWHRokVt2m97+e3IQlWvamF+ltewArf5KxbjH9mp2USGRbJh3wauzrk62OEYPxs2bBg//elPmTx5MiLCmDFjWLBgwXHL3XvvvXznO98hNzeX2NhYnnnmGQB++9vf8t577xEeHs7QoUOZMWMGzz//PA899BCRkZHExcXx7LPPHre98PBwLrjgAhYsWFC3rREjRjBq1CiGDRvGgAEDmDhxYrOxjx49miuuuIKRI0fSr18/zjzzzLp5v/jFLzj11FPp168fOTk5dQniyiuv5Oabb+aRRx6pu7ANzq/4p59+mssuu4zq6mrGjRvHLbfc0urP8aabbmLXrl2MHj0aVSUtLY1XX321xfV69erF/fffz9SpU1FVZs6cycUXB+4BZFaivAPrDOf0Rz4+kt7xvVlyzZI2r9sZ2tceVqK88wql9lmJctMh2IOQjOkaLFmYdsnNyOWrkq84WHYw2KEYY/zIkoVpF7vIHVid+bSxCTxf/r1YsjDtkpPudJm1GlH+Fx0dzcGDBy1hmFZRVQ4ePEh0dLRPthfIm/JMCOoZ15MesT3sukUAZGZmkp+fz4EDB4IdChUVFT77EuqIQqV90dHRZGZm+mRblixMu4iIc5HbakT5XWRkZN3du8GWl5fnk7uCO6pQb9+JsNNQpt1y03PZtH8TNR7/VLs0xgSfJQvTbrkZuZRVlbHz8M5gh2KM8RNLFqbdantE2XULY0KXJQvTbkPThhImYdZ91pgQZsnCtFtMZAyDUgbZkYUxIcyShfEJK/thTGizZGF8Ijcjl88Pf07psdJgh2KM8QNLFsYnai9yb9q/KciRGGP8wZKF8QnrEWVMaLNkYXyiX2I/4rvFW40oY0KUJQvjEyJCTkaOlf0wJkRZsjA+k5vu9IiyqqjGhB6/JQsReUpE9ovIJq9pD4nIVhHZICKviEiS17wfi8gOEdkmIuf5Ky7jP7kZuRRVFJF/JD/YoRhjfMyfRxYLgOkNpi0FhqtqLvAZ8GMAERkKXAkMc9f5g4iE+zE24wd2kduY0OW3ZKGqK4BDDaa9rarV7ugqoLbQ+sXA86paqapfADuA8f6KzfjH8PThgCULY0JRMJ9ncQPwgjvcByd51Mp3px1HROYCcwHS0tLIy8vzY4jBVVpa2unalxGVwbJNyzit5rQWl+2M7WuLUG5fKLcNQr99JyIoyUJEfgpUAwtrJzWyWKNXSVV1PjAfIDs7W6dMmeKPEDuEvLw8Olv7xheMZ+fhna2KuzO2ry1CuX2h3DYI/fadiID3hhKR64ELgGv0m24z+cBJXotlAl8HOjbTfrkZuWwt3EpldWWwQzHG+FBAk4WITAfuBC5S1TKvWa8DV4pIlIj0BwYBqwMZm/GN3IxcarSGLYVbgh2KMcaH/Nl1dhGwEsgWkXwRuRF4FIgHlorIehF5HEBVPwUWA5uBt4DbVNWe0dkJWY8oY0KT365ZqOpVjUz+czPL3wfc5694TGCcnHIy0RHRliyMCTF2B7fxqYiwCIamDbWn5hkTYixZGJ+zByEZE3osWRify03PZW/pXvYf3R/sUIwxPmLJwvhc7UVuK1duTOiwZGF8znpEGRN6LFkYn0vrnkbPuJ72bAtjQoglC+MXuRm5dhrKmBBiycL4RU56Dp8e+JRqT3XLCxtjOjxLFsYvcjNyqaiuYMehHcEOxRjjA5YsjF/YRW5jQoslC+MXQ3oMIVzCLVkYEyIsWRi/iIqIYnCPwZYsjAkRliyM3+Rm5FqNKGNChCUL4zc56TnsKtpFcUVxsEMxxrSTJQvjN7UXuTft3xTkSIwx7WXJwviN9YgyJnRYsjB+k5mQSVJ0kiULY0KAJQvjNyLiPNvCakQZ0+n58xncT4nIfhHZ5DUtRUSWish29z3ZnS4i8oiI7BCRDSIy2l9xmcDKTXdqRKlqsEMxxrSDP48sFgDTG0y7C3hHVQcB77jjADOAQe5rLvBHP8ZlAignI4eSYyXsLt4d7FCMMe3gt2ShqiuAQw0mXww84w4/A8zymv6sOlYBSSLSy1+xmcCxi9zGhIaIAO8vQ1ULAFS1QETS3el9gC+9lst3pxU03ICIzMU5+iAtLY28vDy/BhxMpaWlnb595TXlALy26jUSChLqzQuF9jUnlNsXym2D0G/fiQh0smiKNDKt0ZPcqjofmA+QnZ2tU6ZM8WNYwZWXl0cotG/gpwMpiS05ri2h0r6mhHL7QrltEPrtOxGB7g21r/b0kvu+352eD5zktVwm8HWAYzN+kpuRa6ehjOnkAp0sXgeud4evB17zmn6d2ytqAlBce7rKdH65GblsP7Sd8qryYIdijDlB/uw6uwhYCWSLSL6I3Ag8AJwrItuBc91xgCXATmAH8ARwq7/iMoGXk56DRz1sPrA52KEYY06Q365ZqOpVTcw6u5FlFbjNX7GY4PLuETWm95ggR2OMORF2B7fxuwHJA4iNjLXrFsZ0YpYsjN+Fh4UzPH24lf0wphOzZGECIjfd6RFlZT+M6ZxaTBYiMlZEfiAiD4nIz0XkchFJCURwJnTkZuRSWFbIvqP7gh2KMeYENJksRGSOiKwDfgzEANtw7os4A1gqIs+ISN/AhGk6u5yMHMDKfhjTWTXXG6o7MFFVG+0cLyIjcQr/7fFHYCa05KR/kyymDZwW5GiMMW3VZLJQ1cfAKSuuqg0LAqKq6/0ZmAktqbGp9InvY0cWxnRSrbnA/aGI/E1EZopIYzWcjGkVK/thTOfVmmRxCk7hvmuBHSLyKxE5xb9hmVCUm5HL5gObqaqpCnYoxpg2ajFZuM+YWOrekX0TTk2n1SKyXERO83uEJmTkZuRS5anis4OfBTsUY0wbtabrbKqI3C4ia4E7gO8BPYAfAs/5OT4TQuxBSMZ0Xq05DbUSSABmqer5qvqyqlar6lrgcf+GZ0JJdmo2kWGRliyM6YRaU0gwW5u47VZVH/RxPCaERYZHMiRtiJX9MKYTau6mvPkiktNYohCR7iJyg4hc49/wTKixHlHGdE7NnYb6A/DfIrLF7Tr7BxF5SkTeB/4NxAMvBiRKEzJy03PJP5LP4fLDft/XvtJ9PLP+Ga548QoyfpPBb/79G7/v05hQ1dxNeeuBy0UkDhgL9ALKgS2qui1A8ZkQU3uRe+P+jT7fdo2nhrVfr2XJ9iUs2bGEtV+vBaBnXE/6JvZl3tJ5xHWL45axt/h838aEuhavWahqKZDn/1BMV+DdI2o4w9u9vUPlh/jnjn+yZMcS3trxFoVlhYRJGBMyJ/DLqb9k5qCZjOg5ghpPDZcsvoRb/3ErCVEJXJ1zdbv3bUxX4rcn5RnTmJ5xPUmNSXWSRXzbk4Wqsn7v+rqjh1X5q/Coh9SYVGYMmsHMk2cybeA0UmNT660XFh7G4ssWM3PhTK575Triu8VzYfaFvmqWMSEvKMlCRH6Ac4OfAhuB7+Cc5noeSAHWAdeq6rFgxGf8R0TqLnJfHd+6X/dHKo+wbOcyJ0FsX0JBaQEAY3uP5e4z72bmoJmM7T2W8LDwZrcTHRHNa1e+xtnPns1lf7uMt2a/xZSsKe1tkjFdQrPJQkTCgQdUdZ6vdigifYDvA0NVtVxEFgNXAjOBh1X1eRF5HLgR+KOv9ms6jtyMXJ5Y9wSegZ5G56sqWwq31CWH9/e8T7WnmsSoRKYNnMbMQTOZfvJ0esb1bPO+46PiefOaN5m8YDIXLrqQd657h/F9xre3ScaEvGaTharWiMgYEZGm7rVox35jRKQKiAUKgLOA2p+azwD3YskiJOVm5FJWVUZBRUHdtKPHjvLervfqEsTu4t2AU9r8h6f9kJmDZnJa5mlEhke2e/+psam8fe3bnPn0mcxYOIPlc5YzPL3910+MCWXSUg4Qkf/FeW7F34CjtdNV9eUT3qnI7cB9OL2r3gZuB1ap6snu/JOAN1X1uP/BIjIXmAuQlpY2ZvHixScaRodXWlpKXFxcsMPwuW0l27hl3S3clHkT0VHRfHjoQ9YXradKq4gOi2ZM8hhOTTmVU1NOJT063W9xFJQX8P3130dRfjfyd/SJ6ePT7Yfqvx+Edtsg9Ns3derUj1R1bFvWaU2yeLqRyaqqN7RlR17bSwZeAq4AinCS0EvAPQ2SxRJVzWluW9nZ2bptW+j24s3Ly2PKlCnBDsPnyqrKiL8/Ho86p6GyU7OZOWgmMwfN5My+ZxIVERWwWDYf2MykpyeREJXA+995nz4JvksYofrvB6HdNgj99olIm5NFa7rOfufEQ2rUOcAXqnoAQEReBk4HkkQkQlWrgUzgax/v13QQsZGx/OmCP7FhywZun3E7A1MGBi2WoWlDeWv2W5z1zFlM++s0ls9ZTo/YHkGLx5iOqjVVZzNF5BUR2S8i+0TkJRHJbMc+9wATRCTWfZjS2cBm4D3gUneZ64HX2rEP08HdNPomvt3n20FNFLXG9h7LG1e9wc7DO5mxcAZHKo8EOyRjOpzWVJ19Gngd6A30Ad5wp50QVf0Qp0zIOpxus2E4D1e6E/gvEdkBpAJ/PtF9GNNWk7Mm8+JlL7J+73ouWnQR5VWNPnremC6rNckiTVWfdsuSV6vqAiCtPTtV1XtUdbCqDlfVa1W1UlV3qup4VT1ZVS9T1cr27MOYtjr/lPP5y7f+wordK7jsb5fZE/2M8dKaZFEoIrNFJNx9zQYO+jswY4LhyuFX8vgFj/OP7f/gulevo8ZTE+yQjOkQWnMH9w3Ao8DDOHdc/9udZkxImjtmLsUVxfxo2Y9I6JbA4xc8jnN5zZiuqzV3cF+iqhcFKB5jOoR5E+dRVFHErz74FYnRiTx4zoOWMEyX1po7uC/GOaowpkv55Vm/pKiiiIf+/RDJ0cn8+MwfBzskY4KmNaeh/iUijwIvUP8O7nV+i8qYDkBE+P3M33Pk2BF+8u5PSIxO5NZxtwY7LGOCojXJ4nT3/ede0xSnlpMxIS1MwnjqoqcoqSzhtiW3kRCVwOzc2cEOy5iAa+maRRjwR1UN3QJMxrQgMjyS5y99nvOfO585r84hISqBi7LtMp7pWprtOquqHuA/AxSLMR1WdEQ0r17xKmN7j+Xyv13OOzvfCXZIxgQ8pgWxAAAbqUlEQVRUa+6zWCoid4jISSKSUvvye2TGdDDxUfEsuWYJg1IHcfHzF/Nh/ofBDsmYgGlNsrgBuA1YAXzkvtb6MyhjOqqUmBTenv02PeN6MmPhDDbu2xjskIwJiBaThar2b+Q1IBDBGdMR9YrvxbLrlhEbGcu0v05jx6EdwQ7JGL9rMlmIyI+8hi9rMO9X/gzKmI4uKymLpdcupaqminOePYf8I/nBDskYv2ruyOJKr+GGdyNN90MsxnQqQ9KG8M/Z/+RQ+SHO/cu5HDh6INghGeM3zSULaWK4sXFjuqQxvcfw96v/zq6iXUxfOJ3iiuJgh2SMXzSXLLSJ4cbGjemyJvWbxMuXv8yGfRu4cNGFlFWVBTskY3yuuWQxQkSOiEgJkOsO1443+2xsY7qaGYNmsPDbC/lgzwdcuvhSjtUcC3ZIxvhUk3dwq2p4IAMxprO7fNjlHKk8ws1v3My1r1zLVQlXBTskY3ymNbWhjDGtdNPomyiuKOaOpXewmMWkb0hnePpwctJz6t6Hpg0lPio+2KEa0yZBSRYikgQ8CQzHuf5xA7ANp7JtFrALuFxVDwcjPmPa44en/5DTTjqN55Y/R3l8OZsObOKJdU/Uu5aRlZRVl0Bqk0h2j2y6hXcLYuTGNC1YRxa/A95S1UtFpBsQC/wEeEdVHxCRu4C7gDuDFJ8x7XL6SadzLPMYU6ZMAcCjHnYV7WLT/k1s3LeRTQec9zd3vEm1pxqAiLAIslOz6yWQ4enD6Z/cnzBpTbEFY/wn4MlCRBKAScAcAFU9BhxzH7I0xV3sGSAPSxYmRIRJGAOSBzAgeUC9irXHao7x2cHPnASyfxMb929k9VereeHTF+qWiY2MZVjasPqnszJyyOieYU/vMwEjqoHtBSsiI4H5wGZgBE6tqduBr1Q1yWu5w6qa3Mj6c4G5AGlpaWMWLw7d6umlpaXExcUFOwy/sfY1rbymnC+OfnHc63DVN2dmEyIS6N+9PwO6DyCrexb9YvvRN7YvSZFJfk8i9m/XuU2dOvUjVR3blnWCkSzGAquAiar6oYj8DjgCfK81ycJbdna2btu2zb8BB1FeXl7daYxQZO1ruwNHD7Bp/6a6o5Da4ZJjJXXLpMSkMLjHYIb0GFLvPSspi/Aw33RytH+7zk1E2pwsgnHNIh/IV9Xa+s4v4lyf2CcivVS1QER6AfuDEJsxHVpa9zSm9p/K1P5T66apKnuK97C1cCtbCrfUvb/x2Rv8+eM/1y0XFR7FKamnMCRtCINTBzvvPQZzSuopxEbGBqM5phMJeLJQ1b0i8qWIZKvqNuBsnFNSm4HrgQfc99cCHZsxnZGI0C+pH/2S+nHeyefVm3eo/BBbC7c6CeTAFrYe3MpHX3/Ei5tfxKMeZ32c9Rs7GknrnhaMJpkOKFi9ob4HLHR7Qu0EvoNzN/liEbkR2ANc1sz6xphWSIlJ4fSTTuf0k06vN72iuoLtB7cfdzSyfNdyyqvL65ZLjUmtn0Tco5EarQl0U0yQBSVZqOp6oLHzZWcHOhZjuqLoiGhyMnLIyahfucejnrpTWt5HI69te40nP36ybrlIiaTXJ73oGdeTjO4Z9Izr2eTLTnGFBruD2xhTJ0zCyErKIispi+kn138SwcGyg3VJ5J3179AtpRt7S/eyp3gPq79azf6j+9FGaozGd4s/LoE0lmDSu6cTGR4ZqKaaNrJkYYxpldTYVCb2ncjEvhMZeGTgcb2Fqj3VFJYVsrd0b5OvDfs28Pbnb1Nc2Xgp9x6xPeonke7Oe3JMMtWeao7VHONYzTGqaqrqho/VHKPKU9XocMNl6y3XzDa6STdG7RzFyJ4j617D04cTHREdgE+6Y7JkYYzxiYiwiLov+ZaUV5Wz7+g+9pXuOz6pHHXeP9jzAQUlBVTWVDa5HUHoFt6t3isyPPKb4bDIevNiImOanOc9vnnnZgqlkGc/eZbH1jwGQLiEM7jH4HoJZGTPkfSI7eGzz7Ajs2RhjAm4mMiYutNdzVFVjlQeoaiiqNEk4Kv7RhrKC3fus/Cohy8Of8H6veud1771LN+9nIUbF9Yt2ye+z3EJZEDygJAr0WLJwhjTYYkIidGJJEYnBmX/YRLGwJSBDEwZyCVDL6mbXlhWyCd7P6lLIOv3ruetHW/V9RKL6xbHiIwR9RLIsLRhxETGBLwNHvVw9NhRSo6VcKTyCCWVJS2v1AhLFsYY00Y9Yntw9oCzOXvANx04K6or+HT/p/WOQpo7jVWbTBq7l8WjHkqPlVJS6X7BHyupN1z7pV83r5FpteOlx0ob7XjQVpYsjDHGB6IjohnTewxjeo+pm9aa01i943vTO743JZXffOmXHitt1T4jwiKI7xZPQlQC8VHOe0pMCllJWcR3i6+b5r1MfLd4Lrr3opY33nBfbV7DGGNMq7T2NFZhWSH9k/of98VeO97Yl35CVAJR4VEBqzxsycIYYwKssdNYHV1oXa43xhjjF5YsjDHGtMiShTHGmBZZsjDGGNMiSxbGGGNaZMnCGGNMiyxZGGOMaZElC2OMMS2yZGGMMaZFQUsWIhIuIh+LyN/d8f4i8qGIbBeRF9zncxtjjOkAgnlkcTuwxWv8QeBhVR0EHAZuDEpUxhhjjhOUZCEimcD5wJPuuABnAS+6izwDzApGbMYYY44nqu2vc97mnYq8CNwPxAN3AHOAVap6sjv/JOBNVR3eyLpzgbkAaWlpYxYvXhyosAOutLSUuLi4YIfhN9a+ziuU2wah376pU6d+pKpj27JOwKvOisgFwH5V/UhEptRObmTRRrOYqs4H5gNkZ2drw4fGh5K8POfRjqHK2td5hXLbIPTbdyKCUaJ8InCRiMwEooEE4LdAkohEqGo1kAl8HYTYjDHGNCLg1yxU9ceqmqmqWcCVwLuqeg3wHnCpu9j1wGuBjs0YY0zjOtJ9FncC/yUiO4BU4M9BjscYY4wrqE/KU9U8IM8d3gmMD2Y8xhhjGteRjiyMMcZ0UJYsjDHGtMiShTHGmBZZsjDGGNMiSxbGGGNaZMnCGGNMiyxZGGOMaZElC2OMMS2yZGGMMaZFliyMMca0yJKFMcaYFlmyMMYY0yJLFsYYY1pkycIYY0yLLFkYY4xpkSULY4wxLbJkYYwxpkVBfVKeMcaYwCk7Vs3ug2UntG7Ak4WInAQ8C/QEPMB8Vf2diKQALwBZwC7gclU9HOj4jDGmMyutrGZX4VF2Hyxj18Gj9Yb3l1Se8HaDcWRRDfxQVdeJSDzwkYgsBeYA76jqAyJyF3AXcGcQ4jPGmA7tSEUVuwqPsutgGbtr3w8eZdfBoxSWHqu3bFp8FP1TuzP5lDSyenSnX2osFz7Y9n0GPFmoagFQ4A6XiMgWoA9wMTDFXewZIA9LFsaYLqqo7Ng3SaDQPUo46BwlHDpaPyH0TIimX2osZw/OIKtHd7JSY+mX6iSG7lG++ZoP6jULEckCRgEfAhluIkFVC0QkPYihGWOMX1XXeNhfUsnXReXsOVTmdXRQxq7CoxSXV9VbvndiNFk9unPesJ51yaB/j+70TYklplu43+MVVfX7ThrdsUgcsBy4T1VfFpEiVU3ymn9YVZMbWW8uMBcgLS1tzOLFiwMWc6CVlpYSFxcX7DD8xtrXeYVy28A37SuvVg6VKwcrPBwsVw5WKAfLPe67crhS8Xh9/QqQGiNkxArpsWFkxIaR0d0ZTosRuoVL+xrlZerUqR+p6ti2rBOUZCEikcDfgX+q6v+507YBU9yjil5AnqpmN7ed7Oxs3bZtm/8DDpK8vDymTJkS7DD8xtrXeYVy26Dl9nk8yoHSSr4qKuerw+V8XeS8vioq56uiCr4uKj/uyCAiTOiZGE3vpBj6uK/eSTH0ToomMzmWk1JiiIrw/xECgIi0OVkEozeUAH8GttQmCtfrwPXAA+77a4GOzRhjACprlB37SxskgdqkUEFBcTlVNfV/aMdHR9QlgbH9kp2kkBxDnyQnQaTHRxMe5rujg0ALxjWLicC1wEYRWe9O+wlOklgsIjcCe4DLghCbMaaLUVW+KDzK2l2HWbPrEGt2HWLXwTJYurxumTBxLiL3TophVN8kzk/q5R4hRNMnKZZeSdEkREcGsRX+F4zeUB/gnJ5rzNmBjMUY0/VU13jYXHCE1V8cYu2uw6zdfaiuu2lSbCRj+6UwOqWKSaOH1Z0m6pkQTUR41y54YXdwG2NCWtmxaj7eU1R31PDxniLKjtUAkJkcw6RBaYzNSmFcVjID0+IICxPnmsWoPkGOvGOxZGGMCSmFpZV1p5TW7jrEpq+PUONRRGBwzwQuHZPJuKwUxmYl0ysxJtjhdhqWLIwxnZaqsudQWd0ppTW7D7HzwFEAukWEMTIziVsmD2BsVgqj+yaTGBPa1xX8yZKFMabTqPEoWwqOuEcNztFDbb2jhOgIxmWlcNmYkxjfP5nhfRID1hW1K7BkYYzxGVWlqkap9nioqlFqPEp1jYeq2nd3XnWNUlXjodqjVDcyrarG8810j3Kw9Fjd9YbSymoA+iTFcNrAVMZlpTAuK4VB6c71BuMfliyMMfVUVtdwoKSS/SWV7D9SyYGSirrh/e5wwaEywj5YWvdlX/slX+Px302+2RnxzBrV273ekEKfJLveEEiWLIzpIo5WVrtf+u6Xf4nz5X/gyDfD+0sqKSqrOm5dEUjtHkV6fBTpCVEkUUbfzJ5EhgsRYWHOuzscESZEhLvT6g2HEREuRIaHER4mx02LCHPfa9fzmtc9KoI4HxXEMyfGPn1jOrnqGg9fFB5l75EK99f/N1/8B7yOBmq7i3qLDBfS46NJi48iK7U74/unkB4fXZcUaodTunerd5+BUw4jJ5DNNEFmycKYTsTjUb44eJSN+cV8kl/EhvxiPv26mIoqT73lYruFO1/48dEM65PI1PioRpNAUmwkTgUeY5pnycKYDkpV+bq4gg1fFvFJfjEb8ovY+FUxJRXOBd7oyDCG9U7kqvF9yemT6NYfiiI9IdpO2Rifs78oYzqIwtJKNuQX8cmXxWz8ykkOtWUoIsKEwb3iuXBEb0ZkJpKbmcSg9LguX4LCBI4lC2OC4EhFFZvyi+uOGDbkF/NVUTngXEw+OS2Oyaekk5uZSG5mIkN6JRAdafcMmOCxZGGMn1VU1fDp18V88qWbGL4qrrvLGOCklBhG9k3i+tP7kZuZxPA+iXYayXQ49hdpjA/VeJTP9pXw8Z4i3tpUyYOfvM9n+0rq7j9Ij48iNzOJb43sQ457Oimle7cgR21MyyxZGNMOh44e4+M9h/l4TxHr9hzmky+LOOp2Ue0eCaOzunHW4AHkZiYxIjOJnonRQY7YmBNjycKYVqqu8bB1bwkff1nEx7sPs27PYechOUB4mDC4ZzzfHp3JqL5JjOqbzK6Nq5k69dQgR22Mb1iyMKYJhaWVrNt9mI+/LGLd7sNsyC+mvMo5augR141RfZO5YlxfRvVNIjczkdhu9f877bb7F0wIsWRhDFBV42FLwZFvksOew3x5yOmdFBEmDO2dwBXjTmJU3yRG900mMznGbmYzXYolC9Ml7T9SwTqvaw0b8ouprHbugk6Pj2J032SundCPUX2TyemTaN1WTZfX4ZKFiEwHfgeEA0+q6gNBDsn4WHWNh/KqGkqPKYeOHsOjikcVFDwKijrvqqiCx/sdZ7ozv/48wN3WN8vgbquq2sOWvSV1F6Nr72noFh7GsD4JXHNqP0b3c6419E6MtqMGYxroUMlCRMKBx4BzgXxgjYi8rqqbgxtZ11H7RV5+rMZ5rx1uON7MMhVVNZR5LeM9XlFVQ1WNVxnrd5cGtH29E6MZ1TeZ70zMYlTfZIb1tpvdjGmNDpUsgPHADlXdCSAizwMXA10uWVz/1Go+/Pwo4e++FZD9Kc55+3pf5K0UFRFGTLdwYiLDv3mPDCcuKoK0uKi6adGR4cR6De/a+TmnnDKIMAFECBMQ3HcBESFMBAHCwpx530ynbp6IMz1M6q8ruMsIhIswIC3Ouq4ac4I6WrLoA3zpNZ4P1Ot7KCJzgbnuaKWIbApQbMHQAygMdhB+ZO3rvEK5bRD67ctu6wodLVk0dqK43k9dVZ0PzAcQkbWqOjYQgQWDta9zC+X2hXLboGu0r63rdLSSlfnASV7jmcDXQYrFGGOMq6MlizXAIBHpLyLdgCuB14MckzHGdHkd6jSUqlaLyH8C/8TpOvuUqn7azCrzAxNZ0Fj7OrdQbl8otw2sfccR1bb3fjHGGNO1dLTTUMYYYzogSxbGGGNa1GmThYhMF5FtIrJDRO4Kdjy+JCInich7IrJFRD4VkduDHZOviUi4iHwsIn8Pdiy+JiJJIvKiiGx1/w1PC3ZMviQiP3D/LjeJyCIR6dR3OorIUyKy3/ueLRFJEZGlIrLdfU8OZozt0UT7HnL/PjeIyCsiktTSdjplsvAqCzIDGApcJSJDgxuVT1UDP1TVIcAE4LYQax/A7cCWYAfhJ78D3lLVwcAIQqidItIH+D4wVlWH43REuTK4UbXbAmB6g2l3Ae+o6iDgHXe8s1rA8e1bCgxX1VzgM+DHLW2kUyYLvMqCqOoxoLYsSEhQ1QJVXecOl+B82fQJblS+IyKZwPnAk8GOxddEJAGYBPwZQFWPqWpRcKPyuQggRkQigFg6+b1QqroCONRg8sXAM+7wM8CsgAblQ421T1XfVtVqd3QVzj1tzeqsyaKxsiAh82XqTUSygFHAh8GNxKd+C/wI8AQ7ED8YABwAnnZPsz0pIt2DHZSvqOpXwG+APUABUKyqbwc3Kr/IUNUCcH68AelBjsefbgDebGmhzposWiwLEgpEJA54Cfh/qnok2PH4gohcAOxX1Y+CHYufRACjgT+q6ijgKJ37FEY97rn7i4H+QG+gu4jMDm5U5kSJyE9xTnsvbGnZzposQr4siIhE4iSKhar6crDj8aGJwEUisgvn9OFZIvLX4IbkU/lAvqrWHgm+iJM8QsU5wBeqekBVq4CXgdODHJM/7BORXgDu+/4gx+NzInI9cAFwjbbihrvOmixCuiyIOE/e+TOwRVX/L9jx+JKq/lhVM1U1C+ff7V1VDZlfpqq6F/hSRGqrep5NaJXY3wNMEJFY9+/0bELoAr6X14Hr3eHrgdeCGIvPuQ+ZuxO4SFXLWrNOp0wW7oWZ2rIgW4DFLZQF6WwmAtfi/Ope775mBjso02rfAxaKyAZgJPCrIMfjM+4R04vAOmAjzndIpy6NISKLgJVAtojki8iNwAPAuSKyHedhbJ32iZ1NtO9RIB5Y6n6/PN7idqzchzHGmJZ0yiMLY4wxgWXJwhhjTIssWRhjjGmRJQtjjDEtsmRhjDGmRZYsTKNEREXkf73G7xCRe3207QUicqkvttXCfi5zq76+12B6lohcfYLb/Hcrlnky1Ao/ikipj7bzWxGZ1MIyyzpzlddQZcnCNKUS+LaI9Ah2IN7cisOtdSNwq6pObTA9C2g0WbjF8Zqkqi3erayqN6lqKN2I5xMikgJMcAvbNecvwK0BCMm0gSUL05RqnJutftBwRsMjg9pfnSIyRUSWi8hiEflMRB4QkWtEZLWIbBSRgV6bOUdE3neXu8BdP9yts7/GrbP/H17bfU9EnsO5EaxhPFe5298kIg+6034GnAE8LiIPNVjlAeBM92akH4jIHBH5m4i8AbwtInEi8o6IrHO3e7HXvrzbmiffPLdioXtHM+70sbXLi8h9IvKJiKwSkQx3+kB3fI2I/LypX+4iMtv9/NaLyJ/cz6ifOM9Z6CEiYe7nOM1d/lUR+Uic503M9Y5bRB505y0TkfFunDtF5CJ3mTki8pqIvCXOs2LuaSKmeV7/Rv/jTusuIv9w27lJRK5oZNVLgbfc5RPdfWS744tE5GZ3udeBqxrbtwkiVbWXvY57AaVAArALSATuAO515y0ALvVe1n2fAhQBvYAo4Cvgf9x5twO/9Vr/LZwfK4Nw6ilFA3OBu91looC1OAXrpuAU5OvfSJy9cUpQpOEU8XsXmOXOy8N57kLDdaYAf/can+PGkOKORwAJ7nAPYAff3MDq3dZinLpkYTh3yJ7RcL84BS4vdId/7dW+vwNXucO31G63QZxDgDeASHf8D8B17vBNOHdSzwP+5LVObRtigE1AqlccM9zhV4C3gUic522s9/ocCoBUr/XHNmj3NJwfEeK2++84JdkvAZ7wiiOxkfY8U/tZuOPnup/blTjP//Bedntt7PbqGC87sjBNUqfS7bM4D7tprTXqPI+jEvgc50sJnCOCLK/lFquqR1W3AzuBwThfRNeJyHqckuypOMkEYLWqftHI/sYBeeoUtqutntnsOfEmLFXV2pr/AvxKnHIdy3DK32c0ss5qVc1XVQ+wvkH7ah3D+UIF+MhrmdOAv7nDzzUR09nAGGCN+5mcjVMCHVV9Eqdcwy04ibzW90XkE5xnFJzEN5/fMdxf9Tj/FsvVKQTY8N9lqaoeVNVynCKBZzSIaZr7+hin5Mdgdx8bcY4WHxSRM1W1uJH29MIp347bhqXueo/hJD9v+3F+CJgOotnzs8bgPHtiHfC017Rq3FOY7qmXbl7zKr2GPV7jHur/vTWsM6M4X9LfU9V/es8QkSk4RxaNaaxc/Ynw3v41OEcqY1S1SpwKuY09OtS7rTU0/v+pSt2fys0s0xQBnlHV455iJiKxfPPAmjigxP2czgFOU9UyEcnzits7jrp/F1X1NLhO09i/S8OY7lfVPzUS0xhgJnC/iLytqj9vsEi5VzyISBjO0VM5kIJzdFcr2p1uOgg7sjDNcn9tL8a5WFxrF84vXnCebRB5Apu+zD3fPhDn1/I2nMKQ3xWnPDsicoq0/OCgD4HJ7vn7cJxz3ctbWKcE51d5UxJxnrlRJSJTgX6taE9brcI5dQNNP5b0HeBSEUmHuudC18byIM5R1M+AJ7ziPuwmisE4j+Rtq3Pd/cTgPB3uXw3m/xO4QZxnrSAifUQkXUR6A2Wq+lechyM1VpZ9C3Cy1/gP3GlXAU95/bsL0BPn78x0EHZkYVrjf3Gq/NZ6AnhNRFbjfKE19au/OdtwvtQzgFtUtUJEnsQ5JbLO/cI4QAuPs1TVAhH5MfAezq/eJaraUjnpDUC1e7pmAXC4wfyFwBsishbn9NLWtjSslf4f8FcR+SHwD5zrH/Wo6mYRuRvnonsYUIXzPPYsnNNvE1W1RkQuEZHv4JzOusU9fbYNJyG11Qc4vZFOBp5T1bUNYnpbRIYAK93r+aXAbHf5h0TE48b53Ua2/Q/gP4AnReQUnFNP41W1RERWAHcD9+D8EFml3zz203QAVnXWmCBwTyOVq6qKyJU4F7uD+hx5EZmDc0H7P1tath37+AC4QJt5LrmI/A54XVXf8Vccpu3syMKY4BgDPOoeQRXhPAe5K/gh0BenzU3ZZImi47EjC2OMMS2yC9zGGGNaZMnCGGNMiyxZGGOMaZElC2OMMS2yZGGMMaZF/x+crORhlx2o4wAAAABJRU5ErkJggg==\n",
786 | "text/plain": [
787 | ""
788 | ]
789 | },
790 | "metadata": {},
791 | "output_type": "display_data"
792 | }
793 | ],
794 | "source": [
795 | "plt.plot(np.arange(2,m+1),error_train[1:], label = 'train error')\n",
796 | "plt.plot(np.arange(2,m+1),error_val[1:], label = 'cross validation error', color= 'green')\n",
797 | "plt.axis([0, 12, 0, 160])\n",
798 | "plt.title('Training Set')\n",
799 | "plt.xlabel('Number of training examples (x)')\n",
800 | "plt.ylabel('Error (y)')\n",
801 | "plt.grid()\n",
802 | "plt.legend()\n",
803 | "plt.show()"
804 | ]
805 | },
806 | {
807 | "cell_type": "markdown",
808 | "metadata": {},
809 | "source": [
810 | "In the next section, you will implement **polynomial regression** to fit a better model for this dataset."
811 | ]
812 | }
813 | ],
814 | "metadata": {
815 | "kernelspec": {
816 | "display_name": "Python 3",
817 | "language": "python",
818 | "name": "python3"
819 | },
820 | "language_info": {
821 | "codemirror_mode": {
822 | "name": "ipython",
823 | "version": 3
824 | },
825 | "file_extension": ".py",
826 | "mimetype": "text/x-python",
827 | "name": "python",
828 | "nbconvert_exporter": "python",
829 | "pygments_lexer": "ipython3",
830 | "version": "3.6.6"
831 | }
832 | },
833 | "nbformat": 4,
834 | "nbformat_minor": 2
835 | }
836 |
--------------------------------------------------------------------------------
/Week 6 - Regularized Linear Regression and Bias v.s. Variance/Xtest.csv:
--------------------------------------------------------------------------------
1 | -3.331800399061839357e+01
2 | -3.791216402831621224e+01
3 | -5.120693794827316481e+01
4 | -6.132595848228624469e+00
5 | 2.126118327389842833e+01
6 | -4.031952949030852551e+01
7 | -1.454153167295283566e+01
8 | 3.255976024170858807e+01
9 | 1.339343254732885846e+01
10 | 4.420988594678166805e+01
11 | -1.142677675487955291e+00
12 | -1.276686065227885081e+01
13 | 3.405450538959663476e+01
14 | 3.922350027740379375e+01
15 | 1.974496739238940535e+00
16 | 2.962175509870446888e+01
17 | -2.366962970665967703e+01
18 | -9.011801392661441312e+00
19 | -5.594057090677502231e+01
20 | -3.570859751906493074e+01
21 | 9.510205326576553375e+00
22 |
--------------------------------------------------------------------------------
/Week 6 - Regularized Linear Regression and Bias v.s. Variance/Xval.csv:
--------------------------------------------------------------------------------
1 | -1.674653577802068938e+01
2 | -1.457747074924186848e+01
3 | 3.451575865729932246e+01
4 | -4.701007574320075832e+01
5 | 3.697511904636278501e+01
6 | -4.068611001536747551e+01
7 | -4.472010975766456120e+00
8 | 2.653363489478886095e+01
9 | -4.279768310017955457e+01
10 | 2.537409938352766403e+01
11 | -3.110955397730775118e+01
12 | 2.731176863521341147e+01
13 | -3.263862013656720684e+00
14 | -1.818276487115874751e+00
15 | -4.071966240251615687e+01
16 | -5.001324364549542878e+01
17 | -1.741177154801637883e+01
18 | 3.588193696644128572e+00
19 | 7.085480261970673155e+00
20 | 4.628236901853892959e+01
21 | 1.461228909165653889e+01
22 |
--------------------------------------------------------------------------------
/Week 6 - Regularized Linear Regression and Bias v.s. Variance/ex5data1.mat:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Nikronic/Coursera-Machine-Learning/4bc5de16446b664f9995eba95762b730547b1fce/Week 6 - Regularized Linear Regression and Bias v.s. Variance/ex5data1.mat
--------------------------------------------------------------------------------
/Week 6 - Regularized Linear Regression and Bias v.s. Variance/img/jtrain.JPG:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Nikronic/Coursera-Machine-Learning/4bc5de16446b664f9995eba95762b730547b1fce/Week 6 - Regularized Linear Regression and Bias v.s. Variance/img/jtrain.JPG
--------------------------------------------------------------------------------
/Week 6 - Regularized Linear Regression and Bias v.s. Variance/img/poly.JPG:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Nikronic/Coursera-Machine-Learning/4bc5de16446b664f9995eba95762b730547b1fce/Week 6 - Regularized Linear Regression and Bias v.s. Variance/img/poly.JPG
--------------------------------------------------------------------------------
/Week 6 - Regularized Linear Regression and Bias v.s. Variance/img/rlrc.JPG:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Nikronic/Coursera-Machine-Learning/4bc5de16446b664f9995eba95762b730547b1fce/Week 6 - Regularized Linear Regression and Bias v.s. Variance/img/rlrc.JPG
--------------------------------------------------------------------------------
/Week 6 - Regularized Linear Regression and Bias v.s. Variance/img/rlrg.JPG:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Nikronic/Coursera-Machine-Learning/4bc5de16446b664f9995eba95762b730547b1fce/Week 6 - Regularized Linear Regression and Bias v.s. Variance/img/rlrg.JPG
--------------------------------------------------------------------------------
/Week 6 - Regularized Linear Regression and Bias v.s. Variance/ytest.csv:
--------------------------------------------------------------------------------
1 | 3.316889531756360476e+00
2 | 5.397689520216154335e+00
3 | 1.304298374520143078e-01
4 | 6.192598202383872596e+00
5 | 1.708848711547617327e+01
6 | 7.995080467101720600e-01
7 | 2.824791834378351130e+00
8 | 2.862123333554546889e+01
9 | 1.704639080632776782e+01
10 | 5.538437334220785857e+01
11 | 4.079367333128050888e+00
12 | 8.270397934583032296e+00
13 | 3.132355101950563281e+01
14 | 3.915906103292760321e+01
15 | 8.087279893552164367e+00
16 | 2.411134389353750151e+01
17 | 2.477354802758796826e+00
18 | 6.566064719303037478e+00
19 | 6.038088801572035536e+00
20 | 4.692739558592498916e+00
21 | 1.083004606320572805e+01
22 |
--------------------------------------------------------------------------------
/Week 6 - Regularized Linear Regression and Bias v.s. Variance/yval.csv:
--------------------------------------------------------------------------------
1 | 4.170202008850626640e+00
2 | 4.067262803838972651e+00
3 | 3.187306757578932803e+01
4 | 1.062365618969514536e+01
5 | 3.183602128133837184e+01
6 | 4.959369720997110598e+00
7 | 4.451598803470352195e+00
8 | 2.227631845748983963e+01
9 | -4.387382739157175138e-05
10 | 2.050380157930803549e+01
11 | 3.858344762697331731e+00
12 | 1.936505291560758124e+01
13 | 4.883762805453054412e+00
14 | 1.109715884838211863e+01
15 | 7.461708266154005287e+00
16 | 1.476934642298386446e+00
17 | 2.719163877647038152e+00
18 | 1.092690065536872979e+01
19 | 8.348712346301107701e+00
20 | 5.278192797904378608e+01
21 | 1.335733960607468696e+01
22 |
--------------------------------------------------------------------------------
"
168 | ]
169 | },
170 | {
171 | "cell_type": "code",
172 | "execution_count": 8,
173 | "metadata": {},
174 | "outputs": [
175 | {
176 | "data": {
177 | "text/plain": [
178 | "
\n",
363 | "Notice that by defining `x1 = (waterLevel)`, `x2 = (waterLevel)`2, ... , `xp = (waterLevel)`p, we obtain a linear regression model where the features are the various powers of the original value (waterLevel).
"
828 | ]
829 | },
830 | {
831 | "cell_type": "markdown",
832 | "metadata": {},
833 | "source": [
834 | "### Implementation\n",
835 | "`linear_reg_grad(x, y, theta, lambda_)` computes the gradient of cost of using `theta` as the parameter for linear regression to fit the data points in `x` and `y`. Returns the gradient in `grad`."
836 | ]
837 | },
838 | {
839 | "cell_type": "code",
840 | "execution_count": 24,
841 | "metadata": {},
842 | "outputs": [],
843 | "source": [
844 | "def linear_reg_grad(theta_flatten, x_flatten, y, lambda_, num_of_samples, num_of_features):\n",
845 | " x = x_flatten.reshape(num_of_samples, num_of_features)\n",
846 | " m,n = x.shape\n",
847 | " theta = theta_flatten.reshape(n,1)\n",
848 | " new_theta = np.zeros(shape=(theta.shape))\n",
849 | " loss = hypothesis(x,theta)-y\n",
850 | " gradient = np.dot(x.T,loss)\n",
851 | " new_theta[0:,:] = gradient/m\n",
852 | " new_theta[1:,:] = gradient[1:,:]/m + lambda_*(theta[1:,]/m)\n",
853 | " return new_theta.flatten()"
854 | ]
855 | },
856 | {
857 | "cell_type": "markdown",
858 | "metadata": {},
859 | "source": [
860 | "## 3.C Fitting Linear Regression\n",
861 | "Once your cost function and gradient are working correctly, the next part is to **compute the optimal values** of **θ**.