├── 1.Linear-Regression&Gradient-Descent
├── Gradient_Descent_Algorithm.ipynb
├── Images
│ ├── Cost_GradientDescent.jpg
│ ├── Linear_Regression.jpg
│ └── Math_Linear Regression.jpg
├── LinearRegression.ipynb
└── README.md
├── 2.Multiple-Linear-Regression
├── California_Housing_dataset_scikit-learn.ipynb
├── MultipleRegression_from_scratch.ipynb
├── Multiple_Regression_Normal_Equation.ipynb
├── Multiple_Regression_scikit-learn.ipynb
└── README.md
├── 3.Logistic-Regression
├── LogisticRegression_scikit-learn.ipynb
└── README.md
├── Machine-Learning-by-Stanford
├── README.md
├── Week 1 - Introduction and Linear Regression
│ ├── README.md
│ ├── Week 1 - Introduction and Linear Regression.ipynb
│ └── Week1_Images
│ │ ├── gradient_descent1.png
│ │ ├── gradient_descent2.png
│ │ └── image
├── Week 2 - Linear Regression with Multiple Variables
│ ├── README.md
│ └── Week 2 - Linear Regression with Multiple Variables.ipynb
├── Week 3 - Logistic Regression
│ ├── README.md
│ ├── Week 3 - Logistic Regression.ipynb
│ └── Week3_Images
│ │ ├── SigmoidFunction.png
│ │ ├── cost1.png
│ │ ├── cost2.png
│ │ ├── multiclass1.png
│ │ ├── overfit1.png
│ │ ├── overfit2.png
│ │ ├── overfit3.png
│ │ ├── reg_logistic_gradient_descent.png
│ │ ├── reg_logistic_regression.png
│ │ └── regularization.png
├── Week 4 - Neural Networks
│ ├── README.md
│ ├── Week 4 - Neural Networks.ipynb
│ └── Week4_Images
│ │ ├── XNOR.png
│ │ ├── multiclass_classification.png
│ │ ├── neural_net.png
│ │ └── neural_net_intuition.png
└── Week 5 - Neural Networks: Learning
│ ├── README.md
│ ├── Week 5 - Neural Networks - Learning.ipynb
│ └── Week5_Images
│ ├── backpropagation_algo.png
│ ├── forward_propagation.png
│ └── gradient_computation.png
└── README.md
/1.Linear-Regression&Gradient-Descent/Images/Cost_GradientDescent.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/1.Linear-Regression&Gradient-Descent/Images/Cost_GradientDescent.jpg
--------------------------------------------------------------------------------
/1.Linear-Regression&Gradient-Descent/Images/Linear_Regression.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/1.Linear-Regression&Gradient-Descent/Images/Linear_Regression.jpg
--------------------------------------------------------------------------------
/1.Linear-Regression&Gradient-Descent/Images/Math_Linear Regression.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/1.Linear-Regression&Gradient-Descent/Images/Math_Linear Regression.jpg
--------------------------------------------------------------------------------
/1.Linear-Regression&Gradient-Descent/README.md:
--------------------------------------------------------------------------------
1 | # Linear Regression and Gradient Descent
2 | _**Linear Regression**_ is one of the simplest machine learning algorithms you may encounter. It can be described as:
3 | - Supervised learning algorithm which goal is to to find a line that minimizes the prediction error of all data points.
4 | - Linear approach to modeling the relationship between a dependent variable (Y) and one or more independent variables (X).
5 |
6 | In statistics, machine learning and other data science fields, we optimize a lot of stuff. When we fit a line to data using linear regression, we optimize the _**slope**_ and _**intercept**_. Linear Regression predicts continous numerical values based on the given data points.
7 |
8 |
9 | ## Main ideas
10 | - _**Linear Regression fits a line to data**_
11 | - It is a simple algorithm initially developed in the field of statistics and was studied as a model for understanding the relationship between input and output variables
12 | - Linear Regression tries to find parameters of the line function, so the distance between all points and the line is as small as possible.
13 | - In Linear Regression we want to find the optimal values for the _**slope**_ and _**intercept**_ so that we minimize the error
14 | - In statistcs, linear regression is a linear approach to modeling the relationship between a _dependent variable_ and one or more _independent variables_
15 | - dependent variable = _**y**_
16 | - independent variable = _**x**_
17 |
18 | 
19 |
20 | ## Hypothesis
21 | We define a linear relationship between x and y by using the equation _**Y = mX + b**_.
In this equation, m is the _**slope**_ of the line and b is the _**y-intercept**_.
22 |
23 |
24 |
25 | ## Loss Function (Cost Function)
26 | The loss function, also called as "cost function", is the _error_ in the predicted value of m and b, given _y = mx + b_
27 |
28 |
29 |
30 | ## Mean Squared Error (MSE)
31 | It tells you how close a regression line is to a set of points. It does this by taking the distances from the points to the regression line (these distances are the “errors”) and squaring them. The squaring is necessary to remove any negative signs. It also gives more weight to larger differences. It’s called the mean squared error as you’re finding the average of a set of errors. The lower the MSE, the better the predicted value.
32 |
33 |
34 |
35 | ## Gradient Descent
36 | _"A gradient measures how much the output of a function changes if you change the inputs a little bit." — Lex Fridman (MIT)_
37 |
38 | _**Gradient descent**_ can fit a line to data by finding the optimal values of the _**Intercept**_ and the _**Slope**_.
39 |
40 | Gradient Descent is an optimization algorithm used to find the values of parameters (coefficients) of a function (f) that minimizes a cost function (cost). Here, the cost function is the _MSE_. It minimizes some function (f) by iteratively moving in the direction of steepest descent as defined by the negative of the gradient. In machine learning, we use gradient descent to update the parameters of our model. Parameters refer to coefficients in Linear Regression and weights in neural networks.
41 |
42 | A gradient simply measures the change in all weights with regard to the change in error. You can also think of a gradient as the slope of a function. The higher the gradient, the steeper the slope and the faster a model can learn. But if the slope is zero, the model stops learning. In mathematical terms, a gradient is a partial derivative with respect to its inputs.
43 |
44 |
45 | 
46 |
47 |
48 | ### Resources:
49 | 1. [Linear Regression using Gradient Descent in Python - Machine Learning Basics](https://www.youtube.com/watch?v=4PHI11lX11I&ab_channel=AdarshMenon)
50 | 2. [TowardsDataSciene - Linear Regression using Gradient Descent](https://towardsdatascience.com/linear-regression-using-gradient-descent-97a6c8700931)
51 | 3. [StatQuest: Gradient Descent - Step-by-Step](https://www.youtube.com/watch?v=sDv4f4s2SB8&ab_channel=StatQuestwithJoshStarmer)
52 |
--------------------------------------------------------------------------------
/2.Multiple-Linear-Regression/Multiple_Regression_Normal_Equation.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "id": "0317f4a7-d99f-4f57-9f17-efdab3c76c33",
6 | "metadata": {},
7 | "source": [
8 | "# Multiple Regression using Normal Equation\n",
9 | "\n",
10 | "**Dataset:** [Diabetes dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_diabetes.html) from scikit-learn"
11 | ]
12 | },
13 | {
14 | "cell_type": "markdown",
15 | "id": "1da3b721-7e7e-493f-a224-7cb240f2edcd",
16 | "metadata": {},
17 | "source": [
18 | "## 1. Import libraries"
19 | ]
20 | },
21 | {
22 | "cell_type": "code",
23 | "execution_count": 1,
24 | "id": "e391f757-a5e1-42ed-9448-3904fe9de5a8",
25 | "metadata": {},
26 | "outputs": [],
27 | "source": [
28 | "import numpy as np\n",
29 | "import pandas as pd"
30 | ]
31 | },
32 | {
33 | "cell_type": "markdown",
34 | "id": "ea0e301d-09dc-43b5-93de-f28fe23f4d04",
35 | "metadata": {},
36 | "source": [
37 | "## 2. Build model"
38 | ]
39 | },
40 | {
41 | "cell_type": "code",
42 | "execution_count": 2,
43 | "id": "dd271b25-9726-403e-a4b0-322388233463",
44 | "metadata": {},
45 | "outputs": [],
46 | "source": [
47 | "class MultipleRegression():\n",
48 | " '''\n",
49 | " A class which implements multiple regression model using the normal equation.\n",
50 | " '''\n",
51 | " def __init__(self):\n",
52 | " self.coefficients = []\n",
53 | " \n",
54 | " def fit(self, X, y):\n",
55 | " '''\n",
56 | " Used to calculate the coefficients of the multiple regression model.\n",
57 | " \n",
58 | " :param X: array, features\n",
59 | " :param y: array, true values\n",
60 | " :return: None\n",
61 | " '''\n",
62 | " if len(X.shape) == 1:\n",
63 | " X = self._reshape_X(X)\n",
64 | " \n",
65 | " X = self._concatenate_ones(X)\n",
66 | " self.coefficients = np.linalg.inv(X.transpose().dot(X)).dot(X.transpose()).dot(y)\n",
67 | " \n",
68 | " def predict(self, test_data):\n",
69 | " '''\n",
70 | " Makes predictions using the line equation.\n",
71 | " \n",
72 | " :param X: array, features\n",
73 | " :return: array, predictions\n",
74 | " '''\n",
75 | " bias = self.coefficients[0]\n",
76 | " weights = self.coefficients[1:]\n",
77 | " prediction = bias\n",
78 | " \n",
79 | " for xi, bi in zip(test_data, weights):\n",
80 | " prediction += (xi * bi)\n",
81 | " return prediction\n",
82 | " \n",
83 | " def mean_squared_error(self, y, y_pred):\n",
84 | " '''\n",
85 | " Private method, used to evaluate loss at each iteration.\n",
86 | "\n",
87 | " :param: y - array, true values\n",
88 | " :param: y_hat - array, predicted values\n",
89 | " :return: float\n",
90 | " '''\n",
91 | " error = 0\n",
92 | " for i in range(len(y)):\n",
93 | " error += ((y[i] - y_pred[i])**2)\n",
94 | " return error / len(y)\n",
95 | " \n",
96 | " def _reshape_X(self, X):\n",
97 | " return X.reshape(-1, 1)\n",
98 | " \n",
99 | " def _concatenate_ones(self, X):\n",
100 | " ones = np.ones(shape = X.shape[0]).reshape(-1, 1)\n",
101 | " return np.concatenate((ones, X), 1)"
102 | ]
103 | },
104 | {
105 | "cell_type": "markdown",
106 | "id": "9b6b1230-ff97-44c2-9c27-e1ecd7521e81",
107 | "metadata": {},
108 | "source": [
109 | "## 3. Load diabetes dataset"
110 | ]
111 | },
112 | {
113 | "cell_type": "code",
114 | "execution_count": 3,
115 | "id": "4bb19547-272a-4017-bd0d-4a71cdb799c6",
116 | "metadata": {},
117 | "outputs": [],
118 | "source": [
119 | "from sklearn.datasets import load_diabetes\n",
120 | "\n",
121 | "diabetes = load_diabetes()\n",
122 | "X = diabetes.data\n",
123 | "y = diabetes.target"
124 | ]
125 | },
126 | {
127 | "cell_type": "markdown",
128 | "id": "29f3fd68-7fda-47ae-94cd-c60bf872bbd0",
129 | "metadata": {},
130 | "source": [
131 | "### 3.1. Input and output shape"
132 | ]
133 | },
134 | {
135 | "cell_type": "code",
136 | "execution_count": 4,
137 | "id": "385b6a3e-aa26-4a38-9237-60585c047a49",
138 | "metadata": {},
139 | "outputs": [
140 | {
141 | "name": "stdout",
142 | "output_type": "stream",
143 | "text": [
144 | "(442, 10)\n",
145 | "(442,)\n"
146 | ]
147 | }
148 | ],
149 | "source": [
150 | "print(X.shape)\n",
151 | "print(y.shape)"
152 | ]
153 | },
154 | {
155 | "cell_type": "markdown",
156 | "id": "d5c97128-869a-4a28-b255-fe132c7c72ef",
157 | "metadata": {},
158 | "source": [
159 | "### 3.2. Feature names"
160 | ]
161 | },
162 | {
163 | "cell_type": "code",
164 | "execution_count": 5,
165 | "id": "2f21005b-3611-4904-8dcb-208cff2bd9e7",
166 | "metadata": {},
167 | "outputs": [
168 | {
169 | "data": {
170 | "text/plain": [
171 | "['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']"
172 | ]
173 | },
174 | "execution_count": 5,
175 | "metadata": {},
176 | "output_type": "execute_result"
177 | }
178 | ],
179 | "source": [
180 | "diabetes.feature_names"
181 | ]
182 | },
183 | {
184 | "cell_type": "markdown",
185 | "id": "b3db867c-5925-48d7-b2fe-8ebf37b15107",
186 | "metadata": {},
187 | "source": [
188 | "### 3.3. Train Test split"
189 | ]
190 | },
191 | {
192 | "cell_type": "code",
193 | "execution_count": 6,
194 | "id": "ecafb152-6853-4de7-9787-276e98e259e8",
195 | "metadata": {},
196 | "outputs": [
197 | {
198 | "name": "stdout",
199 | "output_type": "stream",
200 | "text": [
201 | "(353, 10)\n",
202 | "(353,)\n"
203 | ]
204 | }
205 | ],
206 | "source": [
207 | "from sklearn.model_selection import train_test_split\n",
208 | "\n",
209 | "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
210 | "\n",
211 | "print(X_train.shape) # 80% of data\n",
212 | "print(y_train.shape) # 20% of data"
213 | ]
214 | },
215 | {
216 | "cell_type": "markdown",
217 | "id": "2d893498-4843-4c37-a3da-62eb51eb6858",
218 | "metadata": {},
219 | "source": [
220 | "## 4. Create multiple regression model using MultipleRegression class"
221 | ]
222 | },
223 | {
224 | "cell_type": "code",
225 | "execution_count": 7,
226 | "id": "b6c45a9f-7380-4e17-a1bc-d5c738114528",
227 | "metadata": {},
228 | "outputs": [],
229 | "source": [
230 | "model = MultipleRegression()"
231 | ]
232 | },
233 | {
234 | "cell_type": "markdown",
235 | "id": "e82ecd28-a1ad-47e3-8cde-7af604055925",
236 | "metadata": {},
237 | "source": [
238 | "### 4.1. Fit model"
239 | ]
240 | },
241 | {
242 | "cell_type": "code",
243 | "execution_count": 8,
244 | "id": "c2d12b31-54dc-43d3-b7c4-76b1d194bce7",
245 | "metadata": {},
246 | "outputs": [],
247 | "source": [
248 | "model.fit(X_train, y_train)"
249 | ]
250 | },
251 | {
252 | "cell_type": "code",
253 | "execution_count": 9,
254 | "id": "9bb43a75-f5c7-4668-b983-94c96a7197b6",
255 | "metadata": {},
256 | "outputs": [
257 | {
258 | "data": {
259 | "text/plain": [
260 | "array([ 151.34565535, 37.90031426, -241.96624835, 542.42575342,\n",
261 | " 347.70830529, -931.46126093, 518.04405547, 163.40353476,\n",
262 | " 275.31003837, 736.18909839, 48.67112488])"
263 | ]
264 | },
265 | "execution_count": 9,
266 | "metadata": {},
267 | "output_type": "execute_result"
268 | }
269 | ],
270 | "source": [
271 | "model.coefficients"
272 | ]
273 | },
274 | {
275 | "cell_type": "markdown",
276 | "id": "9adecf10-17d2-4e12-a8a5-4435a7113643",
277 | "metadata": {},
278 | "source": [
279 | "## 5. Make predictions"
280 | ]
281 | },
282 | {
283 | "cell_type": "code",
284 | "execution_count": 10,
285 | "id": "28d8a532-11df-4a61-a5f3-00677cce3d41",
286 | "metadata": {},
287 | "outputs": [
288 | {
289 | "data": {
290 | "text/plain": [
291 | "210.74244148429636"
292 | ]
293 | },
294 | "execution_count": 10,
295 | "metadata": {},
296 | "output_type": "execute_result"
297 | }
298 | ],
299 | "source": [
300 | "model.predict(X[0])"
301 | ]
302 | },
303 | {
304 | "cell_type": "code",
305 | "execution_count": 11,
306 | "id": "ab6d39ca-fdbe-4237-8f25-def283731bf1",
307 | "metadata": {},
308 | "outputs": [],
309 | "source": [
310 | "y_pred = []\n",
311 | "\n",
312 | "for row in X_test:\n",
313 | " y_pred.append(model.predict(row))"
314 | ]
315 | },
316 | {
317 | "cell_type": "code",
318 | "execution_count": 12,
319 | "id": "6a21e5a5-ce4e-433f-af54-11c6a1b9828a",
320 | "metadata": {},
321 | "outputs": [
322 | {
323 | "data": {
324 | "text/html": [
325 | "\n",
326 | "\n",
339 | "
\n",
340 | " \n",
341 | " \n",
342 | " | \n",
343 | " actual | \n",
344 | " predicted | \n",
345 | "
\n",
346 | " \n",
347 | " \n",
348 | " \n",
349 | " | 0 | \n",
350 | " 219.0 | \n",
351 | " 139.548313 | \n",
352 | "
\n",
353 | " \n",
354 | " | 1 | \n",
355 | " 70.0 | \n",
356 | " 179.520306 | \n",
357 | "
\n",
358 | " \n",
359 | " | 2 | \n",
360 | " 202.0 | \n",
361 | " 134.041333 | \n",
362 | "
\n",
363 | " \n",
364 | " | 3 | \n",
365 | " 230.0 | \n",
366 | " 291.411936 | \n",
367 | "
\n",
368 | " \n",
369 | " | 4 | \n",
370 | " 111.0 | \n",
371 | " 123.787237 | \n",
372 | "
\n",
373 | " \n",
374 | "
\n",
375 | "
13 |
14 | ## Simple vs. Multiple linear regression
15 | - Simple linear regression solves problems with only one input feature
16 | - Multiple linear regression solves problems with multiple input features
17 |
18 |
19 |
20 | ## Main ideas
21 | - Training a linear regression model means calculating the best coefficients for the line equation formula
22 | - **Gradient descent** can be used to obtain the optimal coefficients
23 | - An iterative optimization algorithm that calculates derivatives wrt. (with relation to) each coefficient, and updates the coefficients on the go
24 | - One additional parameter - *learning rate*, specifies the rate at which coefficients are updated
25 | - High learning rate can lead to "missing" the best values
26 | - Low learning rate can lead to slow optimization
27 |
28 |
29 |
30 | ## Formula of multiple linear regression
31 | The model for multiple linear regression, given _n_ observations, is:
32 | ### _y = β0 + β1x1 + β2x2 + ... + βnxn + ϵ_
33 | where, for _**i = 1, 2, ... , n**_ observations:
34 | - _y_ = the predicted value of the dependent variable
35 | - _β0_ = the y-intercept (value of _y_ when all other parameters are set to 0)
36 | - _β1x1_ = the regression coefficient (_β1_) of the first independent variable (_x1_) (a.k.a. the effect that increasing the value of the independent variable has on the predicted _y_ value)
37 | - _β2x2_ = the regression coefficient (_β2_) of the second independent variable (_x2_)
38 | - … = do the same for however many independent variables you are testing
39 | - _βnxn_ = the regression coefficient of the last independent variable
40 | - _ϵ_ = model error (a.k.a. how much variation there is in our estimate of _y_)
41 |
42 |
43 |
44 | In simple words, the model is expressed as _**DATA = FIT + RESIDUAL**_, where the _"FIT"_ term represents the expression _β0 + β1x1 + β2x2 + ... + βnxn_. The _"RESIDUAL"_ term represents the deviations of the observed values _y_ from their means, which are normally distributed with mean 0 and variance. The notation for the model deviations is ϵ.
45 |
46 |
47 |
48 | ## Example of using Multiple Regression for predicting house prices
49 | We want to predict house prices and gathered a few features, such as, size of the house (sqr. feet), number of bedrooms, and age of the house.
50 |
51 | In this example, our formula can be written as:
52 |
_**price = β0 + m1 * size + m2 * bedrooms + m3 * age**_
53 |
54 | - The price is our dependent variable (_y_), also known as the _**target variable**_ in Machine Learning lingo
55 | - The features are our independent variables (_size, bedrooms, age_)
56 | - _β0_ = y-intercept
57 | - _m_ = regression coefficients
58 | - Every value of the independent variable _x_ is associated with a value of the dependent variable _y_
59 |
60 |
68 |
69 |
70 | ## Simple Regression vs Multiple Regression
71 | Simple regression is just fitting a line to data where you have one dependent variable (_y_) and one independent variable (_x_).
72 |
73 | _**Multiple linear regression (MLR)**_, also known simply as _**multiple regression**_, is a statistical technique that uses several explanatory variables to predict the outcome of a response variable.
74 | - The goal of multiple linear regression is to model the linear relationship between the _**explanatory (independent)**_ variables and _**response (dependent)**_ variable.
75 | - Multiple regression is an extension of ordinary least-squares (OLS) regression because it involves more than one explanatory variable.
76 | - Simple linear regression solves problems with only one input feature
77 | - Multiple linear regression solves problems with multiple input features
78 |
79 |
80 |
81 | ## Multiple Regression vs Multivariate Regression
82 | Multiple Linear Regression differs from Multivariate Regression. Multivariate Regression is a method used to measure the degree at which more than one independent variable (predictors) and more than one dependent variable (responses), are linearly related. The method is broadly used to predict the behavior of the response variables associated to changes in the predictor variables, once a desired degree of relation has been established.
83 |
84 |
85 |
86 | ## What does it mean for a multiple regression to be linear?
87 | In multiple linear regression, the model calculates the _**line of best fit**_ that minimizes the variances of each of the variables included as it relates to the dependent variable. Because it fits a line, it is a linear model. There are also non-linear regression models involving multiple variables, such as logistic regression, quadratic regression, and probit models.
88 |
89 |
90 |
91 | ### Resources
92 | 1. [Scribbr - An introduction to multiple linear regression](https://www.scribbr.com/statistics/multiple-linear-regression/)
93 | 2. [Yale Stats](http://www.stat.yale.edu/Courses/1997-98/101/linmult.htm)
94 | 3. [Investopedia - Multiple Linear Regression (MLR)](https://www.investopedia.com/terms/m/mlr.asp)
95 | 4. [codebasics - Machine Learning Tutorial: Linear Regression Multiple Variables](https://www.youtube.com/watch?v=J_LnPL3Qg70&ab_channel=codebasics)
96 | 5. [daradecic - GitHub repo](https://github.com/daradecic/BDS-articles/blob/main/010_MML_Multiple_Linear_Regression.ipynb)
97 |
--------------------------------------------------------------------------------
/3.Logistic-Regression/LogisticRegression_scikit-learn.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "id": "1aa3e8c5-2691-4ff0-9275-1512223554f7",
6 | "metadata": {},
7 | "source": [
8 | "# Logistic Regreesion with scikit-learn\n",
9 | "\n",
10 | "Logistic regression is another technique borrowed by machine learning from the field of statistics.\n",
11 | "\n",
12 | "It is the go-to method for **binary classification problems** (problems with two class values). \n",
13 | "\n",
14 | "I'll be using the [Iris dataset](https://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html) to perform Logistic Regression.\n",
15 | "\n",
16 | "Scikit-learn's [Logistic Regression documentation](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html)\n",
17 | "\n",
18 | "\n",
19 | "
\n",
20 | "\n",
21 | "#### Classification videos:\n",
22 | "- Data Professor: [Machine Learning in Python: Building a Classification Model](https://www.youtube.com/watch?v=XmSlFPDjKdc&ab_channel=DataProfessor)\n",
23 | "- Codebasics: [Machine Learning Tutorial Python - 8: Logistic Regression (Binary Classification)](https://www.youtube.com/watch?v=zM4VZR0px8E&ab_channel=codebasics)\n",
24 | "\n",
25 | "Learn more about Logistic Regression on the [Machine Learning Mastery](https://machinelearningmastery.com/logistic-regression-for-machine-learning/) website."
26 | ]
27 | },
28 | {
29 | "cell_type": "code",
30 | "execution_count": 1,
31 | "id": "063c05eb-7cc3-4b8c-ae74-5c3b07922cc6",
32 | "metadata": {},
33 | "outputs": [],
34 | "source": [
35 | "# Scikit-learn libraries\n",
36 | "from sklearn import datasets\n",
37 | "from sklearn.linear_model import LogisticRegression\n",
38 | "from sklearn.model_selection import train_test_split"
39 | ]
40 | },
41 | {
42 | "cell_type": "markdown",
43 | "id": "76446eaf-e2a5-42db-8f9a-33b045f262cb",
44 | "metadata": {},
45 | "source": [
46 | "## 1. Load dataset + Create X and Y data matrices"
47 | ]
48 | },
49 | {
50 | "cell_type": "code",
51 | "execution_count": 2,
52 | "id": "0bb71c52-4963-4c4f-94c2-94d3e3f6d4a8",
53 | "metadata": {},
54 | "outputs": [],
55 | "source": [
56 | "# Load iris dataset\n",
57 | "iris = datasets.load_iris()"
58 | ]
59 | },
60 | {
61 | "cell_type": "markdown",
62 | "id": "cffb76d1-ae47-4df8-92e5-7bf4fc1f2bce",
63 | "metadata": {},
64 | "source": [
65 | "## 2. Input vs Output features"
66 | ]
67 | },
68 | {
69 | "cell_type": "code",
70 | "execution_count": 3,
71 | "id": "00a16c46-21d1-4886-be46-a62014ef3660",
72 | "metadata": {},
73 | "outputs": [
74 | {
75 | "name": "stdout",
76 | "output_type": "stream",
77 | "text": [
78 | "['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']\n"
79 | ]
80 | }
81 | ],
82 | "source": [
83 | "# Feature names\n",
84 | "print(iris.feature_names)"
85 | ]
86 | },
87 | {
88 | "cell_type": "code",
89 | "execution_count": 4,
90 | "id": "d06422d0-02d4-4485-92dc-bd4af6e02ba6",
91 | "metadata": {},
92 | "outputs": [
93 | {
94 | "name": "stdout",
95 | "output_type": "stream",
96 | "text": [
97 | "['setosa' 'versicolor' 'virginica']\n"
98 | ]
99 | }
100 | ],
101 | "source": [
102 | "# Target names\n",
103 | "print(iris.target_names)"
104 | ]
105 | },
106 | {
107 | "cell_type": "code",
108 | "execution_count": 5,
109 | "id": "fc810ba7-91e4-467c-8af7-159357ef6981",
110 | "metadata": {},
111 | "outputs": [
112 | {
113 | "data": {
114 | "text/plain": [
115 | "array([[5.1, 3.5, 1.4, 0.2],\n",
116 | " [4.9, 3. , 1.4, 0.2],\n",
117 | " [4.7, 3.2, 1.3, 0.2],\n",
118 | " [4.6, 3.1, 1.5, 0.2],\n",
119 | " [5. , 3.6, 1.4, 0.2],\n",
120 | " [5.4, 3.9, 1.7, 0.4],\n",
121 | " [4.6, 3.4, 1.4, 0.3],\n",
122 | " [5. , 3.4, 1.5, 0.2],\n",
123 | " [4.4, 2.9, 1.4, 0.2],\n",
124 | " [4.9, 3.1, 1.5, 0.1]])"
125 | ]
126 | },
127 | "execution_count": 5,
128 | "metadata": {},
129 | "output_type": "execute_result"
130 | }
131 | ],
132 | "source": [
133 | "# First 10 rows of iris data\n",
134 | "'''\n",
135 | "This is our X variable\n",
136 | "'''\n",
137 | "iris.data[:10, :]"
138 | ]
139 | },
140 | {
141 | "cell_type": "code",
142 | "execution_count": 6,
143 | "id": "d55d5fdd-2508-48c3-8216-57676a4ff381",
144 | "metadata": {},
145 | "outputs": [
146 | {
147 | "data": {
148 | "text/plain": [
149 | "array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
150 | " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
151 | " 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
152 | " 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
153 | " 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,\n",
154 | " 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,\n",
155 | " 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])"
156 | ]
157 | },
158 | "execution_count": 6,
159 | "metadata": {},
160 | "output_type": "execute_result"
161 | }
162 | ],
163 | "source": [
164 | "# Iris target values\n",
165 | "'''\n",
166 | "This is our y variable\n",
167 | "'''\n",
168 | "iris.target"
169 | ]
170 | },
171 | {
172 | "cell_type": "markdown",
173 | "id": "93cb1cee-8e93-4bb3-bff1-2248680d985b",
174 | "metadata": {},
175 | "source": [
176 | "### 2.1 Assigning input and output variables"
177 | ]
178 | },
179 | {
180 | "cell_type": "code",
181 | "execution_count": 7,
182 | "id": "bb60b898-65a4-4a18-b490-8fa531eaf3dd",
183 | "metadata": {},
184 | "outputs": [],
185 | "source": [
186 | "X = iris.data\n",
187 | "y = iris.target"
188 | ]
189 | },
190 | {
191 | "cell_type": "markdown",
192 | "id": "5dc45c67-f405-47d0-a735-f773df74218d",
193 | "metadata": {},
194 | "source": [
195 | "### 2.2 Data dimensions"
196 | ]
197 | },
198 | {
199 | "cell_type": "code",
200 | "execution_count": 8,
201 | "id": "84261376-8630-4ba9-aaea-0c23dccd6511",
202 | "metadata": {},
203 | "outputs": [
204 | {
205 | "name": "stdout",
206 | "output_type": "stream",
207 | "text": [
208 | "X shape: (150, 4)\n",
209 | "y shape: (150,)\n"
210 | ]
211 | }
212 | ],
213 | "source": [
214 | "print('X shape:', X.shape)\n",
215 | "print('y shape:', y.shape)"
216 | ]
217 | },
218 | {
219 | "cell_type": "markdown",
220 | "id": "3f8e1585-b495-46ed-a2d8-7f1ca9492349",
221 | "metadata": {},
222 | "source": [
223 | "## 3. Build Classification Model using Logistic Regression"
224 | ]
225 | },
226 | {
227 | "cell_type": "code",
228 | "execution_count": 9,
229 | "id": "8a7e2838-e9c2-413e-a3e6-8cd862affaf2",
230 | "metadata": {},
231 | "outputs": [],
232 | "source": [
233 | "# Importing logistic regression model and setting number of iterations to 1000\n",
234 | "lr = LogisticRegression(max_iter=1000)"
235 | ]
236 | },
237 | {
238 | "cell_type": "code",
239 | "execution_count": 10,
240 | "id": "575797b9-d557-48ed-837b-3c202f783f32",
241 | "metadata": {},
242 | "outputs": [
243 | {
244 | "data": {
245 | "text/plain": [
246 | "LogisticRegression(max_iter=1000)"
247 | ]
248 | },
249 | "execution_count": 10,
250 | "metadata": {},
251 | "output_type": "execute_result"
252 | }
253 | ],
254 | "source": [
255 | "# Fit model\n",
256 | "lr.fit(X,y)"
257 | ]
258 | },
259 | {
260 | "cell_type": "markdown",
261 | "id": "61b97b10-627d-4d6f-9f69-cf35b31bfbb7",
262 | "metadata": {},
263 | "source": [
264 | "## 4. Coefficients and Intercept"
265 | ]
266 | },
267 | {
268 | "cell_type": "code",
269 | "execution_count": 11,
270 | "id": "a467139f-249c-4330-bb7f-32f1fe5eb21c",
271 | "metadata": {},
272 | "outputs": [
273 | {
274 | "name": "stdout",
275 | "output_type": "stream",
276 | "text": [
277 | "[[-0.42434519 0.96692807 -2.51720846 -1.07938946]\n",
278 | " [ 0.53499003 -0.32132698 -0.20620328 -0.94424639]\n",
279 | " [-0.11064484 -0.64560109 2.72341174 2.02363584]]\n"
280 | ]
281 | }
282 | ],
283 | "source": [
284 | "print(lr.coef_)"
285 | ]
286 | },
287 | {
288 | "cell_type": "code",
289 | "execution_count": 12,
290 | "id": "34a1232c-20ae-473b-b3a7-7cf56e235616",
291 | "metadata": {},
292 | "outputs": [
293 | {
294 | "name": "stdout",
295 | "output_type": "stream",
296 | "text": [
297 | "[ 9.85512128 2.23277161 -12.08789289]\n"
298 | ]
299 | }
300 | ],
301 | "source": [
302 | "print(lr.intercept_)"
303 | ]
304 | },
305 | {
306 | "cell_type": "markdown",
307 | "id": "25db3391-1b2e-4baf-b7a1-1a6080f99528",
308 | "metadata": {},
309 | "source": [
310 | "## 5. Make prediction"
311 | ]
312 | },
313 | {
314 | "cell_type": "code",
315 | "execution_count": 13,
316 | "id": "def842ff-767e-4975-91ee-ac1290d6b6bf",
317 | "metadata": {},
318 | "outputs": [
319 | {
320 | "name": "stdout",
321 | "output_type": "stream",
322 | "text": [
323 | "[5.1 3.5 1.4 0.2]\n"
324 | ]
325 | }
326 | ],
327 | "source": [
328 | "print(X[0])"
329 | ]
330 | },
331 | {
332 | "cell_type": "code",
333 | "execution_count": 14,
334 | "id": "a30f99af-491f-42a0-b98d-de7b192bcfef",
335 | "metadata": {},
336 | "outputs": [
337 | {
338 | "name": "stdout",
339 | "output_type": "stream",
340 | "text": [
341 | "[0]\n"
342 | ]
343 | }
344 | ],
345 | "source": [
346 | "print(lr.predict(X[[0]]))"
347 | ]
348 | },
349 | {
350 | "cell_type": "code",
351 | "execution_count": 15,
352 | "id": "60cc860c-e994-4be8-8795-9f799d414fcf",
353 | "metadata": {},
354 | "outputs": [
355 | {
356 | "name": "stdout",
357 | "output_type": "stream",
358 | "text": [
359 | "[[9.81588489e-01 1.84114969e-02 1.45146963e-08]]\n"
360 | ]
361 | }
362 | ],
363 | "source": [
364 | "print(lr.predict_proba([X[0]]))"
365 | ]
366 | },
367 | {
368 | "cell_type": "markdown",
369 | "id": "a24c2724-647e-4b6a-bb41-b9463cc7264e",
370 | "metadata": {},
371 | "source": [
372 | "### 5.1. Rebuild model to see the names of the labels in the predictions"
373 | ]
374 | },
375 | {
376 | "cell_type": "code",
377 | "execution_count": 16,
378 | "id": "9afa0a9e-5451-4c5a-bd13-8b8225e80d0f",
379 | "metadata": {},
380 | "outputs": [
381 | {
382 | "data": {
383 | "text/plain": [
384 | "LogisticRegression(max_iter=1000)"
385 | ]
386 | },
387 | "execution_count": 16,
388 | "metadata": {},
389 | "output_type": "execute_result"
390 | }
391 | ],
392 | "source": [
393 | "lr.fit(iris.data, iris.target_names[iris.target])"
394 | ]
395 | },
396 | {
397 | "cell_type": "code",
398 | "execution_count": 17,
399 | "id": "c1a077aa-e255-4c74-a6ac-fbd26ae52950",
400 | "metadata": {},
401 | "outputs": [
402 | {
403 | "name": "stdout",
404 | "output_type": "stream",
405 | "text": [
406 | "['setosa']\n",
407 | "['virginica']\n"
408 | ]
409 | }
410 | ],
411 | "source": [
412 | "print(lr.predict(X[[0]]))\n",
413 | "print(lr.predict(X[[77]]))"
414 | ]
415 | },
416 | {
417 | "cell_type": "markdown",
418 | "id": "50291886-c602-446e-a152-4e120ea523bf",
419 | "metadata": {},
420 | "source": [
421 | "## 6. Data split (80/20 ratio)"
422 | ]
423 | },
424 | {
425 | "cell_type": "code",
426 | "execution_count": 18,
427 | "id": "f40d8fd3-6bc7-426d-baf6-63aa811f7dfb",
428 | "metadata": {},
429 | "outputs": [],
430 | "source": [
431 | "# Train/Test split\n",
432 | "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)"
433 | ]
434 | },
435 | {
436 | "cell_type": "code",
437 | "execution_count": 19,
438 | "id": "8c322fa9-c71f-41fd-8d7b-6a0ce3eb1727",
439 | "metadata": {},
440 | "outputs": [
441 | {
442 | "data": {
443 | "text/plain": [
444 | "((120, 4), (120,))"
445 | ]
446 | },
447 | "execution_count": 19,
448 | "metadata": {},
449 | "output_type": "execute_result"
450 | }
451 | ],
452 | "source": [
453 | "# 80% goes into the training set\n",
454 | "X_train.shape, y_train.shape"
455 | ]
456 | },
457 | {
458 | "cell_type": "code",
459 | "execution_count": 20,
460 | "id": "3722b514-bac7-49d1-9827-ecde48efd1ea",
461 | "metadata": {},
462 | "outputs": [
463 | {
464 | "data": {
465 | "text/plain": [
466 | "((30, 4), (30,))"
467 | ]
468 | },
469 | "execution_count": 20,
470 | "metadata": {},
471 | "output_type": "execute_result"
472 | }
473 | ],
474 | "source": [
475 | "# 20% goes into the test set\n",
476 | "X_test.shape, y_test.shape"
477 | ]
478 | },
479 | {
480 | "cell_type": "markdown",
481 | "id": "51e7c9c2-7547-4176-acec-f2f8ea134f9b",
482 | "metadata": {},
483 | "source": [
484 | "## 7. Rebuilding the Logistic Regression model"
485 | ]
486 | },
487 | {
488 | "cell_type": "code",
489 | "execution_count": 21,
490 | "id": "5cc3f146-6529-4d56-b47c-659e59466794",
491 | "metadata": {},
492 | "outputs": [
493 | {
494 | "data": {
495 | "text/plain": [
496 | "LogisticRegression(max_iter=1000)"
497 | ]
498 | },
499 | "execution_count": 21,
500 | "metadata": {},
501 | "output_type": "execute_result"
502 | }
503 | ],
504 | "source": [
505 | "lr.fit(X_train, y_train)"
506 | ]
507 | },
508 | {
509 | "cell_type": "markdown",
510 | "id": "aa0685b1-0839-4395-9cd7-83a8bfe2cd73",
511 | "metadata": {},
512 | "source": [
513 | "### 7.1. Perform prediction on single sample from the data set"
514 | ]
515 | },
516 | {
517 | "cell_type": "code",
518 | "execution_count": 22,
519 | "id": "d2679d42-1d69-427e-90b7-614c398ea98b",
520 | "metadata": {},
521 | "outputs": [
522 | {
523 | "name": "stdout",
524 | "output_type": "stream",
525 | "text": [
526 | "[0]\n"
527 | ]
528 | }
529 | ],
530 | "source": [
531 | "print(lr.predict([X[0]]))"
532 | ]
533 | },
534 | {
535 | "cell_type": "code",
536 | "execution_count": 23,
537 | "id": "989d20f5-6942-4410-9f35-001c528422d1",
538 | "metadata": {},
539 | "outputs": [
540 | {
541 | "name": "stdout",
542 | "output_type": "stream",
543 | "text": [
544 | "[[9.79095901e-01 2.09040102e-02 8.91877882e-08]]\n"
545 | ]
546 | }
547 | ],
548 | "source": [
549 | "print(lr.predict_proba([X[0]]))"
550 | ]
551 | },
552 | {
553 | "cell_type": "markdown",
554 | "id": "cf03066b-3ce8-4088-ad45-e05db1ddba0a",
555 | "metadata": {},
556 | "source": [
557 | "### 7.2. Perform prediction on the test set\n",
558 | "#### _Predicted class labels_"
559 | ]
560 | },
561 | {
562 | "cell_type": "code",
563 | "execution_count": 24,
564 | "id": "8fb2bbdd-1780-41a7-9388-f598da13f966",
565 | "metadata": {},
566 | "outputs": [
567 | {
568 | "name": "stdout",
569 | "output_type": "stream",
570 | "text": [
571 | "[0 1 1 0 2 1 2 0 0 2 1 0 2 1 1 0 1 1 0 0 1 1 2 0 2 1 0 0 1 2]\n"
572 | ]
573 | }
574 | ],
575 | "source": [
576 | "print(lr.predict(X_test))"
577 | ]
578 | },
579 | {
580 | "cell_type": "markdown",
581 | "id": "d1d9860d-1cdf-4484-846a-e1905f8dd2b3",
582 | "metadata": {},
583 | "source": [
584 | "#### _Actual class labels_"
585 | ]
586 | },
587 | {
588 | "cell_type": "code",
589 | "execution_count": 25,
590 | "id": "f167b3b0-e9d6-4706-974e-bfa0b6f9617c",
591 | "metadata": {},
592 | "outputs": [
593 | {
594 | "name": "stdout",
595 | "output_type": "stream",
596 | "text": [
597 | "[0 1 1 0 2 1 2 0 0 2 1 0 2 1 1 0 1 1 0 0 1 1 1 0 2 1 0 0 1 2]\n"
598 | ]
599 | }
600 | ],
601 | "source": [
602 | "print(y_test)"
603 | ]
604 | },
605 | {
606 | "cell_type": "markdown",
607 | "id": "35a0fbb3-21ed-47b2-8b10-2ffe78ee7ee6",
608 | "metadata": {},
609 | "source": [
610 | "## 8. Model Performance"
611 | ]
612 | },
613 | {
614 | "cell_type": "code",
615 | "execution_count": 26,
616 | "id": "01625942-9fc0-4289-8e5e-b74ab4863f8d",
617 | "metadata": {},
618 | "outputs": [
619 | {
620 | "name": "stdout",
621 | "output_type": "stream",
622 | "text": [
623 | "0.9666666666666667\n"
624 | ]
625 | }
626 | ],
627 | "source": [
628 | "print(lr.score(X_test, y_test))"
629 | ]
630 | },
631 | {
632 | "cell_type": "markdown",
633 | "id": "50233dba-491e-45c7-9958-256ea28c38b2",
634 | "metadata": {},
635 | "source": [
636 | "## 9. Comparing with Random Forest Classifier"
637 | ]
638 | },
639 | {
640 | "cell_type": "code",
641 | "execution_count": 27,
642 | "id": "7108efa6-0b82-45fc-8154-4b976da7fdea",
643 | "metadata": {},
644 | "outputs": [
645 | {
646 | "name": "stdout",
647 | "output_type": "stream",
648 | "text": [
649 | "0.9666666666666667\n"
650 | ]
651 | }
652 | ],
653 | "source": [
654 | "from sklearn.ensemble import RandomForestClassifier\n",
655 | "\n",
656 | "# Import model\n",
657 | "rf= RandomForestClassifier()\n",
658 | "\n",
659 | "# Fit model\n",
660 | "rf.fit(X_train, y_train)\n",
661 | "\n",
662 | "# Make prediction\n",
663 | "rf.predict(X_test)\n",
664 | "\n",
665 | "# Model Performance\n",
666 | "print(rf.score(X_test, y_test))"
667 | ]
668 | },
669 | {
670 | "cell_type": "markdown",
671 | "id": "0ec82769-9486-4e00-a949-a7e9229b7d65",
672 | "metadata": {},
673 | "source": [
674 | "# In Summary"
675 | ]
676 | },
677 | {
678 | "cell_type": "code",
679 | "execution_count": 28,
680 | "id": "a0fbdcca-9357-4aa9-9e4b-a7eecb298e33",
681 | "metadata": {},
682 | "outputs": [
683 | {
684 | "name": "stdout",
685 | "output_type": "stream",
686 | "text": [
687 | "Coefficients: [[-0.4345712 0.8243118 -2.35072311 -0.96749421]\n",
688 | " [ 0.61906814 -0.42736808 -0.20574344 -0.83176201]\n",
689 | " [-0.18449694 -0.39694372 2.55646655 1.79925622]]\n",
690 | "Intercept: [ 9.50176119 1.63227362 -11.13403481]\n",
691 | "Score: 0.9666666666666667\n"
692 | ]
693 | }
694 | ],
695 | "source": [
696 | "# Scikit-learn libraries\n",
697 | "from sklearn import datasets\n",
698 | "from sklearn.linear_model import LogisticRegression\n",
699 | "from sklearn.model_selection import train_test_split\n",
700 | "\n",
701 | "# Load iris dataset\n",
702 | "iris = datasets.load_iris()\n",
703 | "\n",
704 | "# Assigning input and output variables\n",
705 | "X = iris.data\n",
706 | "y = iris.target\n",
707 | "\n",
708 | "# TrainTest split\n",
709 | "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)\n",
710 | "\n",
711 | "# Logistic Regression model\n",
712 | "'''\n",
713 | "1. Define the logistic regression model\n",
714 | "2. Fit model\n",
715 | "3. Apply model to make prediction (on test set)\n",
716 | "'''\n",
717 | "lr = LogisticRegression(max_iter=1000) # step 1\n",
718 | "lr.fit(X_train, y_train) # step 2\n",
719 | "lr.predict(X_test) # step 3\n",
720 | "\n",
721 | "# Print results\n",
722 | "print('Coefficients:', lr.coef_)\n",
723 | "print('Intercept:', lr.intercept_)\n",
724 | "print('Score:', lr.score(X_test, y_test))"
725 | ]
726 | }
727 | ],
728 | "metadata": {
729 | "kernelspec": {
730 | "display_name": "Python 3",
731 | "language": "python",
732 | "name": "python3"
733 | },
734 | "language_info": {
735 | "codemirror_mode": {
736 | "name": "ipython",
737 | "version": 3
738 | },
739 | "file_extension": ".py",
740 | "mimetype": "text/x-python",
741 | "name": "python",
742 | "nbconvert_exporter": "python",
743 | "pygments_lexer": "ipython3",
744 | "version": "3.8.8"
745 | }
746 | },
747 | "nbformat": 4,
748 | "nbformat_minor": 5
749 | }
750 |
--------------------------------------------------------------------------------
/3.Logistic-Regression/README.md:
--------------------------------------------------------------------------------
1 | # Logistic Regression
2 |
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/README.md:
--------------------------------------------------------------------------------
1 | # Machine Learning by Stanford University | Coursera
2 | [Andrew Ng](https://en.wikipedia.org/wiki/Andrew_Ng)'s Machine Learning course on [Coursera](https://www.coursera.org/learn/machine-learning).
3 |
4 | Includes lecture notes organized by weekly topics. I highly recommend taking this course if you are interested about Machine Learning. Andrew Ng is an excellent professor and the course has amazing content!
5 |
6 |
7 | ### Lecture Notes
8 | - [Week 1 - Introduction and Linear Regression](https://github.com/murilogustineli/Machine-Learning/blob/main/Machine-Learning-by-Stanford/Week%201%20-%20Introduction%20and%20Linear%20Regression/Week%201%20-%20Introduction%20and%20Linear%20Regression.ipynb)
9 | - [Week 2 - Linear Regression with Multiple Variables](https://github.com/murilogustineli/Machine-Learning/blob/main/Machine-Learning-by-Stanford/Week%202%20-%20Linear%20Regression%20with%20Multiple%20Variables/Week%202%20-%20Linear%20Regression%20with%20Multiple%20Variables.ipynb)
10 | - [Week 3 - Logistic Regression](https://github.com/murilogustineli/Machine-Learning/blob/main/Machine-Learning-by-Stanford/Week%203%20-%20Logistic%20Regression/Week%203%20-%20Logistic%20Regression.ipynb)
11 | - [Week 4 - Neural Networks](https://github.com/murilogustineli/Machine-Learning/blob/main/Machine-Learning-by-Stanford/Week%204%20-%20Neural%20Networks/Week%204%20-%20Neural%20Networks.ipynb)
12 | - [Week 5 - Neural Networks: Learning](https://github.com/murilogustineli/Machine-Learning/blob/main/Machine-Learning-by-Stanford/Week%205%20-%20Neural%20Networks:%20Learning/Week%205%20-%20Neural%20Networks%20-%20Learning.ipynb)
13 |
14 |
22 |
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 1 - Introduction and Linear Regression/README.md:
--------------------------------------------------------------------------------
1 | # Week 1
2 | ## Introduction
3 | Welcome to Machine Learning! This week, we introduce the core idea of teaching a computer to learn concepts using data — without being explicitly programmed.
4 |
5 | We are going to start by covering linear regression with one variable. Linear regression predicts a real-valued output based on an input value. We discuss the application of linear regression to housing price prediction, present the notion of a cost function, and introduce the gradient descent method for learning.
6 |
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 1 - Introduction and Linear Regression/Week 1 - Introduction and Linear Regression.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "id": "3389254d-f5c0-47a2-b6de-e626056830c0",
6 | "metadata": {},
7 | "source": [
8 | "# Machine Learning: Week 1 - Introduction\n",
9 | "## What is Machine Learning?\n",
10 | "Two definitions of Machine Learning are offered. Arthur Samuel described it as: _\"the field of study that gives computers the ability to learn without being explicitly programmed.\"_ This is an older, informal definition.\n",
11 | "\n",
12 | "Tom Mitchell provides a more modern definition: _\"A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.\"_\n",
13 | "\n",
14 | "_**Example: playing checkers.**_\n",
15 | "
$E = $ the experience of playing many games of checkers\n",
16 | "
$T = $ the task of playing checkers.\n",
17 | "
$P = $ the probability that the program will win the next game.\n",
18 | "\n",
19 | "In general, any machine learning problem can be assigned to one of two broad classifications:\n",
20 | "\n",
21 | "**Supervised learning** and **Unsupervised learning**.\n",
22 | "\n",
23 | "## Supervised Learning\n",
24 | "In supervised learning, we are given a data set and already know what our correct output should look like, having the idea that there is a relationship between the input and the output.\n",
25 | "\n",
26 | "Supervised learning problems are categorized into \"regression\" and \"classification\" problems. In a regression problem, we are trying to predict results within a continuous output, meaning that we are trying to map input variables to some continuous function. In a classification problem, we are instead trying to predict results in a discrete output. In other words, we are trying to map input variables into discrete categories. \n",
27 | "\n",
28 | "_**Example 1:**_\n",
29 | "
Given data about the size of houses on the real estate market, try to predict their price. Price as a function of size is a continuous output, so this is a regression problem.\n",
30 | "\n",
31 | "We could turn this example into a classification problem by instead making our output about whether the house \"sells for more or less than the asking price.\" Here we are classifying the houses based on price into two discrete categories.\n",
32 | "\n",
33 | "_**Example 2:**_\n",
34 | "
(a) Regression - Given a picture of a person, we have to predict their age on the basis of the given picture\n",
35 | "\n",
36 | "(b) Classification - Given a patient with a tumor, we have to predict whether the tumor is malignant or benign.\n",
37 | "\n",
38 | "## Unsupervised Learning\n",
39 | "Unsupervised learning allows us to approach problems with little or no idea what our results should look like. We can derive structure from data where we don't necessarily know the effect of the variables.\n",
40 | "\n",
41 | "We can derive this structure by clustering the data based on relationships among the variables in the data.\n",
42 | "\n",
43 | "With unsupervised learning there is no feedback based on the prediction results.\n",
44 | "\n",
45 | "_**Example:**_\n",
46 | "
Clustering: Take a collection of 1,000,000 different genes, and find a way to automatically group these genes into groups that are somehow similar or related by different variables, such as lifespan, location, roles, and so on.\n",
47 | "\n",
48 | "Non-clustering: The \"Cocktail Party Algorithm\", allows you to find structure in a chaotic environment. (i.e. identifying individual voices and music from a mesh of sounds at a [cocktail party](https://en.wikipedia.org/wiki/Cocktail_party_effect)).\n",
49 | "Here is an [answer on Quora](https://www.quora.com/What-is-the-difference-between-supervised-and-unsupervised-learning-algorithms) to enhance your understanding.\n",
50 | "\n",
51 | "
\n"
52 | ]
53 | },
54 | {
55 | "cell_type": "markdown",
56 | "id": "173d7c3d-d1cc-49a5-972b-3f37624b6a52",
57 | "metadata": {},
58 | "source": [
59 | "# Linear Regression with One Variable\n",
60 | "## Model Representation\n",
61 | "Recall that in regression problems, we are taking input variables and trying to fit the output onto a continuous expected result function.\n",
62 | "\n",
63 | "Linear regression with one variable is also known as “univariate linear regression”.\n",
64 | "\n",
65 | "Univariate linear regression is used when you want to predict a _**single output**_ value __*y*__ from a _**single input**_ value __*x*__. We’re doing _**supervised learning**_ here, so that means we already have an idea about what the input/output cause and effect should be.\n",
66 | "\n",
67 | "## The Hypothesis Function\n",
68 | "Our hypothesis function has the general form:\n",
69 | "\n",
70 | "$$\\large \\hat{y} \\ =\\ h_{\\theta }( x) \\ =\\ \\theta _{0} \\ +\\ \\theta _{1} x$$\n",
71 | "\n",
72 | "Note that this is like the equation of a straight line. We give to $h_{\\theta }(x)$ values for ${\\theta}_0$ and ${\\theta}_1$ to get our estimated output $\\hat{y} $. In other words, we are trying to create a function called $h{\\theta }_0$ that is trying to map our input data (the x’s) to our output data (the y’s).\n",
73 | "\n",
74 | "_**Example:**_\n",
75 | "
Suppose we have the following set of training data:\n",
76 | "\n",
77 | "| input _x_ | output _y_ |\n",
78 | "| --------- | ---------- |\n",
79 | "| 1 | 4 |\n",
80 | "| 2 | 7 |\n",
81 | "| 3 | 7 |\n",
82 | "| 4 | 8 |\n",
83 | "\n",
84 | "Now we can make a random guess about our $h_{\\theta }$ function: $\\theta_{0} \\ =\\ 2$ and $\\theta _{1} \\ =\\ 2$. The hypothesis function becomes $h_{\\theta }( x) =2+2x$\n",
85 | "\n",
86 | "So for input of 1 to our hypothesis, y will be 4. This is off by 3. Note that we will be trying out various values of $\\theta_{0}$ and $\\theta_{1}$ to try to find values which provide the best possible “fit” or the most representative “straight line” through the data points mapped on the x-y plane.\n",
87 | "\n",
88 | "## Cost Function\n",
89 | "We can measure the accuracy of our hypothesis function by using a **cost function**. This takes an average difference (actually a fancier version of an average) of all the results of the hypothesis with inputs from x's and the actual output y's.\n",
90 | "\n",
91 | "\n",
92 | "$$\\large\n",
93 | "J( \\theta _{0} \\theta _{1}) \\ =\\ \\frac{1}{2m}\\sum _{i=1}^{m}(\\hat{y}_{i} \\ -y_{i})^{2} \\ \\ =\\ \\frac{1}{2m}\\sum _{i=1}^{m}( h_{\\theta }( x_{i}) \\ -\\ y_{i})^{2}\n",
94 | "$$\n",
95 | "\n",
96 | "To break it apart, it is $\\frac{1}{2}\\overline{x}$, where $\\overline{x}$ is the mean of the squares of $h_{\\theta }( x_{i}) \\ -\\ y_{i}$, or the difference between the predicted value and the actual value.\n",
97 | "\n",
98 | "This function is otherwise called the “Squared error function”, or “Mean squared error”. The mean is halved $\\left(\\frac{1}{2}\\right)$ as a convenience for the computation of the gradient descent, as the derivative term of the square function will cancel out the $\\frac{1}{2}$ term.\n",
99 | "\n",
100 | "Now we are able to concretely measure the accuracy of our predictor function against the correct results we have so that we can predict new results we don’t have.\n",
101 | "\n",
102 | "If we try to think of it in visual terms, our training data set is scattered on the x-y plane. We are trying to make straight line (defined by $h_\\theta(x)$ which passes through this scattered set of data. Our objective is to get the best possible line. The best possible line will be such so that the average squared vertical distances of the scattered points from the line will be the least. In the best case, the line should pass through all the points of our training data set. In such a case the value of $J(\\theta_0, \\theta_1)$ will be 0.\n",
103 | "\n",
104 | "## Gradient Descent\n",
105 | "So we have our hypothesis function and we have a way of measuring how well it fits into the data. Now we need to estimate the parameters in hypothesis function. That’s where gradient descent comes in.\n",
106 | "\n",
107 | "Imagine that we graph our hypothesis function based on its fields ${\\theta}_0$ and ${\\theta}_1$ (actually we are graphing the cost function as a function of the parameter estimates). This can be kind of confusing; we are moving up to a higher level of abstraction. We are not graphing x and y itself, but the parameter range of our hypothesis function and the cost resulting from selecting particular set of parameters.\n",
108 | "\n",
109 | "We put ${\\theta}_0$ on the x axis and ${\\theta}_1$ on the y axis, with the cost function on the vertical z axis. The points on our graph will be the result of the cost function using our hypothesis with those specific theta parameters. The graph below depicts such a setup.\n",
110 | "\n",
111 | "\n",
112 | "\n",
113 | "We will know that we have succeeded when our cost function is at the very bottom of the pits in our graph, i.e. when its value is the minimum.\n",
114 | "\n",
115 | "The way we do this is by taking the derivative (the tangential line to a function) of our cost function. The slope of the tangent is the derivative at that point and it will give us a direction to move towards. We make steps down the cost function in the direction with the steepest descent, and the size of each step is determined by the parameter $\\alpha$, which is called the learning rate.\n",
116 | "\n",
117 | "
\n",
118 | "\n",
119 | "The gradient descent algorithm is:\n",
120 | "\n",
121 | "Repeat until convergence:\n",
122 | "\n",
123 | "$$\\large\n",
124 | "\\theta _{j} :=\\ \\theta _{j} \\ -\\ \\alpha \\frac{\\partial }{\\partial \\theta _{j}} J( \\theta _{0} ,\\theta _{1})\n",
125 | "$$\n",
126 | "\n",
127 | "where $j = 0, 1$ represents the feature index number.\n",
128 | "\n",
129 | "At each iteration $j$, one should simultaneously update the parameters $\\theta_1,\\ \\theta_2,\\ ...,\\ \\theta_n$. Updating a specific parameter prior to calculating another one on the $j^{(th)}$ iteration would yield to a wrong implementation. \n",
130 | "\n",
131 | "Intuitively, this could be thought of as:\n",
132 | "\n",
133 | "repeat until convergence:\n",
134 | "\n",
135 | "$$\\large\n",
136 | "\\theta _{j} :=\\theta _{j} -\\alpha [ Slope\\ of\\ tanget\\ aka\\ derivative\\ in\\ j\\ dimensions][ Slope\\ of\\ tangent\\ aka\\ derivative\\ in\\ j\\ dimension]\n",
137 | "$$\n",
138 | "\n",
139 | "## Gradient Descent for Linear Regression\n",
140 | "When specifically applied to the case of linear regression, a new form of the gradient descent equation can be derived. We can substitute our actual cost function and our actual hypothesis function and modify the equation to:\n",
141 | "\n",
142 | "$$\\large\n",
143 | " \\begin{array}{l}\n",
144 | "repeat\\ until\\ convergence:\\ \\\\\n",
145 | "\\{\\\\\n",
146 | "\\theta _{0} :=\\ \\theta _{0} \\ -\\ \\alpha \\frac{1}{m}\\sum _{i=1}^{m}( h_{\\theta }( x_{i}) \\ -\\ y_{i})\\\\\n",
147 | "\\\\\n",
148 | "\\theta _{1} :=\\ \\theta _{1} \\ -\\ \\alpha \\frac{1}{m}\\sum _{i=1}^{m}(( h_{\\theta }( x_{i}) \\ -\\ y_{i}) x_{i})\\\\\n",
149 | "\\}\n",
150 | "\\end{array}\n",
151 | "$$\n",
152 | "\n",
153 | "where $m$ is the size of the training set, $\\theta_0$ a constant that will be changing simultaneously with $\\theta_1$ and $x_i$, $y_i$ are values of the given training set (data).\n",
154 | "\n",
155 | "Note that we have separated out the two cases for $\\theta_j$ into separate equations for $\\theta_0$ and $\\theta_1$ and that for $\\theta_1$ we are multiplying $x_i$ at the end due to the derivative.\n",
156 | "\n",
157 | "The point of all this is that if we start with a guess for our hypothesis and then repeatedly apply these gradient descent equations, our hypothesis will become more and more accurate!\n",
158 | "\n",
159 | "So, this is simply gradient descent on the original cost function $J$. This method looks at every example in the entire training set on every step, and is called **batch gradient descent**. Note that, while gradient descent can be susceptible to **local minima** in general, the optimization problem we have posed here for linear regression has only one global, and no other local, optima; _**thus gradient descent always converges (assuming the learning rate α is not too large) to the global minimum**_. Indeed, $J$ is a convex quadratic function. Here is an example of gradient descent as it is run to minimize a quadratic function.\n",
160 | "\n",
161 | "\n",
162 | "\n",
163 | "The ellipses shown above are the contours of a quadratic function. Also shown is the trajectory taken by gradient descent, which was initialized at (48,30). The $x$’s in the figure (joined by straight lines) mark the successive values of $\\theta$ that gradient descent went through as it converged to its minimum.\n",
164 | "\n",
165 | "The following video (https://www.youtube.com/watch?v=WnqQrPNYz5Q) may be useful as it visualizes the improvement of the hypothesis as the error function reduces.\n",
166 | "\n",
167 | "
\n"
168 | ]
169 | },
170 | {
171 | "cell_type": "markdown",
172 | "id": "1e7c0f71-e29a-47c4-9327-e63832f8c6f0",
173 | "metadata": {},
174 | "source": [
175 | "# Linear Algebra Review\n",
176 | "Khan Academy has an excellent Linear Algebra course, and it's free! (https://www.khanacademy.org/math/linear-algebra)\n",
177 | "\n",
178 | "__*Matrices*__ are 2-dimensional arrays:\n",
179 | "\n",
180 | "$$\n",
181 | "\\begin{bmatrix}\n",
182 | "a & b & c\\\\\n",
183 | "d & e & f\\\\\n",
184 | "g & h & i\\\\\n",
185 | "j & k & l\n",
186 | "\\end{bmatrix}\n",
187 | "$$\n",
188 | "\n",
189 | "The above matrix has four rows and three columns, so it is a 4 x 3 matrix.\n",
190 | "\n",
191 | "A __*vector*__ is a matrix with one column and many rows:\n",
192 | "\n",
193 | "$$\n",
194 | "\\begin{bmatrix}\n",
195 | "w\\\\\n",
196 | "x\\\\\n",
197 | "y\\\\\n",
198 | "z\n",
199 | "\\end{bmatrix}\n",
200 | "$$\n",
201 | "\n",
202 | "So vectors are a subset of matrices. The above vector is a 4 x 1 matrix.\n",
203 | "\n",
204 | "**Notation and Terms:**\n",
205 | "- $A_{ij}$ refers to the element in the ith row and jth column of matrix A.\n",
206 | "- A vector with ‘n’ rows is referred to as an ‘n’-dimensional vector\n",
207 | "- $v_i$ refers to the element in the ith row of the vector.\n",
208 | "- In general, all our vectors and matrices will be 1-indexed. Note that for some programming languages, the arrays are 0-indexed.\n",
209 | "- Matrices are usually denoted by uppercase names while vectors are lowercase.\n",
210 | "- “Scalar” means that an object is a single value, not a vector or matrix.\n",
211 | "- $\\mathbb{R}$ refers to the set of scalar real numbers\n",
212 | "- $\\mathbb{R}^n$ refers to the set of n-dimensional vectors of real numbers\n",
213 | "\n",
214 | "## Addition and Scalar Multiplication\n",
215 | "Addition and subtraction are element-wise, so you simply add or subtract each corresponding element:\n",
216 | "\n",
217 | "$$\n",
218 | "\\begin{bmatrix}\n",
219 | "a & b\\\\\n",
220 | "c & d\n",
221 | "\\end{bmatrix} + \\begin{bmatrix}\n",
222 | "w & x\\\\\n",
223 | "y & z\n",
224 | "\\end{bmatrix} \\ =\\ \\begin{bmatrix}\n",
225 | "a+w & b+x\\\\\n",
226 | "c+y & d+z\n",
227 | "\\end{bmatrix}\n",
228 | "$$\n",
229 | "\n",
230 | "To add or subtract two matrices, their dimensions must be the same.\n",
231 | "\n",
232 | "In scalar multiplication, we simply multiply every element by the scalar value:\n",
233 | "\n",
234 | "$$\n",
235 | "\\begin{bmatrix}\n",
236 | "a & b\\\\\n",
237 | "c & d\n",
238 | "\\end{bmatrix} *x\\ =\\ \\begin{bmatrix}\n",
239 | "a*x & b*x\\\\\n",
240 | "c*x & d*x\n",
241 | "\\end{bmatrix}\n",
242 | "$$\n",
243 | "\n",
244 | "## Matrix-Vector Multiplication\n",
245 | "We map the column of the vector onto each row of the matrix, multiplying each element and summing the result.\n",
246 | "\n",
247 | "$$\n",
248 | "\\begin{bmatrix}\n",
249 | "a & b\\\\\n",
250 | "c & d\\\\\n",
251 | "e & f\n",
252 | "\\end{bmatrix} \\ *\\ \\begin{bmatrix}\n",
253 | "x\\\\\n",
254 | "y\n",
255 | "\\end{bmatrix} \\ =\\begin{bmatrix}\n",
256 | "a*x+b*y\\\\\n",
257 | "c*x+d*y\\\\\n",
258 | "e*x+f*y\n",
259 | "\\end{bmatrix}\n",
260 | "$$\n",
261 | "\n",
262 | "The result is a vector. The vector must be the second term of the multiplication. The number of columns of the matrix must equal the number of rows of the vector.\n",
263 | "\n",
264 | "An $m x n$ matrix multiplied by an $n x 1$ vector results in an $m x 1$ vector.\n",
265 | "\n",
266 | "## Matrix-Matrix Multiplication\n",
267 | "$$\n",
268 | "\\begin{bmatrix}\n",
269 | "a & b\\\\\n",
270 | "c & d\\\\\n",
271 | "e & f\n",
272 | "\\end{bmatrix} \\ *\\ \\begin{bmatrix}\n",
273 | "w & x\\\\\n",
274 | "y & z\n",
275 | "\\end{bmatrix} \\ =\\begin{bmatrix}\n",
276 | "a*w+b*y & a*x+b*z\\\\\n",
277 | "c*w+d*y & c*x+d*z\\\\\n",
278 | "e*w+f*y & e*x+f*z\n",
279 | "\\end{bmatrix}\n",
280 | "$$\n",
281 | "\n",
282 | "An $m x n$ matrix multiplied by an $n x o$ matrix results in an $m x o$ matrix. In the above example, a $3 x 2$ matrix times a $2 x 2$ matrix resulted in a $3 x 2$ matrix.\n",
283 | "\n",
284 | "To multiply two matrices, the number of **columns** of the first matrix must equal the number of **rows** of the second matrix.\n",
285 | "\n",
286 | "## Matrix Multiplication Properties\n",
287 | "- Matrices are not commutative: $A*B ≠ B*A$\n",
288 | "- Matrices are associative: $(A*B)*C ≠ A*(B*C)$\n",
289 | "\n",
290 | "The identity matrix, when multiplied by any matrix of the same dimensions, results in the original matrix. It’s just like multiplying numbers by 1. The identity matrix simply has 1’s on the diagonal (upper left to lower right diagonal) and 0’s elsewhere.\n",
291 | "\n",
292 | "$$\n",
293 | "\\begin{bmatrix}\n",
294 | "1 & 0 & 0\\\\\n",
295 | "0 & 1 & 0\\\\\n",
296 | "0 & 0 & 1\n",
297 | "\\end{bmatrix}\n",
298 | "$$\n",
299 | "\n",
300 | "When multiplying the identity matrix after some matrix $(A*I)$, the square identity matrix should match the other matrix’s **columns**. When multiplying the identity matrix before some other matrix $(I*A)$, the square identity matrix should match the other matrix’s **rows**.\n",
301 | "\n",
302 | "## Inverse and Transpose\n",
303 | "The **inverse** of a matrix A is denoted $A_-1$. Multiplying by the inverse results in the identity matrix.\n",
304 | "\n",
305 | "A non square matrix does not have an inverse matrix. We can compute inverses of matrices in octave with the $pinv(A)$ function and in matlab with the $inv(A)$ function. Matrices that don’t have an inverse are singular or degenerate.\n",
306 | "\n",
307 | "The transposition of a matrix is like rotating the matrix 90° in clockwise direction and then reversing it. We can compute transposition of matrices in matlab with the transpose(A) function or A’:\n",
308 | "\n",
309 | "$$\n",
310 | "A=\\begin{bmatrix}\n",
311 | "a & b\\\\\n",
312 | "c & d\\\\\n",
313 | "e & f\n",
314 | "\\end{bmatrix}\n",
315 | "$$\n",
316 | "
\n",
317 | "$$\n",
318 | "A^{T} =\\begin{bmatrix}\n",
319 | "a & c & e\\\\\n",
320 | "b & d & f\n",
321 | "\\end{bmatrix}\n",
322 | "$$\n",
323 | "\n",
324 | "In other words:\n",
325 | "\n",
326 | "$$\\large\n",
327 | "A_{ij} = A^{T}_{ji}\n",
328 | "$$\n",
329 | "\n"
330 | ]
331 | }
332 | ],
333 | "metadata": {
334 | "kernelspec": {
335 | "display_name": "Python 3",
336 | "language": "python",
337 | "name": "python3"
338 | },
339 | "language_info": {
340 | "codemirror_mode": {
341 | "name": "ipython",
342 | "version": 3
343 | },
344 | "file_extension": ".py",
345 | "mimetype": "text/x-python",
346 | "name": "python",
347 | "nbconvert_exporter": "python",
348 | "pygments_lexer": "ipython3",
349 | "version": "3.8.8"
350 | }
351 | },
352 | "nbformat": 4,
353 | "nbformat_minor": 5
354 | }
355 |
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 1 - Introduction and Linear Regression/Week1_Images/gradient_descent1.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/Machine-Learning-by-Stanford/Week 1 - Introduction and Linear Regression/Week1_Images/gradient_descent1.png
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 1 - Introduction and Linear Regression/Week1_Images/gradient_descent2.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/Machine-Learning-by-Stanford/Week 1 - Introduction and Linear Regression/Week1_Images/gradient_descent2.png
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 1 - Introduction and Linear Regression/Week1_Images/image:
--------------------------------------------------------------------------------
1 |
2 |
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 2 - Linear Regression with Multiple Variables/README.md:
--------------------------------------------------------------------------------
1 | # Week 2
2 | ## Linear Regression with Multiple Variables
3 | This week we’re covering linear regression with multiple variables. We’ll show how linear regression can be extended to accommodate multiple input features. We also discuss best practices for implementing linear regression.
4 |
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 2 - Linear Regression with Multiple Variables/Week 2 - Linear Regression with Multiple Variables.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "id": "3656dbaf-fc2d-449b-94ca-9e741f8cb4d5",
6 | "metadata": {},
7 | "source": [
8 | "# Machine Learning: Week 2 - Linear Regression with Multiple Variables\n",
9 | "## Multiple Features\n",
10 | "Linear regression with multiple variables is also known as “multivariate linear regression”.\n",
11 | "\n",
12 | "We now introduce notation for equations where we can have any number of input variables.\n",
13 | "- $x_{j}^{( i)}=$ value of feature $j$ in the $i^{th}$ training example\n",
14 | "- $x^{( i)}=$ the column vector of all the feature inputs of the $i^{th}$ training example\n",
15 | "- $m=$ number of training examples\n",
16 | "- $n=|x^{( i)} |;$ (the number of features)\n",
17 | "\n",
18 | "Now define the multivariate form of the hypothesis function as follows, accommodating these multiple features:\n",
19 | "\n",
20 | "$$\\large\n",
21 | "h_{\\theta }( x) =\\theta _{0} +\\theta _{1} x_{1} +\\theta _{2} x_{2} +\\theta _{3} x_{3} +...+\\theta _{n} x_{n}\n",
22 | "$$\n",
23 | "\n",
24 | "In order to develop intuition about this function, we can think about $\\theta_0$ as the basic price of a house, $\\theta_1$ as the price per square meter, $\\theta_2$ as the price per floor, etc. $x_1$ will be the number of square meters in the house, $x_2$ the number of floors, etc.\n",
25 | "\n",
26 | "Using the definition of matrix multiplication, our multivariate hypothesis function can be concisely represented as:\n",
27 | "\n",
28 | "$$\\large\n",
29 | "h_{\\theta }( x) =[ \\theta _{0} \\ \\ \\theta _{1} \\ \\ ...\\ \\theta _{n}]\\begin{bmatrix}\n",
30 | "x_{0}\\\\\n",
31 | "x_{1}\\\\\n",
32 | "...\\\\\n",
33 | "x_{n}\n",
34 | "\\end{bmatrix} =\\theta ^{T} x\n",
35 | "$$\n",
36 | "\n",
37 | "This is a vectorization of our hypothesis function for one training example; see the lessons on vectorization to learn more.\n",
38 | "\n",
39 | "Remark: Note that for convenience reasons in this course we assume $x_{0}^{( i)} =1\\ for\\ ( i\\in 1,...,m)$. This allows us to do matrix operations with theta and x. Hence making the two vectors $\\theta$ and $x^{(i)}$ match each other element-wise (that is, have the same number of elements: $n+1$).\n",
40 | "\n",
41 | "The training examples are stored in X row-wise, like such:\n",
42 | "\n",
43 | "$$\\large\n",
44 | "\\begin{bmatrix}\n",
45 | "x_{( 0)}^{( 1)} & x_{( 1)}^{( 1)}\\\\\n",
46 | "x_{( 0)}^{( 2)} & x_{( 1)}^{( 2)}\\\\\n",
47 | "x_{( 0)}^{( 3)} & x_{( 1)}^{( 3)}\n",
48 | "\\end{bmatrix} \\ ,\\ \\theta =\\begin{bmatrix}\n",
49 | "\\theta _{0}\\\\\n",
50 | "\\theta ^{1}\n",
51 | "\\end{bmatrix}\n",
52 | "$$\n",
53 | "\n",
54 | "You can calculate the hypothesis as a column vector of size $(m x 1)$ with:\n",
55 | "\n",
56 | "$$\\large\n",
57 | "h_{\\theta }( X) =X\\theta \n",
58 | "$$\n",
59 | "\n",
60 | "\n",
61 | "## Cost Function\n",
62 | "For the parameter vector $\\theta$ (of type $\\mathbb{R}^{n+1}$ or in $\\mathbb{R}^{( n+1) \\times 1}$, the cost function is:\n",
63 | "\n",
64 | "$$\\large\n",
65 | "J( \\theta ) =\\frac{1}{2m}\\sum _{i=1}^{m}( h_{\\theta }( x_{i}) \\ -\\ y_{i})^{2}\n",
66 | "$$\n",
67 | "\n",
68 | "The vectorized version is:\n",
69 | "\n",
70 | "$$\\large\n",
71 | "J( \\theta ) =\\frac{1}{2m}( X\\theta -\\overline{y})^{T}( X\\theta -\\overline{y})\n",
72 | "$$\n",
73 | "\n",
74 | "where $\\overline{y}$ denotes the vector of all y values.\n",
75 | "\n",
76 | "\n",
77 | "## Gradient Descent for Multiple Variables\n",
78 | "The gradient descent equation itself is generally the same form; we just have to repeat it for our ‘n’ features:\n",
79 | "\n",
80 | "$$\\large\n",
81 | "\\begin{array}{l}\n",
82 | "repeat\\ until\\ convergence:\\{\\\\\n",
83 | "\\theta _{0} :=\\ \\theta _{0} -\\alpha \\frac{1}{m}\\sum _{i=1}^{m}\\left( h_{\\theta }\\left( x^{( i)}\\right) -y^{( i)}\\right) .x_{0}^{( i)}\\\\\n",
84 | "\\\\\n",
85 | "\\theta _{1} :=\\ \\theta _{1} -\\alpha \\frac{1}{m}\\sum _{i=1}^{m}\\left( h_{\\theta }\\left( x^{( i)}\\right) -y^{( i)}\\right) .x_{1}^{( i)}\\\\\n",
86 | "\\\\\n",
87 | "\\theta _{2} :=\\ \\theta _{2} -\\alpha \\frac{1}{m}\\sum _{i=1}^{m}\\left( h_{\\theta }\\left( x^{( i)}\\right) -y^{( i)}\\right) .x_{2}^{( i)}\\\\\n",
88 | "...\\\\\n",
89 | "\\}\n",
90 | "\\end{array}\n",
91 | "$$\n",
92 | "\n",
93 | "In other words:\n",
94 | "\n",
95 | "$$\\large\n",
96 | "\\begin{array}{l}\n",
97 | "repeat\\ until\\ convergence:\\{\\\\\n",
98 | "\\theta _{j} :=\\ \\theta _{j} -\\alpha \\frac{1}{m}\\sum _{i=1}^{m}\\left( h_{\\theta }\\left( x^{( i)}\\right) -y^{( i)}\\right) .x_{j}^{( i)} \\ \\ \\ \\ \\ for\\ j:=0...n\\\\\n",
99 | "\\}\n",
100 | "\\end{array}\n",
101 | "$$\n",
102 | "\n",
103 | "\n",
104 | "## Feature Scaling\n",
105 | "We can speed up gradient descent by having each of our input values in roughly the same range. This is because $\\theta$ will descend quickly on small ranges and slowly on large ranges, and so will oscillate inefficiently down to the optimum when the variables are very uneven.\n",
106 | "\n",
107 | "The way to prevent this is to modify the ranges of our input variables so that they are all roughly the same. Ideally:\n",
108 | "\n",
109 | "$$\\large\n",
110 | "-1\\leqslant x_{( i)} \\leqslant 1\n",
111 | "$$\n",
112 | "\n",
113 | "or\n",
114 | "\n",
115 | "$$\\large\n",
116 | "-0.5\\leqslant x_{( i)} \\leqslant 0.5\n",
117 | "$$\n",
118 | "\n",
119 | "These aren’t exact requirements; we are only trying to speed things up. The goal is to get all input variables into roughly one of these ranges, give or take a few.\n",
120 | "\n",
121 | "Two techniques to help with this are **feature scaling** and **mean normalization**. Feature scaling involves dividing the input values by the range (i.e. the maximum value minus the minimum value) of the input variable, resulting in a new range of just 1. Mean normalization involves subtracting the average value for an input variable from the values for that input variable, resulting in a new average value for the input variable of just zero. To implement both of these techniques, adjust your input values as shown in this formula:\n",
122 | "\n",
123 | "$$\\large\n",
124 | "x_{i} :=\\frac{x_{i} -\\mu _{i}}{s_{i}}\n",
125 | "$$\n",
126 | "\n",
127 | "Where $\\mu_i$ is the average of all the values for feature $(i)$ and $s_i$ is the range of values $(max - min)$, or si is the standard deviation.\n",
128 | "\n",
129 | "Note that dividing by the range, or dividing by the standard deviation, give different results. The quizzes in this course use range - the programming exercises use standard deviation.\n",
130 | "\n",
131 | "For example, if $x_i$ is housing prices with range of 100 to 2000, with a mean value of 1000. Then, \n",
132 | "\n",
133 | "$$\\large\n",
134 | "x_{i} :=\\frac{price-1000}{1900}\n",
135 | "$$\n",
136 | "\n",
137 | "\n",
138 | "## Learning Rate\n",
139 | "**Debugging gradient descent**. Make a plot with number of iterations on the x-axis. Now plot the cost function, $J(\\theta)$ over the number of iterations of gradient descent. If $J(\\theta)$ ever increases, then you probably need to decrease $\\alpha$.\n",
140 | "\n",
141 | "**Automatic convergence test**. Declare convergence if $J(\\theta)$ decreases by less than E in one iteration, where E is some small value such as $10^{-3}$. However in practice it's difficult to choose this threshold value.\n",
142 | "\n",
143 | "It has been proven that if learning rate α is sufficiently small, then $J(\\theta)$ will decrease on every iteration.\n",
144 | "\n",
145 | "To summarize:\n",
146 | "- If $\\alpha$ is too **small**: slow convergence\n",
147 | "- If $\\alpha$ is too **large**: may not decrease on every iteration and thus may not converge\n",
148 | "\n",
149 | "\n",
150 | "## Features and Polynomial Regression\n",
151 | "We can improve our features and the form of our hypothesis function in a couple different ways.\n",
152 | "\n",
153 | "We can combine multiple features into one. For example, we can combine $x_1$ and $x_2$ and into a new feature $x_3$ by taking $x_1.x_2$.\n",
154 | "\n",
155 | "\n",
156 | "### Polynomial Regression\n",
157 | "Our hypothesis function need not be linear (a straight line) if that does not fit the data well.\n",
158 | "\n",
159 | "We can change the behavior or curve of our hypothesis function by making it a quadratic, cubic or square root function (or any other form).\n",
160 | "\n",
161 | "For example, if our hypothesis function is $h_\\theta(x)=\\theta_0+\\theta_1x_1$ then we can create additional features based on x1, to get the quadratic function $h_\\theta(x)=\\theta_0+\\theta_1x_1+\\theta_2x^{2}_1$ or the cubic function $h_\\theta(x)=\\theta_0+\\theta_1x_1+\\theta_2x^{2}_1+\\theta_3x^{3}_1$\n",
162 | "\n",
163 | "In the cubic version, we have created new features $x_2$ and $x_3$ where $x_2 = x^2_1$ and $x_3=x^3_1$.\n",
164 | "\n",
165 | "To make it a square root function, we could do: $h_{\\theta }( x) =\\theta _{0} +\\theta _{1} x_{1} +\\theta _{2}\\sqrt{x_{1}}$\n",
166 | "\n",
167 | "\n",
168 | "## Normal Equation\n",
169 | "The “Normal Equation” is a method of finding the optimum theta without iteration.\n",
170 | "\n",
171 | "$$\\large\n",
172 | "\\theta =\\left( X^{T} X\\right)^{-1} X^{T} y\n",
173 | "$$\n",
174 | "\n",
175 | "There is **no need** to do feature scaling with the normal equation.\n",
176 | "\n",
177 | "Mathematical proof of the Normal equation requires knowledge of linear algebra and is fairly involved, so you do not need to worry about the details.\n",
178 | "\n",
179 | "The following is a comparison of gradient descent and the normal equation:\n",
180 | "\n",
181 | "| **Gradient Descent** | **Normal Equation** |\n",
182 | "| -------------------- | ------------------- |\n",
183 | "| Need to choose alpha | No need to choose alpha |\n",
184 | "| Needs many iterations | No need to iterate |\n",
185 | "| $O(kn^2)$ | $O(n^3)$, need to calculate inverse of $\\left( X^{T} X\\right)$ |\n",
186 | "| Works well when $n$ is large | Slow if $n$ is very large |\n",
187 | "\n",
188 | "With the normal equation, computing the inversion has complexity $O\\left( n^{3}\\right)$. So if we have a very large number of features, the normal equation will be slow. In practice, when $n$ exceeds 10,000 it might be a good time to go from a normal solution to an iterative process.\n",
189 | "\n",
190 | "\n",
191 | "## Normal Equation Noninvertibility\n",
192 | "When implementing the normal equation in octave we want to use the ‘pinv’ function rather than ‘inv.’\n",
193 | "\n",
194 | "$X^TX$ may be noninvertible. The common causes are:\n",
195 | "- Redundant features, where two features are very closely related (i.e. they are linearly dependent)\n",
196 | "- Too many features (e.g. $m\\leqslant n$). In this case, delete some features or use “regularization” (to be explained in a later lesson).\n",
197 | "\n",
198 | "Solutions to the above problems include deleting a feature that is linearly dependent with another or deleting one or more features when there are too many features.\n",
199 | "\n",
200 | "Here's a great article on [Vectorization Implementation in Machine Learning using NumPy](https://towardsdatascience.com/vectorization-implementation-in-machine-learning-ca652920c55d)\n",
201 | "\n"
202 | ]
203 | }
204 | ],
205 | "metadata": {
206 | "kernelspec": {
207 | "display_name": "Python 3",
208 | "language": "python",
209 | "name": "python3"
210 | },
211 | "language_info": {
212 | "codemirror_mode": {
213 | "name": "ipython",
214 | "version": 3
215 | },
216 | "file_extension": ".py",
217 | "mimetype": "text/x-python",
218 | "name": "python",
219 | "nbconvert_exporter": "python",
220 | "pygments_lexer": "ipython3",
221 | "version": "3.8.8"
222 | }
223 | },
224 | "nbformat": 4,
225 | "nbformat_minor": 5
226 | }
227 |
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/README.md:
--------------------------------------------------------------------------------
1 | # Week 3
2 | ## Logistic Regression
3 | This week, we’ll be covering logistic regression. Logistic regression is a method for classifying data into discrete outcomes. For example, we might use logistic regression to classify an email as spam or not spam. In this module, we introduce the notion of classification, the cost function for logistic regression, and the application of logistic regression to multi-class classification.
4 |
5 | We are also covering regularization. Machine learning models need to generalize well to new examples that the model has not seen in practice. We’ll introduce regularization, which helps prevent models from overfitting the training data.
6 |
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week 3 - Logistic Regression.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "id": "88942d76-5ae0-4a09-aee4-908ea6e69101",
6 | "metadata": {},
7 | "source": [
8 | "# Machine Learning: Week 3 - Logistic Regression\n",
9 | "## Logistic Regression\n",
10 | "Now we are switching from regression problems to classification problems. Don’t be confused by the name “Logistic Regression”; it is named that way for historical reasons and is actually an approach to classification problems, not regression problems.\n",
11 | "\n",
12 | "## Classification\n",
13 | "Instead of our output vector y being a continuous range of values, it will only be 0 or 1.\n",
14 | "\n",
15 | "$$\\large\n",
16 | "y\\in \\{0,1\\}\n",
17 | "$$\n",
18 | "\n",
19 | "Where $0$ is usually taken as the **“negative class”** and $1$ as the **“positive class”**, but you are free to assign any representation to it.\n",
20 | "\n",
21 | "We’re only doing two classes for now, called a _**“Binary Classification Problem”**_.\n",
22 | "\n",
23 | "One method is to use linear regression and map all predictions greater than 0.5 as a 1 and all less than 0.5 as a 0. This method doesn’t work well because classification is not actually a linear function.\n",
24 | "\n",
25 | "For instance, if we are trying to build a spam classifier for email, then $x^{(i)}$ may be some features of a piece of email, and $y$ may be $1$ if it is a piece of spam mail, and $0$ otherwise. Hence, $y\\in \\{0,1\\}$. $0$ is called the negative class, and 1 the positive class, and they are sometimes also denoted by the symbols \"−” and “+” Given $x^{(i)}$, the corresponding $y^{(i)}$ is also called the label for the training example.\n",
26 | "\n",
27 | "\n",
28 | "## Hypothesis Representation\n",
29 | "Our hypothesis should satisfy:\n",
30 | "\n",
31 | "$$\\large\n",
32 | "0\\leqslant h_\\theta(x) \\leqslant 1\n",
33 | "$$\n",
34 | "\n",
35 | "Our new form uses the **“Sigmoid Function”**, also called the **“Logistic Function”:**\n",
36 | "\n",
37 | "\n",
38 | "\n",
39 | "The function $g(z)$, shown here, maps any real number to the $(0, 1)$ interval, making it useful for transforming an arbitrary-valued function into a function better suited for classification. Try playing with an interactive plot of sigmoid function: (https://www.desmos.com/calculator/bgontvxotm).\n",
40 | "\n",
41 | "We start with our old hypothesis (linear regression), except that we want to restrict the range to $0$ and $1$. This is accomplished by plugging $\\theta^Tx$ into the Logistic Function.\n",
42 | "\n",
43 | "$h_\\theta(x)$ will give us the probability that our output is $1$. For example, $h_\\theta(x)=0.7$ gives us the probability of $70\\%$ that our output is $1$.\n",
44 | "\n",
45 | "$$\\large\n",
46 | " \\begin{array}{l}\n",
47 | "h_{\\theta }( x) =P( y=1|x;\\theta ) =1-P( y=0|x;\\theta )\\\\\n",
48 | "P( y=0|x;\\theta ) +P( y=1|x;\\theta ) =1\n",
49 | "\\end{array}\n",
50 | "$$\n",
51 | "\n",
52 | "\n",
53 | "## Decision Boundary\n",
54 | "In order to get our discrete 0 or 1 classification, we can translate the output of the hypothesis function as follows:\n",
55 | "\n",
56 | "$$\\large\n",
57 | " \\begin{array}{l}\n",
58 | "h_{\\theta }( x) \\geqslant 0.5\\rightarrow y=1\\\\\n",
59 | "h_{\\theta }( x) \\ < 0.5\\rightarrow y=0\n",
60 | "\\end{array}\n",
61 | "$$\n",
62 | "\n",
63 | "The way our logistic function g behaves is that when its input is greater than or equal to zero, its output is greater than or equal to 0.5:\n",
64 | "\n",
65 | "$$\\large\n",
66 | " \\begin{array}{l}\n",
67 | "g( z) \\geqslant 0.5\\\\\n",
68 | "when\\ z\\geqslant 0\n",
69 | "\\end{array}\n",
70 | "$$\n",
71 | "\n",
72 | "Remember:\n",
73 | "\n",
74 | "$$\\large\n",
75 | "\\begin{array}{l}\n",
76 | "z=0,\\ e^{0} =1\\Longrightarrow g( z) =\\frac{1}{2}\\\\\n",
77 | "\\\\\n",
78 | "z\\rightarrow \\infty ,e^{-\\infty }\\rightarrow 0\\Longrightarrow g( z) =1\\\\\n",
79 | "\\\\\n",
80 | "z\\rightarrow -\\infty ,\\ e^{\\infty }\\rightarrow \\infty \\Longrightarrow g( z) =0\n",
81 | "\\end{array}\n",
82 | "$$\n",
83 | "\n",
84 | "So if our input to $g$ is $\\theta^TX$, then that means:\n",
85 | "\n",
86 | "$$\\large\n",
87 | " \\begin{array}{l}\n",
88 | "h_{\\theta }( x) \\ =\\ g\\left( \\theta ^{T} X\\right) \\geqslant 0.5\\\\\n",
89 | "when\\ \\ \\theta ^{T} X\\geqslant 0\n",
90 | "\\end{array}\n",
91 | "$$\n",
92 | "\n",
93 | "From these statements we can now say:\n",
94 | "\n",
95 | "$$\\large\n",
96 | " \\begin{array}{l}\n",
97 | "\\theta ^{T} X\\geqslant 0\\Longrightarrow y=1\\ \\\\\n",
98 | "\\theta ^{T} X< 0\\Longrightarrow y=0\n",
99 | "\\end{array}\n",
100 | "$$\n",
101 | "\n",
102 | "The decision boundary is the line that separates the area where $y = 0$ and where $y = 1$. It is created by our hypothesis function.\n",
103 | "\n",
104 | "Example:\n",
105 | "\n",
106 | "$$\\large\n",
107 | "\\theta =\\begin{bmatrix}\n",
108 | "5\\\\\n",
109 | "-1\\\\\n",
110 | "0\n",
111 | "\\end{bmatrix}\n",
112 | "$$\n",
113 | "
\n",
114 | "$$\\large\n",
115 | " \\begin{array}{l}\n",
116 | "h_{\\theta }( x) =\\theta _{0} +\\theta _{1} x_{1} +\\theta _{2} x_{2} +...+\\theta _{n} x_{n}\\\\\n",
117 | "\\\\\n",
118 | "y=1\\ \\ if\\ \\ 5+( -1) x_{1} +0x_{2} \\geqslant 0\\\\\n",
119 | "5-x_{1} \\geqslant 0\\\\\n",
120 | "-x_{1} \\geqslant -5\\\\\n",
121 | "x_{1} \\leqslant 5\n",
122 | "\\end{array}\n",
123 | "$$\n",
124 | "\n",
125 | "In this case, our decision boundary is a straight vertical line placed on the graph where $x1 = 5$, and everything to the left of that denotes $y = 1$, while everything to the right denotes $y = 0$.\n",
126 | "\n",
127 | "Again, the input to the sigmoid function $g(z)$ (e.g. $\\theta^TX$) doesn’t need to be linear, and could be a function that describes a circle (e.g. $z=\\theta_0+\\theta_1x^{2}_1+\\theta_2x^{2}_2$) or any shape to fit our data.\n",
128 | "\n",
129 | "\n",
130 | "## Cost Function\n",
131 | "We cannot use the same cost function that we use for linear regression because the Logistic Function will cause the output to be wavy, causing many local optima. In other words, it will not be a convex function.\n",
132 | "\n",
133 | "Instead, our cost function for logistic regression looks like:\n",
134 | "\n",
135 | "$$\\large\n",
136 | " \\begin{array}{l}\n",
137 | "J( \\theta ) =\\frac{1}{m}\\sum _{i=1}^{m} Cost\\left( h_{\\theta }\\left( x^{( i)}\\right) ,y^{( i)}\\right)\\\\\n",
138 | "Cost( h_{\\theta }( x) ,y) =-log( h_{\\theta }( x)) \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ if\\ \\ y=1\\\\\n",
139 | "Cost( h_{\\theta }( x) ,y) =-log( 1-h_{\\theta }( x)) \\ \\ \\ \\ \\ \\ \\ \\ \\ if\\ \\ y=0\n",
140 | "\\end{array}\n",
141 | "$$\n",
142 | "\n",
143 | "\n",
144 | "\n",
145 | "Similarly, when $y = 0$, we get the following plot for $J(\\theta)$ vs $h_\\theta (x)$\n",
146 | "\n",
147 | "\n",
148 | "\n",
149 | "$$\\large\n",
150 | "\\begin{array}{l}\n",
151 | "Cost( h_{\\theta }( x) ,y) =0\\ \\ if\\ \\ h_{\\theta }( x) =y\\\\\n",
152 | "Cost( h_{\\theta }( x) ,y)\\rightarrow \\infty \\ \\ if\\ \\ y=0\\ \\ and\\ \\ h_{\\theta }( x)\\rightarrow 1\\\\\n",
153 | "Cost( h_{\\theta }( x) ,y)\\rightarrow \\infty \\ \\ if\\ \\ y=1\\ \\ and\\ \\ h_{\\theta }( x)\\rightarrow 0\n",
154 | "\\end{array}\n",
155 | "$$\n",
156 | "\n",
157 | "If our correct answer $y$ is $0$, then the cost function will be $0$ if our hypothesis function also outputs $0$. If our hypothesis approaches $1$, then the cost function will approach infinity.\n",
158 | "\n",
159 | "If our correct answer $y$ is $1$, then the cost function will be $0$ if our hypothesis function outputs $1$. If our hypothesis approaches $0$, then the cost function will approach infinity.\n",
160 | "\n",
161 | "Note that writing the cost function in this way guarantees that $J(\\theta)$ is convex for logistic regression.\n",
162 | "\n",
163 | "\n",
164 | "## Simplified Cost Function and Gradient Descent\n",
165 | "We can compress our cost function's two conditional cases into one case:\n",
166 | "\n",
167 | "$$\\large\n",
168 | "Cost( h_{\\theta }( x) ,y) =-y\\ log( h_{\\theta }( x)) -( 1-y) log( 1-h_{\\theta }( x))\n",
169 | "$$\n",
170 | "\n",
171 | "Notice that when y is equal to 1, then the second term $( 1-y) log( 1-h_{\\theta }( x))$ will be zero and will not affect the result. If y is equal to 0, then the first term $-y\\ log( h_{\\theta }( x))$ will be zero and will not affect the result.\n",
172 | "\n",
173 | "$$\\large\n",
174 | "J( \\theta ) =-\\frac{1}{m}\\sum _{i=1}^{m}\\left[ y^{( i)} log\\left( h_{\\theta }\\left( x^{( i)}\\right)\\right) +\\left( 1-y^{( i)}\\right) log\\left( 1-h_{\\theta }\\left( x^{( i)}\\right)\\right)\\right]\n",
175 | "$$\n",
176 | "\n",
177 | "A vectorized implementation is:\n",
178 | "\n",
179 | "$$\\large\n",
180 | "h=g( X\\theta )\n",
181 | "$$\n",
182 | "
\n",
183 | "$$\\large\n",
184 | "J( \\theta ) =\\frac{1}{m}\\left( -y^{T} log( h) -( 1-y)^{T} log( 1-h)\\right)\n",
185 | "$$\n",
186 | "\n",
187 | "### Gradient Descent\n",
188 | "Remember that the general form of gradient descent is:\n",
189 | "\n",
190 | "$$\\large\n",
191 | " \\begin{array}{l}\n",
192 | "Repeat\\ \\{\\\\\n",
193 | "\\theta _{j} :=\\theta _{j} -\\alpha \\frac{\\partial }{\\partial \\theta _{j}} J( \\theta )\\\\\n",
194 | "\\}\n",
195 | "\\end{array}\n",
196 | "$$\n",
197 | "\n",
198 | "We can work out the derivative part using calculus to get:\n",
199 | "\n",
200 | "$$\\large\n",
201 | " \\begin{array}{l}\n",
202 | "Repeat\\ \\{\\\\\n",
203 | "\\theta _{j} :=\\theta _{j} -\\alpha \\sum _{i=1}^{m}\\left( h_{\\theta }\\left( x^{( i)}\\right) -y^{( i)}\\right) x_{j}^{( i)}\\\\\n",
204 | "\\}\n",
205 | "\\end{array}\n",
206 | "$$\n",
207 | "\n",
208 | "Notice that this algorithm is identical to the one we used in linear regression. We still have to simultaneously update all values in theta.\n",
209 | "\n",
210 | "A vectorized implementation is:\n",
211 | "\n",
212 | "$$\\large\n",
213 | "\\theta :=\\theta -\\frac{\\alpha }{m} X^{T}( g( X\\theta ) -\\vec{y})\n",
214 | "$$\n",
215 | "\n",
216 | "\n",
217 | "## Multiclass Classification: One-vs-all\n",
218 | "Now we will approach the classification of data when we have more than two categories. Instead of $y = \\{0,1\\}$ we will expand our definition so that $y = \\{0,1....n\\}$.\n",
219 | "\n",
220 | "Since $y = \\{0,1....n\\}$, we divide our problem into $n+1$ (+1 because the index starts at 0) binary classification problems; in each one, we predict the probability that 'y' is a member of one of our classes.\n",
221 | "\n",
222 | "$$\\large\n",
223 | "\\begin{array}{l}\n",
224 | "y\\in \\{0,1...n\\}\\\\\n",
225 | "h_{\\theta }^{( 0)}( x) =P( y=0|x;\\theta )\\\\\n",
226 | "h_{\\theta }^{( 1)}( x) =P( y=1|x;\\theta )\\\\\n",
227 | "...\\\\\n",
228 | "h_{\\theta }^{( n)}( x) =P( y=n|x;\\theta )\\\\\n",
229 | "prediction=\\underset{i}{\\max}\\left( h_{\\theta }^{( i)}( x)\\right)\n",
230 | "\\end{array}\n",
231 | "$$\n",
232 | "\n",
233 | "We are basically choosing one class and then lumping all the others into a single second class. We do this repeatedly, applying binary logistic regression to each case, and then use the hypothesis that returned the highest value as our prediction.\n",
234 | "\n",
235 | "The following image shows how one could classify 3 classes:\n",
236 | "\n",
237 | "\n",
238 | "\n",
239 | "**To summarize:** \n",
240 | "- Train a logistic regression classifier $h_\\theta(x)$ for each class to predict the probability that $y=i$. \n",
241 | "- To make a prediction on a new $x$, pick the class that maximizes $h_\\theta (x)$."
242 | ]
243 | },
244 | {
245 | "cell_type": "markdown",
246 | "id": "f64769e7-7cd2-4c72-8173-86296b5f162d",
247 | "metadata": {},
248 | "source": [
249 | "# Regularization\n",
250 | "## The Problem of Overfitting\n",
251 | "Consider the problem of predicting $y$ from $x\\in \\mathbb{R}$. The leftmost figure below shows the result of fitting a $y=\\theta_0+\\theta_1x$ to a dataset. We see that the data doesn’t really lie on straight line, and so the fit is not very good.\n",
252 | "\n",
253 | "\n",
254 | "\n",
255 | "Instead, if we had added an extra feature $x^2$, and fit $y=\\theta_0+\\theta_1x+\\theta_2x^2$, then we obtain a slightly better fit to the data (See middle figure). Naively, it might seem that the more features we add, the better. However, there is also a danger in adding too many features: The rightmost figure is the result of fitting a $5^{th}$ order polynomial $y=\\sum _{j=0}^{5} \\theta _{j} x^{j}$. We see that even though the fitted curve passes through the data perfectly, we would not expect this to be a very good predictor of, say, housing prices (y) for different living areas (x). Without formally defining what these terms mean, we’ll say the figure on the left shows an instance of **underfitting** — in which the data clearly shows structure not captured by the model—and the figure on the right is an example of **overfitting**.\n",
256 | "\n",
257 | "__*Underfitting*__, or high bias, is when the form of our hypothesis function h maps poorly to the trend of the data. It is usually caused by a function that is too simple or uses too few features. At the other extreme, _**overfitting**_, or high variance, is caused by a hypothesis function that fits the available data but does not generalize well to predict new data. It is usually caused by a complicated function that creates a lot of unnecessary curves and angles unrelated to the data.\n",
258 | "\n",
259 | "This terminology is applied to both linear and logistic regression. There are two main options to address the issue of overfitting:\n",
260 | "\n",
261 | "1. Reduce the number of features:\n",
262 | " - Manually select which features to keep.\n",
263 | " - Use a model selection algorithm (studied later in the course).\n",
264 | "\n",
265 | "2. Regularization\n",
266 | " - Keep all the features, but reduce the parameters $\\theta_j$.\n",
267 | " - Regularization works well when we have a lot of slightly useful features.\n",
268 | "\n",
269 | "\n",
270 | "## Cost Function\n",
271 | "If we have overfitting from our hypothesis function, we can reduce the weight that some of the terms in our function carry by increasing their cost.\n",
272 | "\n",
273 | "Say we wanted to make the following function more quadratic:\n",
274 | "\n",
275 | "$$\\large\n",
276 | "\\theta_0 + \\theta_1x_1 + \\theta_2x^2 + \\theta_3x^3 + \\theta_4x^4\n",
277 | "$$\n",
278 | "\n",
279 | "We’ll want to eliminate the influence of $\\theta_3x^3$ and $\\theta_4x^4$. Without actually getting rid of these features or changing the form of our hypothesis, we can instead modify our **cost function**:\n",
280 | "\n",
281 | "$$\\large\n",
282 | "min_{\\theta }\\frac{1}{2m}\\sum _{i=1}^{m}\\left( h_{\\theta }\\left( x^{( i)}\\right) -y^{( i)}\\right)^{2} +1000\\times \\theta _{3}^{2} +1000\\times \\theta _{4}^{2}\n",
283 | "$$\n",
284 | "\n",
285 | "We’ve added two extra terms at the end to inflate the cost of $\\theta_3$ and $\\theta_4$. Now, in order for the cost function to get close to zero, we will have to reduce the values of $\\theta_3$ and $\\theta_4$ to near zero. This will in turn greatly reduce the values of $\\theta_3x^3$ and $\\theta_4x^4$ in our hypothesis function. As a result, we see that the new hypothesis (depicted by the pink curve) looks like a quadratic function but fits the data better due to the extra small terms $\\theta_3$ and $\\theta_4$.\n",
286 | "\n",
287 | "\n",
288 | "\n",
289 | "We could also regularize all of our theta parameters in a single summation as:\n",
290 | "\n",
291 | "$$\\large\n",
292 | "min_{\\theta }\\frac{1}{2m}\\left[\\sum _{i=1}^{m}\\left( h_{\\theta }\\left( x^{( i)}\\right) -y^{( i)}\\right)^{2} +\\lambda \\sum _{j=1}^{n} \\theta _{j}^{2}\\right]\n",
293 | "$$\n",
294 | "\n",
295 | "The $\\lambda$, or lambda, is the __regularization parameter__. It determines how much the costs of our theta parameters are inflated. You can visualize the effect of regularization in this interactive plot : https://www.desmos.com/calculator/1hexc8ntqp\n",
296 | "\n",
297 | "Using the above cost function with the extra summation, we can smooth the output of our hypothesis function to reduce overfitting. If lambda is chosen to be too large, it may smooth out the function too much and cause underfitting.\n",
298 | "\n",
299 | "\n",
300 | "## Regularized Linear Regression\n",
301 | "We can apply regularization to both linear regression and logistic regression. We will approach linear regression first.\n",
302 | "\n",
303 | "### Gradient Descent\n",
304 | "
We will modify our gradient descent function to separate out $\\theta_0$ from the rest of the parameters because we do not want to penalize $\\theta_0$.\n",
305 | "\n",
306 | "$$\\large\n",
307 | " \\begin{array}{l}\n",
308 | "Repeat\\ \\{\\\\\n",
309 | "\\theta _{0} :=\\theta _{0} -\\alpha \\frac{1}{m}\\sum _{i=1}^{m}\\left( h_{\\theta }\\left( x^{( i)}\\right) -y^{( i)}\\right) x_{0}^{( i)}\\\\\n",
310 | "\\theta _{j} :=\\theta _{j} -\\alpha \\left[\\left(\\frac{1}{m}\\sum _{i=1}^{m}\\left( h_{\\theta }\\left( x^{( i)}\\right) -y^{( i)}\\right) x_{j}^{( i)}\\right) +\\frac{\\lambda }{m} \\theta _{j}\\right] \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ j\\in \\{1,2...n\\}\\\\\n",
311 | "\\}\n",
312 | "\\end{array}\n",
313 | "$$\n",
314 | "\n",
315 | "The term $\\frac{\\lambda }{m} \\theta _{j}$ performs our regularization. With some manipulation our update rule can also be represented as:\n",
316 | "\n",
317 | "$$\\large\n",
318 | "\\theta _{j} :=\\theta _{j}\\left( 1-\\alpha \\frac{\\lambda }{m}\\right) -\\alpha \\frac{1}{m}\\sum _{i=1}^{m}\\left( h_{\\theta }\\left( x^{( i)}\\right) -y^{( i)}\\right) x_{j}^{( i)}\n",
319 | "$$\n",
320 | "\n",
321 | "The first term in the above equation, $1-\\frac{\\lambda }{m} \\theta _{j}$ will always be less than $1$. Intuitively you can see it as reducing the value of $\\theta_j$ by some amount on every update.\n",
322 | "\n",
323 | "Notice that the second term is now exactly the same as it was before.\n",
324 | "\n",
325 | "### Normal Equation\n",
326 | "Now let's approach regularization using the alternate method of the non-iterative normal equation.\n",
327 | "\n",
328 | "To add in regularization, the equation is the same as our original, except that we add another term inside the parentheses:\n",
329 | "\n",
330 | "$$\\large\n",
331 | "\\theta =(X^{T}X+\\lambda.{L})^{-1} X^{T}y\n",
332 | "$$\n",
333 | "
\n",
334 | "$$\n",
335 | "where\\ \\ L=\\begin{bmatrix}\n",
336 | "0 & & & & \\\\\n",
337 | " & 1 & & & \\\\\n",
338 | " & & 1 & & \\\\\n",
339 | " & & & ... & \\\\\n",
340 | " & & & & 1\n",
341 | "\\end{bmatrix}\n",
342 | "$$\n",
343 | "\n",
344 | "$L$ is a matrix with $0$ at the top left and $1$’s down the diagonal, with $0$’s everywhere else. It should have dimension $(n+1)\\times(n+1)$. Intuitively, this is the identity matrix (though we are not including $x_0$), multiplied with a single real number $\\lambda$.\n",
345 | "\n",
346 | "Recall that if $m\\leqslant n$, then $X^TX$ is non-invertible. However, when we add the term $\\lambda.L$, then $X^TX + \\lambda.L$ becomes invertible.\n",
347 | "\n",
348 | "\n",
349 | "## Regularized Logistic Regression\n",
350 | "We can regularize logistic regression in a similar way that we regularize linear regression. As a result, we can avoid overfitting. The following image shows how the regularized function, displayed by the pink line, is less likely to overfit than the non-regularized function represented by the blue line: \n",
351 | "\n",
352 | "\n",
353 | "\n",
354 | "### Cost Function\n",
355 | "\n",
356 | "Recall that our cost function for logistic regression was:\n",
357 | "\n",
358 | "$$\\large\n",
359 | "J( \\theta ) =-\\frac{1}{m}\\sum _{i=1}^{m}\\left[ y^{( i)} log\\left( h_{\\theta }\\left( x^{( i)}\\right)\\right) +\\left( 1-y^{( i)}\\right) log\\left( 1-h_{\\theta }\\left( x^{( i)}\\right)\\right)\\right]\n",
360 | "$$\n",
361 | "\n",
362 | "We can regularize this equation by adding a term to the end:\n",
363 | "\n",
364 | "$$\\large\n",
365 | "J( \\theta ) =-\\frac{1}{m}\\sum _{i=1}^{m}\\left[ y^{( i)} log\\left( h_{\\theta }\\left( x^{( i)}\\right)\\right) +\\left( 1-y^{( i)}\\right) log\\left( 1-h_{\\theta }\\left( x^{( i)}\\right)\\right)\\right] +\\frac{\\lambda }{2m}\\sum _{j=1}^{n} \\theta _{j}^{2}\n",
366 | "$$\n",
367 | "\n",
368 | "The second sum, $\\sum _{j=1}^{n} \\theta _{j}^{2}$ **means to explicitly exclude** the bias term, $\\theta_0$. I.e. the $\\theta$ vector is indexed from 0 to n (holding n+1 values, $\\theta_0$ through $\\theta_n$, and this sum explicitly skips $\\theta_0$, by running from $1$ to $n$, skipping $0$. Thus, when computing the equation, we should continuously update the two following equations:\n",
369 | "\n",
370 | "\n",
371 | "\n",
372 | "### Gradient Descent\n",
373 | "Just like with linear regression, we will want to separately update $\\theta_0$ and the rest of the parameters because we do not want to regularize $\\theta_0$.\n",
374 | "\n",
375 | "$$\\large\n",
376 | " \\begin{array}{l}\n",
377 | "Repeat\\ \\{\\\\\n",
378 | "\\theta _{0} :=\\theta _{0} -\\alpha \\frac{1}{m}\\sum _{i=1}^{m}\\left( h_{\\theta }\\left( x^{( i)}\\right) -y^{( i)}\\right) x_{0}^{( i)}\\\\\n",
379 | "\\theta _{j} :=\\theta _{j} -\\alpha \\left[\\left(\\frac{1}{m}\\sum _{i=1}^{m}\\left( h_{\\theta }\\left( x^{( i)}\\right) -y^{( i)}\\right) x_{j}^{( i)}\\right) +\\frac{\\lambda }{m} \\theta _{j}\\right] \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ j\\in \\{1,2...n\\}\\\\\n",
380 | "\\}\n",
381 | "\\end{array}\n",
382 | "$$\n",
383 | "\n",
384 | "This is identical to the gradient descent function presented for linear regression."
385 | ]
386 | }
387 | ],
388 | "metadata": {
389 | "kernelspec": {
390 | "display_name": "Python 3",
391 | "language": "python",
392 | "name": "python3"
393 | },
394 | "language_info": {
395 | "codemirror_mode": {
396 | "name": "ipython",
397 | "version": 3
398 | },
399 | "file_extension": ".py",
400 | "mimetype": "text/x-python",
401 | "name": "python",
402 | "nbconvert_exporter": "python",
403 | "pygments_lexer": "ipython3",
404 | "version": "3.8.8"
405 | }
406 | },
407 | "nbformat": 4,
408 | "nbformat_minor": 5
409 | }
410 |
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week3_Images/SigmoidFunction.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week3_Images/SigmoidFunction.png
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week3_Images/cost1.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week3_Images/cost1.png
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week3_Images/cost2.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week3_Images/cost2.png
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week3_Images/multiclass1.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week3_Images/multiclass1.png
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week3_Images/overfit1.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week3_Images/overfit1.png
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week3_Images/overfit2.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week3_Images/overfit2.png
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week3_Images/overfit3.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week3_Images/overfit3.png
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week3_Images/reg_logistic_gradient_descent.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week3_Images/reg_logistic_gradient_descent.png
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week3_Images/reg_logistic_regression.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week3_Images/reg_logistic_regression.png
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week3_Images/regularization.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/Machine-Learning-by-Stanford/Week 3 - Logistic Regression/Week3_Images/regularization.png
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 4 - Neural Networks/README.md:
--------------------------------------------------------------------------------
1 | # Week 4
2 | ## Neural Networks
3 | This week, we are covering neural networks. Neural networks is a model inspired by how the brain works. It is widely used today in many applications: when your phone interprets and understand your voice commands, it is likely that a neural network is helping to understand your speech; when you cash a check, the machines that automatically read the digits also use neural networks.
4 |
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 4 - Neural Networks/Week 4 - Neural Networks.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "id": "cc2d59ce-e648-4538-8387-bfa27c5a8499",
6 | "metadata": {},
7 | "source": [
8 | "# Machine Learning: Week 4 - Neural Networks\n",
9 | "## Non-linear Hypothesis\n",
10 | "Performing linear regression with a complex set of data with many features is very unwieldy. Say you wanted to create a hypothesis from three (3) features that included all the quadratic terms:\n",
11 | "\n",
12 | "$$\\large\n",
13 | "g( \\theta _{0} +\\theta _{1} x_{1}^{2} +\\theta _{2} x_{1} x_{2} +\\theta _{3} x_{1} x_{3} +\\theta _{4} x_{2}^{2} +_{\\theta }{}_{5} x_{2} x_{3} +\\theta _{6} x_{3}^{2})\n",
14 | "$$\n",
15 | "\n",
16 | "\n",
17 | "That gives us 6 features. The exact way to calculate how many features for all polynomial terms is the combination function with repetition: http://www.mathsisfun.com/combinatorics/combinations-permutations.html $\\frac{( n+r-1) !}{r!( n-1)!}$. In this case we are taking all two-element combinations of three features: $\\frac{( 3+2-1) !}{( 2!( 3-1) !)} =\\frac{4!}{4} =6$\n",
18 | "\n",
19 | "For 100 features, if we wanted to make them quadratic we would get $\\frac{( 100+2-1) !}{( 2!( 100-1) !)} =5050$\n",
20 | "\n",
21 | "We can approximate the growth of the number of new features we get with all quadratic terms with $O(n^2/2)$. And if you wanted to include all cubic terms in your hypothesis, the features would grow asymptotically at $On^3)$. These are very steep growths, so as the number of our features increase, the number of quadratic or cubic features increase very rapidly and becomes quickly impractical.\n",
22 | "\n",
23 | "Example: let our training set be a collection of $50 \\times 50$ pixel black-and-white photographs, and our goal will be to classify which ones are photos of cars. Our feature set size is then $n = 2500$ if we compare every pair of pixels.\n",
24 | "\n",
25 | "Now let’s say we need to make a quadratic hypothesis function. With quadratic features, our growth is $O(n^2/2)$. So our total features will be about $25002/2=3125000$, which is very impractical.\n",
26 | "\n",
27 | "Neural networks offers an alternate way to perform machine learning when we have complex hypotheses with many features.\n",
28 | "\n",
29 | "## Neurons and the Brain\n",
30 | "Neural networks are limited imitations of how our own brains work. They’ve had a big recent resurgence because of advances in computer hardware.\n",
31 | "\n",
32 | "There is evidence that the brain uses only one “learning algorithm” for all its different functions. Scientists have tried cutting (in an animal brain) the connection between the ears and the auditory cortex and rewiring the optical nerve with the auditory cortex to find that the auditory cortex literally learns to see.\n",
33 | "\n",
34 | "This principle is called “neuroplasticity” and has many examples and experimental evidence.\n",
35 | "\n",
36 | "## Model Representation I\n",
37 | "Let’s examine how we will represent a hypothesis function using neural networks.\n",
38 | "\n",
39 | "At a very simple level, neurons are basically computational units that take input (**dendrites**) as electrical input (called “spikes”) that are channeled to outputs (**axons**).\n",
40 | "\n",
41 | "In our model, our dendrites are like the input features $x_1 ... x_n$, and the output is the result of our hypothesis function. In this model our $x_0$ input node is sometimes called the “bias unit.” It is always equal to $1$.\n",
42 | "\n",
43 | "In neural networks, we use the same logistic function as in classification: $\\frac{1}{1+e^{-\\theta ^{T} x}}$. In neural networks however we sometimes call it a **sigmoid (logistic) activation function**. Our “theta” parameters are sometimes instead called _**“weights”**_ in the neural networks model.\n",
44 | "\n",
45 | "Visually, a simplistic representation looks like:\n",
46 | "\n",
47 | "$$\\large\n",
48 | "\\begin{bmatrix}\n",
49 | "x_{0}\\\\\n",
50 | "x_{1}\\\\\n",
51 | "x_{2}\n",
52 | "\\end{bmatrix}\\rightarrow [ \\ \\ \\ ] \\ \\rightarrow h_{\\theta }( x)\n",
53 | "$$\n",
54 | "\n",
55 | "Our input nodes (layer 1) go into another node (layer 2), and are output as the hypothesis function. The first layer is called the “input layer” and the final layer the “output layer,” which gives the final value computed on the hypothesis.\n",
56 | "\n",
57 | "We can have intermediate layers of nodes between the input and output layers called the “hidden layer.”\n",
58 | "\n",
59 | "We label these intermediate or “hidden” layer nodes $a_2^0 ... a_2^n$ and call them “activation units.”\n",
60 | "\n",
61 | "$$\\large\n",
62 | " \\begin{array}{l}\n",
63 | "a_{i}^{( j)} =activation\\ \\ of\\ \\ unit\\ \\ i\\ \\ in\\ \\ layer\\ \\ j\\\\\n",
64 | "\\Theta ^{( j)} =matrix\\ \\ of\\ \\ weights\\ \\ controlling\\ \\ function\\ \\ mapping\\ \\ from\\ \\ layer\\ \\ j\\ \\ to\\ \\ layer\\ \\ j+1\n",
65 | "\\end{array}\n",
66 | "$$\n",
67 | "\n",
68 | "If we had one hidden layer, it would look visually something like:\n",
69 | "\n",
70 | "$$\\large\n",
71 | "\\begin{bmatrix}\n",
72 | "x_{o}\\\\\n",
73 | "x_{1}\\\\\n",
74 | "x_{2}\\\\\n",
75 | "x_{3}\n",
76 | "\\end{bmatrix}\\rightarrow \\begin{bmatrix}\n",
77 | "a_{1}^{( 2)}\\\\\n",
78 | "a_{2}^{( 2)}\\\\\n",
79 | "a_{3}^{( 2)}\n",
80 | "\\end{bmatrix}\\rightarrow h_{\\theta }( x)\n",
81 | "$$\n",
82 | "\n",
83 | "The values for each of the “activation” nodes is obtained as follows:\n",
84 | "\n",
85 | "$$\\large\n",
86 | " \\begin{array}{l}\n",
87 | "a_{1}^{( 2)} =g\\left( \\Theta _{10}^{( 1)} x_{0} +\\Theta _{11}^{( 1)} x_{1} +\\Theta _{12}^{( 1)} x_{2} +\\Theta _{13}^{( 1)} x_{3}\\right)\\\\\n",
88 | "a_{2}^{( 2)} =g\\left( \\Theta _{20}^{( 1)} x_{0} +\\Theta _{21}^{( 1)} x_{1} +\\Theta _{22}^{( 1)} x_{2} +\\Theta _{23}^{( 1)} x_{3}\\right)\\\\\n",
89 | "a_{3}^{( 2)} =g\\left( \\Theta _{30}^{( 1)} x_{0} +\\Theta _{31}^{( 1)} x_{1} +\\Theta _{32}^{( 1)} x_{2} +\\Theta _{33}^{( 1)} x_{3}\\right)\\\\\n",
90 | "h_{\\Theta }( x) =a_{1}^{( 3)} =g\\left( \\Theta _{10}^{( 2)} a_{0}^{( 2)} +\\Theta _{11}^{( 2)} a_{1}^{( 2)} +\\Theta _{12}^{( 2)} a_{2}^{( 2)} +\\Theta _{13}^{( 2)} a_{3}^{( 2)}\\right)\n",
91 | "\\end{array}\n",
92 | "$$\n",
93 | "\n",
94 | "This is saying that we compute our activation nodes by using a $3\\times4$ matrix of parameters. We apply each row of the parameters to our inputs to obtain the value for one activation node. Our hypothesis output is the logistic function applied to the sum of the values of our activation nodes, which have been multiplied by yet another parameter matrix $\\Theta^{(2)}$ containing the weights for our second layer of nodes.\n",
95 | "\n",
96 | "Each layer gets its own matrix of weights, $\\Theta^{(j)}$.\n",
97 | "\n",
98 | "The dimensions of these matrices of weights is determined as follows:\n",
99 | "\n",
100 | "If network has $s_j$ units in layer $j$ and $s_{j+1}$ units in layer $j+1$, then $\\Theta^{(j)}$ will be of dimension $s_{j+1}\\times(s_j+1)$.\n",
101 | "\n",
102 | "The $+1$ comes from the addition in $\\Theta^{(j)}$ of the “bias nodes”, $x_0$ and $\\Theta^{(j)}_0$. In other words the output nodes will not include the bias nodes while the inputs will. The following image summarizes our model representation:\n",
103 | "\n",
104 | "\n",
105 | "\n",
106 | "Example: layer 1 has 2 input nodes and layer 2 has 4 activation nodes. Dimension of $\\Theta^{(1)}$ is going to be $4\\times3$ where $s_j=2$ and $s_{j+1}=4$, so $s_{j+1}\\times(s_j+1)=4\\times3$.\n",
107 | "\n",
108 | "## Model Representation II\n",
109 | "In this section we’ll do a vectorized implementation of the above functions. We’re going to define a new variable $z^{(j)}_k$ that encompasses the parameters inside our $g$ function. In our previous example if we replaced the variable $z$ for all the parameters we would get:\n",
110 | "\n",
111 | "$$\\large\n",
112 | " \\begin{array}{l}\n",
113 | "a_{1}^{( 2)} =g\\left( z_{1}^{( 2)}\\right)\\\\\n",
114 | "a_{2}^{( 2)} =g\\left( z_{2}^{( 2)}\\right)\\\\\n",
115 | "a_{3}^{( 2)} =g\\left( z_{3}^{( 2)}\\right)\n",
116 | "\\end{array}\n",
117 | "$$\n",
118 | "\n",
119 | "In other words, for layer $j=2$ and node $k$, the variable $z$ will be:\n",
120 | "\n",
121 | "$$\\large\n",
122 | "z_{k}^{( 2)} =\\Theta _{k,0}^{( 1)} x_{0} + \\Theta _{k,1}^{( 1)} x_{1} +...+\\Theta _{k,n}^{( 1)} x_{n}\n",
123 | "$$\n",
124 | "\n",
125 | "The vector representation of $x$ and $z^j$ is:\n",
126 | "\n",
127 | "$$\\large\n",
128 | "x=\\begin{bmatrix}\n",
129 | "x_{o}\\\\\n",
130 | "x_{1}\\\\\n",
131 | "...\\\\\n",
132 | "x_{n}\n",
133 | "\\end{bmatrix} \\ \\ \\ \\ \\ z^{( j)} =\\begin{bmatrix}\n",
134 | "z_{1}^{( j)}\\\\\n",
135 | "z_{2}^{( j)}\\\\\n",
136 | "...\\\\\n",
137 | "z_{n}^{( j)}\n",
138 | "\\end{bmatrix}\n",
139 | "$$\n",
140 | "\n",
141 | "Setting $x=a^{(1)}$, we can rewrite the equation as:\n",
142 | "\n",
143 | "$$\\large\n",
144 | "z^{( j)} =\\Theta ^{( j-1)} a^{( j-1)}\n",
145 | "$$\n",
146 | "\n",
147 | "We are multiplying our matrix $\\Theta^{(j-1)}$ with dimensions $s_j\\times(n+1)$ (where $s_j$ is the number of our activation nodes) by our vector $a^{(j-1)}$ with height $(n+1)$. This gives us our vector $z^{(j)}$ with height $s_j$.\n",
148 | "\n",
149 | "Now we can get a vector of our activation nodes for layer $j$ as follows:\n",
150 | "\n",
151 | "$$\\large\n",
152 | "a^{( j)} =g\\left( z^{( j)}\\right)\n",
153 | "$$\n",
154 | "\n",
155 | "Where our function $g$ can be applied element-wise to our vector $z^{(j)}$.\n",
156 | "\n",
157 | "We can then add a bias unit (equal to $1$) to layer $j$ after we have computed $a^{(j)}$. This will be element $a^{(j)}_0$ and will be equal to $1$.\n",
158 | "\n",
159 | "To compute our final hypothesis, let’s first compute another $z$ vector:\n",
160 | "\n",
161 | "$$\\large\n",
162 | "z^{(j+1)} =\\Theta ^{( j)} a^{( j)}\n",
163 | "$$\n",
164 | "\n",
165 | "We get this final z vector by multiplying the next theta matrix after $\\Theta^{(j-1)}$ with the values of all the activation nodes we just got. This last theta matrix $\\Theta^{(j)}$ will have only **one row** so that our result is a single number.\n",
166 | "\n",
167 | "We then get our final result with:\n",
168 | "$$\\large\n",
169 | "h_{\\Theta }( x) =a^{( j+1)} =g\\left( z^{( j+1)}\\right)\n",
170 | "$$\n",
171 | "\n",
172 | "Notice that in this **last step**, between layer $j$ and layer $j+1$, we are doing __exactly the same thing__ as we did in logistic regression.\n",
173 | "\n",
174 | "Adding all these intermediate layers in neural networks allows us to more elegantly produce interesting and more complex non-linear hypotheses.\n",
175 | "\n",
176 | "## Examples and Intuitions I\n",
177 | "A simple example of applying neural networks is by predicting $x_1$ _AND_ $x_2$, which is the logical ‘and’ operator and is only true if both $x_1$ and $x_2$ are $1$.\n",
178 | "\n",
179 | "The graph of our functions will look like:\n",
180 | "\n",
181 | "$$\\large\n",
182 | "\\begin{bmatrix}\n",
183 | "x_{o}\\\\\n",
184 | "x_{1}\\\\\n",
185 | "x_{2}\n",
186 | "\\end{bmatrix}\\rightarrow \\left[ g\\left( z^{( 2)}\\right)\\right]\\rightarrow h_{\\Theta }( x)\n",
187 | "$$\n",
188 | "\n",
189 | "Remember that $x_0$ is our bias variable and is always $1$.\n",
190 | "\n",
191 | "Let’s set our first theta matrix as:\n",
192 | "\n",
193 | "$$\\large\n",
194 | "\\Theta ^{( 1)} =[ -30\\ \\ \\ 20\\ \\ \\ 20]\n",
195 | "$$\n",
196 | "\n",
197 | "This will cause the output of our hypothesis to only be positive if both $x_1$ and $x_2$ are $1$. In other words:\n",
198 | "\n",
199 | "$$\\large\n",
200 | "h_{\\Theta }( x) =g( -30+20x_{1} +20x_{2})\n",
201 | "$$\n",
202 | "
\n",
203 | "$$\\large\n",
204 | " \\begin{array}{l}\n",
205 | "x_{1} =0\\ \\ and\\ \\ x_{2} =0\\ \\ then\\ \\ g( -30) \\approx 0\\\\\n",
206 | "x_{1} =0\\ \\ and\\ \\ x_{2} =1\\ \\ then\\ \\ g( -10) \\approx 0\\\\\n",
207 | "x_{1} =1\\ \\ and\\ \\ x_{2} =0\\ \\ then\\ \\ g( -10) \\approx 0\\\\\n",
208 | "x_{1} =1\\ \\ and\\ \\ x_{2} =1\\ \\ then\\ \\ g( 10) \\approx 1\n",
209 | "\\end{array}\n",
210 | "$$\n",
211 | "\n",
212 | "So we have constructed one of the fundamental operations in computers by using a small neural network rather than using an actual AND gate. Neural networks can also be used to simulate all the other logical gates. The following is an example of the logical operator 'OR', meaning either $x_1$ is true or $x_2$ is true, or both:\n",
213 | "\n",
214 | "\n",
215 | "\n",
216 | "## Examples and Intuitions II\n",
217 | "The $\\Theta^{(1)}$ matrices for AND, NOR, and OR are:\n",
218 | "\n",
219 | "$$\\large\n",
220 | " \\begin{array}{l}\n",
221 | "AND:\\\\\n",
222 | "\\ \\ \\ \\ \\ \\ \\ \\ \\Theta ^{( 1)} =[ -30\\ \\ \\ \\ \\ 20\\ \\ \\ \\ \\ 20]\\\\\n",
223 | "NOR:\\\\\n",
224 | "\\ \\ \\ \\ \\ \\ \\ \\ \\Theta ^{( 1)} =[ 10\\ \\ \\ \\ -20\\ \\ \\ \\ -20]\\\\\n",
225 | "OR:\\\\\n",
226 | "\\ \\ \\ \\ \\ \\ \\ \\ \\Theta ^{( 1)} =[ -10\\ \\ \\ \\ \\ 20\\ \\ \\ \\ \\ 20]\n",
227 | "\\end{array}\n",
228 | "$$\n",
229 | "\n",
230 | "We can combine these to get the XNOR logical operator (which gives 1 if $x_1$ and $x_2$ are both $0$ or both $1$).\n",
231 | "\n",
232 | "$$\\large\n",
233 | "\\begin{bmatrix}\n",
234 | "x_{o}\\\\\n",
235 | "x_{1}\\\\\n",
236 | "x_{2}\n",
237 | "\\end{bmatrix}\\rightarrow \\begin{bmatrix}\n",
238 | "a_{1}^{( 2)}\\\\\n",
239 | "a_{2}^{( 2)}\n",
240 | "\\end{bmatrix}\\rightarrow \\left[ a^{( 3)}\\right]\\rightarrow h_{\\Theta }( x)\n",
241 | "$$\n",
242 | "\n",
243 | "For the transition between the first and second layer, we’ll use a $\\Theta^{(1)}$ matrix that combines the values for AND and NOR:\n",
244 | "\n",
245 | "$$\\large\n",
246 | "\\Theta ^{( 1)} =\\begin{bmatrix}\n",
247 | "-30 & 20 & 20\\\\\n",
248 | "10 & -20 & -20\n",
249 | "\\end{bmatrix}\n",
250 | "$$\n",
251 | "\n",
252 | "For the transition between the second and third layer, we’ll use a $\\Theta^{(2)}$ matrix that uses the value for OR:\n",
253 | "\n",
254 | "$$\\large\n",
255 | "\\Theta ^{( 1)} =[ -10\\ \\ \\ \\ \\ 20\\ \\ \\ \\ \\ 20]\n",
256 | "$$\n",
257 | "\n",
258 | "Let’s write out the values for all our nodes:\n",
259 | "\n",
260 | "$$\\large\n",
261 | " \\begin{array}{l}\n",
262 | "a^{( 2)} =g\\left( \\Theta ^{( 1)} x\\right)\\\\\n",
263 | "a^{( 3)} =g\\left( \\Theta ^{( 2)} a^{( 2)}\\right)\\\\\n",
264 | "h_{\\Theta }( x) =a^{( 3)}\n",
265 | "\\end{array}\n",
266 | "$$\n",
267 | "\n",
268 | "And there we have the XNOR operator using a hidden layer with two nodes! The following summarizes the above algorithm:\n",
269 | "\n",
270 | "\n",
271 | "\n",
272 | "## Multiclass Classification\n",
273 | "To classify data into multiple classes, we let our hypothesis function return a vector of values. Say we wanted to classify our data into one of four categories. We will use the following example to see how this classification is done. This algorithm takes as input an image and classifies it accordingly: \n",
274 | "\n",
275 | "\n",
276 | "\n",
277 | "We can define our set of resulting classes as $y$:\n",
278 | "\n",
279 | "$$\\large\n",
280 | "y^{( i)} =\\begin{bmatrix}\n",
281 | "1\\\\\n",
282 | "0\\\\\n",
283 | "0\\\\\n",
284 | "0\n",
285 | "\\end{bmatrix} \\ ,\\ \\begin{bmatrix}\n",
286 | "0\\\\\n",
287 | "1\\\\\n",
288 | "0\\\\\n",
289 | "0\n",
290 | "\\end{bmatrix} \\ ,\\ \\begin{bmatrix}\n",
291 | "0\\\\\n",
292 | "0\\\\\n",
293 | "1\\\\\n",
294 | "0\n",
295 | "\\end{bmatrix} \\ ,\\ \\begin{bmatrix}\n",
296 | "0\\\\\n",
297 | "0\\\\\n",
298 | "0\\\\\n",
299 | "1\n",
300 | "\\end{bmatrix}\n",
301 | "$$\n",
302 | "\n",
303 | "Each $y^{(i)}$ represents a different image corresponding to either a car, pedestrian, truck, or motorcycle. The inner layers, each provide us with some new information which leads to our final hypothesis function. The setup looks like:\n",
304 | "\n",
305 | "$$\\large\n",
306 | "\\begin{bmatrix}\n",
307 | "x_{o}\\\\\n",
308 | "x_{1}\\\\\n",
309 | "x_{2}\\\\\n",
310 | "...\\\\\n",
311 | "x_{n}\n",
312 | "\\end{bmatrix}\\rightarrow \\begin{bmatrix}\n",
313 | "a_{0}^{( 2)}\\\\\n",
314 | "a_{1}^{( 2)}\\\\\n",
315 | "a_{2}^{( 2)}\\\\\n",
316 | "...\n",
317 | "\\end{bmatrix}\\rightarrow \\begin{bmatrix}\n",
318 | "a_{0}^{( 3)}\\\\\n",
319 | "a_{1}^{( 3)}\\\\\n",
320 | "a_{2}^{( 3)}\\\\\n",
321 | "...\n",
322 | "\\end{bmatrix}\\rightarrow ...\\rightarrow \\begin{bmatrix}\n",
323 | "h_{\\Theta }( x)_{1}\\\\\n",
324 | "h_{\\Theta }( x)_{2}\\\\\n",
325 | "h_{\\Theta }( x)_{3}\\\\\n",
326 | "h_{\\Theta }( x)_{4}\n",
327 | "\\end{bmatrix}\n",
328 | "$$\n",
329 | "\n",
330 | "Our resulting hypothesis for one set of inputs may look like:\n",
331 | "\n",
332 | "$$\\large\n",
333 | "h_{\\Theta }( x) =\\begin{bmatrix}\n",
334 | "0\\\\\n",
335 | "0\\\\\n",
336 | "1\\\\\n",
337 | "0\n",
338 | "\\end{bmatrix}\n",
339 | "$$\n",
340 | "\n",
341 | "In which case our resulting class is the third one down, or $h_\\Theta(x)_3$, which represents a **motorcycle**.\n"
342 | ]
343 | }
344 | ],
345 | "metadata": {
346 | "kernelspec": {
347 | "display_name": "Python 3",
348 | "language": "python",
349 | "name": "python3"
350 | },
351 | "language_info": {
352 | "codemirror_mode": {
353 | "name": "ipython",
354 | "version": 3
355 | },
356 | "file_extension": ".py",
357 | "mimetype": "text/x-python",
358 | "name": "python",
359 | "nbconvert_exporter": "python",
360 | "pygments_lexer": "ipython3",
361 | "version": "3.8.8"
362 | }
363 | },
364 | "nbformat": 4,
365 | "nbformat_minor": 5
366 | }
367 |
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 4 - Neural Networks/Week4_Images/XNOR.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/Machine-Learning-by-Stanford/Week 4 - Neural Networks/Week4_Images/XNOR.png
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 4 - Neural Networks/Week4_Images/multiclass_classification.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/Machine-Learning-by-Stanford/Week 4 - Neural Networks/Week4_Images/multiclass_classification.png
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 4 - Neural Networks/Week4_Images/neural_net.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/Machine-Learning-by-Stanford/Week 4 - Neural Networks/Week4_Images/neural_net.png
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 4 - Neural Networks/Week4_Images/neural_net_intuition.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/Machine-Learning-by-Stanford/Week 4 - Neural Networks/Week4_Images/neural_net_intuition.png
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 5 - Neural Networks: Learning/README.md:
--------------------------------------------------------------------------------
1 | # Week 5
2 | ## Neural Networks: Learning
3 | In Week 5, you will be learning how to train Neural Networks. The Neural Network is one of the most powerful learning algorithms (when a linear classifier doesn't work, this is what I usually turn to), and this week's videos explain the 'backpropagation' algorithm for training these models. In this week's programming assignment, you'll also get to implement this algorithm and see it work for yourself.
4 |
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 5 - Neural Networks: Learning/Week 5 - Neural Networks - Learning.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "id": "edbeb780-7cf6-4c02-a573-69b10047074d",
6 | "metadata": {},
7 | "source": [
8 | "# Machine Learning: Week 5 - Neural Networks: Learning\n",
9 | "## Cost Function\n",
10 | "Let's first define a few variables that we will need to use:\n",
11 | "- $L =$ total number of layers in the network\n",
12 | "- $s_l =$ number of units (not counting bias unit) in layer $l$\n",
13 | "- $K =$ number of output units/classes\n",
14 | "\n",
15 | "Recall that in neural networks, we may have many output nodes. We denote $h_\\Theta(x)_k$ as being a hypothesis that results in the $k^{th}$ output. Our cost function for neural networks is going to be a generalization of the one we used for logistic regression. Recall that the cost function for regularized logistic regression was:\n",
16 | "\n",
17 | "$$\\large\n",
18 | "J( \\theta ) =-\\frac{1}{m}\\sum _{i=1}^{m}\\left[ y^{( i)} log\\left( h_{\\theta }\\left( x^{( i)}\\right)\\right) +\\left( 1-y^{( i)}\\right) log\\left( 1-h_{\\theta }\\left( x^{( i)}\\right)\\right)\\right] +\\frac{\\lambda }{2m}\\sum _{j=1}^{n} \\theta _{j}^{2}\n",
19 | "$$\n",
20 | "\n",
21 | "For neural networks, it is going to be slightly more complicated:\n",
22 | "\n",
23 | "$$\\large\n",
24 | "J( \\Theta ) =-\\frac{1}{m}\\sum _{i=1}^{m}\\sum _{k=1}^{K}\\left[ y_{k}^{( i)} log\\left(\\left( h_{\\theta }\\left( x^{( i)}\\right)\\right)_{k}\\right) +\\left( 1-y_{k}^{( i)}\\right) log\\left( 1-\\left( h_{\\theta }\\left( x^{( i)}\\right)\\right)_{k}\\right)\\right] +\\frac{\\lambda }{2m}\\sum _{l=1}^{L-1}\\sum _{i=1}^{s_{l}}\\sum _{j=1}^{s_{l+1}}\\left( \\Theta _{j,i}^{l}\\right)^{2}\n",
25 | "$$\n",
26 | "\n",
27 | "We have added a few nested summations to account for our multiple output nodes. In the first part of the equation, before the square brackets, we have an additional nested summation that loops through the number of output nodes.\n",
28 | "\n",
29 | "In the regularization part, after the square brackets, we must account for multiple theta matrices. The number of columns in our current theta matrix is equal to the number of nodes in our current layer (including the bias unit). The number of rows in our current theta matrix is equal to the number of nodes in the next layer (excluding the bias unit). As before with logistic regression, we square every term.\n",
30 | "\n",
31 | "Note:\n",
32 | "- the double sum simply adds up the logistic regression costs calculated for each cell in the output layer\n",
33 | "- the triple sum simply adds up the squares of all the individual $\\Theta$s in the entire network.\n",
34 | "- the $i$ in the triple sum does not refer to training example $i$\n",
35 | "\n",
36 | "\n",
37 | "\n",
38 | "## Backpropagation Algorithm\n",
39 | "“Backpropagation” is neural-network terminology for minimizing our cost function, just like what we were doing with gradient descent in logistic and linear regression.\n",
40 | "\n",
41 | "Our goal is to compute:\n",
42 | "\n",
43 | "$$\\large\n",
44 | "min_\\Theta J(\\Theta)\n",
45 | "$$\n",
46 | "\n",
47 | "That is, we want to minimize our cost function J using an optimal set of parameters in theta. In this section we'll look at the equations we use to compute the partial derivative of $J(\\Theta($:\n",
48 | "\n",
49 | "$$\\large\n",
50 | "\\frac{\\partial }{\\partial \\Theta _{ij}^{( l)}} J( \\Theta )\n",
51 | "$$\n",
52 | "\n",
53 | "To do so, we use the following algorithm:\n",
54 | "\n",
55 | "\n",
56 | "\n",
57 | "
\n",
58 | "\n",
59 | "\n",
60 | "\n",
61 | "### Backpropagation Algorithm\n",
62 | "In back propagation we’re going to compute for every node:\n",
63 | "\n",
64 | "$$\\large\n",
65 | "\\delta^{(l)}_j = error\\ \\ of\\ \\ node\\ \\ j\\ \\ in\\ \\ layer\\ \\ l\n",
66 | "$$\n",
67 | "\n",
68 | "Recall that $a^{(l)}_j$ is activation node $j$ in layer $l$.\n",
69 | "\n",
70 | "For the last layer, we can compute the vector of delta values with:\n",
71 | "\n",
72 | "$$\\large\n",
73 | "\\delta^{(L)} = a^{(L)} - y^{(t)}\n",
74 | "$$\n",
75 | "\n",
76 | "Where L is our total number of layers and $a^{(L)}$ is the vector of outputs of the activation units for the last layer. So our “error values” for the last layer are simply the differences of our actual results in the last layer and the correct outputs in $y$.\n",
77 | "\n",
78 | "To get the delta values of the layers before the last layer, we can use an equation that steps us back from right to left:\n",
79 | "\n",
80 | "$$\\large\n",
81 | "g'( u) =g( u) .*( 1-g( u))\n",
82 | "$$\n",
83 | "\n",
84 | "The full back propagation equation for the inner nodes is then:\n",
85 | "\n",
86 | "$$\\large\n",
87 | "\\delta^{(l)} = ((\\Theta^{(l)})^T \\delta^{(l+1)})\\ .*\\ a^{(l)}\\ .*\\ (1 - a^{(l)})\n",
88 | "$$\n",
89 | "\n",
90 | "Andrew Ng states that the derivation and proofs are complicated and involved, but you can still implement the above equations to do back propagation without knowing the details.\n",
91 | "\n",
92 | "We can compute our partial derivative terms by multiplying our activation values and our error values for each training example $t$:\n",
93 | "\n",
94 | "$$\\large\n",
95 | "\\frac{\\partial J( \\Theta )}{\\partial \\Theta _{ij}^{( l)}} =\\frac{1}{m}\\sum _{t=1}^{m} a_{k}^{( t)( l)} \\delta ^{( t)( l-1)}\n",
96 | "$$\n",
97 | "\n",
98 | "Note: $\\delta^{(l+1)}$ and $a^{(l+1)}$ are vectors with $s^{(l+1)}$ elements. Similarly, $(a^{(l)})$ is a vector with sl elements. Multiplying them produces a matrix that is $s^{(l+1)}$ by $s^l$ which is the same dimension as $\\Theta^{(l)}$. That is, the process produces a gradient term for every element in $\\Theta^{(l)}$. (Actually, $\\Theta^{(l)}$ has $s^{(l + 1)}$ column, so the dimensionality is not exactly the same).\n",
99 | "\n",
100 | "We can now take all these equations and put them together into a backpropagation algorithm:\n",
101 | "\n",
102 | "\n",
103 | "### Backpropagation Algorithm\n",
104 | "Given training set ${(x(1),y(1))...(x(m),y(m))}$\n",
105 | "- Set $\\Delta^{(l)}_{i,j}:=0$ for all $(l,i,j)$\n",
106 | "\n",
107 | "For training example t =1 to m:\n",
108 | "- Set $a^{(1)}:=x^{(t)}$\n",
109 | "\n",
110 | "- Perform forward propagation to compute $a^{(l)}$ for $l=2,3,...,L$\n",
111 | "\n",
112 | "- Using $y^{(t)}$, compute $\\delta ^{( L)} =a^{( L)} -y^{( t)}$\n",
113 | "\n",
114 | "- Compute $\\delta ^{( L-1)} ,\\delta ^{( L-2)} ,...,\\delta ^{( 2)}$ using $\\delta ^{( l)} =\\left(\\left( \\Theta ^{( l)}\\right)^{T} \\delta ^{( l+1)}\\right) .*a^{( l)} .*( 1-a)$\n",
115 | "\n",
116 | "- $\\Delta^{(l)}_{i,j}:= \\Delta^{(l)}_{i,j} + a^{(l)}_j \\delta^{(l+1)}_i$ or with vectorization, $\\Delta^{(l)}:=\\Delta^{(l)}+\\delta^{(l+1)}(a^{(l)})T$\n",
117 | "\n",
118 | "- $D_{i,j}^{( l)} :=\\frac{1}{m}\\left( \\Delta _{i,j}^{( l)} +\\lambda \\Theta _{i,j}^{( l)}\\right) \\ \\ if\\ \\ j\\neq 0$\n",
119 | "\n",
120 | "- $D_{i,j}^{( l)} :=\\frac{1}{m} \\Delta _{i,j}^{( l)} \\ \\ if\\ \\ j=0$\n",
121 | "\n",
122 | "The capital-delta matrix is used as an “accumulator” to add up our values as we go along and eventually compute our partial derivative.\n",
123 | "\n",
124 | "The actual proof is quite involved, but, the $\\Delta^{(l)}_{i,j}$ terms are the partial derivatives and the results we are looking for:\n",
125 | "\n",
126 | "$$\\large\n",
127 | "D_{i,j}^{( l)} =\\frac{\\partial J( \\Theta )}{\\partial \\Theta _{i,j}^{( l)}}\n",
128 | "$$\n",
129 | "\n",
130 | "\n",
131 | "## Backpropagation Intuition\n",
132 | "Recall that the cost function for a neural network is:\n",
133 | "\n",
134 | "$$\\large\n",
135 | "J( \\Theta ) =-\\frac{1}{m}\\sum _{i=1}^{m}\\sum _{k=1}^{K}\\left[ y_{k}^{( i)} log\\left(\\left( h_{\\theta }\\left( x^{( i)}\\right)\\right)_{k}\\right) +\\left( 1-y_{k}^{( i)}\\right) log\\left( 1-\\left( h_{\\theta }\\left( x^{( i)}\\right)\\right)_{k}\\right)\\right] +\\frac{\\lambda }{2m}\\sum _{l=1}^{L-1}\\sum _{i=1}^{s_{l}}\\sum _{j=1}^{s_{l+1}}\\left( \\Theta _{j,i}^{l}\\right)^{2}\n",
136 | "$$\n",
137 | "\n",
138 | "If we consider simple non-multiclass classification $(k = 1)$ and disregard regularization, the cost is computed with:\n",
139 | "\n",
140 | "$$\\large\n",
141 | "cost( t) =y^{( t)} log\\left( h_{\\Theta }\\left( x^{( t)}\\right)\\right) +\\left( 1-y^{( t)}\\right) log\\left( 1-h_{\\Theta }\\left( x^{( t)}\\right)\\right)\n",
142 | "$$\n",
143 | "\n",
144 | "Intuitively, $\\delta_j^{(l)}$ is the \"error\" for $a^{(l)}_j$ (unit $j$ in layer $l$). More formally, the delta values are actually the derivative of the cost function:\n",
145 | "\n",
146 | "$$\\large\n",
147 | "\\delta _{j}^{( l)} =\\frac{\\partial }{\\partial z_{j}^{( l)}} cost( t)\n",
148 | "$$\n",
149 | "\n",
150 | "Recall that our derivative is the slope of a line tangent to the cost function, so the steeper the slope the more incorrect we are. Let us consider the following neural network below and see how we could calculate some $\\delta_j^{(l)}$:\n",
151 | "\n",
152 | "\n",
153 | "\n",
154 | "In the image above, to calculate $\\delta_2^{(2)}$, we multiply the weights $\\Theta^{(2)}_{12}$ and $\\Theta^{(2)}_{22}$ by their respective $\\delta$ values found to the right of each edge. So we get $\\delta _{2}^{( 2)} =\\Theta _{12}^{( 2)} *\\delta _{1}^{( 3)} +\\Theta _{22}^{( 2)} *\\delta _{2}^{( 3)}$. To calculate every single possible $\\delta_j^{(l)}$, we could start from the right of our diagram. We can think of our edges as our $\\Theta_{ij}$. Going from right to left, to calculate the value of $\\delta^{(l)}_j$, you can just take the over all sum of each weight times the $\\delta$ it is coming from. Hence, another example would be $\\delta _{2}^{( 3)} =\\Theta _{12}^{( 3)} *\\delta _{1}^{( 4)}$\n",
155 | "\n",
156 | "## Putting it Together\n",
157 | "First, pick a network architecture; choose the layout of your neural network, including how many hidden units in each layer and how many layers total.\n",
158 | "- Number of input units = dimension of features x(i)\n",
159 | "- Number of output units = number of classes\n",
160 | "- Number of hidden units per layer = usually more the better (must balance with cost of computation as it increases with more hidden units)\n",
161 | "- Defaults: 1 hidden layer. If more than 1 hidden layer, then the same number of units in every hidden layer.\n",
162 | "\n",
163 | "### Training a Neural Network\n",
164 | "1. Randomly initialize the weights\n",
165 | "2. Implement forward propagation to get hθ(x(i))\n",
166 | "3. Implement the cost function\n",
167 | "4. Implement backpropagation to compute partial derivatives\n",
168 | "5. Use gradient checking to confirm that your backpropagation works. Then disable gradient checking.\n",
169 | "6. Use gradient descent or a built-in optimization function to minimize the cost function with the weights in theta.\n",
170 | "\n",
171 | "When we perform forward and back propagation, we loop on every training example:"
172 | ]
173 | },
174 | {
175 | "cell_type": "markdown",
176 | "id": "91a8852a-92a9-4b85-83a4-1a3712094b49",
177 | "metadata": {},
178 | "source": [
179 | "```\n",
180 | "for i = 1:m,\n",
181 | " Perform forward propagation and backpropagation using example (x(i),y(i))\n",
182 | " (Get activations a(l) and delta terms d(l) for l = 2,...,L\n",
183 | "```"
184 | ]
185 | },
186 | {
187 | "cell_type": "markdown",
188 | "id": "59a81697-f417-4b94-b6a2-166cd02b09c3",
189 | "metadata": {},
190 | "source": [
191 | "## Implementation"
192 | ]
193 | },
194 | {
195 | "cell_type": "code",
196 | "execution_count": 1,
197 | "id": "ed9ca975-9e31-4fe3-9a13-f6efbec4469f",
198 | "metadata": {},
199 | "outputs": [],
200 | "source": [
201 | "import numpy as np"
202 | ]
203 | },
204 | {
205 | "cell_type": "code",
206 | "execution_count": 2,
207 | "id": "ea6c6f0a-2aae-4b6e-9acb-2f0c5bf7a93b",
208 | "metadata": {},
209 | "outputs": [],
210 | "source": [
211 | "theta1 = np.ones([10,11])\n",
212 | "theta2 = 2*(np.ones([10,11]))\n",
213 | "theta3 = 3*(np.ones([1,11]))"
214 | ]
215 | },
216 | {
217 | "cell_type": "code",
218 | "execution_count": 3,
219 | "id": "c6cfb938-0284-42a9-a0f1-3f3278d7a439",
220 | "metadata": {},
221 | "outputs": [
222 | {
223 | "data": {
224 | "text/plain": [
225 | "array([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
226 | " [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
227 | " [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
228 | " [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
229 | " [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
230 | " [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
231 | " [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
232 | " [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
233 | " [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
234 | " [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]])"
235 | ]
236 | },
237 | "execution_count": 3,
238 | "metadata": {},
239 | "output_type": "execute_result"
240 | }
241 | ],
242 | "source": [
243 | "theta1"
244 | ]
245 | },
246 | {
247 | "cell_type": "code",
248 | "execution_count": 4,
249 | "id": "85c42915-4638-44d9-9691-0e9e30f7f171",
250 | "metadata": {},
251 | "outputs": [
252 | {
253 | "data": {
254 | "text/plain": [
255 | "array([[2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],\n",
256 | " [2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],\n",
257 | " [2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],\n",
258 | " [2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],\n",
259 | " [2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],\n",
260 | " [2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],\n",
261 | " [2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],\n",
262 | " [2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],\n",
263 | " [2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],\n",
264 | " [2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.]])"
265 | ]
266 | },
267 | "execution_count": 4,
268 | "metadata": {},
269 | "output_type": "execute_result"
270 | }
271 | ],
272 | "source": [
273 | "theta2"
274 | ]
275 | },
276 | {
277 | "cell_type": "code",
278 | "execution_count": 5,
279 | "id": "53d0b4fa-92e2-4536-85b2-a5f5759f9302",
280 | "metadata": {},
281 | "outputs": [
282 | {
283 | "data": {
284 | "text/plain": [
285 | "array([[3., 3., 3., 3., 3., 3., 3., 3., 3., 3., 3.]])"
286 | ]
287 | },
288 | "execution_count": 5,
289 | "metadata": {},
290 | "output_type": "execute_result"
291 | }
292 | ],
293 | "source": [
294 | "theta3"
295 | ]
296 | },
297 | {
298 | "cell_type": "code",
299 | "execution_count": 6,
300 | "id": "cd87e604-cfa3-466a-a594-10defd93f559",
301 | "metadata": {},
302 | "outputs": [],
303 | "source": [
304 | "thetaVec = np.array([theta1, theta2, theta3], dtype=object)"
305 | ]
306 | },
307 | {
308 | "cell_type": "code",
309 | "execution_count": 7,
310 | "id": "195dea91-c8b1-4540-84b1-1a7f0f0be3fc",
311 | "metadata": {},
312 | "outputs": [
313 | {
314 | "name": "stdout",
315 | "output_type": "stream",
316 | "text": [
317 | "Number of elements in array: 231\n"
318 | ]
319 | }
320 | ],
321 | "source": [
322 | "count = 0\n",
323 | "for i in thetaVec:\n",
324 | " for j in i:\n",
325 | " count += len(j)\n",
326 | " \n",
327 | "print(\"Number of elements in array:\", count)"
328 | ]
329 | },
330 | {
331 | "cell_type": "code",
332 | "execution_count": 8,
333 | "id": "8bce9f03-e3d1-4a7a-9acd-86b859233289",
334 | "metadata": {},
335 | "outputs": [
336 | {
337 | "data": {
338 | "text/plain": [
339 | "array([array([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
340 | " [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
341 | " [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
342 | " [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
343 | " [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
344 | " [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
345 | " [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
346 | " [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
347 | " [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
348 | " [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]]),\n",
349 | " array([[2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],\n",
350 | " [2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],\n",
351 | " [2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],\n",
352 | " [2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],\n",
353 | " [2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],\n",
354 | " [2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],\n",
355 | " [2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],\n",
356 | " [2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],\n",
357 | " [2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],\n",
358 | " [2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.]]),\n",
359 | " array([[3., 3., 3., 3., 3., 3., 3., 3., 3., 3., 3.]])],\n",
360 | " dtype=object)"
361 | ]
362 | },
363 | "execution_count": 8,
364 | "metadata": {},
365 | "output_type": "execute_result"
366 | }
367 | ],
368 | "source": [
369 | "thetaVec"
370 | ]
371 | },
372 | {
373 | "cell_type": "code",
374 | "execution_count": 9,
375 | "id": "972eec04-9bad-46da-82ca-37c419b29083",
376 | "metadata": {},
377 | "outputs": [
378 | {
379 | "data": {
380 | "text/plain": [
381 | "3.0001000000000055"
382 | ]
383 | },
384 | "execution_count": 9,
385 | "metadata": {},
386 | "output_type": "execute_result"
387 | }
388 | ],
389 | "source": [
390 | "((1+0.01)**3-(1-0.01)**3)/(2*0.01)"
391 | ]
392 | },
393 | {
394 | "cell_type": "code",
395 | "execution_count": 11,
396 | "id": "cbd92d95-ac4d-493d-a19b-f59af593448b",
397 | "metadata": {},
398 | "outputs": [
399 | {
400 | "data": {
401 | "text/plain": [
402 | "75.00009999999904"
403 | ]
404 | },
405 | "execution_count": 11,
406 | "metadata": {},
407 | "output_type": "execute_result"
408 | }
409 | ],
410 | "source": [
411 | "j = 3*1 +2\n",
412 | "((j+0.01)**3-(j-0.01)**3)/(2*0.01)"
413 | ]
414 | },
415 | {
416 | "cell_type": "code",
417 | "execution_count": null,
418 | "id": "4383a4a1-970c-4545-87b8-5203ba5317ec",
419 | "metadata": {},
420 | "outputs": [],
421 | "source": []
422 | }
423 | ],
424 | "metadata": {
425 | "kernelspec": {
426 | "display_name": "Python 3",
427 | "language": "python",
428 | "name": "python3"
429 | },
430 | "language_info": {
431 | "codemirror_mode": {
432 | "name": "ipython",
433 | "version": 3
434 | },
435 | "file_extension": ".py",
436 | "mimetype": "text/x-python",
437 | "name": "python",
438 | "nbconvert_exporter": "python",
439 | "pygments_lexer": "ipython3",
440 | "version": "3.8.8"
441 | }
442 | },
443 | "nbformat": 4,
444 | "nbformat_minor": 5
445 | }
446 |
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 5 - Neural Networks: Learning/Week5_Images/backpropagation_algo.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/Machine-Learning-by-Stanford/Week 5 - Neural Networks: Learning/Week5_Images/backpropagation_algo.png
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 5 - Neural Networks: Learning/Week5_Images/forward_propagation.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/Machine-Learning-by-Stanford/Week 5 - Neural Networks: Learning/Week5_Images/forward_propagation.png
--------------------------------------------------------------------------------
/Machine-Learning-by-Stanford/Week 5 - Neural Networks: Learning/Week5_Images/gradient_computation.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/murilogustineli/machine-learning/d7047508dabbba5ec81b674b4350a4c6cfdc62a6/Machine-Learning-by-Stanford/Week 5 - Neural Networks: Learning/Week5_Images/gradient_computation.png
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Machine Learning
2 | Repo for storing and tracking my self-study progress in Machine Learning
3 |
4 |
5 |
6 | | Machine Learning Resources and Books |
7 | | ------------------------------------ |
8 | | 1. [Machine Learning by Stanford University - Coursera](https://www.coursera.org/learn/machine-learning) |
9 | | 2. [Machine Learning From Scratch by Danny Friedman](https://dafriedman97.github.io/mlbook/content/introduction.html) |
10 | | 3. [Hands On Machine Learning with Scikit Learn, Keras and TensorFlow](https://www.amazon.com/Hands-Machine-Learning-Scikit-Learn-TensorFlow/dp/1492032646/ref=pd_lpo_1?pd_rd_i=1492032646&psc=1) |
11 | | 4. [scikit-learn Tutorials](https://scikit-learn.org/stable/tutorial/#) |
12 | | 5. [Foundations of Machine Learning - Bloomberg ML EDU](https://bloomberg.github.io/foml/#home) |
13 |
14 |
15 |
16 | | Deep Learning Resources and Books |
17 | | --------------------------------- |
18 | | 1. [Deep Learning Specialization - DeepLearning.AI](https://www.deeplearning.ai/program/deep-learning-specialization/) |
19 | | 2. [fast.ai](https://www.fast.ai/) |
20 |
21 |
22 |
23 | | Mathematics and Statistics |
24 | | -------------------------- |
25 | | 1. [Linear Algebra - Khan Academy](https://www.khanacademy.org/math/linear-algebra) |
26 | | 2. [3Blue1Brown - Essence of Linear Algebra](https://www.3blue1brown.com/topics/linear-algebra) |
27 | | 3. [Statistics and probability - Khan Academy](https://www.khanacademy.org/math/statistics-probability) |
28 | | 4. [The Roadmap of Mathematics for Machine Learning - TowardsDataScience](https://towardsdatascience.com/the-roadmap-of-mathematics-for-deep-learning-357b3db8569b) |
29 |
30 |
31 |
32 | | Notebooks and Projects |
33 | | ---------------------- |
34 | | 1. [Linear Regression - Gradient Descent Algorithm](https://github.com/murilogustineli/Machine-Learning/blob/main/1.Linear-Regression%26Gradient-Descent/Gradient_Descent_Algorithm.ipynb) |
35 | | 2. [Linear Regression from scratch](https://github.com/murilogustineli/Machine-Learning/blob/main/1.Linear-Regression%26Gradient-Descent/LinearRegression.ipynb) |
36 | | 3. [Multiple Linear Regression - scikit-learn](https://github.com/murilogustineli/Machine-Learning/blob/main/2.Multiple-Linear-Regression/Multiple_Regression_scikit-learn.ipynb) |
37 | | 4. [California Housing dataset - scikit-learn](https://github.com/murilogustineli/Machine-Learning/blob/main/2.Multiple-Linear-Regression/California_Housing_dataset_scikit-learn.ipynb) |
38 |
39 |
40 |
41 | | Articles and Blog posts |
42 | | ----------------------- |
43 | | 1. [Machine Learning Mastery - Why Machine Learning Does Not Have to Be So Hard](https://machinelearningmastery.com/youre-wrong-machine-learning-not-hard/) |
44 | | 2. [8 Fun Machine Learning Projects for Beginners](https://elitedatascience.com/machine-learning-projects-for-beginners) |
45 | | 3. [Neural Networks - IBM](https://www.ibm.com/cloud/learn/neural-networks) |
46 | | 4. [Forecast KPIs: RMSE, MAE, MAPE & Bias](https://towardsdatascience.com/forecast-kpi-rmse-mae-mape-bias-cdc5703d242d) |
47 | | 5. [Which Evaluation Metric Should You Use in Machine Learning Regression Problems?](https://towardsdatascience.com/which-evaluation-metric-should-you-use-in-machine-learning-regression-problems-20cdaef258e) |
48 |
49 |
50 |
--------------------------------------------------------------------------------