├── .ipynb_checkpoints ├── Binary Naive Bayes-checkpoint.ipynb ├── Gaussian Discriminant Analyses-checkpoint.ipynb ├── Gaussian Naive Bayes-checkpoint.ipynb ├── KMeans-checkpoint.ipynb ├── Linear Regression Implementation from scratch-checkpoint.ipynb ├── Logistic Regression-checkpoint.ipynb ├── Momentum in ML-checkpoint.ipynb ├── Multi Class Gaussian Discriminant Analyses-checkpoint.ipynb ├── Multinomial Naive Bayes-checkpoint.ipynb └── Naive Bayes Implementation-checkpoint.ipynb ├── BayesianClassifier.ipynb ├── Binary Naive Bayes.ipynb ├── Gaussian Discriminant Analyses.ipynb ├── Gaussian Naive Bayes.ipynb ├── ID3.ipynb ├── KMeans.ipynb ├── Linear Regression Implementation from scratch.ipynb ├── Linear Regression with Newtons Method.ipynb ├── Linear_reg.ipynb ├── Logistic Regression with Newtons Method.ipynb ├── Logistic Regression.ipynb ├── Momentum in ML.ipynb ├── Multi Class Gaussian Discriminant Analyses.ipynb ├── Multi Class Logistic Regression with Newtons Method.ipynb ├── Multi Class Logistic Regression.ipynb ├── Multinomial Naive Bayes.ipynb ├── Naive Bayes Implementation.ipynb ├── Perceptron.ipynb ├── README.md ├── README_files ├── 7457498 ├── 15320635 ├── 1f334.png ├── 1f3af.png ├── 1f3e0.png ├── 1f912.png ├── compat-bootstrap-f87ad7f1.js ├── decision_tree_predictions.png ├── frameworks-481a47a96965f6706fb41bae0d14b09a.css ├── frameworks-e4aa17e0.js ├── github-bootstrap-d08823e5.js ├── github-eabfbaded2e91939e805d1a3af34018a.css ├── image_preprocessing.png ├── octocat-spinner-128.gif └── search-key-slash.svg ├── Wrapper Method For Feature Selection - Forward and Backward .ipynb ├── book_rec_knn.ipynb ├── figures ├── MM.png ├── bayes-theorem.png ├── decision_tree.png ├── decision_tree_predictions.png ├── image_preprocessing.png ├── linear_regression.jpg ├── logistic_regression.jpg ├── neural_net.png ├── perceptron_hyperplane.png ├── preprocessing.png ├── regression_tree.png └── softmax_regression.jpg ├── gaussian_naive_bayes.py ├── gda.py ├── kmeans.ipynb ├── kmeans.py ├── lwlr.ipynb ├── naive_bayes.py ├── spam filter.ipynb ├── svm.ipynb ├── train_test_split.pyc ├── utils.py └── utils.pyc /.ipynb_checkpoints/Binary Naive Bayes-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Naive Bayes Classifier Implementation using numpy\n", 8 | "\n", 9 | "Naive Bayes is anoother supervised machine laerning algorithm for classification problem. It makes a strong assumption about the data that **each feature is independent of the value of any other feature**. For example, a fruit may be considered to be an apple if it is red, round, and about 10 cm in diameter. A naive Bayes classifier considers each of these features to contribute independently to the probability that this fruit is an apple, regardless of any possible correlations between the color, roundness, and diameter features.\n", 10 | "\n", 11 | "In Naive bayes classifier what we are trying to find is the probability that a given data point belogs to a specific class, we are going to have prediction for all the class in our target.\n", 12 | "\n", 13 | "\n", 14 | "![title](figures/bayes-theorem.png)\n", 15 | "\n", 16 | "This is bernolli naive bayes impementation, which we expecting the features to be true or false (1 or 0)." 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": 72, 22 | "metadata": {}, 23 | "outputs": [], 24 | "source": [ 25 | "# %load naive_bayes.py\n", 26 | "import numpy as np\n", 27 | "\n", 28 | "class NaiveBayesBinaryClassifier:\n", 29 | " \n", 30 | " def fit(self, X, y):\n", 31 | " self.y_classes, y_counts = np.unique(y, return_counts=True)\n", 32 | " self.phi_y = 1.0 * y_counts/y_counts.sum()\n", 33 | " self.phi_x = [1.0 * X[y==k].mean(axis=0) for k in self.y_classes]\n", 34 | " return self\n", 35 | " \n", 36 | " def predict(self, X):\n", 37 | " return np.apply_along_axis(lambda x: self.compute_probs(x), 1, X)\n", 38 | " \n", 39 | " def compute_probs(self, x):\n", 40 | " probs = [self.compute_prob(x, y) for y in range(len(self.y_classes))]\n", 41 | " return self.y_classes[np.argmax(probs)]\n", 42 | " \n", 43 | " def compute_prob(self, x, y):\n", 44 | " res = 1\n", 45 | " for j in range(len(x)):\n", 46 | " Pxy = self.phi_x[y][j] # p(xj=1|y)\n", 47 | " res *= (Pxy**x[j])*((1-Pxy)**(1-x[j])) # p(xj=0|y)\n", 48 | " return res * self.phi_y[y]\n", 49 | " \n", 50 | " def score(self, X, y):\n", 51 | " return (self.predict(X) == y).mean()" 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": null, 57 | "metadata": {}, 58 | "outputs": [], 59 | "source": [] 60 | } 61 | ], 62 | "metadata": { 63 | "kernelspec": { 64 | "display_name": "python3", 65 | "language": "python", 66 | "name": "python3" 67 | }, 68 | "language_info": { 69 | "codemirror_mode": { 70 | "name": "ipython", 71 | "version": 2 72 | }, 73 | "file_extension": ".py", 74 | "mimetype": "text/x-python", 75 | "name": "python", 76 | "nbconvert_exporter": "python", 77 | "pygments_lexer": "ipython2", 78 | "version": "2.7.16" 79 | } 80 | }, 81 | "nbformat": 4, 82 | "nbformat_minor": 2 83 | } 84 | -------------------------------------------------------------------------------- /.ipynb_checkpoints/Gaussian Discriminant Analyses-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Gaussian Descriminant Analysis Implementation using numpy\n", 8 | "\n", 9 | "When we have a classification problem in which the input features x are\n", 10 | "continuous-valued random variables, we can then use the Gaussian Discriminant Analysis (GDA) model, which models p(x|y) using a multivariate normal distribution.\n", 11 | "\n", 12 | "GDA is a family of generative algorithm that try to model p(x|y) (and p(y)). After modeling p(y) (called the class priors) and p(x|y), our algorithm can then use Bayes rule to derive the posterior distribution on y given x:\n", 13 | "\n", 14 | "\n", 15 | "![title](figures/bayes-theorem.png)\n", 16 | "\n", 17 | "GDA makes stronger modeling assumptions, and is more data efficient (i.e., requires less training data to learn “well”) when the modeling assumptions are correct or at least approximately correct. Comparing to Logistic regression that makes weaker assumptions, and is significantly more robust to deviations from modeling assumptions\n", 18 | "\n", 19 | "### Our motivation\n", 20 | "To gain better understand on how the algorithm works" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": 4, 26 | "metadata": {}, 27 | "outputs": [], 28 | "source": [ 29 | "# %load gda.py\n", 30 | "import numpy as np\n", 31 | "\n", 32 | "class GDABinaryClassifier:\n", 33 | " \n", 34 | " def fit(self, X, y):\n", 35 | " self.fi = y.mean()\n", 36 | " self.u = np.array([ X[y==k].mean(axis=0) for k in [0,1]])\n", 37 | " X_u = X.copy()\n", 38 | " for k in [0,1]: X_u[y==k] -= self.u[k]\n", 39 | " self.E = X_u.T.dot(X_u) / len(y)\n", 40 | " self.invE = np.linalg.pinv(self.E)\n", 41 | " return self\n", 42 | " \n", 43 | " def predict(self, X):\n", 44 | " return np.argmax([self.compute_prob(X, i) for i in range(len(self.u))], axis=0)\n", 45 | " \n", 46 | " def compute_prob(self, X, i):\n", 47 | " u, phi = self.u[i], ((self.fi)**i * (1 - self.fi)**(1 - i))\n", 48 | " return np.exp(-1.0 * np.sum((X-u).dot(self.invE)*(X-u), axis=1)) * phi\n", 49 | " \n", 50 | " def score(self, X, y):\n", 51 | " return (self.predict(X) == y).mean()" 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": 5, 57 | "metadata": {}, 58 | "outputs": [ 59 | { 60 | "data": { 61 | "text/plain": [ 62 | "0.9666080843585237" 63 | ] 64 | }, 65 | "execution_count": 5, 66 | "metadata": {}, 67 | "output_type": "execute_result" 68 | } 69 | ], 70 | "source": [ 71 | "from sklearn.datasets import load_breast_cancer\n", 72 | "X,y = load_breast_cancer(return_X_y=True)\n", 73 | "model = GDABinaryClassifier().fit(X,y)\n", 74 | "pre = model.predict(X)\n", 75 | "model.score(X,y)" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": null, 81 | "metadata": {}, 82 | "outputs": [], 83 | "source": [] 84 | } 85 | ], 86 | "metadata": { 87 | "kernelspec": { 88 | "display_name": "python3", 89 | "language": "python", 90 | "name": "python3" 91 | }, 92 | "language_info": { 93 | "codemirror_mode": { 94 | "name": "ipython", 95 | "version": 2 96 | }, 97 | "file_extension": ".py", 98 | "mimetype": "text/x-python", 99 | "name": "python", 100 | "nbconvert_exporter": "python", 101 | "pygments_lexer": "ipython2", 102 | "version": "2.7.16" 103 | } 104 | }, 105 | "nbformat": 4, 106 | "nbformat_minor": 2 107 | } 108 | -------------------------------------------------------------------------------- /.ipynb_checkpoints/Gaussian Naive Bayes-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# %load gaussian_naive_bayes.py\n", 10 | "import numpy as np\n", 11 | " \n", 12 | "class GaussianNB:\n", 13 | " \n", 14 | " def fit(self, X, y, epsilon = 1e-10):\n", 15 | " self.y_classes, y_counts = np.unique(y, return_counts=True)\n", 16 | " self.x_classes = np.array([np.unique(x) for x in X.T])\n", 17 | " self.phi_y = 1.0 * y_counts/y_counts.sum()\n", 18 | " self.u = np.array([X[y==k].mean(axis=0) for k in self.y_classes])\n", 19 | " self.var_x = np.array([X[y==k].var(axis=0) + epsilon for k in self.y_classes])\n", 20 | " return self\n", 21 | " \n", 22 | " def predict(self, X):\n", 23 | " return np.apply_along_axis(lambda x: self.compute_probs(x), 1, X)\n", 24 | " \n", 25 | " def compute_probs(self, x):\n", 26 | " probs = np.array([self.compute_prob(x, y) for y in range(len(self.y_classes))])\n", 27 | " return self.y_classes[np.argmax(probs)]\n", 28 | " \n", 29 | " def compute_prob(self, x, y):\n", 30 | " c = 1.0 /np.sqrt(2.0 * np.pi * (self.var_x[y]))\n", 31 | " return np.prod(c * np.exp(-1.0 * np.square(x - self.u[y]) / (2.0 * self.var_x[y])))\n", 32 | " \n", 33 | " def evaluate(self, X, y):\n", 34 | " return (self.predict(X) == y).mean()" 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": 11, 40 | "metadata": {}, 41 | "outputs": [ 42 | { 43 | "data": { 44 | "text/plain": [ 45 | "0.96" 46 | ] 47 | }, 48 | "execution_count": 11, 49 | "metadata": {}, 50 | "output_type": "execute_result" 51 | } 52 | ], 53 | "source": [ 54 | "from sklearn import datasets\n", 55 | "from utils import accuracy_score\n", 56 | "iris = datasets.load_iris()\n", 57 | "X = iris.data \n", 58 | "y = iris.target\n", 59 | "GaussianNB().fit(X, y).evaluate(X, y)" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 12, 65 | "metadata": {}, 66 | "outputs": [ 67 | { 68 | "name": "stdout", 69 | "output_type": "stream", 70 | "text": [ 71 | "[1 1 1 2 2 2]\n" 72 | ] 73 | } 74 | ], 75 | "source": [ 76 | "X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])\n", 77 | "Y = np.array([1, 1, 1, 2, 2, 2])\n", 78 | "clf = GaussianNB().fit(X, Y)\n", 79 | "print(clf.predict(X))" 80 | ] 81 | }, 82 | { 83 | "cell_type": "code", 84 | "execution_count": 13, 85 | "metadata": {}, 86 | "outputs": [ 87 | { 88 | "data": { 89 | "text/plain": [ 90 | "0.8091263216471898" 91 | ] 92 | }, 93 | "execution_count": 13, 94 | "metadata": {}, 95 | "output_type": "execute_result" 96 | } 97 | ], 98 | "source": [ 99 | "from sklearn import datasets\n", 100 | "digits = datasets.load_digits()\n", 101 | "X = digits.data\n", 102 | "y = digits.target\n", 103 | "GaussianNB().fit(X, y).evaluate(X, y)" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": null, 109 | "metadata": {}, 110 | "outputs": [], 111 | "source": [] 112 | } 113 | ], 114 | "metadata": { 115 | "kernelspec": { 116 | "display_name": "python3", 117 | "language": "python", 118 | "name": "python3" 119 | }, 120 | "language_info": { 121 | "codemirror_mode": { 122 | "name": "ipython", 123 | "version": 2 124 | }, 125 | "file_extension": ".py", 126 | "mimetype": "text/x-python", 127 | "name": "python", 128 | "nbconvert_exporter": "python", 129 | "pygments_lexer": "ipython2", 130 | "version": "2.7.16" 131 | } 132 | }, 133 | "nbformat": 4, 134 | "nbformat_minor": 2 135 | } 136 | -------------------------------------------------------------------------------- /.ipynb_checkpoints/Logistic Regression-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Logistic Regression in Python and Numpy" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "In logistic regression, we are trying to model the outcome of a **binary variable** given a **linear combination of input features**. For example, we could try to predict the outcome of an election (win/lose) using information about how much money a candidate spent campaigning, how much time she/he spent campaigning, etc.\n", 15 | "\n", 16 | "### Model \n", 17 | "\n", 18 | "Logistic regression works as follows.\n", 19 | "\n", 20 | "**Given:** \n", 21 | "- dataset $\\{(\\boldsymbol{x}^{(1)}, y^{(1)}), ..., (\\boldsymbol{x}^{(m)}, y^{(m)})\\}$\n", 22 | "- with $\\boldsymbol{x}^{(i)}$ being a $d-$dimensional vector $\\boldsymbol{x}^{(i)} = (x^{(i)}_1, ..., x^{(i)}_d)$\n", 23 | "- $y^{(i)}$ being a binary target variable, $y^{(i)} \\in \\{0,1\\}$\n", 24 | "\n", 25 | "The logistic regression model can be interpreted as a very **simple neural network:**\n", 26 | "- it has a real-valued weight vector $\\boldsymbol{w}= (w^{(1)}, ..., w^{(d)})$\n", 27 | "- it has a real-valued bias $b$\n", 28 | "- it uses a sigmoid function as its activation function\n", 29 | "\n", 30 | "![title](figures/logistic_regression.jpg)" 31 | ] 32 | }, 33 | { 34 | "cell_type": "markdown", 35 | "metadata": {}, 36 | "source": [ 37 | "### Training\n", 38 | "\n", 39 | "Different to [linear regression](linear_regression.ipynb), logistic regression has no closed form solution. But the cost function is convex, so we can train the model using gradient descent. In fact, **gradient descent** (or any other optimization algorithm) is guaranteed to find the global minimum (if the learning rate is small enough and enough training iterations are used). \n", 40 | "\n", 41 | "Training a logistic regression model has different steps. In the beginning (step 0) the parameters are initialized. The other steps are repeated for a specified number of training iterations or until convergence of the parameters.\n", 42 | "\n", 43 | "* * * \n", 44 | "**Step 0: ** Initialize the weight vector and bias with zeros (or small random values).\n", 45 | "* * *\n", 46 | "\n", 47 | "**Step 1: ** Compute a linear combination of the input features and weights. This can be done in one step for all training examples, using vectorization and broadcasting:\n", 48 | "$\\boldsymbol{a} = \\boldsymbol{X} \\cdot \\boldsymbol{w} + b $\n", 49 | "\n", 50 | "where $\\boldsymbol{X}$ is a matrix of shape $(n_{samples}, n_{features})$ that holds all training examples, and $\\cdot$ denotes the dot product.\n", 51 | "* * *\n", 52 | "\n", 53 | "**Step 2: ** Apply the sigmoid activation function, which returns values between 0 and 1:\n", 54 | "\n", 55 | "$\\boldsymbol{\\hat{y}} = \\sigma(\\boldsymbol{a}) = \\frac{1}{1 + \\exp(-\\boldsymbol{a})}$\n", 56 | "* * *\n", 57 | "\n", 58 | "** Step 3: ** Compute the cost over the whole training set. We want to model the probability of the target values being 0 or 1. So during training we want to adapt our parameters such that our model outputs high values for examples with a positive label (true label being 1) and small values for examples with a negative label (true label being 0). This is reflected in the cost function:\n", 59 | "\n", 60 | "$J(\\boldsymbol{w},b) = - \\frac{1}{m} \\sum_{i=1}^m \\Big[ y^{(i)} \\log(\\hat{y}^{(i)}) + (1 - y^{(i)}) \\log(1 - \\hat{y}^{(i)}) \\Big]$\n", 61 | "* * *\n", 62 | "\n", 63 | "** Step 4: ** Compute the gradient of the cost function with respect to the weight vector and bias. A detailed explanation of this derivation can be found [here](https://stats.stackexchange.com/questions/278771/how-is-the-cost-function-from-logistic-regression-derivated).\n", 64 | "\n", 65 | "The general formula is given by:\n", 66 | "\n", 67 | "$ \\frac{\\partial J}{\\partial w_j} = \\frac{1}{m}\\sum_{i=1}^m\\left[\\hat{y}^{(i)}-y^{(i)}\\right]\\,x_j^{(i)}$\n", 68 | "\n", 69 | "For the bias, the inputs $x_j^{(i)}$ will be given 1.\n", 70 | "* * *\n", 71 | "\n", 72 | "** Step 5: ** Update the weights and bias\n", 73 | "\n", 74 | "$\\boldsymbol{w} = \\boldsymbol{w} - \\eta \\, \\nabla_w J$ \n", 75 | "\n", 76 | "$b = b - \\eta \\, \\nabla_b J$\n", 77 | "\n", 78 | "where $\\eta$ is the learning rate." 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": 18, 84 | "metadata": {}, 85 | "outputs": [], 86 | "source": [ 87 | "import numpy as np\n", 88 | "class LogisticRegression:\n", 89 | " \n", 90 | " def fit(self, X, y, lr = 0.001, epochs=10000, verbose=True, batch_size=1):\n", 91 | " self.classes = np.unique(y)\n", 92 | " y = (y==self.classes[1]) * 1\n", 93 | " X = self.add_bias(X)\n", 94 | " self.weights = np.zeros(X.shape[1])\n", 95 | " self.loss = []\n", 96 | " for i in range(epochs):\n", 97 | " self.loss.append(self.cross_entropy(X,y))\n", 98 | " if i % 1000 == 0 and verbose: \n", 99 | " print('Iterations: %d - Error : %.4f' %(i, self.loss[i]))\n", 100 | " idx = np.random.choice(X.shape[0], batch_size)\n", 101 | " X_batch, y_batch = X[idx], y[idx]\n", 102 | " self.weights -= lr * self.get_gradient(X_batch, y_batch)\n", 103 | " return self\n", 104 | " \n", 105 | " def get_gradient(self, X, y):\n", 106 | " return -1.0 * (y - self.predict_(X)).dot(X) / len(X)\n", 107 | " \n", 108 | " def predict_(self, X):\n", 109 | " return self.sigmoid(np.dot(X, self.weights))\n", 110 | " \n", 111 | " def predict(self, X):\n", 112 | " return self.predict_(self.add_bias(X))\n", 113 | " \n", 114 | " def sigmoid(self, z):\n", 115 | " return 1.0/(1 + np.exp(-z))\n", 116 | " \n", 117 | " def predict_classes(self, X):\n", 118 | " return self.predict_classes_(self.add_bias(X))\n", 119 | "\n", 120 | " def predict_classes_(self, X):\n", 121 | " return np.vectorize(lambda c: self.classes[1] if c>=0.5 else self.classes[0])(self.predict_(X))\n", 122 | " \n", 123 | " def cross_entropy(self, X, y):\n", 124 | " p = self.predict_(X)\n", 125 | " return (-1 / len(y)) * (y * np.log(p)).sum()\n", 126 | "\n", 127 | " def add_bias(self,X):\n", 128 | " return np.insert(X, 0, 1, axis=1)\n", 129 | "\n", 130 | " def score(self, X, y):\n", 131 | " return self.cross_entropy(self.add_bias(X), y)\n" 132 | ] 133 | }, 134 | { 135 | "cell_type": "code", 136 | "execution_count": 19, 137 | "metadata": {}, 138 | "outputs": [], 139 | "source": [ 140 | "from sklearn.metrics import accuracy_score\n", 141 | "def train_model(X, y, model):\n", 142 | " model.fit(X, y, lr=0.1)\n", 143 | " pre = model.predict_classes(X)\n", 144 | " print('Accuracy :: ', accuracy_score(y, pre))" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": 20, 150 | "metadata": {}, 151 | "outputs": [ 152 | { 153 | "name": "stdout", 154 | "output_type": "stream", 155 | "text": [ 156 | "Iterations: 0 - Error : 69.3147\n", 157 | "Iterations: 1000 - Error : 0.4746\n", 158 | "Iterations: 2000 - Error : 0.3847\n", 159 | "Iterations: 3000 - Error : 0.1645\n", 160 | "Iterations: 4000 - Error : 0.1280\n", 161 | "Iterations: 5000 - Error : 0.1126\n", 162 | "Iterations: 6000 - Error : 0.0783\n", 163 | "Iterations: 7000 - Error : 0.0674\n", 164 | "Iterations: 8000 - Error : 0.0621\n", 165 | "Iterations: 9000 - Error : 0.0664\n", 166 | "('Accuracy :: ', 1.0)\n" 167 | ] 168 | } 169 | ], 170 | "source": [ 171 | "from sklearn.datasets import load_iris\n", 172 | "X, y = load_iris(return_X_y=True)\n", 173 | "lr = LogisticRegression()\n", 174 | "train_model(X,(y !=0 )*1, lr)" 175 | ] 176 | }, 177 | { 178 | "cell_type": "code", 179 | "execution_count": 21, 180 | "metadata": {}, 181 | "outputs": [ 182 | { 183 | "data": { 184 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfgAAAGDCAYAAADHzQJ9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi40LCBodHRwOi8vbWF0cGxvdGxpYi5vcmcv7US4rQAAIABJREFUeJzt3XmcJXV97//Xu7tngQEZBkbCpoOIJi6J4mhQs+BylRgVc39G8XoVo5FrzGqSqxIT482qMVFjTMw1omhUlKCJmLgRhBhvFBxcWF1GdhxgWIRhmbU/vz/q23BseqZ7Zrr7TBev5+NxHl31rW/V+Zzqmn6fWqYqVYUkSeqXkWEXIEmSZp8BL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8NJ2JDk3yS8Pu475lOThSb6RZEOS3xh2PQtdkgcluSPJ6Gz2lWbCgNceLcmVSe5ugfODJP+V5FVJ3HanMAtfSl4LnFNV+1bVO2erru1JcmqSP5nr99kVSV6W5Eu7s4yqurqq9qmqbbPZV5oJ/0hqIXhOVe0LPBh4M/A64JThltRbDwYuGXYR8y3J2C7O59629lgGvBaMqrqtqs4EXgicmORRAEmWJPnLJFcnuSHJ3yfZq027LMmzJ5aRZCzJ+iRHt/Fj2lGBHyT5ZpJjp3rvJCNJfj/JVUluTPLBJPu1aauSVJKTknw/ybokvzsw75uS/FOSD7UjERcleViSk9uyrknyjIH++yU5pS3nuiR/MhEkE3uV7fPemuSKJD/Xpv0p8NPAu9qh3ndt57M8N8kl7TOfm+THWvsXgKcMzP+wKeZdkeT97XPemuRfBqa9MsnaJLckOTPJIa09Sd7ePuvt7fM/KslJwIuB17b3+9R26n1Skq8mua39fFJrf2GSNZP6vibJmTPYLo5Ncm2S1yW5Hnj/pOX8GPD3wBNbbT9o7acmeXeSTye5E3hKkp9P8vX22a5J8qaB5UxsG2Nt/Nwkf5zk/7Vt4fNJDtzZvm36S9v2eHOSP0h3tOvpU61D3U9VlS9fe+wLuBJ4+hTtVwO/0obfDpwJrAD2BT4F/Hmb9kbgwwPz/TxwWRs+FLgZeBbdl93/1sZXtunnAr/chl8OrAUeAuwDfAL4xzZtFVDAacAy4NHA+om6gTcBG4FnAmPAB4ErgDcAi4BXAlcM1PjPwP9ty3ogcD7wv9q0lwFb2jyjwK8A3wcyuebtrM+HAXe2z7qI7pD8WmDxDOf/N+BjwP5t/p9t7U8FbgKOBpYAfwN8sU17JnABsBwI8GPAwW3aqcCf7OD9VgC3Ai9p6+5FbfwAYG9gA3DUQP+vAifMYLs4FtgKvKXVu9cU7/0y4EuT2k4FbgOeTLfNLG3LenQb/3HgBuB5k7aNsYH1+732e9irjb95F/o+ArgD+ClgMfCXbbu4z78VX/ff19AL8OVrRy+2H/BfoQvItMA6cmDaE2mBCTy0hcDebfzDwBvb8OtoIT0w7+eAE9vwudwb8GcDrx7o9/D2B3Vs4A/zjw5M/wvglDb8JuCsgWnPaX+cR9v4vm3+5cBBwKbBwKELtXPa8MuAtQPT9m7z/sjkmrezPv8AOH1gfAS4Djh2uvmBg4FxYP8ppp0C/MXA+D5t/ayiC//vAMcAI5PmO5UdB/xLgPMntX0ZeFkb/tDA7/Ooid/1DLaLY4HNwNIdvPfLmDrgPzjNNvsO4O1teGLbGAzt3x/o+2rgs7vQ943AaZO2g80Y8L4GXrt03knaAxwK3AKspPvjdkGSiWmh27ulqtYmuQx4TjsE/Fzgsa3fg4FfTPKcgeUuAs6Z4v0OAa4aGL+KLtwPGmi7ZtL0Rw+M3zAwfDdwU917MdXd7ec+7X0WAesGPs/IpGVfPzFQVXe1fvtMUfNUfuhzVNV4kmvo1ud0Dgduqapbt7Pcrw0s944kNwOHVtUX2umCvwUenOQTwO9W1e07W29z1UC9HwH+Cvgj4H8A/9LWyQPZwXbRrK+qjTOoYbLB3wVJfpLu2pBH0e1NLwH+aQfzXz8wfBc7/t1tr+8hg3W0z3zztJXrfsVz8Fpwkjye7g/8l+gOC98NPLKqlrfXflU1+EfzNLq94OOBS6tqbWu/hm4PfvnAa1lVvXmKt/0+3ReCCQ+iO8Q7GNyHT5r+/V34eNfQ7cEfOFDTA6rqkTOcf7rHQ/7Q50iXfofT7cXPpLYVSZbPYLnL6A6jXwdQVe+sqsfRHVp+GPC/d6Xe5kED9Z4FrEzyGLrf8Uda+0y2i+nee3vTJ7d/hO5UwOFVtR/dufvcZ67ZtQ44bGKkXVtwwBy/pxYYA14LRpIHpLtg7qPAh6rqoqoaB/4BeHvbayPJoUmeOTDrR4Fn0J2v/shA+4fo9uyfmWQ0ydJ28dVh3NdpwGuSHJFkH+DPgI9V1daBPn+QZO8kjwR+ie5c9U6pqnXA54G/ap93JMmRSX52hou4ge46ge05Hfj5JE9Lsgj4HbovFP81w9o+A/xdkv2TLEryM23yacAvJXlMkiV06+e8qroyyeOT/GR7vzvprkcYn2G9nwYeluR/pLtA8oV0XxL+tdW0hW5v+a1059rPau0z2S6mcwNwWJLF0/Tbl+7IxsYkT6A7kjDXzqDbdp/U6nsTc/+lQguMAa+F4FNJNtDtQb4BeBtdgE54Hd2FYl9Jcjvw73TnyIF7gunLwJMYCN2quoZur/736C6Ku4Zuz3KqfxfvA/4R+CLdBXIbgV+f1Oc/Wh1nA39ZVZ/ftY/LS+kO9V5Kd0HZGXTnv2fir4Hnp7vC/T7/j72qvg38T7qL4G6iux7gOVW1eYbLfwndufVvATcCv9WW++905/c/Trd3eSRwQpvnAXRheyvd4fWb6QIZunP3j0h3Rf89V+QP1Hsz8Gy6LyI3010U+Oyqummg20eApwP/NOkL1w63ixn4At1/Gbw+yU076Pdq4I/aNvpGui9Rc6qqLqHb/j5Kt77voPt9bJrr99bCMXHlraRdlGQVXegvmhQw0rxoR5V+QPc/Cq4Ydj3aM7gHL0kLUJLntFNCy+j+m9xFdP/rRAIMeElaqI6nuwjx+3T/RfCE8pCsBniIXpKkHnIPXpKkHjLgJUnqoQV9J7sDDzywVq1aNewyJEmaNxdccMFNVbVyun4LOuBXrVrFmjVrpu8oSVJPJJl8++YpeYhekqQeMuAlSeohA16SpB4y4CVJ6iEDXpKkHjLgJUnqIQNekqQemrOAT/K+JDcmuXiKab+TpJIc2MaT5J1J1ia5MMnRc1WXJEn3B3O5B38qcNzkxiSHA88Arh5o/jm6pyEdBZwEvHsO65IkqffmLOCr6ovALVNMejvwWmDwMXbHAx+szleA5UkOnqvaJEnqu3k9B5/keOC6qvrmpEmHAtcMjF/b2iRJ0i6Yt4BPsjfwe8Abd3M5JyVZk2TN+vXrZ6e4Zu2Nd1BV03eUJGkPN5978EcCRwDfTHIlcBjwtSQ/AlwHHD7Q97DWdh9V9Z6qWl1Vq1eunPZhOjN2wVW38vS3/Qen/teVs7ZMSZKGZd4CvqouqqoHVtWqqlpFdxj+6Kq6HjgTeGm7mv4Y4LaqWjdftQFcdfOdAFx47W3z+baSJM2JufxvcqcBXwYenuTaJK/YQfdPA5cDa4F/AF49V3VJknR/MGfPg6+qF00zfdXAcAG/Ole1SJJ0f+Od7CRJ6iEDXpKkHjLgJUnqIQNekqQeMuAlSeohA16SpB4y4CVJ6iEDXpKkHjLgJUnqIQNekqQeMuAlSeohA16SpB4y4CVJ6iEDXpKkHjLgJUnqIQNekqQeMuAlSeohA16SpB4y4CVJ6iEDXpKkHjLgJUnqIQNekqQeMuAnqaphlyBJ0m4z4Jtk2BVIkjR7DHhJknrIgG88Mi9J6hMDfpJ4rF6S1AMGvCRJPWTAS5LUQwa8JEk9ZMBLktRDBrwkST1kwEuS1EMGvCRJPTRnAZ/kfUluTHLxQNtbk3wryYVJ/jnJ8oFpJydZm+TbSZ45V3VJknR/MJd78KcCx01qOwt4VFX9OPAd4GSAJI8ATgAe2eb5uySjc1ibJEm9NmcBX1VfBG6Z1Pb5qtraRr8CHNaGjwc+WlWbquoKYC3whLmqTZKkvhvmOfiXA59pw4cC1wxMu7a13UeSk5KsSbJm/fr1c1yiJEkL01ACPskbgK3Ah3d23qp6T1WtrqrVK1eunP3iJEnqgbH5fsMkLwOeDTyt6p5nuF0HHD7Q7bDWJkmSdsG87sEnOQ54LfDcqrprYNKZwAlJliQ5AjgKOH8+a5MkqU/mbA8+yWnAscCBSa4F/pDuqvklwFntsaxfqapXVdUlSU4HLqU7dP+rVbVtrmqTJKnv5izgq+pFUzSfsoP+fwr86VzVI0nS/Yl3spMkqYcMeEmSesiAlySphwx4SZJ6yICXJKmHDHhJknrIgJ/k3pvrSZK0cBnwTXffHUmS+sGAlySphwx4SZJ6yICXJKmHDPjGa+skSX1iwE8Sr7aTJPWAAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8JNU1bBLkCRptxnwTTLsCiRJmj1zFvBJ3pfkxiQXD7StSHJWku+2n/u39iR5Z5K1SS5McvRc1SVJ0v3BXO7BnwocN6nt9cDZVXUUcHYbB/g54Kj2Ogl49xzWJUlS781ZwFfVF4FbJjUfD3ygDX8AeN5A+wer8xVgeZKD56q2qYyPdz8vXXf7fL6tJElzYr7PwR9UVeva8PXAQW34UOCagX7Xtrb7SHJSkjVJ1qxfv37WCrv+9o0AfOeGO2ZtmZIkDcvQLrKr7nL1nb5kvareU1Wrq2r1ypUr56AySZIWvvkO+BsmDr23nze29uuAwwf6HdbaJEnSLpjvgD8TOLENnwh8cqD9pe1q+mOA2wYO5UuSpJ00NlcLTnIacCxwYJJrgT8E3gycnuQVwFXAC1r3TwPPAtYCdwG/NFd1SZJ0fzBnAV9VL9rOpKdN0beAX52rWiRJur/xTnaSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfBN97wbSZL6wYCXJKmHDHhJknrIgJckqYcM+CbJsEuQJGnWGPCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8E1VDbsESZJmjQEvSVIPDSXgk7wmySVJLk5yWpKlSY5Icl6StUk+lmTxMGqTJKkP5j3gkxwK/AawuqoeBYwCJwBvAd5eVQ8FbgVeMd+1SZLUF8M6RD8G7JVkDNgbWAc8FTijTf8A8Lwh1SZJ0oI37wFfVdcBfwlcTRfstwEXAD+oqq2t27XAofNdmyRJfTGMQ/T7A8cDRwCHAMuA43Zi/pOSrEmyZv369bNZ16wtS5KkYRvGIfqnA1dU1fqq2gJ8AngysLwdsgc4DLhuqpmr6j1VtbqqVq9cuXJ+KpYkaYEZRsBfDRyTZO90u81PAy4FzgGe3/qcCHxyCLVJktQLwzgHfx7dxXRfAy5qNbwHeB3w20nWAgcAp8x3bZIk9cXY9F1mX1X9IfCHk5ovB54whHIkSeod72QnSVIPGfCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPXQjAI+yT/OpE2SJO0ZZroH/8jBkSSjwONmvxxJkjQbdhjwSU5OsgH48SS3t9cG4EZ8GIwkSXusHQZ8Vf15Ve0LvLWqHtBe+1bVAVV18jzVKEmSdtJMD9H/a5JlAEn+Z5K3JXnwHNYlSZJ2w0wD/t3AXUl+Avgd4HvAB+esKkmStFtmGvBbq6qA44F3VdXfAvvOXVnzr/t4kiT1w0yfB78hycnAS4CfTjICLJq7siRJ0u6Y6R78C4FNwMur6nrgMOCtc1bVENy4YdOwS5AkadbMKOBbqH8Y2C/Js4GNVdWrc/DfW3/HsEuQJGnWzPROdi8Azgd+EXgBcF6S589lYZIkadfN9Bz8G4DHV9WNAElWAv8OnDFXhUmSpF0303PwIxPh3ty8E/NKkqR5NtM9+M8m+RxwWht/IfDpuSlJkiTtrh0GfJKHAgdV1f9O8t+Bn2qTvkx30Z0kSdoDTbcH/w7gZICq+gTwCYAkj27TnjOn1UmSpF0y3Xn0g6rqosmNrW3VnFQkSZJ223QBv3wH0/aazUIkSdLsmS7g1yR55eTGJL8MXDA3JUmSpN013Tn43wL+OcmLuTfQVwOLgV+Yy8IkSdKu22HAV9UNwJOSPAV4VGv+t6r6wpxXJkmSdtmM/h98VZ0DnDPHtUiSpFni3egkSeohA16SpB4y4JuQYZcgSdKsMeAlSeqhoQR8kuVJzkjyrSSXJXlikhVJzkry3fZz/2HUJklSHwxrD/6vgc9W1Y8CPwFcBrweOLuqjgLObuOSJGkXzHvAJ9kP+BngFICq2lxVPwCOBz7Qun0AeN581yZJUl8MYw/+CGA98P4kX0/y3iTL6B5ss671uR44aKqZk5yUZE2SNevXr5+1ouI1dpKkHhlGwI8BRwPvrqrHAncy6XB8VRVQU81cVe+pqtVVtXrlypVzXqwkSQvRMAL+WuDaqjqvjZ9BF/g3JDkYoP28cQi1SZLUC/Me8FV1PXBNkoe3pqcBlwJnAie2thOBT853bZIk9cWM7kU/B34d+HCSxcDlwC/Rfdk4PckrgKuAFwypNkmSFryhBHxVfYPusbOTPW2+a5EkqY+8k11TU17SJ0nSwmTAS5LUQwa8JEk9ZMBLktRDBrwkST1kwEuS1EMGvCRJPWTAS5LUQwa8JEk9ZMBLktRDBrwkST1kwDfJsCuQJGn2GPCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAHfxMvoJUk9YsBLktRDBrwkST1kwEuS1EMGvCRJPWTAN1U17BIkSZo1BrwkST1kwEuS1EMGvCRJPWTAS5LUQwb8FO7evG3YJUiStFsM+Cls2Lhl2CVIkrRbDHhJknrIgJckqYcMeEmSesiAlySph4YW8ElGk3w9yb+28SOSnJdkbZKPJVk8rNrw0fCSpAVumHvwvwlcNjD+FuDtVfVQ4FbgFUOpCsDb0kuSFrihBHySw4CfB97bxgM8FTijdfkA8Lxh1CZJUh8Maw/+HcBrgfE2fgDwg6ra2savBQ6dz4K67xiSJPXDvAd8kmcDN1bVBbs4/0lJ1iRZs379+tmra7sjkiQtPMPYg38y8NwkVwIfpTs0/9fA8iRjrc9hwHVTzVxV76mq1VW1euXKlfNRryRJC868B3xVnVxVh1XVKuAE4AtV9WLgHOD5rduJwCfnuzZJkvpiT/p/8K8DfjvJWrpz8qcMrRKvopckLXBj03eZO1V1LnBuG74ceMLQahnWG0uSNAf2pD34PYcX2UmSFjgDXpKkHjLgJUnqIQNekqQeMuAbT7tLkvrEgJckqYcMeEmSesiAlySphwx4SZJ6yICfQrzkTpK0wBnwkiT1kAE/hfLO9JKkBc6AlySphwx4SZJ6yICfghfZSZIWOgO+iZkuSeoRA16SpB4y4JvywnlJUo8Y8JIk9ZABL0lSDxnwkiT1kAHfeBW9JKlPDHhJknrIgJ+Ce/OSpIXOgJ+C/2VOkrTQGfCSJPWQAT+FW+/aPOwSJEnaLQb8FL703ZuGXYIkSbvFgJckqYcM+Cl4jZ0kaaEz4CVJ6iEDXpKkHjLgJUnqIQO+8eZ1kqQ+mfeAT3J4knOSXJrkkiS/2dpXJDkryXfbz/3nuzZJkvpiGHvwW4HfqapHAMcAv5rkEcDrgbOr6ijg7DYuSZJ2wbwHfFWtq6qvteENwGXAocDxwAdatw8Az5vPuuITZiRJPTLUc/BJVgGPBc4DDqqqdW3S9cBB25nnpCRrkqxZv379rNVSA0+Y2bx1fNaWK0nSMAwt4JPsA3wc+K2qun1wWnVpO+X9ZqrqPVW1uqpWr1y5ck5qe8tnv8WnL1o3fUdJkvZQQwn4JIvowv3DVfWJ1nxDkoPb9IOBG4dR24TXfOwbw3x7SZJ2yzCuog9wCnBZVb1tYNKZwIlt+ETgk/Nd26BNHqaXJC1gY0N4zycDLwEuSjKxm/x7wJuB05O8ArgKeMEQapMkqRfmPeCr6kts/74yT5vPWiRJ6ivvZCdJUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCN96KXJPWJAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8I3PkpMk9YkBL0lSDxnwO3DLnZuHXYIkSbvEgN+Bo//4LM67/Ga+fvWtVNWwy5EkacYM+Gn8n09dyi/83X/xqQvXDbsUSZJmzICfxhU33QnA9268Y8iVSJI0cwb8NEZHuuvrz/zm99mybXzI1UiSNDMG/DRavnPFTXfy7nO/N9xiJEmaIQN+Grdv3HrP8I0bNg6xEkmSZs6A3wkf+srV/Fu72O5tn/82F11725ArkiRpagb8Tvrt07/Bqtf/G+/8wlqe864vDbscSZKmZMDvpE1bvdBOkrTnM+CbRaO7tipO/+o1s1yJJEm7z4Bvfu7RP7JL87324xdy1+at92nfuGXb7pYkSdIuM+CbZz364F2e91Uf+hrv+sJ32bJtnH/88pWc/ImL+NE/+Cz/9b2b7ulTVVN+EZAkaS5kT7vHepLjgL8GRoH3VtWbt9d39erVtWbNmll5323jxZG/92kAPvObP80D913Cqz50AV+98lZee9zDeeVPP4Tvrb+D497xnzu97O/92bPuWfbLn3wERz5wGc9/3GHcsXEr+y5dxJ99+jLWb9jETx91IE888gAeuO9SbrpjE4ev2HtWPpskqT+SXFBVq6fttycFfJJR4DvAfwOuBb4KvKiqLp2q/2wGPMA/fPFynvzQA3nEIQ8A4O7N2/jMxev4hcceSnLvE+PHx4tnvfM/+db1G2btvaeyeGyEhx20D9vG4XmPOYRvXPMDbtywiX2XjrFi2WKuv20jd27exmXfv50fO+QBrH7w/ixdNMLVt9zNJd+/jUcesh9POGIFowlX3XInS8ZG2TY+zs13bObym+7k/CtuAeCYh6zgd57xcEYSNm3dxt2bt3HHpq3dPQCqWDQ6wkjC+js2cdj+e3HwfnvdcwOgzVvHuebWu7hr8zauvuUuqmC/vRaxeGyEJWMjbBsvNm4ZZ8WyRTz0gfuyeCxs2VZs3VYsWzLKAcuWsGTRCPvttah9nq1s3DLO6Ei4a9NWliwaYemiURaNjrBh41a2bBvnwQfszf57L6YKbrpjE9vGi/Eqxgug+zleRQ38HB0JDztoX7aNF0XXVgXV+lcVBdQ490wfb23jXcd7hqu6zz2SMDYa9l48yvK9F9/ze+uO1mzj1rs2MzoSRhIesHQRey0endPtRdL9w0IN+CcCb6qqZ7bxkwGq6s+n6j/bAb8rrrnlLv703y5jn6VjHLp8Lx6/agVv+tQlfPxVT+K7N27g+X//5Xv6vvrYI/m7dje8/fdexK13bQHgCatWcP6Vt/zQckdHwrbxH/7d7Lt0jAet2Jvv3LCBbePF0kWjPGjF3rv8RWPvxaPctdlrBWbD0kUjjI2MUFVs2jrO1vH7/rtaumiEJWOj7LVolK3jRdW9X0wm/h2OtYs97/nC0RYzMT7xReM+0ye+tNzTpxvfVkWAkXRfNEh3d8aJ8QAJjIwMjoeR3Hub5h2Zvgc/9OV4V8xk9un6ZAaVTr+MmdSx417TLmMmn3U3a5jZMmZSxzSfdfd+7W0Zu7k+Z1DHfHzW0155DMuWjE3/RjO0UAP++cBxVfXLbfwlwE9W1a8N9DkJOAngQQ960OOuuuqqodS6M6pqyg11fLxItr8Rbxsvto0XN9+5ifGCg/ZdwtjoCBu3bCOBJWOj9yzn6lvuYnQk/Mh+S1k0OsJdm7dy2brbOWDZEhaPjbBl2zgH7rOEvRePMl73/vHeuGUbl6+/k3W33c3oSLoAWjzKvkvH2GfJGCMJt9y5mUWj4YB9lnDdrXdz852b7tnDXTw6wuEr9mbpolFWLFvM6EgYHy82bxtn87Zx7trUHQ245Pu3ccCyJWyrYtFI2FbV9r5hw8YtbNi4lUOX78WyJWMsGRthvIq9Fo+yZVuxccs2Nm8dZ9+lXT3fvn4DW8bHCWHFskUsGRu9Zz1OhFkycZvhLqzW3bbxns+RNj3c229i+N5l3Bt0TFpu6I6uVMHW8XHW3baRDQN3PFwy1h2R2G+vRQBsGS9u2rCJu7dsY+OW7gjJ2OgIoyMDwdtMPO9gsL6J0B1sv3eYH6qbSfONjuSHjlRMHIGo2v74eHXb1HjVDv9wzeRPx3RdpltGTbuE6d9kJn/hpvs7OLNl7F4dM/lbPG2PGa2uaT7rTJYxC7+33V9f077FtEuZnW14+oX87YuPZu/FBvy0AT9oT9iDlyRpPs004Pe0q+ivAw4fGD+stUmSpJ2wpwX8V4GjkhyRZDFwAnDmkGuSJGnBmb2TArOgqrYm+TXgc3T/Te59VXXJkMuSJGnB2aMCHqCqPg18eth1SJK0kO1ph+glSdIsMOAlSeohA16SpB4y4CVJ6iEDXpKkHjLgJUnqIQNekqQeMuAlSeohA16SpB7ao54mt7OSrAdm83mxBwI3zeLy7q9cj7vPdbj7XIe7z3W4++ZiHT64qlZO12lBB/xsS7JmJo/g0465Hnef63D3uQ53n+tw9w1zHXqIXpKkHjLgJUnqIQP+h71n2AX0hOtx97kOd5/rcPe5Dnff0Nah5+AlSeoh9+AlSeohA75JclySbydZm+T1w65nT5Lk8CTnJLk0ySVJfrO1r0hyVpLvtp/7t/YkeWdblxcmOXpgWSe2/t9NcuKwPtOwJBlN8vUk/9rGj0hyXltXH0uyuLUvaeNr2/RVA8s4ubV/O8kzh/NJhiPJ8iRnJPlWksuSPNGQRT1RAAAIGElEQVTtcOckeU37d3xxktOSLHU7nF6S9yW5McnFA22ztu0leVySi9o870yS3S66qu73L2AU+B7wEGAx8E3gEcOua095AQcDR7fhfYHvAI8A/gJ4fWt/PfCWNvws4DNAgGOA81r7CuDy9nP/Nrz/sD/fPK/L3wY+AvxrGz8dOKEN/z3wK2341cDft+ETgI+14Ue07XMJcETbbkeH/bnmcf19APjlNrwYWO52uFPr71DgCmCvge3vZW6HM1p3PwMcDVw80DZr2x5wfuubNu/P7W7N7sF3ngCsrarLq2oz8FHg+CHXtMeoqnVV9bU2vAG4jO4PxfF0f3BpP5/Xho8HPlidrwDLkxwMPBM4q6puqapbgbOA4+bxowxVksOAnwfe28YDPBU4o3WZvA4n1u0ZwNNa/+OBj1bVpqq6AlhLt/32XpL96P7IngJQVZur6ge4He6sMWCvJGPA3sA63A6nVVVfBG6Z1Dwr216b9oCq+kp1af/BgWXtMgO+cyhwzcD4ta1Nk7RDdI8FzgMOqqp1bdL1wEFteHvr8/6+nt8BvBYYb+MHAD+oqq1tfHB93LOu2vTbWv/78zo8AlgPvL+d5nhvkmW4Hc5YVV0H/CVwNV2w3wZcgNvhrpqtbe/QNjy5fbcY8JqxJPsAHwd+q6puH5zWvnX6XzK2I8mzgRur6oJh17KAjdEdIn13VT0WuJPusOg93A53rJ0jPp7uy9IhwDLuX0cv5syeuO0Z8J3rgMMHxg9rbWqSLKIL9w9X1Sda8w3t0BLt542tfXvr8/68np8MPDfJlXSngJ4K/DXdobux1mdwfdyzrtr0/YCbuX+vw2uBa6vqvDZ+Bl3gux3O3NOBK6pqfVVtAT5Bt226He6a2dr2rmvDk9t3iwHf+SpwVLuSdDHdxSRnDrmmPUY753YKcFlVvW1g0pnAxFWgJwKfHGh/abuS9BjgtnYY63PAM5Ls3/YkntHaeq+qTq6qw6pqFd329YWqejFwDvD81m3yOpxYt89v/au1n9Cubj4COIru4pzeq6rrgWuSPLw1PQ24FLfDnXE1cEySvdu/64l16Ha4a2Zl22vTbk9yTPu9vHRgWbtu2Fcm7ikvuqsev0N3Negbhl3PnvQCforu0NOFwDfa61l05+LOBr4L/DuwovUP8LdtXV4ErB5Y1svpLshZC/zSsD/bkNbnsdx7Ff1D6P4wrgX+CVjS2pe28bVt+kMG5n9DW7ffZhautF1IL+AxwJq2Lf4L3ZXIboc7tw7/D/At4GLgH+muhHc7nH69nUZ33cIWuqNJr5jNbQ9Y3X4n3wPeRbsR3e68vJOdJEk95CF6SZJ6yICXJKmHDHhJknrIgJckqYcMeEmSesiAl+ZRkkryVwPjv5vkTbO07FOTPH/6nrv9Pr+Y7klu50xqPyTJGW34MUmeNYvvuTzJq6d6L0lTM+Cl+bUJ+O9JDhx2IYMG7mI2E68AXllVTxlsrKrvV9XEF4zH0N0rYbZqWE73ZLOp3kvSFAx4aX5tBd4DvGbyhMl74EnuaD+PTfIfST6Z5PIkb07y4iTnt+dHHzmwmKcnWZPkO+3+9xPPoH9rkq+2Z1P/r4Hl/meSM+nuZja5nhe15V+c5C2t7Y10Nz46JclbJ/Vf1fouBv4IeGGSbyR5YZJl6Z6nfX57UMzxbZ6XJTkzyReAs5Psk+TsJF9r7z3xVMc3A0e25b114r3aMpYmeX/r//UkTxlY9ieSfDbds7f/YmB9nNpqvSjJfX4XUh/szLd2SbPjb4ELJwJnhn4C+DG6x1VeDry3qp6Q5DeBXwd+q/VbRffYziOBc5I8lO62l7dV1eOTLAH+X5LPt/5HA4+q7pGf90hyCPAW4HHArcDnkzyvqv4oyVOB362qNVMVWlWb2xeB1VX1a215f0Z3m9OXJ1kOnJ/k3wdq+PGquqXtxf9CVd3ejnJ8pX0BeX2r8zFteasG3vJXu7etRyf50Vbrw9q0x9A9/XAT8O0kfwM8EDi0qh7VlrV8mnUvLUjuwUvzrLon8X0Q+I2dmO2rVbWuqjbR3cpyIqAvogv1CadX1XhVfZfui8CP0t3v+qVJvkH3mN8D6O4dDnD+5HBvHg+cW91DSbYCH6Z7Fvuuegbw+lbDuXS3QH1Qm3ZWVU08ZzvAnyW5kO7Wn4dy7yM4t+engA8BVNW3gKuAiYA/u6puq6qNdEcpHky3Xh6S5G+SHAfcPsUypQXPPXhpON4BfA14/0DbVtqX7iQjwOKBaZsGhscHxsf54X/Hk+89XXSh+etV9UMPVElyLN0jV+dDgP+vqr49qYafnFTDi4GVwOOqaku6p+8t3Y33HVxv24Cxqro1yU8AzwReBbyA7v7gUq+4By8NQdtjPZ3ugrUJV9IdEgd4LrBoFxb9i0lG2nn5h9A9CORzwK+ke+QvSR6WZNk0yzkf+NkkByYZBV4E/MdO1LEB2Hdg/HPAr7cnZZHksduZbz/gxhbuT6Hb455qeYP+k+6LAe3Q/IPoPveU2qH/kar6OPD7dKcIpN4x4KXh+Stg8Gr6f6AL1W8CT2TX9q6vpgvnzwCvaoem30t3ePpr7cK0/8s0R++qe3zl6+keI/pN4IKq2pnHV54DPGLiIjvgj+m+sFyY5JI2PpUPA6uTXER37cC3Wj030107cPHki/uAvwNG2jwfA17WTmVsz6HAue10wYeAk3fic0kLhk+TkySph9yDlySphwx4SZJ6yICXJKmHDHhJknrIgJckqYcMeEmSesiAlySphwx4SZJ66P8HvudQiwOC+6QAAAAASUVORK5CYII=\n", 185 | "text/plain": [ 186 | "
" 187 | ] 188 | }, 189 | "metadata": { 190 | "needs_background": "light" 191 | }, 192 | "output_type": "display_data" 193 | } 194 | ], 195 | "source": [ 196 | "import matplotlib.pyplot as plt\n", 197 | "fig = plt.figure(figsize=(8,6))\n", 198 | "plt.plot(np.arange(len(lr.loss)), lr.loss)\n", 199 | "plt.title(\"Development of cost over training\")\n", 200 | "plt.xlabel(\"Number of iterations\")\n", 201 | "plt.ylabel(\"Cost\")\n", 202 | "plt.show()" 203 | ] 204 | }, 205 | { 206 | "cell_type": "code", 207 | "execution_count": null, 208 | "metadata": {}, 209 | "outputs": [], 210 | "source": [] 211 | } 212 | ], 213 | "metadata": { 214 | "kernelspec": { 215 | "display_name": "python3", 216 | "language": "python", 217 | "name": "python3" 218 | }, 219 | "language_info": { 220 | "codemirror_mode": { 221 | "name": "ipython", 222 | "version": 2 223 | }, 224 | "file_extension": ".py", 225 | "mimetype": "text/x-python", 226 | "name": "python", 227 | "nbconvert_exporter": "python", 228 | "pygments_lexer": "ipython2", 229 | "version": "2.7.16" 230 | } 231 | }, 232 | "nbformat": 4, 233 | "nbformat_minor": 2 234 | } 235 | -------------------------------------------------------------------------------- /.ipynb_checkpoints/Multi Class Gaussian Discriminant Analyses-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np \n", 10 | "\n", 11 | "class GDAClassifier:\n", 12 | " \n", 13 | " def fit(self, X, y, epsilon = 1e-10):\n", 14 | " self.y_classes, y_counts = np.unique(y, return_counts=True)\n", 15 | " self.phi_y = 1.0 * y_counts/len(y)\n", 16 | " self.u = np.array([ X[y==k].mean(axis=0) for k in self.y_classes])\n", 17 | " self.E = self.compute_sigma(X, y)\n", 18 | " self.E += np.ones_like(self.E) * epsilon # fix zero overflow\n", 19 | " self.invE = np.linalg.pinv(self.E)\n", 20 | " return self\n", 21 | " \n", 22 | " def compute_sigma(self,X, y):\n", 23 | " X_u = X.copy().astype('float64')\n", 24 | " for i in range(len(self.u)):\n", 25 | " X_u[y==self.y_classes[i]] -= self.u[i]\n", 26 | " return X_u.T.dot(X_u) / len(y)\n", 27 | "\n", 28 | " def predict(self, X):\n", 29 | " return np.apply_along_axis(self.get_prob, 1, X)\n", 30 | " \n", 31 | " def score(self, X, y):\n", 32 | " return (self.predict(X) == y).mean()\n", 33 | " \n", 34 | " def get_prob(self, x):\n", 35 | " p = np.exp(-1.0 * np.sum((x - self.u).dot(self.invE) * (x - self.u), axis =1)) * self.phi_y\n", 36 | " return np.argmax(p)" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 2, 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "from utils import train_test_split\n", 46 | "from sklearn.datasets import load_iris\n", 47 | "X,y = load_iris(return_X_y=True)\n", 48 | "X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.8)\n", 49 | "model = GDAClassifier().fit(X_train,y_train)" 50 | ] 51 | }, 52 | { 53 | "cell_type": "code", 54 | "execution_count": 3, 55 | "metadata": {}, 56 | "outputs": [ 57 | { 58 | "data": { 59 | "text/plain": [ 60 | "0.975" 61 | ] 62 | }, 63 | "execution_count": 3, 64 | "metadata": {}, 65 | "output_type": "execute_result" 66 | } 67 | ], 68 | "source": [ 69 | "model.score(X_test,y_test)" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 5, 75 | "metadata": {}, 76 | "outputs": [ 77 | { 78 | "data": { 79 | "text/plain": [ 80 | "0.9296703296703297" 81 | ] 82 | }, 83 | "execution_count": 5, 84 | "metadata": {}, 85 | "output_type": "execute_result" 86 | } 87 | ], 88 | "source": [ 89 | "from sklearn.datasets import load_breast_cancer\n", 90 | "X,y = load_breast_cancer(return_X_y=True)\n", 91 | "X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.8)\n", 92 | "model = GDAClassifier().fit(X_train,y_train)\n", 93 | "model.score(X_test,y_test)" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": 6, 99 | "metadata": {}, 100 | "outputs": [ 101 | { 102 | "data": { 103 | "text/plain": [ 104 | "0.9510022271714922" 105 | ] 106 | }, 107 | "execution_count": 6, 108 | "metadata": {}, 109 | "output_type": "execute_result" 110 | } 111 | ], 112 | "source": [ 113 | "from sklearn.datasets import load_digits\n", 114 | "digits = load_digits()\n", 115 | "X = digits.data\n", 116 | "y = digits.target\n", 117 | "X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.5)\n", 118 | "model = GDAClassifier().fit(X_train,y_train)\n", 119 | "model.score(X_test,y_test)" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": null, 125 | "metadata": {}, 126 | "outputs": [], 127 | "source": [] 128 | } 129 | ], 130 | "metadata": { 131 | "kernelspec": { 132 | "display_name": "python3", 133 | "language": "python", 134 | "name": "python3" 135 | }, 136 | "language_info": { 137 | "codemirror_mode": { 138 | "name": "ipython", 139 | "version": 2 140 | }, 141 | "file_extension": ".py", 142 | "mimetype": "text/x-python", 143 | "name": "python", 144 | "nbconvert_exporter": "python", 145 | "pygments_lexer": "ipython2", 146 | "version": "2.7.16" 147 | } 148 | }, 149 | "nbformat": 4, 150 | "nbformat_minor": 2 151 | } 152 | -------------------------------------------------------------------------------- /.ipynb_checkpoints/Multinomial Naive Bayes-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 4, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# %load naive_bayes.py\n", 10 | "import numpy as np\n", 11 | "\n", 12 | "class NaiveBayesBinaryClassifier:\n", 13 | " \n", 14 | " def fit(self, X, y):\n", 15 | " self.y_classes, y_counts = np.unique(y, return_counts=True)\n", 16 | " self.phi_y = 1.0 * y_counts/y_counts.sum()\n", 17 | " self.phi_x = [1.0 * X[y==k].mean(axis=0) for k in self.y_classes]\n", 18 | " return self\n", 19 | " \n", 20 | " def predict(self, X):\n", 21 | " return np.apply_along_axis(lambda x: self.compute_probs(x), 1, X)\n", 22 | " \n", 23 | " def compute_probs(self, x):\n", 24 | " probs = np.array([self.compute_prob(x, y) for y in range(len(self.y_classes))])\n", 25 | " return self.y_classes[np.argmax(probs)]\n", 26 | " \n", 27 | " def compute_prob(self, x, y):\n", 28 | " res = 1\n", 29 | " for j in range(len(x)):\n", 30 | " Pxy = self.phi_x[y][j] # p(xj=1|y)\n", 31 | " res *= (Pxy**x[j])*((1-Pxy)**(1-x[j])) # p(xj=0|y)\n", 32 | " return res * self.phi_y[y]\n", 33 | " \n", 34 | " def evaluate(self, X, y):\n", 35 | " return (self.predict(X) == y).mean()\n", 36 | " \n", 37 | "class MultinomialNB:\n", 38 | " \n", 39 | " def fit(self, X, y):\n", 40 | " self.y_classes, y_counts = np.unique(y, return_counts=True)\n", 41 | " self.x_classes = np.array([np.unique(x) for x in X.T])\n", 42 | " self.phi_y = 1.0 * y_counts/y_counts.sum()\n", 43 | " self.phi_x = self.mean_x(X, y)\n", 44 | " return self\n", 45 | " \n", 46 | " def mean_x(self, X, y):\n", 47 | " return [[(X[:,j][y==k].reshape(-1,1) == self.x_classes[j]).mean(axis=0)\n", 48 | " for j in range(len(self.x_classes))]\n", 49 | " for k in self.y_classes]\n", 50 | " \n", 51 | " def predict(self, X):\n", 52 | " return np.apply_along_axis(lambda x: self.compute_probs(x), 1, X)\n", 53 | " \n", 54 | " def compute_probs(self, x):\n", 55 | " probs = np.array([self.compute_prob(x, y) for y in range(len(self.y_classes))])\n", 56 | " return self.y_classes[np.argmax(probs)]\n", 57 | " \n", 58 | " def compute_prob(self, x, y):\n", 59 | " Pxy = 1\n", 60 | " for j in range(len(x)):\n", 61 | " i = list(self.x_classes[j]).index(x[j])\n", 62 | " Pxy *= self.phi_x[y][j][i] # p(xj|y)\n", 63 | " return Pxy * self.phi_y[y]\n", 64 | " \n", 65 | " def evaluate(self, X, y):\n", 66 | " return (self.predict(X) == y).mean()\n", 67 | " \n", 68 | " \n", 69 | "class GaussianNB:\n", 70 | " \n", 71 | " def fit(self, X, y, epsilon = 1e-10):\n", 72 | " self.y_classes, y_counts = np.unique(y, return_counts=True)\n", 73 | " self.x_classes = np.array([np.unique(x) for x in X.T])\n", 74 | " self.phi_y = 1.0 * y_counts/y_counts.sum()\n", 75 | " self.u = np.array([X[y==k].mean(axis=0) for k in self.y_classes])\n", 76 | " self.var_x = np.array([X[y==k].var(axis=0) + epsilon for k in self.y_classes])\n", 77 | " return self\n", 78 | " \n", 79 | " def predict(self, X):\n", 80 | " return np.apply_along_axis(lambda x: self.compute_probs(x), 1, X)\n", 81 | " \n", 82 | " def compute_probs(self, x):\n", 83 | " probs = np.array([self.compute_prob(x, y) for y in range(len(self.y_classes))])\n", 84 | " return self.y_classes[np.argmax(probs)]\n", 85 | " \n", 86 | " def compute_prob(self, x, y):\n", 87 | " c = 1.0 /np.sqrt(2.0 * np.pi * (self.var_x[y]))\n", 88 | " return np.prod(c * np.exp(-1.0 * np.square(x - self.u[y]) / (2.0 * self.var_x[y])))\n", 89 | " \n", 90 | " def evaluate(self, X, y):\n", 91 | " return (self.predict(X) == y).mean()\n", 92 | " " 93 | ] 94 | }, 95 | { 96 | "cell_type": "code", 97 | "execution_count": 5, 98 | "metadata": {}, 99 | "outputs": [ 100 | { 101 | "data": { 102 | "text/plain": [ 103 | "0.9666666666666667" 104 | ] 105 | }, 106 | "execution_count": 5, 107 | "metadata": {}, 108 | "output_type": "execute_result" 109 | } 110 | ], 111 | "source": [ 112 | "from sklearn import datasets\n", 113 | "from utils import accuracy_score\n", 114 | "iris = datasets.load_iris()\n", 115 | "X = iris.data \n", 116 | "y = iris.target\n", 117 | "MultinomialNB().fit(X, y).evaluate(X, y)" 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": 72, 123 | "metadata": {}, 124 | "outputs": [ 125 | { 126 | "data": { 127 | "text/plain": [ 128 | "array([5.1, 3.5, 1.4, 0.2])" 129 | ] 130 | }, 131 | "execution_count": 72, 132 | "metadata": {}, 133 | "output_type": "execute_result" 134 | } 135 | ], 136 | "source": [ 137 | "X [0]" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": 65, 143 | "metadata": {}, 144 | "outputs": [ 145 | { 146 | "data": { 147 | "text/plain": [ 148 | "0.96" 149 | ] 150 | }, 151 | "execution_count": 65, 152 | "metadata": {}, 153 | "output_type": "execute_result" 154 | } 155 | ], 156 | "source": [ 157 | "from sklearn import datasets\n", 158 | "from utils import accuracy_score\n", 159 | "iris = datasets.load_iris()\n", 160 | "X = iris.data\n", 161 | "y = iris.target\n", 162 | "GaussianNB().fit(X, y).evaluate(X, y)" 163 | ] 164 | }, 165 | { 166 | "cell_type": "code", 167 | "execution_count": 66, 168 | "metadata": {}, 169 | "outputs": [ 170 | { 171 | "data": { 172 | "text/plain": [ 173 | "0.7533333333333333" 174 | ] 175 | }, 176 | "execution_count": 66, 177 | "metadata": {}, 178 | "output_type": "execute_result" 179 | } 180 | ], 181 | "source": [ 182 | "from sklearn import datasets\n", 183 | "from utils import accuracy_score\n", 184 | "\n", 185 | "iris = datasets.load_iris()\n", 186 | "X = iris.data \n", 187 | "X = (X > X.mean(axis=0))*1 # turn our feature to categorical using mean\n", 188 | "y = iris.target\n", 189 | "NaiveBayesBinaryClassifier().fit(X, y).evaluate(X, y)" 190 | ] 191 | }, 192 | { 193 | "cell_type": "code", 194 | "execution_count": 67, 195 | "metadata": {}, 196 | "outputs": [ 197 | { 198 | "name": "stdout", 199 | "output_type": "stream", 200 | "text": [ 201 | "[1 1 1 2 2 2]\n" 202 | ] 203 | } 204 | ], 205 | "source": [ 206 | "X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])\n", 207 | "Y = np.array([1, 1, 1, 2, 2, 2])\n", 208 | "clf = MultinomialNB().fit(X, Y)\n", 209 | "print(clf.predict(X))" 210 | ] 211 | }, 212 | { 213 | "cell_type": "code", 214 | "execution_count": 68, 215 | "metadata": {}, 216 | "outputs": [ 217 | { 218 | "data": { 219 | "text/plain": [ 220 | "0.9833055091819699" 221 | ] 222 | }, 223 | "execution_count": 68, 224 | "metadata": {}, 225 | "output_type": "execute_result" 226 | } 227 | ], 228 | "source": [ 229 | "from sklearn import datasets\n", 230 | "digits = datasets.load_digits()\n", 231 | "X = digits.data\n", 232 | "y = digits.target\n", 233 | "MultinomialNB().fit(X, y).evaluate(X, y)" 234 | ] 235 | }, 236 | { 237 | "cell_type": "code", 238 | "execution_count": 69, 239 | "metadata": {}, 240 | "outputs": [ 241 | { 242 | "data": { 243 | "text/plain": [ 244 | "0.8091263216471898" 245 | ] 246 | }, 247 | "execution_count": 69, 248 | "metadata": {}, 249 | "output_type": "execute_result" 250 | } 251 | ], 252 | "source": [ 253 | "from sklearn import datasets\n", 254 | "digits = datasets.load_digits()\n", 255 | "X = digits.data\n", 256 | "y = digits.target\n", 257 | "GaussianNB().fit(X, y).evaluate(X, y)" 258 | ] 259 | }, 260 | { 261 | "cell_type": "code", 262 | "execution_count": 70, 263 | "metadata": {}, 264 | "outputs": [], 265 | "source": [ 266 | "b = np.array([0, 1])\n", 267 | "a = np.array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", 268 | " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", 269 | " 0, 0, 0, 0, 0, 0])" 270 | ] 271 | }, 272 | { 273 | "cell_type": "code", 274 | "execution_count": 61, 275 | "metadata": {}, 276 | "outputs": [ 277 | { 278 | "data": { 279 | "text/plain": [ 280 | "0.9666666666666667" 281 | ] 282 | }, 283 | "execution_count": 61, 284 | "metadata": {}, 285 | "output_type": "execute_result" 286 | } 287 | ], 288 | "source": [ 289 | "from sklearn import datasets\n", 290 | "from utils import accuracy_score, train_test_split\n", 291 | "iris = datasets.load_iris()\n", 292 | "X = iris.data\n", 293 | "# X = (X-X.min(axis=0)).astype(int)\n", 294 | "y = iris.target\n", 295 | "X_tr, X_ts, y_tr, y_ts = train_test_split(X,y)\n", 296 | "GaussianNB().fit(X_tr, y_tr).evaluate(X_ts, y_ts)" 297 | ] 298 | }, 299 | { 300 | "cell_type": "markdown", 301 | "metadata": {}, 302 | "source": [ 303 | "### Naive Bayes" 304 | ] 305 | }, 306 | { 307 | "cell_type": "code", 308 | "execution_count": 40, 309 | "metadata": {}, 310 | "outputs": [ 311 | { 312 | "data": { 313 | "text/plain": [ 314 | "array([[0, 1, 0, 0],\n", 315 | " [2, 1, 3, 1],\n", 316 | " [2, 1, 3, 1],\n", 317 | " [0, 1, 0, 0],\n", 318 | " [1, 0, 3, 1],\n", 319 | " [2, 0, 3, 1],\n", 320 | " [2, 1, 4, 2],\n", 321 | " [2, 1, 3, 1],\n", 322 | " [1, 0, 3, 1],\n", 323 | " [2, 1, 5, 2]])" 324 | ] 325 | }, 326 | "execution_count": 40, 327 | "metadata": {}, 328 | "output_type": "execute_result" 329 | } 330 | ], 331 | "source": [ 332 | "X_tr[:10]" 333 | ] 334 | }, 335 | { 336 | "cell_type": "code", 337 | "execution_count": 41, 338 | "metadata": {}, 339 | "outputs": [ 340 | { 341 | "data": { 342 | "text/plain": [ 343 | "array([0, 1, 1, 0, 1, 1, 2, 1, 1, 2])" 344 | ] 345 | }, 346 | "execution_count": 41, 347 | "metadata": {}, 348 | "output_type": "execute_result" 349 | } 350 | ], 351 | "source": [ 352 | "y_classes = [0 1 2]\n", 353 | "P(y) = []" 354 | ] 355 | }, 356 | { 357 | "cell_type": "code", 358 | "execution_count": null, 359 | "metadata": {}, 360 | "outputs": [], 361 | "source": [ 362 | "p(y|x) = p(x|y) * p(y)\n", 363 | "x_uniques = [0 1 2]" 364 | ] 365 | }, 366 | { 367 | "cell_type": "code", 368 | "execution_count": null, 369 | "metadata": {}, 370 | "outputs": [], 371 | "source": [ 372 | "p(y|x) = p(x|y) * p(y) / p(x)" 373 | ] 374 | }, 375 | { 376 | "cell_type": "code", 377 | "execution_count": null, 378 | "metadata": {}, 379 | "outputs": [], 380 | "source": [ 381 | "p(x|y) " 382 | ] 383 | }, 384 | { 385 | "cell_type": "code", 386 | "execution_count": null, 387 | "metadata": {}, 388 | "outputs": [], 389 | "source": [ 390 | "p(y) = [0.2, 0.6, 0.2]" 391 | ] 392 | }, 393 | { 394 | "cell_type": "code", 395 | "execution_count": null, 396 | "metadata": {}, 397 | "outputs": [], 398 | "source": [ 399 | "p(x0|y=0) = [1 0 0]\n", 400 | "p(x0|y=1) = [0 0.33 0.6]\n", 401 | "p(x0|y=2) = [0 0.33 0.6]" 402 | ] 403 | } 404 | ], 405 | "metadata": { 406 | "kernelspec": { 407 | "display_name": "python3", 408 | "language": "python", 409 | "name": "python3" 410 | }, 411 | "language_info": { 412 | "codemirror_mode": { 413 | "name": "ipython", 414 | "version": 2 415 | }, 416 | "file_extension": ".py", 417 | "mimetype": "text/x-python", 418 | "name": "python", 419 | "nbconvert_exporter": "python", 420 | "pygments_lexer": "ipython2", 421 | "version": "2.7.16" 422 | } 423 | }, 424 | "nbformat": 4, 425 | "nbformat_minor": 2 426 | } 427 | -------------------------------------------------------------------------------- /.ipynb_checkpoints/Naive Bayes Implementation-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Naive Bayes Classifier Implementation using numpy\n", 8 | "\n", 9 | "Naive Bayes is anoother supervised machine laerning algorithm for classification problem. It makes a strong assumption about the data that **each feature is independent of the value of any other feature**. For example, a fruit may be considered to be an apple if it is red, round, and about 10 cm in diameter. A naive Bayes classifier considers each of these features to contribute independently to the probability that this fruit is an apple, regardless of any possible correlations between the color, roundness, and diameter features.\n", 10 | "\n", 11 | "In Naive bayes classifier what we are trying to find is the probability that a given data point belogs to a specific class, we are going to have prediction for all the class in our target.\n", 12 | "\n", 13 | "\n", 14 | "![title](figures/bayes-theorem.png)\n", 15 | "\n", 16 | "### Our motivation\n", 17 | "To gain better understand on how the algorithm works" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 3, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "from sklearn import datasets\n", 27 | "digits = datasets.load_digits()\n", 28 | "X = digits.data\n", 29 | "y = digits.target" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": null, 35 | "metadata": {}, 36 | "outputs": [], 37 | "source": [ 38 | "p(y|x) = p(x|y) * p(y)" 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": null, 44 | "metadata": {}, 45 | "outputs": [], 46 | "source": [ 47 | "class MultinomialNB:\n", 48 | " \n", 49 | " def fit(self, X, y):\n", 50 | " unique_y, y_counts = np.unique(y, return_counts=True)\n", 51 | " self.prob_y = y_counts / len(y)\n", 52 | " self.unique_y = unique_y\n", 53 | " self.unique_x = np.array([np.unique(x_t) for x_t in X.T])\n", 54 | " self.prob_x = self.get_prob_x(X, y)\n", 55 | " return self\n", 56 | " \n", 57 | " def get_prob_x(X, y):\n", 58 | " return []\n", 59 | " \n", 60 | " def predict(self, X):\n", 61 | " pass\n", 62 | " \n", 63 | " def score(self, X, y):\n", 64 | " " 65 | ] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": null, 70 | "metadata": {}, 71 | "outputs": [], 72 | "source": [ 73 | "model = MultinomialNB().fit(X, y)\n", 74 | "model.score(X, y)" 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "execution_count": 13, 80 | "metadata": {}, 81 | "outputs": [ 82 | { 83 | "data": { 84 | "text/plain": [ 85 | "array([[ 0., 0., 5.],\n", 86 | " [ 0., 0., 0.],\n", 87 | " [ 0., 0., 0.],\n", 88 | " [ 0., 0., 7.],\n", 89 | " [ 0., 0., 0.],\n", 90 | " [ 0., 0., 12.],\n", 91 | " [ 0., 0., 0.],\n", 92 | " [ 0., 0., 7.],\n", 93 | " [ 0., 0., 9.],\n", 94 | " [ 0., 0., 11.]])" 95 | ] 96 | }, 97 | "execution_count": 13, 98 | "metadata": {}, 99 | "output_type": "execute_result" 100 | } 101 | ], 102 | "source": [ 103 | "X[:10, :3]" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": 8, 109 | "metadata": {}, 110 | "outputs": [ 111 | { 112 | "data": { 113 | "text/plain": [ 114 | "array([0, 1])" 115 | ] 116 | }, 117 | "execution_count": 8, 118 | "metadata": {}, 119 | "output_type": "execute_result" 120 | } 121 | ], 122 | "source": [ 123 | "y[:2]" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": null, 129 | "metadata": {}, 130 | "outputs": [], 131 | "source": [] 132 | } 133 | ], 134 | "metadata": { 135 | "kernelspec": { 136 | "display_name": "python3", 137 | "language": "python", 138 | "name": "python3" 139 | }, 140 | "language_info": { 141 | "codemirror_mode": { 142 | "name": "ipython", 143 | "version": 2 144 | }, 145 | "file_extension": ".py", 146 | "mimetype": "text/x-python", 147 | "name": "python", 148 | "nbconvert_exporter": "python", 149 | "pygments_lexer": "ipython2", 150 | "version": "2.7.16" 151 | } 152 | }, 153 | "nbformat": 4, 154 | "nbformat_minor": 2 155 | } 156 | -------------------------------------------------------------------------------- /BayesianClassifier.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 3, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "name": "stdout", 10 | "output_type": "stream", 11 | "text": [ 12 | "COLUMN\tVALUE\tOUTPUT\tPROBABILITY\n", 13 | "Temp--->Hot--->Rainy--->0.4\n", 14 | "Temp--->Mild--->Rainy--->0.4\n", 15 | "Temp--->Cool--->Rainy--->0.2\n", 16 | "Humidity--->High--->Rainy--->0.6\n", 17 | "Humidity--->Normal--->Rainy--->0.4\n", 18 | "Windy--->Low--->Rainy--->0.6\n", 19 | "Windy--->High--->Rainy--->0.4\n", 20 | "Cloudy--->Yes--->Rainy--->0.6\n", 21 | "Cloudy--->No--->Rainy--->0.4\n", 22 | "Temp--->Hot--->Overcast--->0.5\n", 23 | "Temp--->Mild--->Overcast--->0.25\n", 24 | "Temp--->Cool--->Overcast--->0.25\n", 25 | "Humidity--->High--->Overcast--->0.5\n", 26 | "Humidity--->Normal--->Overcast--->0.5\n", 27 | "Windy--->Low--->Overcast--->0.5\n", 28 | "Windy--->High--->Overcast--->0.5\n", 29 | "Cloudy--->Yes--->Overcast--->0.5\n", 30 | "Cloudy--->No--->Overcast--->0.5\n", 31 | "Temp--->Hot--->Sunny--->0.0\n", 32 | "Temp--->Mild--->Sunny--->0.6\n", 33 | "Temp--->Cool--->Sunny--->0.4\n", 34 | "Humidity--->High--->Sunny--->0.4\n", 35 | "Humidity--->Normal--->Sunny--->0.6\n", 36 | "Windy--->Low--->Sunny--->0.6\n", 37 | "Windy--->High--->Sunny--->0.4\n", 38 | "Cloudy--->Yes--->Sunny--->0.2\n", 39 | "Cloudy--->No--->Sunny--->0.8\n", 40 | "{'Rainy': 5, 'Overcast': 4, 'Sunny': 5}\n", 41 | "{'Rainy': 0.6162624821683309, 'Overcast': 0.17831669044222537, 'Sunny': 0.20542082738944364}\n" 42 | ] 43 | } 44 | ], 45 | "source": [ 46 | "import pandas as pd\n", 47 | "\n", 48 | "class BayesianClassifier:\n", 49 | "\n", 50 | " def __init__(self, path, col):\n", 51 | " self.cols = {}\n", 52 | " self.op_cols = {}\n", 53 | " self.col = col\n", 54 | " self.data = pd.read_csv(path)\n", 55 | " self.dataM = self.data\n", 56 | " self.op = self.data[col]\n", 57 | " self.data = self.data.drop(col, axis=1)\n", 58 | " self.total = len(self.dataM)\n", 59 | " \n", 60 | " def get_probability_table(self):\n", 61 | " for i in self.op.unique():\n", 62 | " self.cols[i] = {}\n", 63 | " for j in self.data.columns:\n", 64 | " self.cols[i][j] = {}\n", 65 | " for k in self.data[j]:\n", 66 | " if k not in self.cols[i][j]:\n", 67 | " self.cols[i][j][k] = \"\"\n", 68 | " dfs = []\n", 69 | " for i in self.op.unique():\n", 70 | " for j in self.data.columns:\n", 71 | " for k in self.data[j].unique():\n", 72 | " dfs.append(self.dataM[self.dataM[self.col] == i])\n", 73 | "\n", 74 | " \n", 75 | " print(\"COLUMN\\tVALUE\\tOUTPUT\\tPROBABILITY\")\n", 76 | " for x in self.cols:\n", 77 | " for y in self.cols[x]:\n", 78 | " for z in self.cols[x][y]:\n", 79 | " self.op_cols[x] = len(self.dataM[self.dataM[self.col] == x])\n", 80 | " total = len(self.dataM[(self.dataM[self.col] == x) & (self.dataM[y])])\n", 81 | " p = len(self.dataM[(self.dataM[y] == z) & (self.dataM[self.col] == x)]) / total\n", 82 | " self.cols[x][y][z] = p\n", 83 | " print(\"{}--->{}--->{}--->{}\".format(y,z,x,p))\n", 84 | " print(self.op_cols)\n", 85 | " \n", 86 | " def classify(self, values):\n", 87 | " p = {}\n", 88 | " tot = 0\n", 89 | " for i in self.op.unique():\n", 90 | " a = 1\n", 91 | " for key, value in values.items():\n", 92 | " a *= self.cols[i][key][value]\n", 93 | " p[i] = a*self.op_cols[i]/self.total\n", 94 | " tot += a*self.op_cols[i]/self.total\n", 95 | " \n", 96 | " for i in p:\n", 97 | " p[i] = p[i] / tot\n", 98 | " \n", 99 | " return p\n", 100 | " \n", 101 | "b = BayesianClassifier(\"new.csv\", \"Weather\")\n", 102 | "\n", 103 | "b.get_probability_table()\n", 104 | "print(b.classify({\"Temp\": 'Mild', \"Windy\": \"Low\", \"Humidity\": \"High\", \"Cloudy\": \"Yes\"}))" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": null, 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [] 113 | } 114 | ], 115 | "metadata": { 116 | "kernelspec": { 117 | "display_name": "Python 3", 118 | "language": "python", 119 | "name": "python3" 120 | }, 121 | "language_info": { 122 | "codemirror_mode": { 123 | "name": "ipython", 124 | "version": 3 125 | }, 126 | "file_extension": ".py", 127 | "mimetype": "text/x-python", 128 | "name": "python", 129 | "nbconvert_exporter": "python", 130 | "pygments_lexer": "ipython3", 131 | "version": "3.7.3" 132 | } 133 | }, 134 | "nbformat": 4, 135 | "nbformat_minor": 2 136 | } 137 | -------------------------------------------------------------------------------- /Binary Naive Bayes.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Naive Bayes Classifier Implementation using numpy\n", 8 | "\n", 9 | "Naive Bayes is anoother supervised machine laerning algorithm for classification problem. It makes a strong assumption about the data that **each feature is independent of the value of any other feature**. For example, a fruit may be considered to be an apple if it is red, round, and about 10 cm in diameter. A naive Bayes classifier considers each of these features to contribute independently to the probability that this fruit is an apple, regardless of any possible correlations between the color, roundness, and diameter features.\n", 10 | "\n", 11 | "In Naive bayes classifier what we are trying to find is the probability that a given data point belogs to a specific class, we are going to have prediction for all the class in our target.\n", 12 | "\n", 13 | "\n", 14 | "![title](figures/bayes-theorem.png)\n", 15 | "\n", 16 | "This is bernolli naive bayes impementation, which we expecting the features to be true or false (1 or 0)." 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": 72, 22 | "metadata": {}, 23 | "outputs": [], 24 | "source": [ 25 | "# %load naive_bayes.py\n", 26 | "import numpy as np\n", 27 | "\n", 28 | "class NaiveBayesBinaryClassifier:\n", 29 | " \n", 30 | " def fit(self, X, y):\n", 31 | " self.y_classes, y_counts = np.unique(y, return_counts=True)\n", 32 | " self.phi_y = 1.0 * y_counts/y_counts.sum()\n", 33 | " self.phi_x = [1.0 * X[y==k].mean(axis=0) for k in self.y_classes]\n", 34 | " return self\n", 35 | " \n", 36 | " def predict(self, X):\n", 37 | " return np.apply_along_axis(lambda x: self.compute_probs(x), 1, X)\n", 38 | " \n", 39 | " def compute_probs(self, x):\n", 40 | " probs = [self.compute_prob(x, y) for y in range(len(self.y_classes))]\n", 41 | " return self.y_classes[np.argmax(probs)]\n", 42 | " \n", 43 | " def compute_prob(self, x, y):\n", 44 | " res = 1\n", 45 | " for j in range(len(x)):\n", 46 | " Pxy = self.phi_x[y][j] # p(xj=1|y)\n", 47 | " res *= (Pxy**x[j])*((1-Pxy)**(1-x[j])) # p(xj=0|y)\n", 48 | " return res * self.phi_y[y]\n", 49 | " \n", 50 | " def score(self, X, y):\n", 51 | " return (self.predict(X) == y).mean()" 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": null, 57 | "metadata": {}, 58 | "outputs": [], 59 | "source": [] 60 | } 61 | ], 62 | "metadata": { 63 | "kernelspec": { 64 | "display_name": "python3", 65 | "language": "python", 66 | "name": "python3" 67 | }, 68 | "language_info": { 69 | "codemirror_mode": { 70 | "name": "ipython", 71 | "version": 2 72 | }, 73 | "file_extension": ".py", 74 | "mimetype": "text/x-python", 75 | "name": "python", 76 | "nbconvert_exporter": "python", 77 | "pygments_lexer": "ipython2", 78 | "version": "2.7.16" 79 | } 80 | }, 81 | "nbformat": 4, 82 | "nbformat_minor": 2 83 | } 84 | -------------------------------------------------------------------------------- /Gaussian Discriminant Analyses.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Gaussian Descriminant Analysis Implementation using numpy\n", 8 | "\n", 9 | "When we have a classification problem in which the input features x are\n", 10 | "continuous-valued random variables, we can then use the Gaussian Discriminant Analysis (GDA) model, which models p(x|y) using a multivariate normal distribution.\n", 11 | "\n", 12 | "GDA is a family of generative algorithm that try to model p(x|y) (and p(y)). After modeling p(y) (called the class priors) and p(x|y), our algorithm can then use Bayes rule to derive the posterior distribution on y given x:\n", 13 | "\n", 14 | "\n", 15 | "![title](figures/bayes-theorem.png)\n", 16 | "\n", 17 | "GDA makes stronger modeling assumptions, and is more data efficient (i.e., requires less training data to learn “well”) when the modeling assumptions are correct or at least approximately correct. Comparing to Logistic regression that makes weaker assumptions, and is significantly more robust to deviations from modeling assumptions\n", 18 | "\n", 19 | "### Our motivation\n", 20 | "To gain better understand on how the algorithm works" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": 4, 26 | "metadata": {}, 27 | "outputs": [], 28 | "source": [ 29 | "# %load gda.py\n", 30 | "import numpy as np\n", 31 | "\n", 32 | "class GDABinaryClassifier:\n", 33 | " \n", 34 | " def fit(self, X, y):\n", 35 | " self.fi = y.mean()\n", 36 | " self.u = np.array([ X[y==k].mean(axis=0) for k in [0,1]])\n", 37 | " X_u = X.copy()\n", 38 | " for k in [0,1]: X_u[y==k] -= self.u[k]\n", 39 | " self.E = X_u.T.dot(X_u) / len(y)\n", 40 | " self.invE = np.linalg.pinv(self.E)\n", 41 | " return self\n", 42 | " \n", 43 | " def predict(self, X):\n", 44 | " return np.argmax([self.compute_prob(X, i) for i in range(len(self.u))], axis=0)\n", 45 | " \n", 46 | " def compute_prob(self, X, i):\n", 47 | " u, phi = self.u[i], ((self.fi)**i * (1 - self.fi)**(1 - i))\n", 48 | " return np.exp(-1.0 * np.sum((X-u).dot(self.invE)*(X-u), axis=1)) * phi\n", 49 | " \n", 50 | " def score(self, X, y):\n", 51 | " return (self.predict(X) == y).mean()" 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": 5, 57 | "metadata": {}, 58 | "outputs": [ 59 | { 60 | "data": { 61 | "text/plain": [ 62 | "0.9666080843585237" 63 | ] 64 | }, 65 | "execution_count": 5, 66 | "metadata": {}, 67 | "output_type": "execute_result" 68 | } 69 | ], 70 | "source": [ 71 | "from sklearn.datasets import load_breast_cancer\n", 72 | "X,y = load_breast_cancer(return_X_y=True)\n", 73 | "model = GDABinaryClassifier().fit(X,y)\n", 74 | "pre = model.predict(X)\n", 75 | "model.score(X,y)" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": null, 81 | "metadata": {}, 82 | "outputs": [], 83 | "source": [] 84 | } 85 | ], 86 | "metadata": { 87 | "kernelspec": { 88 | "display_name": "python3", 89 | "language": "python", 90 | "name": "python3" 91 | }, 92 | "language_info": { 93 | "codemirror_mode": { 94 | "name": "ipython", 95 | "version": 2 96 | }, 97 | "file_extension": ".py", 98 | "mimetype": "text/x-python", 99 | "name": "python", 100 | "nbconvert_exporter": "python", 101 | "pygments_lexer": "ipython2", 102 | "version": "2.7.16" 103 | } 104 | }, 105 | "nbformat": 4, 106 | "nbformat_minor": 2 107 | } 108 | -------------------------------------------------------------------------------- /Gaussian Naive Bayes.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# %load gaussian_naive_bayes.py\n", 10 | "import numpy as np\n", 11 | " \n", 12 | "class GaussianNB:\n", 13 | " \n", 14 | " def fit(self, X, y, epsilon = 1e-10):\n", 15 | " self.y_classes, y_counts = np.unique(y, return_counts=True)\n", 16 | " self.x_classes = np.array([np.unique(x) for x in X.T])\n", 17 | " self.phi_y = 1.0 * y_counts/y_counts.sum()\n", 18 | " self.u = np.array([X[y==k].mean(axis=0) for k in self.y_classes])\n", 19 | " self.var_x = np.array([X[y==k].var(axis=0) + epsilon for k in self.y_classes])\n", 20 | " return self\n", 21 | " \n", 22 | " def predict(self, X):\n", 23 | " return np.apply_along_axis(lambda x: self.compute_probs(x), 1, X)\n", 24 | " \n", 25 | " def compute_probs(self, x):\n", 26 | " probs = np.array([self.compute_prob(x, y) for y in range(len(self.y_classes))])\n", 27 | " return self.y_classes[np.argmax(probs)]\n", 28 | " \n", 29 | " def compute_prob(self, x, y):\n", 30 | " c = 1.0 /np.sqrt(2.0 * np.pi * (self.var_x[y]))\n", 31 | " return np.prod(c * np.exp(-1.0 * np.square(x - self.u[y]) / (2.0 * self.var_x[y])))\n", 32 | " \n", 33 | " def evaluate(self, X, y):\n", 34 | " return (self.predict(X) == y).mean()" 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": 11, 40 | "metadata": {}, 41 | "outputs": [ 42 | { 43 | "data": { 44 | "text/plain": [ 45 | "0.96" 46 | ] 47 | }, 48 | "execution_count": 11, 49 | "metadata": {}, 50 | "output_type": "execute_result" 51 | } 52 | ], 53 | "source": [ 54 | "from sklearn import datasets\n", 55 | "from utils import accuracy_score\n", 56 | "iris = datasets.load_iris()\n", 57 | "X = iris.data \n", 58 | "y = iris.target\n", 59 | "GaussianNB().fit(X, y).evaluate(X, y)" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 12, 65 | "metadata": {}, 66 | "outputs": [ 67 | { 68 | "name": "stdout", 69 | "output_type": "stream", 70 | "text": [ 71 | "[1 1 1 2 2 2]\n" 72 | ] 73 | } 74 | ], 75 | "source": [ 76 | "X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])\n", 77 | "Y = np.array([1, 1, 1, 2, 2, 2])\n", 78 | "clf = GaussianNB().fit(X, Y)\n", 79 | "print(clf.predict(X))" 80 | ] 81 | }, 82 | { 83 | "cell_type": "code", 84 | "execution_count": 13, 85 | "metadata": {}, 86 | "outputs": [ 87 | { 88 | "data": { 89 | "text/plain": [ 90 | "0.8091263216471898" 91 | ] 92 | }, 93 | "execution_count": 13, 94 | "metadata": {}, 95 | "output_type": "execute_result" 96 | } 97 | ], 98 | "source": [ 99 | "from sklearn import datasets\n", 100 | "digits = datasets.load_digits()\n", 101 | "X = digits.data\n", 102 | "y = digits.target\n", 103 | "GaussianNB().fit(X, y).evaluate(X, y)" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": null, 109 | "metadata": {}, 110 | "outputs": [], 111 | "source": [] 112 | } 113 | ], 114 | "metadata": { 115 | "kernelspec": { 116 | "display_name": "python3", 117 | "language": "python", 118 | "name": "python3" 119 | }, 120 | "language_info": { 121 | "codemirror_mode": { 122 | "name": "ipython", 123 | "version": 2 124 | }, 125 | "file_extension": ".py", 126 | "mimetype": "text/x-python", 127 | "name": "python", 128 | "nbconvert_exporter": "python", 129 | "pygments_lexer": "ipython2", 130 | "version": "2.7.16" 131 | } 132 | }, 133 | "nbformat": 4, 134 | "nbformat_minor": 2 135 | } 136 | -------------------------------------------------------------------------------- /ID3.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "data": { 10 | "text/plain": [ 11 | "'\\nID3 Algorithm\\nMuskan Pandey\\n'" 12 | ] 13 | }, 14 | "execution_count": 1, 15 | "metadata": {}, 16 | "output_type": "execute_result" 17 | } 18 | ], 19 | "source": [ 20 | "'''\n", 21 | "ID3 Algorithm\n", 22 | "Muskan Pandey\n", 23 | "'''" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": 2, 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "import csv\n", 33 | "import math\n", 34 | "import os" 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": 3, 40 | "metadata": {}, 41 | "outputs": [], 42 | "source": [ 43 | "def load_csv_to_header_data(filename):\n", 44 | " fpath = os.path.join(os.getcwd(), filename)\n", 45 | " fs = csv.reader(open(fpath, newline='\\n'))\n", 46 | "\n", 47 | " all_row = []\n", 48 | " for r in fs:\n", 49 | " all_row.append(r)\n", 50 | "\n", 51 | " headers = all_row[0]\n", 52 | " idx_to_name, name_to_idx = get_header_name_to_idx_maps(headers)\n", 53 | "\n", 54 | " data = {\n", 55 | " 'header': headers,\n", 56 | " 'rows': all_row[1:],\n", 57 | " 'name_to_idx': name_to_idx,\n", 58 | " 'idx_to_name': idx_to_name\n", 59 | " }\n", 60 | " return data" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": 4, 66 | "metadata": {}, 67 | "outputs": [], 68 | "source": [ 69 | "def get_header_name_to_idx_maps(headers):\n", 70 | " name_to_idx = {}\n", 71 | " idx_to_name = {}\n", 72 | " for i in range(0, len(headers)):\n", 73 | " name_to_idx[headers[i]] = i\n", 74 | " idx_to_name[i] = headers[i]\n", 75 | " return idx_to_name, name_to_idx" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": 5, 81 | "metadata": {}, 82 | "outputs": [], 83 | "source": [ 84 | "def project_columns(data, columns_to_project):\n", 85 | " data_h = list(data['header'])\n", 86 | " data_r = list(data['rows'])\n", 87 | "\n", 88 | " all_cols = list(range(0, len(data_h)))\n", 89 | "\n", 90 | " columns_to_project_ix = [data['name_to_idx'][name] for name in columns_to_project]\n", 91 | " columns_to_remove = [cidx for cidx in all_cols if cidx not in columns_to_project_ix]\n", 92 | "\n", 93 | " for delc in sorted(columns_to_remove, reverse=True):\n", 94 | " del data_h[delc]\n", 95 | " for r in data_r:\n", 96 | " del r[delc]\n", 97 | "\n", 98 | " idx_to_name, name_to_idx = get_header_name_to_idx_maps(data_h)\n", 99 | "\n", 100 | " return {'header': data_h, 'rows': data_r,\n", 101 | " 'name_to_idx': name_to_idx,\n", 102 | " 'idx_to_name': idx_to_name}" 103 | ] 104 | }, 105 | { 106 | "cell_type": "code", 107 | "execution_count": 6, 108 | "metadata": {}, 109 | "outputs": [], 110 | "source": [ 111 | "def get_uniq_values(data):\n", 112 | " idx_to_name = data['idx_to_name']\n", 113 | " idxs = idx_to_name.keys()\n", 114 | "\n", 115 | " val_map = {}\n", 116 | " for idx in iter(idxs):\n", 117 | " val_map[idx_to_name[idx]] = set()\n", 118 | "\n", 119 | " for data_row in data['rows']:\n", 120 | " for idx in idx_to_name.keys():\n", 121 | " att_name = idx_to_name[idx]\n", 122 | " val = data_row[idx]\n", 123 | " if val not in val_map.keys():\n", 124 | " val_map[att_name].add(val)\n", 125 | " return val_map" 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": 7, 131 | "metadata": {}, 132 | "outputs": [], 133 | "source": [ 134 | "def get_class_labels(data, target_attribute):\n", 135 | " rows = data['rows']\n", 136 | " col_idx = data['name_to_idx'][target_attribute]\n", 137 | " labels = {}\n", 138 | " for r in rows:\n", 139 | " val = r[col_idx]\n", 140 | " if val in labels:\n", 141 | " labels[val] = labels[val] + 1\n", 142 | " else:\n", 143 | " labels[val] = 1\n", 144 | " return labels" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": 8, 150 | "metadata": {}, 151 | "outputs": [], 152 | "source": [ 153 | "def entropy(n, labels):\n", 154 | " ent = 0\n", 155 | " for label in labels.keys():\n", 156 | " p_x = labels[label] / n\n", 157 | " ent += - p_x * math.log(p_x, 2)\n", 158 | " return ent" 159 | ] 160 | }, 161 | { 162 | "cell_type": "code", 163 | "execution_count": 9, 164 | "metadata": {}, 165 | "outputs": [], 166 | "source": [ 167 | "def partition_data(data, group_att):\n", 168 | " partitions = {}\n", 169 | " data_rows = data['rows']\n", 170 | " partition_att_idx = data['name_to_idx'][group_att]\n", 171 | " for row in data_rows:\n", 172 | " row_val = row[partition_att_idx]\n", 173 | " if row_val not in partitions.keys():\n", 174 | " partitions[row_val] = {\n", 175 | " 'name_to_idx': data['name_to_idx'],\n", 176 | " 'idx_to_name': data['idx_to_name'],\n", 177 | " 'rows': list()\n", 178 | " }\n", 179 | " partitions[row_val]['rows'].append(row)\n", 180 | " return partitions" 181 | ] 182 | }, 183 | { 184 | "cell_type": "code", 185 | "execution_count": 10, 186 | "metadata": {}, 187 | "outputs": [], 188 | "source": [ 189 | "def avg_entropy_w_partitions(data, splitting_att, target_attribute):\n", 190 | " # find uniq values of splitting att\n", 191 | " data_rows = data['rows']\n", 192 | " n = len(data_rows)\n", 193 | " partitions = partition_data(data, splitting_att)\n", 194 | "\n", 195 | " avg_ent = 0\n", 196 | "\n", 197 | " for partition_key in partitions.keys():\n", 198 | " partitioned_data = partitions[partition_key]\n", 199 | " partition_n = len(partitioned_data['rows'])\n", 200 | " partition_labels = get_class_labels(partitioned_data, target_attribute)\n", 201 | " partition_entropy = entropy(partition_n, partition_labels)\n", 202 | " avg_ent += partition_n / n * partition_entropy\n", 203 | "\n", 204 | " return avg_ent, partitions" 205 | ] 206 | }, 207 | { 208 | "cell_type": "code", 209 | "execution_count": 11, 210 | "metadata": {}, 211 | "outputs": [], 212 | "source": [ 213 | "def most_common_label(labels):\n", 214 | " mcl = max(labels, key=lambda k: labels[k])\n", 215 | " return mcl" 216 | ] 217 | }, 218 | { 219 | "cell_type": "code", 220 | "execution_count": 12, 221 | "metadata": {}, 222 | "outputs": [], 223 | "source": [ 224 | "def id3(data, uniqs, remaining_atts, target_attribute):\n", 225 | " labels = get_class_labels(data, target_attribute)\n", 226 | "\n", 227 | " node = {}\n", 228 | "\n", 229 | " if len(labels.keys()) == 1:\n", 230 | " node['label'] = next(iter(labels.keys()))\n", 231 | " return node\n", 232 | "\n", 233 | " if len(remaining_atts) == 0:\n", 234 | " node['label'] = most_common_label(labels)\n", 235 | " return node\n", 236 | "\n", 237 | " n = len(data['rows'])\n", 238 | " ent = entropy(n, labels)\n", 239 | "\n", 240 | " max_info_gain = None\n", 241 | " max_info_gain_att = None\n", 242 | " max_info_gain_partitions = None\n", 243 | "\n", 244 | " for remaining_att in remaining_atts:\n", 245 | " avg_ent, partitions = avg_entropy_w_partitions(data, remaining_att, target_attribute)\n", 246 | " info_gain = ent - avg_ent\n", 247 | " if max_info_gain is None or info_gain > max_info_gain:\n", 248 | " max_info_gain = info_gain\n", 249 | " max_info_gain_att = remaining_att\n", 250 | " max_info_gain_partitions = partitions\n", 251 | "\n", 252 | " if max_info_gain is None:\n", 253 | " node['label'] = most_common_label(labels)\n", 254 | " return node\n", 255 | "\n", 256 | " node['attribute'] = max_info_gain_att\n", 257 | " node['nodes'] = {}\n", 258 | "\n", 259 | " remaining_atts_for_subtrees = set(remaining_atts)\n", 260 | " remaining_atts_for_subtrees.discard(max_info_gain_att)\n", 261 | "\n", 262 | " uniq_att_values = uniqs[max_info_gain_att]\n", 263 | "\n", 264 | " for att_value in uniq_att_values:\n", 265 | " if att_value not in max_info_gain_partitions.keys():\n", 266 | " node['nodes'][att_value] = {'label': most_common_label(labels)}\n", 267 | " continue\n", 268 | " partition = max_info_gain_partitions[att_value]\n", 269 | " node['nodes'][att_value] = id3(partition, uniqs, remaining_atts_for_subtrees, target_attribute)\n", 270 | "\n", 271 | " return node" 272 | ] 273 | }, 274 | { 275 | "cell_type": "code", 276 | "execution_count": 13, 277 | "metadata": {}, 278 | "outputs": [], 279 | "source": [ 280 | "def pretty_print_tree(root):\n", 281 | " stack = []\n", 282 | " rules = set()\n", 283 | "\n", 284 | " def traverse(node, stack, rules):\n", 285 | " if 'label' in node:\n", 286 | " stack.append(' THEN ' + node['label'])\n", 287 | " rules.add(''.join(stack))\n", 288 | " stack.pop()\n", 289 | " elif 'attribute' in node:\n", 290 | " ifnd = 'IF ' if not stack else ' AND '\n", 291 | " stack.append(ifnd + node['attribute'] + ' EQUALS ')\n", 292 | " for subnode_key in node['nodes']:\n", 293 | " stack.append(subnode_key)\n", 294 | " traverse(node['nodes'][subnode_key], stack, rules)\n", 295 | " stack.pop()\n", 296 | " stack.pop()\n", 297 | "\n", 298 | " traverse(root, stack, rules)\n", 299 | " print(os.linesep.join(rules))" 300 | ] 301 | }, 302 | { 303 | "cell_type": "code", 304 | "execution_count": 16, 305 | "metadata": {}, 306 | "outputs": [ 307 | { 308 | "name": "stdout", 309 | "output_type": "stream", 310 | "text": [ 311 | "IF Outlook EQUALS Rainy AND Windy EQUALS False THEN Yes\n", 312 | "IF Outlook EQUALS Rainy AND Windy EQUALS True THEN No\n", 313 | "IF Outlook EQUALS Sunny AND Humidity EQUALS Normal THEN Yes\n", 314 | "IF Outlook EQUALS Sunny AND Humidity EQUALS High THEN No\n", 315 | "IF Outlook EQUALS Overcast THEN Yes\n" 316 | ] 317 | } 318 | ], 319 | "source": [ 320 | "def main():\n", 321 | " config = {'data_file': 'tennis.csv', 'data_mappers': [], 'data_project_columns': ['Outlook', 'Temperature', 'Humidity', 'Windy', 'PlayTennis'], 'target_attribute': 'PlayTennis'}\n", 322 | "\n", 323 | " data = load_csv_to_header_data(config['data_file'])\n", 324 | " data = project_columns(data, config['data_project_columns'])\n", 325 | "\n", 326 | " target_attribute = config['target_attribute']\n", 327 | " remaining_attributes = set(data['header'])\n", 328 | " remaining_attributes.remove(target_attribute)\n", 329 | "\n", 330 | " uniqs = get_uniq_values(data)\n", 331 | "\n", 332 | " root = id3(data, uniqs, remaining_attributes, target_attribute)\n", 333 | "\n", 334 | " pretty_print_tree(root)\n", 335 | "\n", 336 | "\n", 337 | "if __name__ == \"__main__\": main()" 338 | ] 339 | } 340 | ], 341 | "metadata": { 342 | "kernelspec": { 343 | "display_name": "Python 3", 344 | "language": "python", 345 | "name": "python3" 346 | }, 347 | "language_info": { 348 | "codemirror_mode": { 349 | "name": "ipython", 350 | "version": 3 351 | }, 352 | "file_extension": ".py", 353 | "mimetype": "text/x-python", 354 | "name": "python", 355 | "nbconvert_exporter": "python", 356 | "pygments_lexer": "ipython3", 357 | "version": "3.7.3" 358 | } 359 | }, 360 | "nbformat": 4, 361 | "nbformat_minor": 2 362 | } 363 | -------------------------------------------------------------------------------- /Linear_reg.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# Implement linear regression and perform the following operations:\n", 10 | "# 1. Generate a proper 2D dataset of n points, split it into training and testing data and perform linear regression using R2 method." 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 2, 16 | "metadata": {}, 17 | "outputs": [], 18 | "source": [ 19 | "import pandas as pd\n", 20 | "import numpy as np" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": 3, 26 | "metadata": { 27 | "scrolled": true 28 | }, 29 | "outputs": [ 30 | { 31 | "name": "stdout", 32 | "output_type": "stream", 33 | "text": [ 34 | "(84, 2)\n", 35 | "Index(['SAT', 'GPA'], dtype='object')\n" 36 | ] 37 | } 38 | ], 39 | "source": [ 40 | "data = pd.read_csv('slr.csv')\n", 41 | "print(data.shape)\n", 42 | "print(data.columns)" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 4, 48 | "metadata": {}, 49 | "outputs": [], 50 | "source": [ 51 | "msk = np.random.rand(len(data)) < 0.8\n", 52 | "train = data[msk]\n", 53 | "test = data[~msk]" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": 5, 59 | "metadata": {}, 60 | "outputs": [ 61 | { 62 | "data": { 63 | "text/html": [ 64 | "
\n", 65 | "\n", 78 | "\n", 79 | " \n", 80 | " \n", 81 | " \n", 82 | " \n", 83 | " \n", 84 | " \n", 85 | " \n", 86 | " \n", 87 | " \n", 88 | " \n", 89 | " \n", 90 | " \n", 91 | " \n", 92 | " \n", 93 | " \n", 94 | " \n", 95 | " \n", 96 | " \n", 97 | " \n", 98 | " \n", 99 | " \n", 100 | " \n", 101 | " \n", 102 | " \n", 103 | " \n", 104 | " \n", 105 | " \n", 106 | " \n", 107 | " \n", 108 | " \n", 109 | " \n", 110 | " \n", 111 | " \n", 112 | " \n", 113 | "
SATGPA
017142.40
116642.52
217602.54
316852.74
617643.00
\n", 114 | "
" 115 | ], 116 | "text/plain": [ 117 | " SAT GPA\n", 118 | "0 1714 2.40\n", 119 | "1 1664 2.52\n", 120 | "2 1760 2.54\n", 121 | "3 1685 2.74\n", 122 | "6 1764 3.00" 123 | ] 124 | }, 125 | "execution_count": 5, 126 | "metadata": {}, 127 | "output_type": "execute_result" 128 | } 129 | ], 130 | "source": [ 131 | "train.head()" 132 | ] 133 | }, 134 | { 135 | "cell_type": "code", 136 | "execution_count": 6, 137 | "metadata": {}, 138 | "outputs": [ 139 | { 140 | "name": "stdout", 141 | "output_type": "stream", 142 | "text": [ 143 | "63 21\n" 144 | ] 145 | } 146 | ], 147 | "source": [ 148 | "print(len(train), len(test))" 149 | ] 150 | }, 151 | { 152 | "cell_type": "code", 153 | "execution_count": 7, 154 | "metadata": {}, 155 | "outputs": [], 156 | "source": [ 157 | "X_train = train['SAT']\n", 158 | "Y_train = train['GPA']\n", 159 | "X_test = test['SAT']\n", 160 | "Y_test = test['GPA']" 161 | ] 162 | }, 163 | { 164 | "cell_type": "code", 165 | "execution_count": 8, 166 | "metadata": {}, 167 | "outputs": [ 168 | { 169 | "name": "stdout", 170 | "output_type": "stream", 171 | "text": [ 172 | "1843.936507936508 3.314603174603175\n" 173 | ] 174 | } 175 | ], 176 | "source": [ 177 | "x_mean = sum(X_train)/len(X_train)\n", 178 | "y_mean = sum(Y_train)/len(Y_train)\n", 179 | "print(x_mean, y_mean)" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": 9, 185 | "metadata": {}, 186 | "outputs": [], 187 | "source": [ 188 | "b1 = sum((X_train - x_mean)*(Y_train - y_mean)) / sum((X_train - x_mean)**2)\n", 189 | "b0 = y_mean - b1*x_mean" 190 | ] 191 | }, 192 | { 193 | "cell_type": "code", 194 | "execution_count": 10, 195 | "metadata": { 196 | "scrolled": true 197 | }, 198 | "outputs": [ 199 | { 200 | "name": "stdout", 201 | "output_type": "stream", 202 | "text": [ 203 | "-0.16705420536473436 0.0018881655441944278\n" 204 | ] 205 | } 206 | ], 207 | "source": [ 208 | "print(b0,b1)" 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": 11, 214 | "metadata": {}, 215 | "outputs": [], 216 | "source": [ 217 | "import matplotlib.pyplot as plt" 218 | ] 219 | }, 220 | { 221 | "cell_type": "code", 222 | "execution_count": 12, 223 | "metadata": {}, 224 | "outputs": [ 225 | { 226 | "data": { 227 | "text/plain": [ 228 | "" 229 | ] 230 | }, 231 | "execution_count": 12, 232 | "metadata": {}, 233 | "output_type": "execute_result" 234 | }, 235 | { 236 | "data": { 237 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAY1UlEQVR4nO3de5BkZXnH8e9vLwpTIOAyWgSYGVJCoiGyyASp0kQLFDfEaBKNhRnxWplSUhZ4SXlZiwqYrYqaECulSI0hBYFJ0AilK2Wiq1njpWRxdl0WdhcVlF0RlAXv2QrK7pM/zhmZaU7fpk/3uf0+VV19+u13Tr995vTTb7/nOe9RRGBmZtW3qugGmJlZPhzQzcxqwgHdzKwmHNDNzGrCAd3MrCbWFPXCxx9/fExNTRX18mZmlbR9+/aHImI867nCAvrU1BQLCwtFvbyZWSVJ2tfuOQ+5mJnVhAO6mVlNOKCbmdWEA7qZWU04oJuZ1YQDuplZTTigm5nVhAO6mdmIzM/D1BSsWpXcz8/nu/6uAV3SEZJuk3S7pN2SLs+oMyFpq6RvSNol6YJ8m2lmVm3z8zA7C/v2QURyPzubb1DvpYf+CHBuRJwBrAc2SDqnpc57gI9HxJnAhcBV+TXRzKz6Nm6EgweXlx08mJTnpeup/5Fc0ugX6cO16a31MkcBPCldPga4P68GmpnVwf79/ZWvRE9j6JJWS9oJPAhsiYhtLVX+BniVpPuAzwBvzq+JZmbVNzHRX/lK9BTQI+JQRKwHTgLOlnR6S5VXAtdGxEnABcD1kh63bkmzkhYkLRw4cGDQtpuZVcamTTA2trxsbCwpz0tfWS4R8RNgK7Ch5ak3AB9P63wNOAI4PuPv5yJiOiKmx8czZ380M6ulmRmYm4PJSZCS+7m5pDwvvWS5jEs6Nl0+EnghcFdLtf3AeWmdp5MEdHfBzezXhp2yVwUzM3DvvXD4cHKfZzCH3uZDPwG4TtJqki+Aj0fELZKuABYiYjPwNuCjkt5CcoD0tenBVDOzX6fsLWZ5LKbsQf5BrclUVNydnp4OX+DCrBmmppIg3mpyMumpWu8kbY+I6aznfKaomQ3dKFL2zAHdzEYg75Q9j8dnc0A3s6HLM2VvFKfQV5UDupkNXZ4pe6M4hb6qHNDNlvBP+eXy3B55pex5PL69XtIWzRrBqXXLlXV7TExkZ8zkeQp9VbmHbpbyT/nlyro9RnEKfVU5oJul/FN+ubJuj1GcQl9VHnIxS/mn/HJl3h4zMw7gWdxDN0v5p/xy3h7V44BulvJP+eW8ParHc7mYmVWI53IxM2sAB3Qzs5pwQDezWmny2b5OWzSz2ijr2a2j4h66mdVGWc9uHRUHdDOrjbKe3ToqDuhmVht5XkijimPxXQO6pCMk3Sbpdkm7JV3ept4rJO1J6/xb/k01M+ssr7Nbq3oRjV566I8A50bEGcB6YIOkc5ZWkHQq8C7gORHxO8ClubfUzEamir1TyO/s1qqOxXfNconkVNJfpA/XprfW00v/EvhwRPw4/ZsH82ykmY1O1TNF8pi4q6pj8T2NoUtaLWkn8CCwJSK2tVQ5DThN0lcl3SppQ5v1zEpakLRw4MCBwVpuViJV7dFmueSSavZO85T3Ra1HpaeAHhGHImI9cBJwtqTTW6qsAU4Fng+8EviopGMz1jMXEdMRMT0+Pj5Yy81KoqrjrVnm5+Hhh7OfK3vvNE9VnWmyryyXiPgJsBVo7YHfB2yOiF9FxHeBb5EEeLPaq+p4a5ZObS577zRPVZ1pspcsl/HF3rakI4EXAne1VPskSe8cSceTDMF8J9eWmpVUVcdbs3Rqc9l7p3nL66LWo9RLD/0EYKukXcDXScbQb5F0haSXpHU+CzwsaQ9JD/6vI6LNDzezeqnqeGuWdm1et64aAa3pugb0iNgVEWdGxDMj4vSIuCItvywiNqfLERFvjYhnRMTvRsSNw264WVlUdbw1S9Z7WVTFYwJN4zNFzQZU1fHWLIvvZd265eUPP1zdA71N4isWmdnjTE1lXyB6cjIZT7bi+IpFZqmV5IvXKce8V3U60NskDujWGCvJFy9DjnkRXyh1OtA7aoV2ACKikNtZZ50VZqM0ORmRhOXlt8nJfP8mTzfcEDE2tvy1x8aS8jq+btWNYrsBC9EmrnoM3Rpj1arkI9ZKSnKN8/qbPBU5lj0/n5xotH9/0jPftKmaB3pHaRT/L4+hWy0M+lN2JcMIRQ89ZAWHTuV5quKJNUUr+tiDA7pVQh5j2SvJFy86x3z16v7KrVhFdwAc0K0S8pgvZSX54kXmmM/Pw6FD2c+1K7diFd0B8EFRqwQp++CkVHTL8nfDDRHr1mW/31EflF2JG25I2icl9007kDrs90+Hg6JdL3BhVgYTE9njxnVLo2u9uESWMk8rUPWLY+QhjwtsrJSHXKwSCv8pOyJZQ0utyjytQJ2mEq4iB3SrhDrNl9JJt2yIyclyv+eiszyazgHdKqNuaXRZaZidhpCq8Iuk6CyPpnNANytAuzTMCy7Inr523bpq/CJpytBYWTmgmxWg3VjzZz7z+KGlG26Ahx4qfzCH5gyNlVWlAnoTZ72z8sljP+w01lz1oaWqt7/KKhPQyzDrnY1OWb+8s/bDV70Kjj++vzZ6rLk3Zd0PSqtdgvriDTgCuA24HdgNXN6h7suAAKa7rbffE4uKnvXORqfMM/212w/7bWOZ32NZeBtlo8OJRb0EdAFHpctrgW3AORn1jga+BNw6jIDepDMFmy6PL+9hna3Xbj8sUxv7Mcw2DLpud+KyDRTQl1WGMWAH8OyM5z4I/BHwRffQOyvDB7nMBv3yHmbPrlMPvWodjGFupzzW3ct+0MTP0sABHVgN7AR+Abwv4/lnATely0MJ6HX5+VWX9zFMg3555/3lvzRorFsX8YQn5NNDL9owO0l5rLvbOpr6Wcqzh34ssBU4fUnZqjSIT0WXgA7MAgvAwsTERN9vpA7fxnX6pTEsg35Q8xyey2rL2rURRx31+PVXLZgMcxgzj3V32w+a+lnKLaAn6+Iy4O1LHh8DPATcm97+D7i/Wy+9qbMt1v1YQF5fuoOsJ88Peqd1ZbWxSp2OsvfQIzpvz7p/ltoZ9KDoOHBsunwk8GXgxR3qD2XIpS7q3Ksoy0/gPNvRT9Aoy/vvVdnH0Lup82epk0ED+jOBbwC7gDuBy9LyK4CXZNR3QO+gah/6fpTpA5ZXT7mf91Sm99+rMme59LL+un6WOsl1yCWvW1MDekS1fpb3o44/gfsJGnV8/2VX189SJ50CemXOFK2Tup4aXcezH/uZm6SO73+lRnWGZ10/SyvlgG65qetMe70Gjbq+/355mo7iOKBbbpo+017T3/8iX7WoOEqGZEZveno6FhYWCnltMxueVauSnnkrKfmVY4ORtD0iprOecw/drAvP+NcfH0sojgO6WQceD+6fjyUUxwHdrAOPB/fPxxKK44Buuanj0ES7Kwvt2zf4+6zj9lrkdMJiOKDbirQGo4svLsfQRFa7BgmancZ9B3mfHsqxYXCWi/VtMRgtHYqQsjMbJieTHlpR7Wo1Ntbfz/9e1gn9v8+pqSSID7oea55OWS4O6Na3dsEoyyhT1XptV79Bc34+GTPfvz/7Swv6f59O7bOVctqi5arduHKWUaaq9dquftoPy8eDJyez6/T7Pp3aZ8PggG59axd0pOWPR52q1mswHCRo5pWS59Q+GwYHdOtbu2D0xjcWm6qW1a5WgwbNvFLynNpnQ9FuGsZh35o8fW4dlHXa0tZ2velN5WynZSvrflUmdJg+1wdFzawUsjKK+s1KagIfFDWz0vNZuYNzQDezUmiXfdRvVlKTOaDnpM6ncZuNglM5B9c1oEs6QtJtkm6XtFvS5Rl13ippj6Rdkr4gqU22bj35NG6zwTmVc3C99NAfAc6NiDOA9cAGSee01PkGMB0RzwQ+Abw/32aW2yjH/kb9SyDP1/OvGOvEqZw5aJf+knUDxoAdwLM71DkT+Gq3ddUpbXFUV3vv5wr0ZXu9UbfdrK4YNG1R0mpgO/A04MMR8Y4OdT8E/CAi/jbjuVlgFmBiYuKsfb1OCFJyo5poadQTOuX5ep6MyiwfA6ctRsShiFgPnAScLen0Ni/0KmAa+ECb9cxFxHRETI+Pj/fW+hLoNlQwqrG/UWcB5Pl6VclgGNawkIebbCTadd3b3YDLgLdnlL8A2As8pZf1VGXIpdehglGc4TY5mT20MzmZ/2vl/XqjbvtKDGtYyMNNlic6DLn0EsDHgWPT5SOBLwMvbqlzJnAPcGq39S3eqhLQyxSIPIY+XMP6X5dpH7LqGzSgP5Mki2UXcCdwWVp+BfCSdPnzwA+Bneltc7f1ViWgj+qAZ69GPddFnq9X9nk6hvW/Lts+ZNXWKaB7LpcufDCvOYb1v/Y+ZHnyXC4D8MkOzTGs/7X3IRsVB/QufLJDcwzrf+19yEbFQy42sKXX3JyYSHqeDlZmw9FpyGXNqBtj9dI6h/XiPDbgoG42ah5ysYF4Dmuz8nBAt4FU5QxQsyZwQLeBeA5rs/JwQLeBOCXPrDwc0Esmj0mcRjkRlFPyzMrDAb1E8rjyURFXT5qZSc54vP765PFFF3lGQbMiOA+9RPI4Rbyo08xb0xchGXpxb90sX53y0B3QS2TVqqRX3UqCw4dHt46V8HwlZqPhuVwqIo+MkaKyTpy+aFY8B/QSySNjpKisE6cvmhXPAb1E8sgYKSrrxOmLZsXzGLrlxpN0mQ2fx9AbpqgLEi+mLx4+nNw7mJuNlgN6zRSRh75SRX3x9Ksq7TTrGtAlHSHpNkm3S9ot6fKMOk+U9DFJd0vaJmlqGI217qoy+2FVvniq0k4z6K2H/ghwbkScAawHNkg6p6XOG4AfR8TTgH8E3pdvM61Xw0wfvPhiWLMmOdi6Zk3yeKWq8sVTlXaaQQ8BPb3Q9C/Sh2vTW+uR1JcC16XLnwDOk6TcWmk9G1b64MUXw0c+AocOJY8PHUoerzSoVyVvvSrtNIMex9AlrZa0E3gQ2BIR21qqnAh8DyAiHgV+CqzLs6HWm2GlD87N9VfeTVXy1qvSTjPoMaBHxKGIWA+cBJwt6fSVvJikWUkLkhYOHDiwklVYF8PKQ1/smfda3k1V8tar0k4z6DPLJSJ+AmwFNrQ89X3gZABJa4BjgIcz/n4uIqYjYnp8fHxlLbauhpE+uHp1f+XdVGXa3aq00wx6y3IZl3Rsunwk8ELgrpZqm4HXpMsvB/47ijpjyYZi8cLPvZb3oip561Vpp9maHuqcAFwnaTXJF8DHI+IWSVcACxGxGbgGuF7S3cCPgAuH1mIrxFVXJfdzc8kwy+rVSTBfLDez4vnUfzOzCvGp/2ZmDeCAbmZWEw7oZmY14YBuZlYTDuhmZjXhgG5mVhMO6GZmNeGAbmZWEw7oZmY14YBuZlYTDuhmZjXhgG5mVhMO6GZmNeGAbmZWEw7oZmY14YBuZlYTDuhmZjXhgG5mVhMO6GZmNdE1oEs6WdJWSXsk7ZZ0SUadYyR9WtLtaZ3XDae5ZmbWzpoe6jwKvC0idkg6GtguaUtE7FlS56+APRHxx5LGgW9Kmo+IXw6j0WZm9nhde+gR8UBE7EiXfw7sBU5srQYcLUnAUcCPSL4IzMxsRPoaQ5c0BZwJbGt56kPA04H7gTuASyLicMbfz0pakLRw4MCBFTXYzMyy9RzQJR0F3ARcGhE/a3n6RcBO4DeA9cCHJD2pdR0RMRcR0xExPT4+PkCzzcysVU8BXdJakmA+HxE3Z1R5HXBzJO4Gvgv8dn7NNDOzbnrJchFwDbA3Iq5sU20/cF5a/6nAbwHfyauRZmbWXS9ZLs8BLgLukLQzLXs3MAEQEVcD7wWulXQHIOAdEfHQENprZmZtdA3oEfEVkiDdqc79wPl5Napq5udh40bYvx8mJmDTJpiZKbpVZtY0vfTQrYP5eZidhYMHk8f79iWPwUHdzEbLp/4PaOPGx4L5ooMHk3Izs1FyQB/Q/v39lZuZDYsD+oAmJvorNzMbFgf0AW3aBGNjy8vGxpJyM7NRckAf0MwMzM3B5CRIyf3cnA+ImtnoOcslBzMzDuBmVjz30M3MasIB3cysJhzQzcxqwgHdzKwmHNDNzGrCAd3MrCYc0M3MasIB3cysJhzQzcxqwgHdzKwmHNDNzGqil4tEnyxpq6Q9knZLuqRNvedL2pnW+Z/8m2pmZp30MjnXo8DbImKHpKOB7ZK2RMSexQqSjgWuAjZExH5JTxlSe83MrI2uPfSIeCAidqTLPwf2Aie2VPsL4OaI2J/WezDvhpqZWWd9jaFLmgLOBLa1PHUacJykL0raLunV+TTPzMx61fN86JKOAm4CLo2In2Ws5yzgPOBI4GuSbo2Ib7WsYxaYBZjwNdrMzHLVUw9d0lqSYD4fETdnVLkP+GxE/G9EPAR8CTijtVJEzEXEdERMj4+PD9JuMzNr0UuWi4BrgL0RcWWbap8CnitpjaQx4NkkY+1mZjYivQy5PAe4CLhD0s607N3ABEBEXB0ReyX9F7ALOAz8c0TcOYwGm5lZtq4BPSK+AqiHeh8APpBHo8zMrH8+U9Q6mp+HqSlYtSq5n58vukVm1k7PWS7WPPPzMDsLBw8mj/ftSx4DzMwU1y4zy+YeurW1ceNjwXzRwYNJuZmVjwM6HlZoZ//+/srNrFiND+iLwwr79kHEY8MKDurQ7twvnxNmVk6ND+geVmhv0yYYG1teNjaWlJtZ+TQ+oHtYob2ZGZibg8lJkJL7uTkfEDUrq8ZnuUxMJMMsWeWWBG8HcLNqaHwP3cMKZlYXjQ/oHlYws7po/JALeFjBzOqh8T10M7O6cEA3M6sJB3Qzs5pwQDczqwkHdDOzmnBANzOrCQd0M7OacEA3M6uJrgFd0smStkraI2m3pEs61P09SY9Kenm+zWwez9FuZv3q5UzRR4G3RcQOSUcD2yVtiYg9SytJWg28D/jcENrZKL70m5mtRNceekQ8EBE70uWfA3uBEzOqvhm4CXgw1xY2kOdoN7OV6GsMXdIUcCawraX8ROBPgY90+ftZSQuSFg4cONBfSxvEc7Sb2Ur0HNAlHUXSA780In7W8vQHgXdExOFO64iIuYiYjojp8fHx/lvbEL70m5mtRE8BXdJakmA+HxE3Z1SZBm6UdC/wcuAqSX+SWysbxnO0m9lKdD0oKknANcDeiLgyq05EnLKk/rXALRHxybwa2TSLBz43bkyGWSYmkmDuA6Jm1kkvWS7PAS4C7pC0My17NzABEBFXD6ltjeY52s2sX10DekR8BVCvK4yI1w7SIDMzWxmfKWpmVhMO6GZmNeGAbmZWEw7oZmY14YBuZlYTDuhmZjWhiCjmhaUDwL5CXrwcjgceKroRJedt1J23UXd120aTEZE5d0phAb3pJC1ExHTR7Sgzb6PuvI26a9I28pCLmVlNOKCbmdWEA3px5opuQAV4G3XnbdRdY7aRx9DNzGrCPXQzs5pwQDczqwkH9JxI+hdJD0q6c0nZxyTtTG/3LplPHknvknS3pG9KetGS8g1p2d2S3jnq9zFMbbbRekm3pttoQdLZabkk/VO6HXZJetaSv3mNpG+nt9cU8V6Gqc12OkPS1yTdIenTkp605LlG7UuSTpa0VdIeSbslXZKWP1nSlnS/2CLpuLS8OftSRPiWww34A+BZwJ1tnv8H4LJ0+RnA7cATgVOAe4DV6e0e4DeBJ6R1nlH0exvmNgI+B/xhunwB8MUly/9JMhf/OcC2tPzJwHfS++PS5eOKfm8j2E5fB56XLr8eeG9T9yXgBOBZ6fLRwLfS7fB+4J1p+TuB9zVtX3IPPScR8SXgR1nPpZfxewXw72nRS4EbI+KRiPgucDdwdnq7OyK+ExG/BG5M69ZCm20UwGJv8xjg/nT5pcC/RuJW4FhJJwAvArZExI8i4sfAFmDD8Fs/Om2202nAl9LlLcDL0uXG7UsR8UBE7EiXfw7sBU4keX/XpdWuAxava9yYfckBfTR+H/hhRHw7fXwi8L0lz9+XlrUrr7NLgQ9I+h7w98C70nJvo+V281hA/nPg5HS50dtJ0hRwJrANeGpEPJA+9QPgqelyY7aRA/povJLHeue23JuAt0TEycBbSC5Ibo/3euBiSdtJhhl+WXB7CifpKOAm4NKI+NnS5yIZU2lcTrYD+pBJWgP8GfCxJcXf57EeFsBJaVm78jp7DXBzuvwfJEMF4G20TETcFRHnR8RZJJ2De9KnGrmdJK0lCebzEbG4//wwHUohvX8wLW/MNnJAH74XAHdFxH1LyjYDF0p6oqRTgFOB20gOfJ0q6RRJTwAuTOvW2f3A89Llc4HFYanNwKvTDIVzgJ+mP6c/C5wv6bg0i+H8tKzWJD0lvV8FvAe4On2qcftSekzqGmBvRFy55KnNJB0E0vtPLSlvxr5U9FHZutxIek0PAL8iGYt7Q1p+LfDGjPobSXpZ3yTN8kjLLyA5an8PsLHo9zXsbQQ8F9hOkoWxDTgrrSvgw+l2uAOYXrKe15Mc/LsbeF3R72tE2+mSdL/4FvB3pGd5N3FfSveZAHYBO9PbBcA64AsknYLPA09u2r7kU//NzGrCQy5mZjXhgG5mVhMO6GZmNeGAbmZWEw7oZmY14YBuZlYTDuhmZjXx/7K3+j4YNOSxAAAAAElFTkSuQmCC\n", 238 | "text/plain": [ 239 | "
" 240 | ] 241 | }, 242 | "metadata": { 243 | "needs_background": "light" 244 | }, 245 | "output_type": "display_data" 246 | } 247 | ], 248 | "source": [ 249 | "plt.scatter(X_train, Y_train, color = \"b\", marker = \"o\")" 250 | ] 251 | }, 252 | { 253 | "cell_type": "code", 254 | "execution_count": 13, 255 | "metadata": {}, 256 | "outputs": [], 257 | "source": [ 258 | "Y_pred = b0 + b1*X_test" 259 | ] 260 | }, 261 | { 262 | "cell_type": "code", 263 | "execution_count": 14, 264 | "metadata": {}, 265 | "outputs": [ 266 | { 267 | "data": { 268 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3yV1eHH8c/JIAkQEkLCkC0BZLijgmJRVJy4Z7VqseJordqfLQ4c1bql1tbR0lq3VcSFAxUVyhCwoCKIiiAgCjLC3iE5vz/Ovd6d3CR35/t+vZ7XfXKe5z45uVy+97nnOc85xlqLiIikv6xkV0BERGJDgS4ikiEU6CIiGUKBLiKSIRToIiIZIidZv7i0tNR269YtWb9eRCQtzZkzZ621tizctqQFerdu3Zg9e3ayfr2ISFoyxiyLtE1NLiIiGUKBLiKSIRToIiIZQoEuIpIhFOgiIhlCgS4ikiEU6CIiGSJp/dBForZpE9xzD7RpA6Wl7tF/vbgYsnRuIqJAl9S3ejXcdx9UV4ffnpUFrVsHhn2k8PdfcnMT+3eIxJkCXVJfeTlUVbkz9bVrobLSt/j/7F3/7jv49FP3844dkY/bqlV04e//c/Pmifu7RepJgS7pwRgoKnJLjx7RP2/bttrD3//nr79265s2RT5efn704e/9uVUrV3+ROFOgS2Zr3twtnTtH/5xdu2DdutrD37s+d677ed06iDSdY04OlJREF/7e9ZISyM6OzWsgTYYCXSRYs2bQvr1bolVTAxs21B7+3p8XLYJZs9x6VVX44xnjLvZGE/7+63l5sXkNJC0p0EViISvLnVWXlET/HGthy5bovgmsWAHz5rn1rVsjH7NFi/pdHC4tdc9Rk1BGUKCLJIsxUFjolvrMDbBjR2Dw13Z9YMkS9/OGDZGP16xZ/S8Oq6toSlKgi6Sb/Hzo2NEt0dq9G9avj+7i8IIFvp9r6ypaUlK/JqGSEnUVjTMFukhTkJMDZWVuiZa1sHFjdE1CS5fCnDluva6uovW5OFxaCgUFjf7zmwoFuoiE570wW1xc/66i0Vwc9nYVXbsWNm+OfLyCgvpfHG6iXUUV6CISW82bQ5cubomWf1fRum4eW77crdfVVTT4zuC6rg+0bp32XUUV6CKSfA3pKlpd7S72RnNxeNEimDnTrUfTVbQ+N4+lUFdRBbqIpKfsbF+oRsu/q2hdF4ej7SrasmX97hz2dhWNAwW6iDQdsegqWleT0OLFbj1CV9Hd2Ybt771N4ZDjYvM3+VGgi4jUpaFdRT3XBTatWMKF/zyRR9+ClrssYyrf4zoU6CIi6SEnh8qW2ZQ+1heA83fBHlvgyhPgkTNHx+dXxuWoIiJN2Jqta2j7QNuffi7ZBg++C3bgAB59Y3rculQq0EVEYmTVllW0Hx3aU2ft6oswu56DMf+M65AJCnQRkUb6ccuPdBjdIaS8+pZqsiZNhtuOghtvhP7941oPBbqISAOt2LyCjn/2u1BqoaAKtlzxHVkbN8H0j+Cyy9ysW6NGxb0+CnQREX81NW4Mmw0b3IBm3ke/9S2rvmf8rKdpvQNmbIfiHdB6B7StaobZtQvu8rtLNisLJk5MyJg0CnQRyTy7doWEcLhgDru+cWPkIQWA3Qa2F0BFPmzIh/UFcMigczDFxW74AO+jd33PPes3Fk4j1Bnoxph8YAqQ59l/nLX21jD7nQ3cBlhgrrX257Gtqog0Gd47OsMFbzTBvH177ccvKAgM3Y4dXft2cBh71ldkbeOQV45nfT5sbQZ4OqnU3FKDSaFBwKI5Q98JDLHWbjHG5ALTjDETrLUzvTsYY3oCNwCHWWvXG2PaRjqYiDQR3rFWIgVvbcG8YYO7Mac2RUWB4du7d/gz5OD14uKox19ZumEp3R/q7vl9vvJUC3KvOgPdWmuBLZ4fcz1L8PeRS4FHrLXrPc9ZHctKikgMWOtC1rvs3h3+Mdqy4G3z5sHNNyfu79m40S1Ll4bf3qOHG5SrARavW0z538pDylM1yL2iakM3xmQDc4ByXHDPCtqll2e/6UA2cJu19p0wxxkBjADoUp+hNUUisdZdxKpv+MRiW6J+T6zqUFOT7H+txOrcud5PWbRuET3/1jOk3N4auU09lUQV6NbaamA/Y0wx8Koxpr+1dn7QcXoCRwCdgCnGmL2ttRuCjjMGGANQUVGRHq+QhLd6NTz4oBuFLtkhl6qys9243NnZgeu1lYXblpvrxhKp7/Mauy1RvycF5iZdWLmQ3g/3DijLMllU35LC768w6tXLxVq7wRgzCTgO8A/074FZ1toqYIkxZiEu4P8Xs5pKalmyBP76Vzc7TV2Ki93Ids2aBf6nbtYsfUKnvtuysprkjDnp5qu1X9HnkT4BZfk5+Wy/qY6Lqikqml4uZUCVJ8wLgGOAe4N2ew04D3jCGFOKa4L5NtaVlRRyyCGwaRN8/71rp1y0CL75xre+eLFvbskNG+DLL133rfJyt/Ts6Vvv0iXtZ4qR9LJgzQL6PdovoKxls5ZsvqGWqfDSQDRn6B2Apzzt6FnAWGvtm8aY24HZ1trxwLvAUGPMAqAa+L21tjJutZbUkJ0NXbu65aijArfV1MAPP/gC3j/w338/sFtZbm5g2PsHfteu7qxXJAbmr57P3o/tHVDWOr8160auS1KNYsvYWjrQx1NFRYWdPXt2Un63JJm1sHJl4Bm9f+D7zw6Tk+Oaa/zP6L2B362b+zAQqcPnqz5n37/vG1DWrkU7frzuxyTVqOGMMXOstRXhtunURxLPGNhjD7cMHhy4zVpYtSp82E+bFjg7vPcbQnATTs+e0L27a6OXJu3TlZ9ywJgDAso6terE8muXJ6lG8aVAl9RijG+y4MMPD9xmLaxZE9pev2gRPPus65PslZXl2uaDw7683DXv5Ocn9u+ShHpr4Vuc9J+TAsq6FnVl6TVLk1OhBFGgS/owBtq2dcuhhwZus9Y3u3tw2L/wgrsD0f84nTuHv0Dbo0dCBlGS+Hjtq9c47cXTAsp6tenF17/5Okk1SiwFumQGY3wzqg8YELp93brwvXFeftl9EPjr1Cn8BdoePeI2W7s0zrgF4zjrpbNCytPlhqBYUaBL01BSAgcf7JZg69e7bpbBYT9+vLuByt8ee0QO+8LCxPwt8pMX57/IuS+fG1Le1ILcS4Eu0ro1VFS4JdjGjb6w9w/8t9+GH4N6SLRrF743Tnk5tGqVmL+libh76t3c+OGNIeVNNci91G1RpKE2bw4f9osWwYoVgfuWlYW/QNuzp7uTVqIy/PXhPPHZEyHlTSnI1W1RJB4KC2G//dwSbOvWwLD3Bv6HH8LTTwfu26ZN+Au0PXu6piLh/FfO5/l5z4eUN6Ugj4YCXSQeWrSAffZxS7Dt28OH/dSp8PzzgbPltG4dPuzLy90F4AwfL+asl85i3IJxIeUK8vAU6CKJVlDgZscJNwP8jh1u4LPgrpczZsCLLwYOgVtUFL69vrzcde1M47A/+T8n88bCN0LKFeS1U6CLpJL8fOjTxy3Bdu50kzkEh/3//gcvvRQY9oWFgWHvH/jt26ds2A99ZigTv50YUq4gj44CXSRd5OW5adZ69w7dtmsXLFsWenH2s8/g1VcDp3Nr3jzyBdoOHZIyPvkRTx7Bf5f9N6RcQV4/CnSRTNCsmQvknj3h+OMDt1VVwXffhfbGmT/f9bWvqvLtW1Dg+tQHh315ubvhKsZhP/Dxgcz8fmZIuYK8YRToIpkuN9eFdI8ecOyxgdt274bly0PD/quv4K233Jm/V16eO0a4s/vOnes1pv0B/ziAT3/8NKRcQd44CnSRpiwnx41M2b07HHNM4Lbq6sAJTPwD/733fBOYgPuGUNsEJp4x7fs+0pcv134ZUg0FeWwo0EUkvIZOYPLBByETmHzbGr4squKyElhUAt94HheN3qUx7WNIgS4i9ZeV5ZpZOneGI48M3OY3gcn/PXoKpSs2Ur4OelbC4KXQ0q/JnkcLQicw8a5756GVqCnQRSS2jKH02X2o3F4JfXGLh72lJvIEJtOnB05gkpXlvh2E643Tq1dSeuOkOo3lIiKRVVe7tvKdO90Sbt2v7JKx58POXeTthvzdkFftHm8dMLLu5+/Y4S7QrlpVd71uvRVuuy3uf34q0lguIumkpsYXelGEaFy2e9erq+tV9ccjbZj5F9dLJj/fPYZbLyx0TTiRtvs/DhvW6Jc5EynQRcC1++7aldwQ9T769wtvjNzcyIHoXS8urn17Hc8f8sIJ7MyBHTmwM9vzmAPLr1/l27dZs5S9MzXTKNAleax1/aBT4Ux0587Y/E1ZWXWHY8uWboTFKAKzUdvj2MZs/ugJ6D0Dy9X9MLkU6Klo3Tq46y7YsiXZNYmON5gbEqj+4480RjThV1QUv/D0rudk9n+pn4I8iII8NWT2uy9dVVbCuHHupo56tmHWqqgofrPdh/t6n5fnzkTjFZ7ex9xcfaWPMwV5elCgp6KePd2oeta6KdAqK6Nf1q6FbdvCH3fjRtc+26ZN4FJaGlrmvxQVKTCbKAV5elGgpzJj3EWr4mI3hka0duyI/gNg+XL3uG5d4MQK/rKz3cw5tYV+8AdESYluCkljCvL0pEDPRPn50LGjW6JVUwMbNgSe6Uf6EFi6FObMcev+43kEKyyM/kPAuxQW6ttAEinI05sCXZysLHdWXVLimnyitW1b3U1A3vXFi93jhg2Rj5ebG/nbQKSmoZKSjL8YGW8K8syg/wXSOM2bu6Vz5+ifs3s3rF8fXZPQokUwc6Zbr61/dlFR/b8NtGjR5L8NKMgziwJdEi8nB8rK3BIta103zmgvDH/9tVvftCnyMb29cOqztG5dr3G/U5WCPDMp0CU9GOPa1wsL3Sh80aqqchd867ouUFkJCxb41iN1F/VeqK7ronBwWUFBTF6GxlKQZzYFumS23Fxo184t0bLWndlHc11g5Uo3lVtlJWzdGvmYBQX1/zZQXByzuz3DBXmWyaL6lhje5yBJp0AXCWaMa5MvKnKz8ERr587ou4vOnese16+PfLdsVpZr4onmXgH/JS/P96eECfIWuS3YcmOa3IUs9aJAF4mVvDzYYw+3RCu4u2hty3ffwaefunX/GYGCtWjBspytVBbAe82hsgAqm8O2Vvn8ftg90KGDa17q1Uu9gzKM/jVFkqmh3UW3bw97XWDUS1fSZvtW2myDNtuhzTYo35RN9+pW7tvAB9f4jtGsGfTtC/vsA3vv7R732cc1TzXx3j/pSoEuko4KCqBTJ7fg17Qy2LdL9+LufHv1t76C6moX6t9/79r9P/8c5s2D99+Hp5/27VdWFhjwe+8N/fqlzIVdiazOQDfG5ANTgDzP/uOstbdG2PcMYBxwkLVW0xGJxFm4NvI+pX1Y8OsFoTtnZ7u2+NJS2G+/wG1r17pwnzfPBf3nn8M//uFr2snKct8ggs/mu3bVVHApJJoz9J3AEGvtFmNMLjDNGDPBWjvTfydjTCFwNTArDvUUET/hgnz/9vvzyWWfNOyApaVusmf/CZ+rq+Hbb30BP28efPIJvPSSb5/CQujf3xfw3sAvKmpYPaRR6gx06yYd9V4Sz/Us4Tqt3gHcC/w+ZrUTkQDhgnxApwHMuGRG7H9ZdrY7K+/ZE844w1e+ZYtrsvE/mx871p3Re3XpEno2r4uwcRfVq2uMyQbmAOXAI9baWUHbDwA6W2vfMsZEDHRjzAhgBECXLl0aXGmRpiZckA/uOpjJF09OfGVatoQBA9ziZS388EPg2fznn8M777ihHiDwIqx/2OsibMwYG2nI1HA7G1MMvApcZa2d7ynLAj4ELrbWLjXGTAauq6sNvaKiws6erWZ2kdqEC/KhPYby7gXvJqE2DbBzpxuGwRv03rBfscK3T1lZ6Nl83766CBuBMWaOtbYi3LZ6ff+x1m4wxkwCjgPme4oLgf7AZOM+ZdsD440xJ+vCqEjDhAvyYb2GMf688UmoTSPk5flC2l99LsL6h70uwtaqzjN0Y0wZUOUJ8wLgPeBea+2bEfafjM7QRRokXJCf2fdMXjrrpTB7p6Hqaje+Trhl50746is31v4nn7jHVasCn+9/EfbAA+Hii93wDk1IY8/QOwBPedrRs4Cx1to3jTG3A7OttWl2yiCSesIF+c/3/jnPnfasC8Ht2yMHYTot9WjiDWvzZpgxwy0AffrAoEGNO2YGqVcbeizpDF2SYsUKN/BWsoPNs3y5ch651ZBbw0+PLWhGc3J8+yVKTo47241mqc++8VoKC6F9+8S9PikiZm3oImlryRL4wx9g3LjYHrchQdS8OW8ueY+qbKhqi3vMgvJ2ezGox5HJCcecHPU0yQAKdMlsmzfD3XfDn//sLqbddJO7jT0WIZidXe8Q/Klp5VBf2VUHX8Vfj/9rDP9oaaoU6JKZamrc+CQ33AA//gjnnw/33PPT2CeJFq6N/LqB13H/0PuTUBvJVAp0yTzTpsE117heEoccAq++GngTTAKFC/IbBt3AXUfdlYTaSKZToEvmWLYMRo6EF1+Ejh3h2WfhvPOS0m85XJDfOvhWbjvitoTXRZoOBbqkvy1b4N574YEH3M+33OIugLZokfCqhAvyO4fcyY2H35jwukjTo0CX9FVTA889B9df77ojnneeaydPwjhB4YL8/mPu57pDr0t4XaTpUqBLepoxw7WTf/wxHHSQG9L10EPrfl6MhQvyvxz7F64ecHXC6yKiQJf0sny5OyN//nk3N+ZTT8EFFyS8nTxckD9ywiNcedCVCa2HiD8FuqSHbdvgvvvcYi2MGuUugLZsmdBqhAvyMSeN4dIDL01oPUTCUaBLarPWnY1ff72bC/Occ9wF0K5dE1qNcEH+5ClPctF+FyW0HiK1UaBLavvVr+Df/3brXbq4M/J//MONoV1WBm3bBj42axazX22tJev20KacZ097lvP3OT9mv0ckVhToktqOOcY1t6xZA6tXw1tvufXq6vD7FxVFDvvgstLSsB8AkYL8xTNf5Ox+Z8f6LxSJGY22KOmnpgY2bPCF/Jo1gevBZWvXRv4AKC7+Kext27b887vXWN0C1rSANc1hdQu44bTRHDXgPPcB0MTG3pbUU9toiwp0yXw1NbB+fcTgt2vWMGnWi5Rtg7ZboXQbZEf6b9G6df2+AWhSZIkxDZ8rTVtWFrRp45a99vqpuMbWkH17NpQCfXy7TzjvLY5rc0jkM37v48KFMH26+wZQUxP+d5eURP8B0KaNPgCkUfTukSbnpyAPMvEXEzl6z6N9BW3auBlx6lJd7b4B1Nb8s3q1m15t6lT3ARDum7Ex9f8AyA79O6TpUqBLk1FdU03OHaFv+UkXTeKIbkc0/MDZ2a55pbQ0yopUw7p1dbf/L1jgHisra/8ACA7+SB8AJSX6AMhwCnTJeLtrdpN7R+jFzKm/nMqgLkmYjzI72xe40aiudqFe24XfNWtg/nz3uG5d5A+ANm3q9wGQhJEqpeEU6JKxqqqraPan0G6JMy6ZwYBOyRkfvUGys13Atm3rZluqy+7doR8A4T4M5s1z6+vWhT+O99pDtB8ArVvrAyDJFOjSeFVV8M038MUX7izR+1hZGbrv2rUJq1YuELazym0DE1aHtFZT4/swkNj617/gkktiflgFukSvuhq+/TY0uL/+2jc7fVYWlJe7M8n27UPn3Hz00cTXWyTVVITtddhoCnQJVVMD330XGtxffgk7dvj2697dBfdJJ7nH/v1dt8D8/MjHfuSRmFd3e9V2mt/VPKT888s/Z+92e8f894mkKgV6U2atmxgiOLgXLHCzAHl16uQCe8gQX3D36ZPwkQ6DbavaRou7Qmcl+uLKL+hb1jcJNRJJLgV6U+HtBeEf3F984W6h92rXzgX28OG+4O7b190en0K27NpC4d2FIeVf/forepf2TkKNRFKDAj3TrF/vgjr4rNv/wlbr1i6szz3XPfbv7wI82n7USbJ552Za3dMqpHzRVYvoUdIjCTUSSS0K9HS1ebNrGgkO7hUrfPsUFrqgPvlkX2j37x/+YmUK27hjI8X3hn5LWHL1EroVd0t8hURSlAI91W3f7i5GBgf3smW+fQoKXNPI0UcHBnfnzmkV3MEiNa0su2YZXYoSPxG0SKpToKeKXbvcYE/BbdyLF/sGfmrWzPUiOfRQGDHCF9zdumXULd2bdm6i6J6ikPLvr/2ejq06JqFGIulBgZ5ou3e7kA4O7oUL3TZw4dyrF+y7L5x/vi+4y8szejS+DTs20Pre1iHl6/6wjtYFoeUiEihz0yHZampg6dLQ4P7yS3c2Dq45ZM89XVifeqrvAmWvXpCXl9TqJ9L67espua8ktHzkeorzU6uHjUgqU6A3lrVu8uLg4F6wwE2d5tWliwvroUN97dx9+kDz0BtimorKbZWU3h/as2bj9RtplRfam0VEaqdAj5a1sGpV6MXJL76ATZt8+3Xo4AJ7xAhfcPftC60UUF5rtq6h7QNtQ8o3Xb+JwrzQi6AiEh0FejiVleGD23+wqTZtYO+94Re/8AV3v35uyFEJK1LTypYbttCiWegdnyJSP0070DdtCh/cP/7o26eoyAX1GWf4Lk726+eGC03jLoGJtG77Otrc1yakfOuNW2me23SbnERirWkE+tat7mJkcHAvX+7bp0UL1zRy/PGBwd2xo4K7gSK1ke8ctZNm2aHjlItI49QZ6MaYfGAKkOfZf5y19tagfX4H/ArYDawBhltrlwUfK+527HBDuXpD2xvcS5b4ZnDJy3MXIwcPDgzurl01OH+MRGoj3zVqF7nZoTMHiUhsRHOGvhMYYq3dYozJBaYZYyZYa2f67fMpUGGt3WaMuQK4DzgnDvV1Ik2osGiRG7MbXH/t3r3hoIPg4ot9wd2jR0bdhJNKVm1ZRfvR7UPKFeQiiVFnoFtrLeAdSzWXMBPBWGsn+f04E7ggVhUMMWmSaxbZuTN0W5s2cOSRbjnsMDfsq6bFirsft/xIh9EdQsqrbq4iJ6tptOqJpIKo/rcZY7KBOUA58Ii1dlYtu18CTIhwnBHACIAuXRo4FkevXnDFFbBypZsP0busXet6oYwb5xYv74S83jkZ61paqLdFtFZsXkHHP4feir/75t1kZ+lbkEiiGRtudvBIOxtTDLwKXGWtnR9m+wXAb4DB1towp9A+FRUVdvbs2fWsbi28M6P7h3xty+bN4Y/TvHn04V9aCrlNrykhUvdDBblI/Blj5lhrw85hV6/vw9baDcaYScBxQECgG2OOBm4iijCPC/+Z0aOxfbtvBvRIyw8/wKefunXvnJnBSkqi/wAoLk7rHjORuh9W31JNllGzlkiyRdPLpQyo8oR5AXAMcG/QPvsD/wCOs9aujktNY62gwN2OH03Tj7WwcWPdZ/1ffOHa+MPNdg/ubL4+zT8FBbH9mxsoXPfDPqV9+OLKLzBp/AElkmmiOUPvADzlaUfPAsZaa980xtwOzLbWjgfuB1oCL3n+g39nrT05XpVOOGPc2XVxsWvDr0tVVXTNP9984x63bg1/nJYtow//Nm1iPhJjuO6He7fdm7mXz1WQi6SgerWhx1LM29DT2datdTf/+C/erpn+jHGhXlvot28PBx9cZ7fN1VtX0+6BdgFl+7Xfj09GfKIgF0mymLWhS5y0aOGWbt3q3remxk3sXFfoz53rHtevD3z+E0+4fvlhhOtHXrFHBR//6mMFuUgaUKCnm6wsdyG2pMTNXlSXTZvgnHPgnXfcmOtnnhmyS7h+5AM6DeCj4R8pyEXSiAI9ky1fDqefDrNnwx//CKNGBdxkFa4f+aAug5hy8RQFuUgaUqBnqmnT3AiR27fD66/Dyb5r1D9s+oFOD3YK2H1w18FMvnhygispIrGkQM9Ef/87XHUVdO8Okye7wciA5RuX0+Uvgd00h3QfwgcXfpCESopIrCnQM8nOnfDb38KYMW68m+efh+Ji5qyYQ8U/Ay+KD+0xlHcveDdJFRWReFCgZ4qVK90Fz48+ghtvhNtv538/fsLBDx0csNu+7fbls8s/S1IlRSSeFOiZ4OOP4bTTXHfGsWOZObAzA/8U+k9rb03OPQcikhgK9HT3xBNw+eXQsSOfvfoY+884GxYE7qIgF2kaFOjpqqoK/u//4G9/Y/1hB1J+2BzWzbgoYBcFuUjTokBPR2vWwNlnw+TJjB4II4fModrvbn4FuUjTpEBPN59+yvaTjoPVq7n0NHhuX98mBblI06ZATyOf/+UGyn9/D5XN4dTh8MkerlxBLiKgQE8LE756k3mXDOMPH8GULnDW2bC6pYJcRAJpmpkU9ubCNykZacg6yYX5wwfB0IsMq+63CnMRCaEz9BT0ycpPOHDMgfRbBf97ATpvhCtOyeGx16r4TbIrJyIpS4GeQrxBDnDaAnj6Vdicb2g2bTqPDRyY5NqJSKpToKeA2Stmc9A/DwLA1MAfJ8PNU4BDDqHlK6/AHnsktX4ikh4U6En08Q8fc8i/Dvnp51Y7YMakHvSdtRiGD4dHH4W8vCTWUETSiQI9CWZ+P5OBjwc2oYzuOoLf3fNfWLwYHn4YrrzSzRMqIhIlBXoCfbT8Iw7792EBZaMOH8Ud2w6B8893Z+Pvvw+DByephiKSztRtMQGmfTcN80cTEOa3Dr4Ve0sNd8zId7MJ9ejhpopTmItIA+kMPY6mLJvC4CcDA/r2I27n5sE3w5YtcNZZ8PLL7ux8zBho3jxJNRWRTKBAj4NJSyYx5OkhAWV3DrmTGw+/0f2weDGceiosWAAPPAC/+53ay0Wk0RToMfTBtx9w9DNHB5Tdc9Q9jBw00lcwcSKcc45bf+cdOOaYBNZQRDKZAj0GJi6eyNBnhwaU3X/M/Vx36HW+Amth9GgYORL69YPXXoM990xwTUUkkynQG+GdRe9w/HPHB5T9eeifuXbgtYE7btsGl17qJm0+80w3y1DLlgmsqYg0BQr0Bnj7m7c58fkTA8oeOu4hfnvIb0N3XrbMzff52Wdw551www1qLxeRuFCg18ObC99k2H+GBZQ9fPzD/PrgX4d/wuTJrifLrl3wxhtw4onh9xMRiQEFehTGfz2eU144JaDssRMf4/KKy8M/wVp45BG45hooL4fXX4fevRNQUxFpyhTotXj1y+CI068AAA07SURBVFc5fezpAWVjThrDpQdeGvlJO3bAr38N//43DBsGzzwDRUVxrqmIiAI9rIdmPsQ1714TUPb4yY8zfP/htT9xxQo4/XSYNQtuvhluuw2ydDOuiCSGAt3P6I9Gc93E6wLKnjjlCS7e7+K6nzxjhgvzzZvd3Z+nn173c0REYkiBDtw3/T5Gvj8yoOxPR/6Jm352U3QH+Ne/3OiIXbq4G4f6949DLUVEatekA/3uqXdz44c3BpYddTfXD7o+ugPs2uUufD72GAwdCi+8AK1bx6GmIiJ1a5KBfsd/7+CWybcElIXc2VmXVatcl8SpU+EPf4C77oLs7BjXVEQkenUGujEmH5gC5Hn2H2etvTVonzzgaeBAoBI4x1q7NOa1baRbJ93K7VNuDygLe2dnXWbPdjcLVVbCf/4D554bw1qKiDRMNGfoO4Eh1totxphcYJoxZoK1dqbfPpcA66215caYc4F7gXPiUN8GGfXhKO6cemdAWcQ7O+vyzDPuNv527WD6dNh//xjVUkSkceoMdGutBbZ4fsz1LDZot1OA2zzr44CHjTHG89ykGTlxJPd9dF9AWa13dtZm927XtPLgg3DEETB2LJSVxaaiIiIxEFUbujEmG5gDlAOPWGtnBe3SEVgOYK3dbYzZCLQB1gYdZwQwAqBLly6Nq3ktrnvvOkbPGB1QVuudnXVZu9YNefvhh/Db37oxzHNzY1BTEZHYiSrQrbXVwH7GmGLgVWNMf2vt/Pr+MmvtGGAMQEVFRczP3q955xoemvVQQNk/h/2TXx3wq4YfdO5cNxnFypVulMSLL25cJUVE4qRevVystRuMMZOA4wD/QP8B6Ax8b4zJAYpwF0cTYsI3Ezjh+RMCyqK+Iag2Y8fCL3/puiJOmQIHH9y444mIxFE0vVzKgCpPmBcAx+AuevobD1wEzADOBD5MRPt5uNEPnzr1KS7c98LGHbi6GkaNgnvugUMPdXd+tm/fuGOKiMRZNGfoHYCnPO3oWcBYa+2bxpjbgdnW2vHA48AzxphFwDogrv343vj6DU5+4eSAsrmXz2Wfdvs0/uAbNsDPfw4TJsCIEfC3v0GzZo0/rohInEXTy+VzIKRvnrX2Fr/1HcBZsa1aeJOWTAoI83lXzKN/2xjdar9ggWsvX7oU/v53uOyy2BxXRCQB0u5O0b1K9+KifS/i94f+nn5t+8XuwK+/DhdcAC1auN4sgwbF7tgiIgmQdmO7dijswJOnPhm7MK+pgT/+0Z2Z77WXuwtUYS4iaSjtztBjatMmuPBCd3Z+0UWumSU/P9m1EhFpkKYb6N98A6ecAgsXwl/+4m4Y0uTNIpLGmmagT5gA550HOTnw3nswZEiyayQi0mhp14beKNa6vuUnngjdurn2coW5iGSIpnOGvnUrDB/u7v485xx4/HHXo0VEJEM0jTP0JUvcHZ8vvQT33uvGMFeYi0iGyfwz9A8+gLPPdt0T334bjjsu2TUSEYmLzD1Dt9b1Xjn2WDcOy8cfK8xFJKNlZqBv3+6Gub32Whg2DGbOhJ49k10rEZG4yrxAX74cfvYzePppdwfoyy9DYWGyayUiEneZ1YY+bRqccYY7Q3/9dTj55LqfIyKSITIn0CdNgqFDoXt3mDwZ+vRJdo1ERBIqc5pcWrb0zfNZUJDcuoiIJEHmBPpBB8H778OaNXDYYTC/3lOeioiktcwJdHA3D02d6tYPPxw++ii59RERSaDMCnSA/v1h+nQoK4Ojj3Y3E4mINAGZF+jgBt6aNs1dGD3lFHj22WTXSEQk7jIz0AHatnU9X372M/jFL9xdoyIiGSxzAx2gVSt46y04/XR31+hNN7khAUREMlBmBzq4KeXGjoURI+Cuu+Cyy6C6Otm1EhGJucy5sag22dluvtCyMrjzTqishOee0/yhIpJRMv8M3csY+NOfXFv6K6/ACSe4SaJFRDJE0wl0r6uvdr1epk6FI4+E1auTXSMRkZhoeoEOcP75MH48fPmlu6t06dJk10hEpNGaZqADHH+8m82ostLdYTpvXrJrJCLSKE030AEGDnRNL8a4/urTpye7RiIiDda0Ax2gXz835kvbtnDMMa7fuohIGlKgA3Tt6oYK6NvXDRXwzDPJrpGISL0p0L3KytxQAYMHw4UXwoMPJrtGIiL1okD3V1joRmc880z43e/g5puTXSMRkagp0IPl5cELL8AVV0Dv3smujYhI1JrGrf/1lZ0Njz6a7FqIiNSLztBFRDJEnYFujOlsjJlkjFlgjPnCGHN1mH2KjDFvGGPmevb5ZXyqKyIikUTT5LIb+D9r7SfGmEJgjjFmorV2gd8+vwYWWGuHGWPKgK+NMc9Za3fFo9IiIhKqzjN0a+1Ka+0nnvXNwJdAx+DdgEJjjAFaAutwHwQiIpIg9booaozpBuwPzAra9DAwHlgBFALnWGtrYlA/ERGJUtQXRY0xLYGXgWustcEDiR8LfAbsAewHPGyMaRXmGCOMMbONMbPXrFnTiGqLiEiwqALdGJOLC/PnrLWvhNnll8Ar1lkELAH2Ct7JWjvGWlthra0oKytrTL1FRCRINL1cDPA48KW19s8RdvsOOMqzfzugN/BtrCopIiJ1M9ba2ncwZhAwFZgHeNvFbwS6AFhr/26M2QN4EugAGOAea+2zdRx3DbDMr6gUWFv/PyEp0qWu6VJPUF3jIV3qCaprfXS11oZt4qgz0BPFGDPbWluR7HpEI13qmi71BNU1HtKlnqC6xoruFBURyRAKdBGRDJFKgT4m2RWoh3Spa7rUE1TXeEiXeoLqGhMp04YuIiKNk0pn6CIi0ggKdBGRDBHXQDfG/NsYs9oYMz+o/CpjzFeeoXbv85R1M8ZsN8Z85ln+7rf/gcaYecaYRcaYv3pudoprPY0xL/rVZakx5jO/bTd46vK1MeZYv/LjPGWLjDHXx7KODalrMl/TWuq6nzFmpqc+s40xB3vKjacei4wxnxtjDvB7zkXGmG88y0VJrucRxpiNfq/pLX7PSda//77GmBmef883/IfdSNZ7tT71TIH3adghwo0xJcaYiZ733URjTGtPedLeq3Wy1sZtAX4GHADM9ys7EngfyPP83Nbz2M1/v6DjfAwMwN20NAE4Pt71DNo+GrjFs94XmAvkAd2BxUC2Z1kM7Ak08+zTNxGvaS11TdprWsu//3ve3wWcAEz2W5/gqc8AYJanvAR313EJ0Nqz3jqJ9TwCeDPMMZL27w/8DxjsWR8O3JHs92o965ns92kH4ADPeiGw0PPa3Qdc7ym/Hrg32e/Vupa4nqFba6fghtL1dwXuTtKdnn1W13YMY0wHoJW1dqZ1r9rTwKkJqKf39xvgbOA/nqJTgBestTuttUuARcDBnmWRtfZb68aBf8Gzb0zVs65hJeI1raWuFvCeQRbhRugE91o9bZ2ZQLGnnscCE62166y164GJwHFJrGckyfz37wVM8axPBM7wrCftvVrPeoaVwPdppCHCTwGe8uz2lN/vTtp7tS7JaEPvBRxujJlljPmvMeYgv23djTGfesoP95R1BL732+d7Qsdjj6fDgVXW2m/86rM8TH0ilSdScF0h9V7Ta4D7jTHLgQeAG/zqlEqva6R6Agw0bnauCcaYfp6yZP77f4EvkM8COtdRp2TVNVI9IUXepyZwiPB21tqVnk0/Au386pVKr+tPkhHoObivJAOA3wNjPWeWK4Eu1tr9gd8Bz5swQ/AmwXnUccabQoLrmoqv6RXAtdbazsC1uIHfUlGken6CG0tjX+BvwGtJqp+/4cCVxpg5uCaDVJ0pLFI9U+J9amoZItzzDSHl+3gnI9C/xzfU7se4Ab9KPV8LKwGstXNwbXy9gB+ATn7P7+QpiztjTA5wOvCiX/EPBJ5ZeOsTqTwhwtU1FV9T4CLAOwTzS7iv/5B6r2vYelprN1lrt3jW3wZyjTGlSawn1tqvrLVDrbUH4j7QF3s2pdRrGqmeqfA+NeGHCF/laUrxNv94m4dT6nX1l4xAfw13YRRjTC/cRZm1xpgyY0y2p3xPoCfwrecrzyZjzADPmfyFwOsJquvRwFfWWv+vfeOBc40xecaY7p56foy74NPTGNPdGNMMONezb6KE1DVFX9MVwGDP+hDA2zw0HrjQ04NgALDRU893gaHGmNaeXgZDPWVJqacxpr23p4VxPV+ygEqS+O9vjGnrecwCRgHeXiIp9V6NVM9kv089xw43RPh43Ac7nsfX/cpT6b3qE88rrrhP4ZVAFe7M/BJcgD8LzMd9fR3i2fcMXBvbZ57yYX7HqfDsvxg33Z2Jdz095U8Cl4fZ/yZPXb7G76o77ur3Qs+2mxL1mkaqazJf01r+/QcBc3A9K2YBB3r2NcAjnvrMAyr8jjMcd0FvEfDLJNfzN57XdC4wEzg02f/+wNWe37sQuMf/3zJZ79X61DMF3qeDcM0pn3vq8Jnn9WkDfID7MH8fKEn2e7WuRbf+i4hkCN0pKiKSIRToIiIZQoEuIpIhFOgiIhlCgS4ikiEU6CIiGUKBLiKSIf4fwAoT7O/wHJ0AAAAASUVORK5CYII=\n", 269 | "text/plain": [ 270 | "
" 271 | ] 272 | }, 273 | "metadata": { 274 | "needs_background": "light" 275 | }, 276 | "output_type": "display_data" 277 | } 278 | ], 279 | "source": [ 280 | "plt.plot(X_test, Y_pred, color=\"g\")\n", 281 | "plt.plot()\n", 282 | "plt.plot(X_test, Y_test, color=\"r\")\n", 283 | "plt.plot()\n", 284 | "plt.show()" 285 | ] 286 | }, 287 | { 288 | "cell_type": "code", 289 | "execution_count": 15, 290 | "metadata": {}, 291 | "outputs": [ 292 | { 293 | "name": "stdout", 294 | "output_type": "stream", 295 | "text": [ 296 | "0.8829116848754562\n" 297 | ] 298 | } 299 | ], 300 | "source": [ 301 | "R_2 = sum((Y_pred - y_mean)**2) / sum((Y_test - y_mean)**2)\n", 302 | "print(R_2)" 303 | ] 304 | }, 305 | { 306 | "cell_type": "code", 307 | "execution_count": null, 308 | "metadata": {}, 309 | "outputs": [], 310 | "source": [] 311 | } 312 | ], 313 | "metadata": { 314 | "kernelspec": { 315 | "display_name": "Python 3", 316 | "language": "python", 317 | "name": "python3" 318 | }, 319 | "language_info": { 320 | "codemirror_mode": { 321 | "name": "ipython", 322 | "version": 3 323 | }, 324 | "file_extension": ".py", 325 | "mimetype": "text/x-python", 326 | "name": "python", 327 | "nbconvert_exporter": "python", 328 | "pygments_lexer": "ipython3", 329 | "version": "3.7.3" 330 | } 331 | }, 332 | "nbformat": 4, 333 | "nbformat_minor": 2 334 | } 335 | -------------------------------------------------------------------------------- /Logistic Regression with Newtons Method.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 4, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np\n", 10 | "class LogisticRegression:\n", 11 | " \n", 12 | " def fit(self, X, y, epochs=10, lr=0.1):\n", 13 | " X = np.insert(X, 0, 1, axis=1)\n", 14 | " self.weights = np.zeros(X.shape[1])\n", 15 | " self.loss = []\n", 16 | " Xs = X.T.dot(X)\n", 17 | " for i in range(epochs):\n", 18 | " h = self.sigmoid(X.dot(self.weights))\n", 19 | " self.loss.append(self.get_loss(h,y))\n", 20 | " invH = np.linalg.pinv(Xs * h.dot(1-h))\n", 21 | " gradient = (h - y).dot(X)\n", 22 | " self.weights -= invH.dot(gradient)\n", 23 | " return self\n", 24 | " \n", 25 | " def predict(self, X):\n", 26 | " return self.sigmoid(np.insert(X, 0, 1, axis=1).dot(self.weights))\n", 27 | " \n", 28 | " def get_loss(self, h, y):\n", 29 | " return np.abs(y.dot(np.log(h)) + (1 - y).dot(np.log(1 - h)))\n", 30 | " \n", 31 | " def sigmoid(self, z):\n", 32 | " return 1/(1 + np.exp(-1 * z))\n", 33 | " \n", 34 | " def predict_classes(self, X):\n", 35 | " return (self.predict(X) >= 0.5) * 1" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": null, 41 | "metadata": {}, 42 | "outputs": [], 43 | "source": [] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 7, 48 | "metadata": {}, 49 | "outputs": [], 50 | "source": [ 51 | "from sklearn.metrics import accuracy_score, f1_score\n", 52 | "def train_model(X, y, model):\n", 53 | " model.fit(X, y, lr=0.1)\n", 54 | " print(model.predict(X[:2, :]))\n", 55 | " print(model.predict_classes(X[:2, :]))\n", 56 | " pre = model.predict_classes(X)\n", 57 | " print('Accuracy :: ', model.get_loss(model.predict(X), y))\n", 58 | " print('F1 Score :: ', f1_score(y, pre))" 59 | ] 60 | }, 61 | { 62 | "cell_type": "code", 63 | "execution_count": 8, 64 | "metadata": {}, 65 | "outputs": [ 66 | { 67 | "name": "stdout", 68 | "output_type": "stream", 69 | "text": [ 70 | "[0.46901795 0.47771344]\n", 71 | "[0 0]\n", 72 | "('Accuracy :: ', 95.3393291345393)\n", 73 | "('F1 Score :: ', 1.0)\n" 74 | ] 75 | } 76 | ], 77 | "source": [ 78 | "from sklearn.datasets import load_iris\n", 79 | "X, y = load_iris(return_X_y=True)\n", 80 | "train_model(X,(y !=0 )*1, LogisticRegression())" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": 256, 86 | "metadata": {}, 87 | "outputs": [ 88 | { 89 | "name": "stdout", 90 | "output_type": "stream", 91 | "text": [ 92 | "[0.49048766 0.49403437]\n", 93 | "[0 0]\n", 94 | "('Accuracy :: ', 0.9648506151142355)\n", 95 | "('F1 Score :: ', 0.9726027397260274)\n" 96 | ] 97 | } 98 | ], 99 | "source": [ 100 | "from sklearn.datasets import load_breast_cancer\n", 101 | "X,y = load_breast_cancer(return_X_y=True)\n", 102 | "model = LogisticRegression()\n", 103 | "train_model(X,y, model)" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": 258, 109 | "metadata": {}, 110 | "outputs": [ 111 | { 112 | "data": { 113 | "text/plain": [ 114 | "array([ 1.75896854e-01, 1.51896300e-02, -3.17047148e-04, -1.65585845e-03,\n", 115 | " -2.21690168e-05, -5.90715949e-03, 2.94487743e-01, -9.75106585e-02,\n", 116 | " -1.49393191e-01, -7.16398499e-03, -2.31986997e-03, -3.03382659e-02,\n", 117 | " 4.71406827e-04, 1.57079149e-03, 6.43946929e-05, -1.10584177e+00,\n", 118 | " -4.52706299e-03, 2.48692110e-01, -7.37116644e-01, -1.18389729e-01,\n", 119 | " 4.98465538e-01, -1.36140562e-02, -4.99368104e-04, 1.69845087e-04,\n", 120 | " 7.05330073e-05, -3.78643487e-02, -4.68430831e-03, -2.65881514e-02,\n", 121 | " -3.23856876e-02, -3.88360534e-02, -3.00168736e-01])" 122 | ] 123 | }, 124 | "execution_count": 258, 125 | "metadata": {}, 126 | "output_type": "execute_result" 127 | } 128 | ], 129 | "source": [ 130 | "model.weights" 131 | ] 132 | }, 133 | { 134 | "cell_type": "code", 135 | "execution_count": null, 136 | "metadata": {}, 137 | "outputs": [], 138 | "source": [] 139 | } 140 | ], 141 | "metadata": { 142 | "kernelspec": { 143 | "display_name": "python3", 144 | "language": "python", 145 | "name": "python3" 146 | }, 147 | "language_info": { 148 | "codemirror_mode": { 149 | "name": "ipython", 150 | "version": 2 151 | }, 152 | "file_extension": ".py", 153 | "mimetype": "text/x-python", 154 | "name": "python", 155 | "nbconvert_exporter": "python", 156 | "pygments_lexer": "ipython2", 157 | "version": "2.7.16" 158 | } 159 | }, 160 | "nbformat": 4, 161 | "nbformat_minor": 2 162 | } 163 | -------------------------------------------------------------------------------- /Logistic Regression.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Logistic Regression in Python and Numpy" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "In logistic regression, we are trying to model the outcome of a **binary variable** given a **linear combination of input features**. For example, we could try to predict the outcome of an election (win/lose) using information about how much money a candidate spent campaigning, how much time she/he spent campaigning, etc.\n", 15 | "\n", 16 | "### Model \n", 17 | "\n", 18 | "Logistic regression works as follows.\n", 19 | "\n", 20 | "**Given:** \n", 21 | "- dataset $\\{(\\boldsymbol{x}^{(1)}, y^{(1)}), ..., (\\boldsymbol{x}^{(m)}, y^{(m)})\\}$\n", 22 | "- with $\\boldsymbol{x}^{(i)}$ being a $d-$dimensional vector $\\boldsymbol{x}^{(i)} = (x^{(i)}_1, ..., x^{(i)}_d)$\n", 23 | "- $y^{(i)}$ being a binary target variable, $y^{(i)} \\in \\{0,1\\}$\n", 24 | "\n", 25 | "The logistic regression model can be interpreted as a very **simple neural network:**\n", 26 | "- it has a real-valued weight vector $\\boldsymbol{w}= (w^{(1)}, ..., w^{(d)})$\n", 27 | "- it has a real-valued bias $b$\n", 28 | "- it uses a sigmoid function as its activation function\n", 29 | "\n", 30 | "![title](figures/logistic_regression.jpg)" 31 | ] 32 | }, 33 | { 34 | "cell_type": "markdown", 35 | "metadata": {}, 36 | "source": [ 37 | "### Training\n", 38 | "\n", 39 | "Different to [linear regression](linear_regression.ipynb), logistic regression has no closed form solution. But the cost function is convex, so we can train the model using gradient descent. In fact, **gradient descent** (or any other optimization algorithm) is guaranteed to find the global minimum (if the learning rate is small enough and enough training iterations are used). \n", 40 | "\n", 41 | "Training a logistic regression model has different steps. In the beginning (step 0) the parameters are initialized. The other steps are repeated for a specified number of training iterations or until convergence of the parameters.\n", 42 | "\n", 43 | "* * * \n", 44 | "**Step 0: ** Initialize the weight vector and bias with zeros (or small random values).\n", 45 | "* * *\n", 46 | "\n", 47 | "**Step 1: ** Compute a linear combination of the input features and weights. This can be done in one step for all training examples, using vectorization and broadcasting:\n", 48 | "$\\boldsymbol{a} = \\boldsymbol{X} \\cdot \\boldsymbol{w} + b $\n", 49 | "\n", 50 | "where $\\boldsymbol{X}$ is a matrix of shape $(n_{samples}, n_{features})$ that holds all training examples, and $\\cdot$ denotes the dot product.\n", 51 | "* * *\n", 52 | "\n", 53 | "**Step 2: ** Apply the sigmoid activation function, which returns values between 0 and 1:\n", 54 | "\n", 55 | "$\\boldsymbol{\\hat{y}} = \\sigma(\\boldsymbol{a}) = \\frac{1}{1 + \\exp(-\\boldsymbol{a})}$\n", 56 | "* * *\n", 57 | "\n", 58 | "** Step 3: ** Compute the cost over the whole training set. We want to model the probability of the target values being 0 or 1. So during training we want to adapt our parameters such that our model outputs high values for examples with a positive label (true label being 1) and small values for examples with a negative label (true label being 0). This is reflected in the cost function:\n", 59 | "\n", 60 | "$J(\\boldsymbol{w},b) = - \\frac{1}{m} \\sum_{i=1}^m \\Big[ y^{(i)} \\log(\\hat{y}^{(i)}) + (1 - y^{(i)}) \\log(1 - \\hat{y}^{(i)}) \\Big]$\n", 61 | "* * *\n", 62 | "\n", 63 | "** Step 4: ** Compute the gradient of the cost function with respect to the weight vector and bias. A detailed explanation of this derivation can be found [here](https://stats.stackexchange.com/questions/278771/how-is-the-cost-function-from-logistic-regression-derivated).\n", 64 | "\n", 65 | "The general formula is given by:\n", 66 | "\n", 67 | "$ \\frac{\\partial J}{\\partial w_j} = \\frac{1}{m}\\sum_{i=1}^m\\left[\\hat{y}^{(i)}-y^{(i)}\\right]\\,x_j^{(i)}$\n", 68 | "\n", 69 | "For the bias, the inputs $x_j^{(i)}$ will be given 1.\n", 70 | "* * *\n", 71 | "\n", 72 | "** Step 5: ** Update the weights and bias\n", 73 | "\n", 74 | "$\\boldsymbol{w} = \\boldsymbol{w} - \\eta \\, \\nabla_w J$ \n", 75 | "\n", 76 | "$b = b - \\eta \\, \\nabla_b J$\n", 77 | "\n", 78 | "where $\\eta$ is the learning rate." 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": 18, 84 | "metadata": {}, 85 | "outputs": [], 86 | "source": [ 87 | "import numpy as np\n", 88 | "class LogisticRegression:\n", 89 | " \n", 90 | " def fit(self, X, y, lr = 0.001, epochs=10000, verbose=True, batch_size=1):\n", 91 | " self.classes = np.unique(y)\n", 92 | " y = (y==self.classes[1]) * 1\n", 93 | " X = self.add_bias(X)\n", 94 | " self.weights = np.zeros(X.shape[1])\n", 95 | " self.loss = []\n", 96 | " for i in range(epochs):\n", 97 | " self.loss.append(self.cross_entropy(X,y))\n", 98 | " if i % 1000 == 0 and verbose: \n", 99 | " print('Iterations: %d - Error : %.4f' %(i, self.loss[i]))\n", 100 | " idx = np.random.choice(X.shape[0], batch_size)\n", 101 | " X_batch, y_batch = X[idx], y[idx]\n", 102 | " self.weights -= lr * self.get_gradient(X_batch, y_batch)\n", 103 | " return self\n", 104 | " \n", 105 | " def get_gradient(self, X, y):\n", 106 | " return -1.0 * (y - self.predict_(X)).dot(X) / len(X)\n", 107 | " \n", 108 | " def predict_(self, X):\n", 109 | " return self.sigmoid(np.dot(X, self.weights))\n", 110 | " \n", 111 | " def predict(self, X):\n", 112 | " return self.predict_(self.add_bias(X))\n", 113 | " \n", 114 | " def sigmoid(self, z):\n", 115 | " return 1.0/(1 + np.exp(-z))\n", 116 | " \n", 117 | " def predict_classes(self, X):\n", 118 | " return self.predict_classes_(self.add_bias(X))\n", 119 | "\n", 120 | " def predict_classes_(self, X):\n", 121 | " return np.vectorize(lambda c: self.classes[1] if c>=0.5 else self.classes[0])(self.predict_(X))\n", 122 | " \n", 123 | " def cross_entropy(self, X, y):\n", 124 | " p = self.predict_(X)\n", 125 | " return (-1 / len(y)) * (y * np.log(p)).sum()\n", 126 | "\n", 127 | " def add_bias(self,X):\n", 128 | " return np.insert(X, 0, 1, axis=1)\n", 129 | "\n", 130 | " def score(self, X, y):\n", 131 | " return self.cross_entropy(self.add_bias(X), y)\n" 132 | ] 133 | }, 134 | { 135 | "cell_type": "code", 136 | "execution_count": 19, 137 | "metadata": {}, 138 | "outputs": [], 139 | "source": [ 140 | "from sklearn.metrics import accuracy_score\n", 141 | "def train_model(X, y, model):\n", 142 | " model.fit(X, y, lr=0.1)\n", 143 | " pre = model.predict_classes(X)\n", 144 | " print('Accuracy :: ', accuracy_score(y, pre))" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": 20, 150 | "metadata": {}, 151 | "outputs": [ 152 | { 153 | "name": "stdout", 154 | "output_type": "stream", 155 | "text": [ 156 | "Iterations: 0 - Error : 69.3147\n", 157 | "Iterations: 1000 - Error : 0.4746\n", 158 | "Iterations: 2000 - Error : 0.3847\n", 159 | "Iterations: 3000 - Error : 0.1645\n", 160 | "Iterations: 4000 - Error : 0.1280\n", 161 | "Iterations: 5000 - Error : 0.1126\n", 162 | "Iterations: 6000 - Error : 0.0783\n", 163 | "Iterations: 7000 - Error : 0.0674\n", 164 | "Iterations: 8000 - Error : 0.0621\n", 165 | "Iterations: 9000 - Error : 0.0664\n", 166 | "('Accuracy :: ', 1.0)\n" 167 | ] 168 | } 169 | ], 170 | "source": [ 171 | "from sklearn.datasets import load_iris\n", 172 | "X, y = load_iris(return_X_y=True)\n", 173 | "lr = LogisticRegression()\n", 174 | "train_model(X,(y !=0 )*1, lr)" 175 | ] 176 | }, 177 | { 178 | "cell_type": "code", 179 | "execution_count": 21, 180 | "metadata": {}, 181 | "outputs": [ 182 | { 183 | "data": { 184 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfgAAAGDCAYAAADHzQJ9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi40LCBodHRwOi8vbWF0cGxvdGxpYi5vcmcv7US4rQAAIABJREFUeJzt3XmcJXV97//Xu7tngQEZBkbCpoOIJi6J4mhQs+BylRgVc39G8XoVo5FrzGqSqxIT482qMVFjTMw1omhUlKCJmLgRhBhvFBxcWF1GdhxgWIRhmbU/vz/q23BseqZ7Zrr7TBev5+NxHl31rW/V+Zzqmn6fWqYqVYUkSeqXkWEXIEmSZp8BL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8NJ2JDk3yS8Pu475lOThSb6RZEOS3xh2PQtdkgcluSPJ6Gz2lWbCgNceLcmVSe5ugfODJP+V5FVJ3HanMAtfSl4LnFNV+1bVO2erru1JcmqSP5nr99kVSV6W5Eu7s4yqurqq9qmqbbPZV5oJ/0hqIXhOVe0LPBh4M/A64JThltRbDwYuGXYR8y3J2C7O59629lgGvBaMqrqtqs4EXgicmORRAEmWJPnLJFcnuSHJ3yfZq027LMmzJ5aRZCzJ+iRHt/Fj2lGBHyT5ZpJjp3rvJCNJfj/JVUluTPLBJPu1aauSVJKTknw/ybokvzsw75uS/FOSD7UjERcleViSk9uyrknyjIH++yU5pS3nuiR/MhEkE3uV7fPemuSKJD/Xpv0p8NPAu9qh3ndt57M8N8kl7TOfm+THWvsXgKcMzP+wKeZdkeT97XPemuRfBqa9MsnaJLckOTPJIa09Sd7ePuvt7fM/KslJwIuB17b3+9R26n1Skq8mua39fFJrf2GSNZP6vibJmTPYLo5Ncm2S1yW5Hnj/pOX8GPD3wBNbbT9o7acmeXeSTye5E3hKkp9P8vX22a5J8qaB5UxsG2Nt/Nwkf5zk/7Vt4fNJDtzZvm36S9v2eHOSP0h3tOvpU61D3U9VlS9fe+wLuBJ4+hTtVwO/0obfDpwJrAD2BT4F/Hmb9kbgwwPz/TxwWRs+FLgZeBbdl93/1sZXtunnAr/chl8OrAUeAuwDfAL4xzZtFVDAacAy4NHA+om6gTcBG4FnAmPAB4ErgDcAi4BXAlcM1PjPwP9ty3ogcD7wv9q0lwFb2jyjwK8A3wcyuebtrM+HAXe2z7qI7pD8WmDxDOf/N+BjwP5t/p9t7U8FbgKOBpYAfwN8sU17JnABsBwI8GPAwW3aqcCf7OD9VgC3Ai9p6+5FbfwAYG9gA3DUQP+vAifMYLs4FtgKvKXVu9cU7/0y4EuT2k4FbgOeTLfNLG3LenQb/3HgBuB5k7aNsYH1+732e9irjb95F/o+ArgD+ClgMfCXbbu4z78VX/ff19AL8OVrRy+2H/BfoQvItMA6cmDaE2mBCTy0hcDebfzDwBvb8OtoIT0w7+eAE9vwudwb8GcDrx7o9/D2B3Vs4A/zjw5M/wvglDb8JuCsgWnPaX+cR9v4vm3+5cBBwKbBwKELtXPa8MuAtQPT9m7z/sjkmrezPv8AOH1gfAS4Djh2uvmBg4FxYP8ppp0C/MXA+D5t/ayiC//vAMcAI5PmO5UdB/xLgPMntX0ZeFkb/tDA7/Ooid/1DLaLY4HNwNIdvPfLmDrgPzjNNvsO4O1teGLbGAzt3x/o+2rgs7vQ943AaZO2g80Y8L4GXrt03knaAxwK3AKspPvjdkGSiWmh27ulqtYmuQx4TjsE/Fzgsa3fg4FfTPKcgeUuAs6Z4v0OAa4aGL+KLtwPGmi7ZtL0Rw+M3zAwfDdwU917MdXd7ec+7X0WAesGPs/IpGVfPzFQVXe1fvtMUfNUfuhzVNV4kmvo1ud0Dgduqapbt7Pcrw0s944kNwOHVtUX2umCvwUenOQTwO9W1e07W29z1UC9HwH+Cvgj4H8A/9LWyQPZwXbRrK+qjTOoYbLB3wVJfpLu2pBH0e1NLwH+aQfzXz8wfBc7/t1tr+8hg3W0z3zztJXrfsVz8Fpwkjye7g/8l+gOC98NPLKqlrfXflU1+EfzNLq94OOBS6tqbWu/hm4PfvnAa1lVvXmKt/0+3ReCCQ+iO8Q7GNyHT5r+/V34eNfQ7cEfOFDTA6rqkTOcf7rHQ/7Q50iXfofT7cXPpLYVSZbPYLnL6A6jXwdQVe+sqsfRHVp+GPC/d6Xe5kED9Z4FrEzyGLrf8Uda+0y2i+nee3vTJ7d/hO5UwOFVtR/dufvcZ67ZtQ44bGKkXVtwwBy/pxYYA14LRpIHpLtg7qPAh6rqoqoaB/4BeHvbayPJoUmeOTDrR4Fn0J2v/shA+4fo9uyfmWQ0ydJ28dVh3NdpwGuSHJFkH+DPgI9V1daBPn+QZO8kjwR+ie5c9U6pqnXA54G/ap93JMmRSX52hou4ge46ge05Hfj5JE9Lsgj4HbovFP81w9o+A/xdkv2TLEryM23yacAvJXlMkiV06+e8qroyyeOT/GR7vzvprkcYn2G9nwYeluR/pLtA8oV0XxL+tdW0hW5v+a1059rPau0z2S6mcwNwWJLF0/Tbl+7IxsYkT6A7kjDXzqDbdp/U6nsTc/+lQguMAa+F4FNJNtDtQb4BeBtdgE54Hd2FYl9Jcjvw73TnyIF7gunLwJMYCN2quoZur/736C6Ku4Zuz3KqfxfvA/4R+CLdBXIbgV+f1Oc/Wh1nA39ZVZ/ftY/LS+kO9V5Kd0HZGXTnv2fir4Hnp7vC/T7/j72qvg38T7qL4G6iux7gOVW1eYbLfwndufVvATcCv9WW++905/c/Trd3eSRwQpvnAXRheyvd4fWb6QIZunP3j0h3Rf89V+QP1Hsz8Gy6LyI3010U+Oyqummg20eApwP/NOkL1w63ixn4At1/Gbw+yU076Pdq4I/aNvpGui9Rc6qqLqHb/j5Kt77voPt9bJrr99bCMXHlraRdlGQVXegvmhQw0rxoR5V+QPc/Cq4Ydj3aM7gHL0kLUJLntFNCy+j+m9xFdP/rRAIMeElaqI6nuwjx+3T/RfCE8pCsBniIXpKkHnIPXpKkHjLgJUnqoQV9J7sDDzywVq1aNewyJEmaNxdccMFNVbVyun4LOuBXrVrFmjVrpu8oSVJPJJl8++YpeYhekqQeMuAlSeohA16SpB4y4CVJ6iEDXpKkHjLgJUnqIQNekqQemrOAT/K+JDcmuXiKab+TpJIc2MaT5J1J1ia5MMnRc1WXJEn3B3O5B38qcNzkxiSHA88Arh5o/jm6pyEdBZwEvHsO65IkqffmLOCr6ovALVNMejvwWmDwMXbHAx+szleA5UkOnqvaJEnqu3k9B5/keOC6qvrmpEmHAtcMjF/b2iRJ0i6Yt4BPsjfwe8Abd3M5JyVZk2TN+vXrZ6e4Zu2Nd1BV03eUJGkPN5978EcCRwDfTHIlcBjwtSQ/AlwHHD7Q97DWdh9V9Z6qWl1Vq1eunPZhOjN2wVW38vS3/Qen/teVs7ZMSZKGZd4CvqouqqoHVtWqqlpFdxj+6Kq6HjgTeGm7mv4Y4LaqWjdftQFcdfOdAFx47W3z+baSJM2JufxvcqcBXwYenuTaJK/YQfdPA5cDa4F/AF49V3VJknR/MGfPg6+qF00zfdXAcAG/Ole1SJJ0f+Od7CRJ6iEDXpKkHjLgJUnqIQNekqQeMuAlSeohA16SpB4y4CVJ6iEDXpKkHjLgJUnqIQNekqQeMuAlSeohA16SpB4y4CVJ6iEDXpKkHjLgJUnqIQNekqQeMuAlSeohA16SpB4y4CVJ6iEDXpKkHjLgJUnqIQNekqQeMuAnqaphlyBJ0m4z4Jtk2BVIkjR7DHhJknrIgG88Mi9J6hMDfpJ4rF6S1AMGvCRJPWTAS5LUQwa8JEk9ZMBLktRDBrwkST1kwEuS1EMGvCRJPTRnAZ/kfUluTHLxQNtbk3wryYVJ/jnJ8oFpJydZm+TbSZ45V3VJknR/MJd78KcCx01qOwt4VFX9OPAd4GSAJI8ATgAe2eb5uySjc1ibJEm9NmcBX1VfBG6Z1Pb5qtraRr8CHNaGjwc+WlWbquoKYC3whLmqTZKkvhvmOfiXA59pw4cC1wxMu7a13UeSk5KsSbJm/fr1c1yiJEkL01ACPskbgK3Ah3d23qp6T1WtrqrVK1eunP3iJEnqgbH5fsMkLwOeDTyt6p5nuF0HHD7Q7bDWJkmSdsG87sEnOQ54LfDcqrprYNKZwAlJliQ5AjgKOH8+a5MkqU/mbA8+yWnAscCBSa4F/pDuqvklwFntsaxfqapXVdUlSU4HLqU7dP+rVbVtrmqTJKnv5izgq+pFUzSfsoP+fwr86VzVI0nS/Yl3spMkqYcMeEmSesiAlySphwx4SZJ6yICXJKmHDHhJknrIgJ/k3pvrSZK0cBnwTXffHUmS+sGAlySphwx4SZJ6yICXJKmHDPjGa+skSX1iwE8Sr7aTJPWAAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8JNU1bBLkCRptxnwTTLsCiRJmj1zFvBJ3pfkxiQXD7StSHJWku+2n/u39iR5Z5K1SS5McvRc1SVJ0v3BXO7BnwocN6nt9cDZVXUUcHYbB/g54Kj2Ogl49xzWJUlS781ZwFfVF4FbJjUfD3ygDX8AeN5A+wer8xVgeZKD56q2qYyPdz8vXXf7fL6tJElzYr7PwR9UVeva8PXAQW34UOCagX7Xtrb7SHJSkjVJ1qxfv37WCrv+9o0AfOeGO2ZtmZIkDcvQLrKr7nL1nb5kvareU1Wrq2r1ypUr56AySZIWvvkO+BsmDr23nze29uuAwwf6HdbaJEnSLpjvgD8TOLENnwh8cqD9pe1q+mOA2wYO5UuSpJ00NlcLTnIacCxwYJJrgT8E3gycnuQVwFXAC1r3TwPPAtYCdwG/NFd1SZJ0fzBnAV9VL9rOpKdN0beAX52rWiRJur/xTnaSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfBN97wbSZL6wYCXJKmHDHhJknrIgJckqYcM+CbJsEuQJGnWGPCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8E1VDbsESZJmjQEvSVIPDSXgk7wmySVJLk5yWpKlSY5Icl6StUk+lmTxMGqTJKkP5j3gkxwK/AawuqoeBYwCJwBvAd5eVQ8FbgVeMd+1SZLUF8M6RD8G7JVkDNgbWAc8FTijTf8A8Lwh1SZJ0oI37wFfVdcBfwlcTRfstwEXAD+oqq2t27XAofNdmyRJfTGMQ/T7A8cDRwCHAMuA43Zi/pOSrEmyZv369bNZ16wtS5KkYRvGIfqnA1dU1fqq2gJ8AngysLwdsgc4DLhuqpmr6j1VtbqqVq9cuXJ+KpYkaYEZRsBfDRyTZO90u81PAy4FzgGe3/qcCHxyCLVJktQLwzgHfx7dxXRfAy5qNbwHeB3w20nWAgcAp8x3bZIk9cXY9F1mX1X9IfCHk5ovB54whHIkSeod72QnSVIPGfCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPXQjAI+yT/OpE2SJO0ZZroH/8jBkSSjwONmvxxJkjQbdhjwSU5OsgH48SS3t9cG4EZ8GIwkSXusHQZ8Vf15Ve0LvLWqHtBe+1bVAVV18jzVKEmSdtJMD9H/a5JlAEn+Z5K3JXnwHNYlSZJ2w0wD/t3AXUl+Avgd4HvAB+esKkmStFtmGvBbq6qA44F3VdXfAvvOXVnzr/t4kiT1w0yfB78hycnAS4CfTjICLJq7siRJ0u6Y6R78C4FNwMur6nrgMOCtc1bVENy4YdOwS5AkadbMKOBbqH8Y2C/Js4GNVdWrc/DfW3/HsEuQJGnWzPROdi8Azgd+EXgBcF6S589lYZIkadfN9Bz8G4DHV9WNAElWAv8OnDFXhUmSpF0303PwIxPh3ty8E/NKkqR5NtM9+M8m+RxwWht/IfDpuSlJkiTtrh0GfJKHAgdV1f9O8t+Bn2qTvkx30Z0kSdoDTbcH/w7gZICq+gTwCYAkj27TnjOn1UmSpF0y3Xn0g6rqosmNrW3VnFQkSZJ223QBv3wH0/aazUIkSdLsmS7g1yR55eTGJL8MXDA3JUmSpN013Tn43wL+OcmLuTfQVwOLgV+Yy8IkSdKu22HAV9UNwJOSPAV4VGv+t6r6wpxXJkmSdtmM/h98VZ0DnDPHtUiSpFni3egkSeohA16SpB4y4JuQYZcgSdKsMeAlSeqhoQR8kuVJzkjyrSSXJXlikhVJzkry3fZz/2HUJklSHwxrD/6vgc9W1Y8CPwFcBrweOLuqjgLObuOSJGkXzHvAJ9kP+BngFICq2lxVPwCOBz7Qun0AeN581yZJUl8MYw/+CGA98P4kX0/y3iTL6B5ss671uR44aKqZk5yUZE2SNevXr5+1ouI1dpKkHhlGwI8BRwPvrqrHAncy6XB8VRVQU81cVe+pqtVVtXrlypVzXqwkSQvRMAL+WuDaqjqvjZ9BF/g3JDkYoP28cQi1SZLUC/Me8FV1PXBNkoe3pqcBlwJnAie2thOBT853bZIk9cWM7kU/B34d+HCSxcDlwC/Rfdk4PckrgKuAFwypNkmSFryhBHxVfYPusbOTPW2+a5EkqY+8k11TU17SJ0nSwmTAS5LUQwa8JEk9ZMBLktRDBrwkST1kwEuS1EMGvCRJPWTAS5LUQwa8JEk9ZMBLktRDBrwkST1kwDfJsCuQJGn2GPCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAHfxMvoJUk9YsBLktRDBrwkST1kwEuS1EMGvCRJPWTAN1U17BIkSZo1BrwkST1kwEuS1EMGvCRJPWTAS5LUQwb8FO7evG3YJUiStFsM+Cls2Lhl2CVIkrRbDHhJknrIgJckqYcMeEmSesiAlySph4YW8ElGk3w9yb+28SOSnJdkbZKPJVk8rNrw0fCSpAVumHvwvwlcNjD+FuDtVfVQ4FbgFUOpCsDb0kuSFrihBHySw4CfB97bxgM8FTijdfkA8Lxh1CZJUh8Maw/+HcBrgfE2fgDwg6ra2savBQ6dz4K67xiSJPXDvAd8kmcDN1bVBbs4/0lJ1iRZs379+tmra7sjkiQtPMPYg38y8NwkVwIfpTs0/9fA8iRjrc9hwHVTzVxV76mq1VW1euXKlfNRryRJC868B3xVnVxVh1XVKuAE4AtV9WLgHOD5rduJwCfnuzZJkvpiT/p/8K8DfjvJWrpz8qcMrRKvopckLXBj03eZO1V1LnBuG74ceMLQahnWG0uSNAf2pD34PYcX2UmSFjgDXpKkHjLgJUnqIQNekqQeMuAbT7tLkvrEgJckqYcMeEmSesiAlySphwx4SZJ6yICfQrzkTpK0wBnwkiT1kAE/hfLO9JKkBc6AlySphwx4SZJ6yICfghfZSZIWOgO+iZkuSeoRA16SpB4y4JvywnlJUo8Y8JIk9ZABL0lSDxnwkiT1kAHfeBW9JKlPDHhJknrIgJ+Ce/OSpIXOgJ+C/2VOkrTQGfCSJPWQAT+FW+/aPOwSJEnaLQb8FL703ZuGXYIkSbvFgJckqYcM+Cl4jZ0kaaEz4CVJ6iEDXpKkHjLgJUnqIQO+8eZ1kqQ+mfeAT3J4knOSXJrkkiS/2dpXJDkryXfbz/3nuzZJkvpiGHvwW4HfqapHAMcAv5rkEcDrgbOr6ijg7DYuSZJ2wbwHfFWtq6qvteENwGXAocDxwAdatw8Az5vPuuITZiRJPTLUc/BJVgGPBc4DDqqqdW3S9cBB25nnpCRrkqxZv379rNVSA0+Y2bx1fNaWK0nSMAwt4JPsA3wc+K2qun1wWnVpO+X9ZqrqPVW1uqpWr1y5ck5qe8tnv8WnL1o3fUdJkvZQQwn4JIvowv3DVfWJ1nxDkoPb9IOBG4dR24TXfOwbw3x7SZJ2yzCuog9wCnBZVb1tYNKZwIlt+ETgk/Nd26BNHqaXJC1gY0N4zycDLwEuSjKxm/x7wJuB05O8ArgKeMEQapMkqRfmPeCr6kts/74yT5vPWiRJ6ivvZCdJUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCN96KXJPWJAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8JIk9ZABL0lSDxnwkiT1kAEvSVIPGfCSJPWQAS9JUg8Z8I3PkpMk9YkBL0lSDxnwO3DLnZuHXYIkSbvEgN+Bo//4LM67/Ga+fvWtVNWwy5EkacYM+Gn8n09dyi/83X/xqQvXDbsUSZJmzICfxhU33QnA9268Y8iVSJI0cwb8NEZHuuvrz/zm99mybXzI1UiSNDMG/DRavnPFTXfy7nO/N9xiJEmaIQN+Grdv3HrP8I0bNg6xEkmSZs6A3wkf+srV/Fu72O5tn/82F11725ArkiRpagb8Tvrt07/Bqtf/G+/8wlqe864vDbscSZKmZMDvpE1bvdBOkrTnM+CbRaO7tipO/+o1s1yJJEm7z4Bvfu7RP7JL87324xdy1+at92nfuGXb7pYkSdIuM+CbZz364F2e91Uf+hrv+sJ32bJtnH/88pWc/ImL+NE/+Cz/9b2b7ulTVVN+EZAkaS5kT7vHepLjgL8GRoH3VtWbt9d39erVtWbNmll5323jxZG/92kAPvObP80D913Cqz50AV+98lZee9zDeeVPP4Tvrb+D497xnzu97O/92bPuWfbLn3wERz5wGc9/3GHcsXEr+y5dxJ99+jLWb9jETx91IE888gAeuO9SbrpjE4ev2HtWPpskqT+SXFBVq6fttycFfJJR4DvAfwOuBb4KvKiqLp2q/2wGPMA/fPFynvzQA3nEIQ8A4O7N2/jMxev4hcceSnLvE+PHx4tnvfM/+db1G2btvaeyeGyEhx20D9vG4XmPOYRvXPMDbtywiX2XjrFi2WKuv20jd27exmXfv50fO+QBrH7w/ixdNMLVt9zNJd+/jUcesh9POGIFowlX3XInS8ZG2TY+zs13bObym+7k/CtuAeCYh6zgd57xcEYSNm3dxt2bt3HHpq3dPQCqWDQ6wkjC+js2cdj+e3HwfnvdcwOgzVvHuebWu7hr8zauvuUuqmC/vRaxeGyEJWMjbBsvNm4ZZ8WyRTz0gfuyeCxs2VZs3VYsWzLKAcuWsGTRCPvttah9nq1s3DLO6Ei4a9NWliwaYemiURaNjrBh41a2bBvnwQfszf57L6YKbrpjE9vGi/Eqxgug+zleRQ38HB0JDztoX7aNF0XXVgXV+lcVBdQ490wfb23jXcd7hqu6zz2SMDYa9l48yvK9F9/ze+uO1mzj1rs2MzoSRhIesHQRey0endPtRdL9w0IN+CcCb6qqZ7bxkwGq6s+n6j/bAb8rrrnlLv703y5jn6VjHLp8Lx6/agVv+tQlfPxVT+K7N27g+X//5Xv6vvrYI/m7dje8/fdexK13bQHgCatWcP6Vt/zQckdHwrbxH/7d7Lt0jAet2Jvv3LCBbePF0kWjPGjF3rv8RWPvxaPctdlrBWbD0kUjjI2MUFVs2jrO1vH7/rtaumiEJWOj7LVolK3jRdW9X0wm/h2OtYs97/nC0RYzMT7xReM+0ye+tNzTpxvfVkWAkXRfNEh3d8aJ8QAJjIwMjoeR3Hub5h2Zvgc/9OV4V8xk9un6ZAaVTr+MmdSx417TLmMmn3U3a5jZMmZSxzSfdfd+7W0Zu7k+Z1DHfHzW0155DMuWjE3/RjO0UAP++cBxVfXLbfwlwE9W1a8N9DkJOAngQQ960OOuuuqqodS6M6pqyg11fLxItr8Rbxsvto0XN9+5ifGCg/ZdwtjoCBu3bCOBJWOj9yzn6lvuYnQk/Mh+S1k0OsJdm7dy2brbOWDZEhaPjbBl2zgH7rOEvRePMl73/vHeuGUbl6+/k3W33c3oSLoAWjzKvkvH2GfJGCMJt9y5mUWj4YB9lnDdrXdz852b7tnDXTw6wuEr9mbpolFWLFvM6EgYHy82bxtn87Zx7trUHQ245Pu3ccCyJWyrYtFI2FbV9r5hw8YtbNi4lUOX78WyJWMsGRthvIq9Fo+yZVuxccs2Nm8dZ9+lXT3fvn4DW8bHCWHFskUsGRu9Zz1OhFkycZvhLqzW3bbxns+RNj3c229i+N5l3Bt0TFpu6I6uVMHW8XHW3baRDQN3PFwy1h2R2G+vRQBsGS9u2rCJu7dsY+OW7gjJ2OgIoyMDwdtMPO9gsL6J0B1sv3eYH6qbSfONjuSHjlRMHIGo2v74eHXb1HjVDv9wzeRPx3RdpltGTbuE6d9kJn/hpvs7OLNl7F4dM/lbPG2PGa2uaT7rTJYxC7+33V9f077FtEuZnW14+oX87YuPZu/FBvy0AT9oT9iDlyRpPs004Pe0q+ivAw4fGD+stUmSpJ2wpwX8V4GjkhyRZDFwAnDmkGuSJGnBmb2TArOgqrYm+TXgc3T/Te59VXXJkMuSJGnB2aMCHqCqPg18eth1SJK0kO1ph+glSdIsMOAlSeohA16SpB4y4CVJ6iEDXpKkHjLgJUnqIQNekqQeMuAlSeohA16SpB7ao54mt7OSrAdm83mxBwI3zeLy7q9cj7vPdbj7XIe7z3W4++ZiHT64qlZO12lBB/xsS7JmJo/g0465Hnef63D3uQ53n+tw9w1zHXqIXpKkHjLgJUnqIQP+h71n2AX0hOtx97kOd5/rcPe5Dnff0Nah5+AlSeoh9+AlSeohA75JclySbydZm+T1w65nT5Lk8CTnJLk0ySVJfrO1r0hyVpLvtp/7t/YkeWdblxcmOXpgWSe2/t9NcuKwPtOwJBlN8vUk/9rGj0hyXltXH0uyuLUvaeNr2/RVA8s4ubV/O8kzh/NJhiPJ8iRnJPlWksuSPNGQRT1RAAAIGElEQVTtcOckeU37d3xxktOSLHU7nF6S9yW5McnFA22ztu0leVySi9o870yS3S66qu73L2AU+B7wEGAx8E3gEcOua095AQcDR7fhfYHvAI8A/gJ4fWt/PfCWNvws4DNAgGOA81r7CuDy9nP/Nrz/sD/fPK/L3wY+AvxrGz8dOKEN/z3wK2341cDft+ETgI+14Ue07XMJcETbbkeH/bnmcf19APjlNrwYWO52uFPr71DgCmCvge3vZW6HM1p3PwMcDVw80DZr2x5wfuubNu/P7W7N7sF3ngCsrarLq2oz8FHg+CHXtMeoqnVV9bU2vAG4jO4PxfF0f3BpP5/Xho8HPlidrwDLkxwMPBM4q6puqapbgbOA4+bxowxVksOAnwfe28YDPBU4o3WZvA4n1u0ZwNNa/+OBj1bVpqq6AlhLt/32XpL96P7IngJQVZur6ge4He6sMWCvJGPA3sA63A6nVVVfBG6Z1Dwr216b9oCq+kp1af/BgWXtMgO+cyhwzcD4ta1Nk7RDdI8FzgMOqqp1bdL1wEFteHvr8/6+nt8BvBYYb+MHAD+oqq1tfHB93LOu2vTbWv/78zo8AlgPvL+d5nhvkmW4Hc5YVV0H/CVwNV2w3wZcgNvhrpqtbe/QNjy5fbcY8JqxJPsAHwd+q6puH5zWvnX6XzK2I8mzgRur6oJh17KAjdEdIn13VT0WuJPusOg93A53rJ0jPp7uy9IhwDLuX0cv5syeuO0Z8J3rgMMHxg9rbWqSLKIL9w9X1Sda8w3t0BLt542tfXvr8/68np8MPDfJlXSngJ4K/DXdobux1mdwfdyzrtr0/YCbuX+vw2uBa6vqvDZ+Bl3gux3O3NOBK6pqfVVtAT5Bt226He6a2dr2rmvDk9t3iwHf+SpwVLuSdDHdxSRnDrmmPUY753YKcFlVvW1g0pnAxFWgJwKfHGh/abuS9BjgtnYY63PAM5Ls3/YkntHaeq+qTq6qw6pqFd329YWqejFwDvD81m3yOpxYt89v/au1n9Cubj4COIru4pzeq6rrgWuSPLw1PQ24FLfDnXE1cEySvdu/64l16Ha4a2Zl22vTbk9yTPu9vHRgWbtu2Fcm7ikvuqsev0N3Negbhl3PnvQCforu0NOFwDfa61l05+LOBr4L/DuwovUP8LdtXV4ErB5Y1svpLshZC/zSsD/bkNbnsdx7Ff1D6P4wrgX+CVjS2pe28bVt+kMG5n9DW7ffZhautF1IL+AxwJq2Lf4L3ZXIboc7tw7/D/At4GLgH+muhHc7nH69nUZ33cIWuqNJr5jNbQ9Y3X4n3wPeRbsR3e68vJOdJEk95CF6SZJ6yICXJKmHDHhJknrIgJckqYcMeEmSesiAl+ZRkkryVwPjv5vkTbO07FOTPH/6nrv9Pr+Y7klu50xqPyTJGW34MUmeNYvvuTzJq6d6L0lTM+Cl+bUJ+O9JDhx2IYMG7mI2E68AXllVTxlsrKrvV9XEF4zH0N0rYbZqWE73ZLOp3kvSFAx4aX5tBd4DvGbyhMl74EnuaD+PTfIfST6Z5PIkb07y4iTnt+dHHzmwmKcnWZPkO+3+9xPPoH9rkq+2Z1P/r4Hl/meSM+nuZja5nhe15V+c5C2t7Y10Nz46JclbJ/Vf1fouBv4IeGGSbyR5YZJl6Z6nfX57UMzxbZ6XJTkzyReAs5Psk+TsJF9r7z3xVMc3A0e25b114r3aMpYmeX/r//UkTxlY9ieSfDbds7f/YmB9nNpqvSjJfX4XUh/szLd2SbPjb4ELJwJnhn4C+DG6x1VeDry3qp6Q5DeBXwd+q/VbRffYziOBc5I8lO62l7dV1eOTLAH+X5LPt/5HA4+q7pGf90hyCPAW4HHArcDnkzyvqv4oyVOB362qNVMVWlWb2xeB1VX1a215f0Z3m9OXJ1kOnJ/k3wdq+PGquqXtxf9CVd3ejnJ8pX0BeX2r8zFteasG3vJXu7etRyf50Vbrw9q0x9A9/XAT8O0kfwM8EDi0qh7VlrV8mnUvLUjuwUvzrLon8X0Q+I2dmO2rVbWuqjbR3cpyIqAvogv1CadX1XhVfZfui8CP0t3v+qVJvkH3mN8D6O4dDnD+5HBvHg+cW91DSbYCH6Z7Fvuuegbw+lbDuXS3QH1Qm3ZWVU08ZzvAnyW5kO7Wn4dy7yM4t+engA8BVNW3gKuAiYA/u6puq6qNdEcpHky3Xh6S5G+SHAfcPsUypQXPPXhpON4BfA14/0DbVtqX7iQjwOKBaZsGhscHxsf54X/Hk+89XXSh+etV9UMPVElyLN0jV+dDgP+vqr49qYafnFTDi4GVwOOqaku6p+8t3Y33HVxv24Cxqro1yU8AzwReBbyA7v7gUq+4By8NQdtjPZ3ugrUJV9IdEgd4LrBoFxb9i0lG2nn5h9A9CORzwK+ke+QvSR6WZNk0yzkf+NkkByYZBV4E/MdO1LEB2Hdg/HPAr7cnZZHksduZbz/gxhbuT6Hb455qeYP+k+6LAe3Q/IPoPveU2qH/kar6OPD7dKcIpN4x4KXh+Stg8Gr6f6AL1W8CT2TX9q6vpgvnzwCvaoem30t3ePpr7cK0/8s0R++qe3zl6+keI/pN4IKq2pnHV54DPGLiIjvgj+m+sFyY5JI2PpUPA6uTXER37cC3Wj030107cPHki/uAvwNG2jwfA17WTmVsz6HAue10wYeAk3fic0kLhk+TkySph9yDlySphwx4SZJ6yICXJKmHDHhJknrIgJckqYcMeEmSesiAlySphwx4SZJ66P8HvudQiwOC+6QAAAAASUVORK5CYII=\n", 185 | "text/plain": [ 186 | "
" 187 | ] 188 | }, 189 | "metadata": { 190 | "needs_background": "light" 191 | }, 192 | "output_type": "display_data" 193 | } 194 | ], 195 | "source": [ 196 | "import matplotlib.pyplot as plt\n", 197 | "fig = plt.figure(figsize=(8,6))\n", 198 | "plt.plot(np.arange(len(lr.loss)), lr.loss)\n", 199 | "plt.title(\"Development of cost over training\")\n", 200 | "plt.xlabel(\"Number of iterations\")\n", 201 | "plt.ylabel(\"Cost\")\n", 202 | "plt.show()" 203 | ] 204 | }, 205 | { 206 | "cell_type": "code", 207 | "execution_count": null, 208 | "metadata": {}, 209 | "outputs": [], 210 | "source": [] 211 | } 212 | ], 213 | "metadata": { 214 | "kernelspec": { 215 | "display_name": "python3", 216 | "language": "python", 217 | "name": "python3" 218 | }, 219 | "language_info": { 220 | "codemirror_mode": { 221 | "name": "ipython", 222 | "version": 2 223 | }, 224 | "file_extension": ".py", 225 | "mimetype": "text/x-python", 226 | "name": "python", 227 | "nbconvert_exporter": "python", 228 | "pygments_lexer": "ipython2", 229 | "version": "2.7.16" 230 | } 231 | }, 232 | "nbformat": 4, 233 | "nbformat_minor": 2 234 | } 235 | -------------------------------------------------------------------------------- /Multinomial Naive Bayes.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 31, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np\n", 10 | " \n", 11 | "# class MultinomialNB:\n", 12 | " \n", 13 | "# def fit(self, X, y):\n", 14 | "# self.y_classes, y_counts = np.unique(y, return_counts=True)\n", 15 | "# self.x_classes = np.array([np.unique(x) for x in X.T])\n", 16 | "# self.phi_y = 1.0 * y_counts/y_counts.sum()\n", 17 | "# self.phi_x = self.mean_x(X, y)\n", 18 | "# return self\n", 19 | " \n", 20 | "# def mean_x(self, X, y):\n", 21 | "# return [[(X[:,j][y==k].reshape(-1,1) == self.x_classes[j]).mean(axis=0)\n", 22 | "# for j in range(len(self.x_classes))]\n", 23 | "# for k in self.y_classes]\n", 24 | " \n", 25 | "# def predict(self, X):\n", 26 | "# return np.apply_along_axis(lambda x: self.compute_probs(x), 1, X)\n", 27 | " \n", 28 | "# def compute_probs(self, x):\n", 29 | "# probs = np.array([self.compute_prob(x, y) for y in range(len(self.y_classes))])\n", 30 | "# return self.y_classes[np.argmax(probs)]\n", 31 | " \n", 32 | "# def compute_prob(self, x, y):\n", 33 | "# Pxy = 1\n", 34 | "# for j in range(len(x)):\n", 35 | "# i = list(self.x_classes[j]).index(x[j])\n", 36 | "# Pxy *= self.phi_x[y][j][i] # p(xj|y)\n", 37 | "# return Pxy * self.phi_y[y]\n", 38 | " \n", 39 | "# def evaluate(self, X, y):\n", 40 | "# return (self.predict(X) == y).mean()\n", 41 | "\n", 42 | "class MultinomialNB:\n", 43 | " \n", 44 | " def fit(self, X, y, ls=0.01):\n", 45 | " self.ls = ls\n", 46 | " self.y_classes, y_counts = np.unique(y, return_counts=True)\n", 47 | " self.x_classes = [np.unique(x) for x in X.T]\n", 48 | " self.phi_y = 1.0 * y_counts/y_counts.sum()\n", 49 | " self.phi_x = self.mean_X(X, y)\n", 50 | " self.c_x = self.count_x(X, y)\n", 51 | " return self\n", 52 | " \n", 53 | " def mean_X(self, X, y):\n", 54 | " return [[self.ls_mean_x(X, y, k, j) for j in range(len(self.x_classes))] for k in self.y_classes]\n", 55 | " \n", 56 | " def ls_mean_x(self, X, y, k, j):\n", 57 | " x_data = (X[:,j][y==k].reshape(-1,1) == self.x_classes[j])\n", 58 | " return (x_data.sum(axis=0) + self.ls ) / (len(x_data) + (len(self.x_classes) * self.ls))\n", 59 | " \n", 60 | " def get_mean_x(self, y, j):\n", 61 | " return 1 + self.ls / (self.c_x[y][j] + (len(self.x_classes) * self.ls))\n", 62 | " \n", 63 | " def count_x(self, X, y):\n", 64 | " return [[len(X[:,j][y==k].reshape(-1,1) == self.x_classes[j])\n", 65 | " for j in range(len(self.x_classes))]\n", 66 | " for k in self.y_classes]\n", 67 | "\n", 68 | " def predict(self, X):\n", 69 | " return np.apply_along_axis(lambda x: self.compute_probs(x), 1, X)\n", 70 | " \n", 71 | " def compute_probs(self, x):\n", 72 | " probs = np.array([self.compute_prob(x, y) for y in range(len(self.y_classes))])\n", 73 | " return self.y_classes[np.argmax(probs)]\n", 74 | " \n", 75 | " def compute_prob(self, x, y):\n", 76 | " Pxy = 1\n", 77 | " for j in range(len(x)):\n", 78 | " x_clas = self.x_classes[j]\n", 79 | " if x[j] in x_clas:\n", 80 | " i = list(x_clas).index(x[j])\n", 81 | " p_x_j_y = self.phi_x[y][j][i] # p(xj|y)\n", 82 | " Pxy *= p_x_j_y\n", 83 | " else:\n", 84 | " Pxy *= get_mean_x(y, j)\n", 85 | " return Pxy * self.phi_y[y]\n", 86 | " \n", 87 | " def evaluate(self, X, y):\n", 88 | " return (self.predict(X) == y).mean()" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": 32, 94 | "metadata": {}, 95 | "outputs": [ 96 | { 97 | "data": { 98 | "text/plain": [ 99 | "0.9666666666666667" 100 | ] 101 | }, 102 | "execution_count": 32, 103 | "metadata": {}, 104 | "output_type": "execute_result" 105 | } 106 | ], 107 | "source": [ 108 | "from sklearn import datasets\n", 109 | "from utils import accuracy_score\n", 110 | "iris = datasets.load_iris()\n", 111 | "X = iris.data \n", 112 | "y = iris.target\n", 113 | "nb = MultinomialNB().fit(X, y)\n", 114 | "nb.evaluate(X, y)" 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": 33, 120 | "metadata": {}, 121 | "outputs": [ 122 | { 123 | "data": { 124 | "text/plain": [ 125 | "array([0.33333333, 0.33333333, 0.33333333])" 126 | ] 127 | }, 128 | "execution_count": 33, 129 | "metadata": {}, 130 | "output_type": "execute_result" 131 | } 132 | ], 133 | "source": [ 134 | "nb.phi_y" 135 | ] 136 | }, 137 | { 138 | "cell_type": "code", 139 | "execution_count": 34, 140 | "metadata": {}, 141 | "outputs": [ 142 | { 143 | "data": { 144 | "text/plain": [ 145 | "[[array([0.02018385, 0.06015188, 0.02018385, 0.08013589, 0.04016787,\n", 146 | " 0.1001199 , 0.08013589, 0.16007194, 0.16007194, 0.06015188,\n", 147 | " 0.02018385, 0.1001199 , 0.04016787, 0.00019984, 0.04016787,\n", 148 | " 0.02018385, 0.00019984, 0.00019984, 0.00019984, 0.00019984,\n", 149 | " 0.00019984, 0.00019984, 0.00019984, 0.00019984, 0.00019984,\n", 150 | " 0.00019984, 0.00019984, 0.00019984, 0.00019984, 0.00019984,\n", 151 | " 0.00019984, 0.00019984, 0.00019984, 0.00019984, 0.00019984]),\n", 152 | " array([0.00019984, 0.00019984, 0.02018385, 0.00019984, 0.00019984,\n", 153 | " 0.00019984, 0.00019984, 0.00019984, 0.02018385, 0.12010392,\n", 154 | " 0.08013589, 0.1001199 , 0.04016787, 0.18005596, 0.12010392,\n", 155 | " 0.06015188, 0.06015188, 0.08013589, 0.04016787, 0.02018385,\n", 156 | " 0.02018385, 0.02018385, 0.02018385]),\n", 157 | " array([2.01838529e-02, 2.01838529e-02, 4.01678657e-02, 1.40087930e-01,\n", 158 | " 2.59992006e-01, 2.59992006e-01, 1.40087930e-01, 8.01358913e-02,\n", 159 | " 4.01678657e-02, 1.99840128e-04, 1.99840128e-04, 1.99840128e-04,\n", 160 | " 1.99840128e-04, 1.99840128e-04, 1.99840128e-04, 1.99840128e-04,\n", 161 | " 1.99840128e-04, 1.99840128e-04, 1.99840128e-04, 1.99840128e-04,\n", 162 | " 1.99840128e-04, 1.99840128e-04, 1.99840128e-04, 1.99840128e-04,\n", 163 | " 1.99840128e-04, 1.99840128e-04, 1.99840128e-04, 1.99840128e-04,\n", 164 | " 1.99840128e-04, 1.99840128e-04, 1.99840128e-04, 1.99840128e-04,\n", 165 | " 1.99840128e-04, 1.99840128e-04, 1.99840128e-04, 1.99840128e-04,\n", 166 | " 1.99840128e-04, 1.99840128e-04, 1.99840128e-04, 1.99840128e-04,\n", 167 | " 1.99840128e-04, 1.99840128e-04, 1.99840128e-04]),\n", 168 | " array([1.00119904e-01, 5.79736211e-01, 1.40087930e-01, 1.40087930e-01,\n", 169 | " 2.01838529e-02, 2.01838529e-02, 1.99840128e-04, 1.99840128e-04,\n", 170 | " 1.99840128e-04, 1.99840128e-04, 1.99840128e-04, 1.99840128e-04,\n", 171 | " 1.99840128e-04, 1.99840128e-04, 1.99840128e-04, 1.99840128e-04,\n", 172 | " 1.99840128e-04, 1.99840128e-04, 1.99840128e-04, 1.99840128e-04,\n", 173 | " 1.99840128e-04, 1.99840128e-04])],\n", 174 | " [array([0.00019984, 0.00019984, 0.00019984, 0.00019984, 0.00019984,\n", 175 | " 0.00019984, 0.02018385, 0.04016787, 0.02018385, 0.02018385,\n", 176 | " 0.00019984, 0.02018385, 0.1001199 , 0.1001199 , 0.1001199 ,\n", 177 | " 0.06015188, 0.04016787, 0.08013589, 0.08013589, 0.04016787,\n", 178 | " 0.06015188, 0.04016787, 0.02018385, 0.04016787, 0.06015188,\n", 179 | " 0.02018385, 0.02018385, 0.02018385, 0.00019984, 0.00019984,\n", 180 | " 0.00019984, 0.00019984, 0.00019984, 0.00019984, 0.00019984]),\n", 181 | " array([0.02018385, 0.04016787, 0.06015188, 0.06015188, 0.08013589,\n", 182 | " 0.06015188, 0.1001199 , 0.12010392, 0.14008793, 0.16007194,\n", 183 | " 0.06015188, 0.06015188, 0.02018385, 0.02018385, 0.00019984,\n", 184 | " 0.00019984, 0.00019984, 0.00019984, 0.00019984, 0.00019984,\n", 185 | " 0.00019984, 0.00019984, 0.00019984]),\n", 186 | " array([0.00019984, 0.00019984, 0.00019984, 0.00019984, 0.00019984,\n", 187 | " 0.00019984, 0.00019984, 0.00019984, 0.00019984, 0.02018385,\n", 188 | " 0.04016787, 0.04016787, 0.02018385, 0.02018385, 0.02018385,\n", 189 | " 0.06015188, 0.1001199 , 0.06015188, 0.08013589, 0.04016787,\n", 190 | " 0.08013589, 0.14008793, 0.06015188, 0.1001199 , 0.04016787,\n", 191 | " 0.04016787, 0.02018385, 0.02018385, 0.00019984, 0.00019984,\n", 192 | " 0.00019984, 0.00019984, 0.00019984, 0.00019984, 0.00019984,\n", 193 | " 0.00019984, 0.00019984, 0.00019984, 0.00019984, 0.00019984,\n", 194 | " 0.00019984, 0.00019984, 0.00019984]),\n", 195 | " array([1.99840128e-04, 1.99840128e-04, 1.99840128e-04, 1.99840128e-04,\n", 196 | " 1.99840128e-04, 1.99840128e-04, 1.40087930e-01, 6.01518785e-02,\n", 197 | " 1.00119904e-01, 2.59992006e-01, 1.40087930e-01, 2.00039968e-01,\n", 198 | " 6.01518785e-02, 2.01838529e-02, 2.01838529e-02, 1.99840128e-04,\n", 199 | " 1.99840128e-04, 1.99840128e-04, 1.99840128e-04, 1.99840128e-04,\n", 200 | " 1.99840128e-04, 1.99840128e-04])],\n", 201 | " [array([0.00019984, 0.00019984, 0.00019984, 0.00019984, 0.00019984,\n", 202 | " 0.00019984, 0.02018385, 0.00019984, 0.00019984, 0.00019984,\n", 203 | " 0.00019984, 0.00019984, 0.00019984, 0.02018385, 0.02018385,\n", 204 | " 0.06015188, 0.02018385, 0.04016787, 0.04016787, 0.04016787,\n", 205 | " 0.12010392, 0.1001199 , 0.08013589, 0.00019984, 0.1001199 ,\n", 206 | " 0.04016787, 0.06015188, 0.00019984, 0.02018385, 0.06015188,\n", 207 | " 0.02018385, 0.02018385, 0.02018385, 0.08013589, 0.02018385]),\n", 208 | " array([1.99840128e-04, 2.01838529e-02, 1.99840128e-04, 1.99840128e-04,\n", 209 | " 8.01358913e-02, 4.01678657e-02, 8.01358913e-02, 1.60071942e-01,\n", 210 | " 4.01678657e-02, 2.40007994e-01, 8.01358913e-02, 1.00119904e-01,\n", 211 | " 6.01518785e-02, 4.01678657e-02, 1.99840128e-04, 2.01838529e-02,\n", 212 | " 1.99840128e-04, 4.01678657e-02, 1.99840128e-04, 1.99840128e-04,\n", 213 | " 1.99840128e-04, 1.99840128e-04, 1.99840128e-04]),\n", 214 | " array([0.00019984, 0.00019984, 0.00019984, 0.00019984, 0.00019984,\n", 215 | " 0.00019984, 0.00019984, 0.00019984, 0.00019984, 0.00019984,\n", 216 | " 0.00019984, 0.00019984, 0.00019984, 0.00019984, 0.00019984,\n", 217 | " 0.00019984, 0.00019984, 0.00019984, 0.00019984, 0.00019984,\n", 218 | " 0.00019984, 0.02018385, 0.00019984, 0.00019984, 0.04016787,\n", 219 | " 0.06015188, 0.06015188, 0.14008793, 0.04016787, 0.04016787,\n", 220 | " 0.04016787, 0.06015188, 0.12010392, 0.06015188, 0.06015188,\n", 221 | " 0.04016787, 0.04016787, 0.06015188, 0.02018385, 0.02018385,\n", 222 | " 0.02018385, 0.04016787, 0.02018385]),\n", 223 | " array([1.99840128e-04, 1.99840128e-04, 1.99840128e-04, 1.99840128e-04,\n", 224 | " 1.99840128e-04, 1.99840128e-04, 1.99840128e-04, 1.99840128e-04,\n", 225 | " 1.99840128e-04, 1.99840128e-04, 2.01838529e-02, 4.01678657e-02,\n", 226 | " 2.01838529e-02, 2.01838529e-02, 2.20023981e-01, 1.00119904e-01,\n", 227 | " 1.20103917e-01, 1.20103917e-01, 6.01518785e-02, 1.60071942e-01,\n", 228 | " 6.01518785e-02, 6.01518785e-02])]]" 229 | ] 230 | }, 231 | "execution_count": 34, 232 | "metadata": {}, 233 | "output_type": "execute_result" 234 | } 235 | ], 236 | "source": [ 237 | "nb.phi_x" 238 | ] 239 | }, 240 | { 241 | "cell_type": "code", 242 | "execution_count": 28, 243 | "metadata": {}, 244 | "outputs": [ 245 | { 246 | "name": "stdout", 247 | "output_type": "stream", 248 | "text": [ 249 | "[1 1 1 2 2 2]\n" 250 | ] 251 | } 252 | ], 253 | "source": [ 254 | "X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])\n", 255 | "Y = np.array([1, 1, 1, 2, 2, 2])\n", 256 | "clf = MultinomialNB().fit(X, Y)\n", 257 | "print(clf.predict(X))" 258 | ] 259 | }, 260 | { 261 | "cell_type": "code", 262 | "execution_count": 30, 263 | "metadata": {}, 264 | "outputs": [ 265 | { 266 | "data": { 267 | "text/plain": [ 268 | "0.9755147468002225" 269 | ] 270 | }, 271 | "execution_count": 30, 272 | "metadata": {}, 273 | "output_type": "execute_result" 274 | } 275 | ], 276 | "source": [ 277 | "from sklearn import datasets\n", 278 | "digits = datasets.load_digits()\n", 279 | "X = digits.data\n", 280 | "y = digits.target\n", 281 | "MultinomialNB().fit(X, y).evaluate(X, y)" 282 | ] 283 | }, 284 | { 285 | "cell_type": "code", 286 | "execution_count": null, 287 | "metadata": {}, 288 | "outputs": [], 289 | "source": [] 290 | } 291 | ], 292 | "metadata": { 293 | "kernelspec": { 294 | "display_name": "python3", 295 | "language": "python", 296 | "name": "python3" 297 | }, 298 | "language_info": { 299 | "codemirror_mode": { 300 | "name": "ipython", 301 | "version": 2 302 | }, 303 | "file_extension": ".py", 304 | "mimetype": "text/x-python", 305 | "name": "python", 306 | "nbconvert_exporter": "python", 307 | "pygments_lexer": "ipython2", 308 | "version": "2.7.16" 309 | } 310 | }, 311 | "nbformat": 4, 312 | "nbformat_minor": 2 313 | } 314 | -------------------------------------------------------------------------------- /Naive Bayes Implementation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Naive Bayes Classifier Implementation using numpy\n", 8 | "\n", 9 | "Naive Bayes is anoother supervised machine laerning algorithm for classification problem. It makes a strong assumption about the data that **each feature is independent of the value of any other feature**. For example, a fruit may be considered to be an apple if it is red, round, and about 10 cm in diameter. A naive Bayes classifier considers each of these features to contribute independently to the probability that this fruit is an apple, regardless of any possible correlations between the color, roundness, and diameter features.\n", 10 | "\n", 11 | "In Naive bayes classifier what we are trying to find is the probability that a given data point belogs to a specific class, we are going to have prediction for all the class in our target.\n", 12 | "\n", 13 | "\n", 14 | "![title](figures/bayes-theorem.png)\n", 15 | "\n", 16 | "### Our motivation\n", 17 | "To gain better understand on how the algorithm works" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 3, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "from sklearn import datasets\n", 27 | "digits = datasets.load_digits()\n", 28 | "X = digits.data\n", 29 | "y = digits.target" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": null, 35 | "metadata": {}, 36 | "outputs": [], 37 | "source": [ 38 | "p(y|x) = p(x|y) * p(y)" 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": null, 44 | "metadata": {}, 45 | "outputs": [], 46 | "source": [ 47 | "class MultinomialNB:\n", 48 | " \n", 49 | " def fit(self, X, y):\n", 50 | " unique_y, y_counts = np.unique(y, return_counts=True)\n", 51 | " self.prob_y = y_counts / len(y)\n", 52 | " self.unique_y = unique_y\n", 53 | " self.unique_x = np.array([np.unique(x_t) for x_t in X.T])\n", 54 | " self.prob_x = self.get_prob_x(X, y)\n", 55 | " return self\n", 56 | " \n", 57 | " def get_prob_x(X, y):\n", 58 | " return []\n", 59 | " \n", 60 | " def predict(self, X):\n", 61 | " pass\n", 62 | " \n", 63 | " def score(self, X, y):\n", 64 | " " 65 | ] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": null, 70 | "metadata": {}, 71 | "outputs": [], 72 | "source": [ 73 | "model = MultinomialNB().fit(X, y)\n", 74 | "model.score(X, y)" 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "execution_count": 13, 80 | "metadata": {}, 81 | "outputs": [ 82 | { 83 | "data": { 84 | "text/plain": [ 85 | "array([[ 0., 0., 5.],\n", 86 | " [ 0., 0., 0.],\n", 87 | " [ 0., 0., 0.],\n", 88 | " [ 0., 0., 7.],\n", 89 | " [ 0., 0., 0.],\n", 90 | " [ 0., 0., 12.],\n", 91 | " [ 0., 0., 0.],\n", 92 | " [ 0., 0., 7.],\n", 93 | " [ 0., 0., 9.],\n", 94 | " [ 0., 0., 11.]])" 95 | ] 96 | }, 97 | "execution_count": 13, 98 | "metadata": {}, 99 | "output_type": "execute_result" 100 | } 101 | ], 102 | "source": [ 103 | "X[:10, :3]" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": 8, 109 | "metadata": {}, 110 | "outputs": [ 111 | { 112 | "data": { 113 | "text/plain": [ 114 | "array([0, 1])" 115 | ] 116 | }, 117 | "execution_count": 8, 118 | "metadata": {}, 119 | "output_type": "execute_result" 120 | } 121 | ], 122 | "source": [ 123 | "y[:2]" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": null, 129 | "metadata": {}, 130 | "outputs": [], 131 | "source": [] 132 | } 133 | ], 134 | "metadata": { 135 | "kernelspec": { 136 | "display_name": "python3", 137 | "language": "python", 138 | "name": "python3" 139 | }, 140 | "language_info": { 141 | "codemirror_mode": { 142 | "name": "ipython", 143 | "version": 2 144 | }, 145 | "file_extension": ".py", 146 | "mimetype": "text/x-python", 147 | "name": "python", 148 | "nbconvert_exporter": "python", 149 | "pygments_lexer": "ipython2", 150 | "version": "2.7.16" 151 | } 152 | }, 153 | "nbformat": 4, 154 | "nbformat_minor": 2 155 | } 156 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Basic Machine Learning Implementation with Python and Numpy 2 | 3 | This repository contains implementations of basic machine learning algorithms in Python and Numpy. All algorithms are implemented from scratch without using additional machine learning libraries. The intention of these notebooks is to provide a basic understanding of the algorithms and their underlying structure, not to provide the most efficient implementations. 4 | 5 | 1. [Linear Regression ](https://github.com/bamtak/machine-learning-implemetation-python/blob/master/Linear%20Regression%20Implementation%20from%20scratch.ipynb) 6 | 7 | 2. [Logistic Regression ](https://github.com/bamtak/machine-learning-implemetation-python/blob/master/Logistic%20Regression.ipynb) 8 | 9 | 3. [Multi class Logisic Regression](https://github.com/bamtak/machine-learning-implemetation-python/blob/master/Multi%20Class%20Logistic%20Regression.ipynb) 10 | 11 | 4. [Linear Regression with newton's method](https://github.com/bamtak/machine-learning-implemetation-python/blob/master/Linear%20Regression%20with%20Newtons%20Method.ipynb) 12 | 13 | 5. [Logistic Regression with newtons methods](https://github.com/bamtak/machine-learning-implemetation-python/blob/master/Logistic%20Regression%20with%20Newtons%20Method.ipynb) 14 | 15 | 6. [Multiclass Logistic Regression with newtons methods](https://github.com/bamtak/machine-learning-implemetation-python/blob/master/Multi%20Class%20Logistic%20Regression%20with%20Newtons%20Method.ipynb) 16 | 17 | 7. [Perceptron](https://github.com/bamtak/machine-learning-implemetation-python/blob/master/Perceptron.ipynb) 18 | 19 | 8. [Binary Naive Bayes](https://github.com/bamtak/machine-learning-implemetation-python/blob/master/Binary%20Naive%20Bayes.ipynb) 20 | 21 | 9. [Multinomial Naive Bayes](https://github.com/bamtak/machine-learning-implemetation-python/blob/master/Multinomial%20Naive%20Bayes.ipynb) 22 | 23 | 10. [Gaussian Naive Bayes](https://github.com/bamtak/machine-learning-implemetation-python/blob/master/Gaussian%20Naive%20Bayes.ipynb) 24 | 25 | 11. [Gaussian Discriminat Analyses](https://github.com/bamtak/machine-learning-implemetation-python/blob/master/Gaussian%20Discriminant%20Analyses.ipynb) 26 | 27 | 12. [KMeans](https://github.com/bamtak/machine-learning-implemetation-python/blob/master/KMeans.ipynb) 28 | 29 | 13. [Wrapper methods implementation - Forward and Backward](https://github.com/bamtak/machine-learning-implemetation-python/blob/master/Wrapper%20Method%20For%20Feature%20Selection%20-%20Forward%20and%20Backward%20.ipynb) 30 | 31 | 14. [Multiclass Gaussian Discriminat Analyses](https://github.com/bamtak/machine-learning-implemetation-python/blob/master/Multi%20Class%20Gaussian%20Discriminant%20Analyses.ipynb) 32 | -------------------------------------------------------------------------------- /README_files/15320635: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/README_files/15320635 -------------------------------------------------------------------------------- /README_files/1f334.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/README_files/1f334.png -------------------------------------------------------------------------------- /README_files/1f3af.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/README_files/1f3af.png -------------------------------------------------------------------------------- /README_files/1f3e0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/README_files/1f3e0.png -------------------------------------------------------------------------------- /README_files/1f912.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/README_files/1f912.png -------------------------------------------------------------------------------- /README_files/7457498: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/README_files/7457498 -------------------------------------------------------------------------------- /README_files/decision_tree_predictions.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/README_files/decision_tree_predictions.png -------------------------------------------------------------------------------- /README_files/image_preprocessing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/README_files/image_preprocessing.png -------------------------------------------------------------------------------- /README_files/octocat-spinner-128.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/README_files/octocat-spinner-128.gif -------------------------------------------------------------------------------- /README_files/search-key-slash.svg: -------------------------------------------------------------------------------- 1 | 3 | 4 | 5 | 6 | -------------------------------------------------------------------------------- /Wrapper Method For Feature Selection - Forward and Backward .ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 2, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "from sklearn.linear_model import LogisticRegression\n", 10 | "\n", 11 | "class FeatureSelection:\n", 12 | " \n", 13 | " def forward_feature_selection(self, X, y, n_features=2):\n", 14 | " results = {'best':[]}\n", 15 | " feature_list = [] #\n", 16 | " while len(feature_list) < n_features:\n", 17 | " best = None\n", 18 | " for i in range(X.shape[1]):\n", 19 | " if i not in feature_list: \n", 20 | " cur_model_result = train_model(X, y, feature_list + [i])\n", 21 | " best = get_best(best, cur_model_result)\n", 22 | " feature_list = best['features']\n", 23 | " results['best'].append(best)\n", 24 | " self.result = compute_best_features(results, n_features)\n", 25 | " return results\n", 26 | "\n", 27 | " def backward_feature_selection(self, X, y, n_features=2):\n", 28 | " results = {'best':[]}\n", 29 | " feature_list = range(X.shape[1])\n", 30 | " while len(feature_list) > 1:\n", 31 | " best = None\n", 32 | " for i in range(len(feature_list)):\n", 33 | " idx = feature_list[0:i] + feature_list[i+1:]\n", 34 | " best = get_best(best, train_model(X, y, idx))\n", 35 | " feature_list = best['features']\n", 36 | " results['best'].append(best)\n", 37 | " self.result = compute_best_features(results, n_features)\n", 38 | " return results\n", 39 | "\n", 40 | "def compute_best_features(results, n_features):\n", 41 | " best_result = None\n", 42 | " for result in results['best']:\n", 43 | " if len(result['features']) <= n_features:\n", 44 | " best_result = get_best(best_result, result)\n", 45 | " results['best_features'] = best_result['features']\n", 46 | " \n", 47 | "def train_model(X, y, idx):\n", 48 | " X_ = X[:, idx]\n", 49 | " model = LogisticRegression(random_state=0, solver='lbfgs',multi_class='multinomial').fit(X_, y)\n", 50 | " acc = accuracy_score(y, model.predict(X_))\n", 51 | " return {'score': acc, 'features': idx}\n", 52 | "\n", 53 | "def get_best(b1, b2):\n", 54 | " return b2 if b1 == None or b2['score'] > b1['score'] else b1" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": 3, 60 | "metadata": {}, 61 | "outputs": [ 62 | { 63 | "name": "stdout", 64 | "output_type": "stream", 65 | "text": [ 66 | "**********Forward*********\n", 67 | "{'best_features': [3, 2], 'best': [{'score': 0.96, 'features': [3]}, {'score': 0.9666666666666667, 'features': [3, 2]}]}\n", 68 | "\n", 69 | "\n", 70 | "**********Backward*********\n", 71 | "{'best_features': [2, 3], 'best': [{'score': 0.9733333333333334, 'features': [1, 2, 3]}, {'score': 0.9666666666666667, 'features': [2, 3]}, {'score': 0.96, 'features': [3]}]}\n" 72 | ] 73 | } 74 | ], 75 | "source": [ 76 | "from sklearn import datasets\n", 77 | "iris = datasets.load_iris()\n", 78 | "X = iris.data \n", 79 | "Y = iris.target\n", 80 | "print('**********Forward*********')\n", 81 | "print(FeatureSelection().forward_feature_selection(X, Y))\n", 82 | "print('\\n')\n", 83 | "print('**********Backward*********')\n", 84 | "print(FeatureSelection().backward_feature_selection(X, Y))" 85 | ] 86 | }, 87 | { 88 | "cell_type": "code", 89 | "execution_count": null, 90 | "metadata": {}, 91 | "outputs": [], 92 | "source": [] 93 | } 94 | ], 95 | "metadata": { 96 | "kernelspec": { 97 | "display_name": "python3", 98 | "language": "python", 99 | "name": "python3" 100 | }, 101 | "language_info": { 102 | "codemirror_mode": { 103 | "name": "ipython", 104 | "version": 2 105 | }, 106 | "file_extension": ".py", 107 | "mimetype": "text/x-python", 108 | "name": "python", 109 | "nbconvert_exporter": "python", 110 | "pygments_lexer": "ipython2", 111 | "version": "2.7.16" 112 | } 113 | }, 114 | "nbformat": 4, 115 | "nbformat_minor": 2 116 | } 117 | -------------------------------------------------------------------------------- /book_rec_knn.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 6, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "name": "stdout", 10 | "output_type": "stream", 11 | "text": [ 12 | "The Karate Kid\n", 13 | "A Brilliant Young Mind\n", 14 | "Finding Forrester\n" 15 | ] 16 | } 17 | ], 18 | "source": [ 19 | "from collections import Counter\n", 20 | "import math\n", 21 | "\n", 22 | "def knn(data, query, k, choice_fn):\n", 23 | " neighbor_distances_and_indices = []\n", 24 | " \n", 25 | " for index, example in enumerate(data):\n", 26 | " \n", 27 | " distance = euclidean_distance(example[:-1], query)\n", 28 | " \n", 29 | " neighbor_distances_and_indices.append((distance, index))\n", 30 | " \n", 31 | " sorted_neighbor_distances_and_indices = sorted(neighbor_distances_and_indices)\n", 32 | " \n", 33 | " k_nearest_distances_and_indices = sorted_neighbor_distances_and_indices[:k]\n", 34 | " \n", 35 | " k_nearest_labels = [data[i][1] for distance, i in k_nearest_distances_and_indices]\n", 36 | "\n", 37 | " return k_nearest_distances_and_indices , choice_fn(k_nearest_labels)\n", 38 | "\n", 39 | "def mean(labels):\n", 40 | " return sum(labels) / len(labels)\n", 41 | "\n", 42 | "def mode(labels):\n", 43 | " return Counter(labels).most_common(1)[0][0]\n", 44 | "\n", 45 | "def euclidean_distance(point1, point2):\n", 46 | " sum_squared_distance = 0\n", 47 | " for i in range(len(point1)):\n", 48 | " sum_squared_distance += math.pow(point1[i] - point2[i], 2)\n", 49 | " return math.sqrt(sum_squared_distance)\n", 50 | "def recommend_books(book_query, k_recommendations):\n", 51 | " raw_books_data = []\n", 52 | " with open('books_recommendation_data.csv', 'r') as md:\n", 53 | " \n", 54 | " next(md)\n", 55 | "\n", 56 | " \n", 57 | " for line in md.readlines():\n", 58 | " data_row = line.strip().split(',')\n", 59 | " raw_books_data.append(data_row)\n", 60 | "\n", 61 | " \n", 62 | " books_recommendation_data = []\n", 63 | " for row in raw_books_data:\n", 64 | " data_row = list(map(float, row[2:]))\n", 65 | " books_recommendation_data.append(data_row)\n", 66 | "\n", 67 | "\n", 68 | " recommendation_indices, _ = knn(\n", 69 | " books_recommendation_data, book_query, k=k_recommendations, choice_fn=lambda x: None\n", 70 | " )\n", 71 | "\n", 72 | " book_recommendations = []\n", 73 | " for _, index in recommendation_indices:\n", 74 | " book_recommendations.append(raw_books_data[index])\n", 75 | "\n", 76 | " return book_recommendations\n", 77 | "\n", 78 | "if __name__ == '__main__':\n", 79 | " the_post = [6.5, 0, 1, 0, 1, 0, 0, 0, 0] # feature vector for The Post\n", 80 | " # Rating,Biography,Drama,Thriller,Comedy,Crime,Mystery,History,Label\n", 81 | " recommended_books = recommend_books(book_query=the_post, k_recommendations=3)\n", 82 | "\n", 83 | " # Print recommended book titles\n", 84 | " for recommendation in recommended_books:\n", 85 | " print(recommendation[1])\n" 86 | ] 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": null, 91 | "metadata": {}, 92 | "outputs": [], 93 | "source": [] 94 | } 95 | ], 96 | "metadata": { 97 | "kernelspec": { 98 | "display_name": "Python 3", 99 | "language": "python", 100 | "name": "python3" 101 | }, 102 | "language_info": { 103 | "codemirror_mode": { 104 | "name": "ipython", 105 | "version": 3 106 | }, 107 | "file_extension": ".py", 108 | "mimetype": "text/x-python", 109 | "name": "python", 110 | "nbconvert_exporter": "python", 111 | "pygments_lexer": "ipython3", 112 | "version": "3.7.3" 113 | } 114 | }, 115 | "nbformat": 4, 116 | "nbformat_minor": 2 117 | } 118 | -------------------------------------------------------------------------------- /figures/MM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/figures/MM.png -------------------------------------------------------------------------------- /figures/bayes-theorem.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/figures/bayes-theorem.png -------------------------------------------------------------------------------- /figures/decision_tree.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/figures/decision_tree.png -------------------------------------------------------------------------------- /figures/decision_tree_predictions.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/figures/decision_tree_predictions.png -------------------------------------------------------------------------------- /figures/image_preprocessing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/figures/image_preprocessing.png -------------------------------------------------------------------------------- /figures/linear_regression.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/figures/linear_regression.jpg -------------------------------------------------------------------------------- /figures/logistic_regression.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/figures/logistic_regression.jpg -------------------------------------------------------------------------------- /figures/neural_net.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/figures/neural_net.png -------------------------------------------------------------------------------- /figures/perceptron_hyperplane.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/figures/perceptron_hyperplane.png -------------------------------------------------------------------------------- /figures/preprocessing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/figures/preprocessing.png -------------------------------------------------------------------------------- /figures/regression_tree.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/figures/regression_tree.png -------------------------------------------------------------------------------- /figures/softmax_regression.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/figures/softmax_regression.jpg -------------------------------------------------------------------------------- /gaussian_naive_bayes.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | class GaussianNB: 4 | 5 | def fit(self, X, y, epsilon = 1e-10): 6 | self.y_classes, y_counts = np.unique(y, return_counts=True) 7 | self.x_classes = np.array([np.unique(x) for x in X.T]) 8 | self.phi_y = 1.0 * y_counts/y_counts.sum() 9 | self.u = np.array([X[y==k].mean(axis=0) for k in self.y_classes]) 10 | self.var_x = np.array([X[y==k].var(axis=0) + epsilon for k in self.y_classes]) 11 | return self 12 | 13 | def predict(self, X): 14 | return np.apply_along_axis(lambda x: self.compute_probs(x), 1, X) 15 | 16 | def compute_probs(self, x): 17 | probs = np.array([self.compute_prob(x, y) for y in range(len(self.y_classes))]) 18 | return self.y_classes[np.argmax(probs)] 19 | 20 | def compute_prob(self, x, y): 21 | c = 1.0 /np.sqrt(2.0 * np.pi * (self.var_x[y])) 22 | return np.prod(c * np.exp(-1.0 * np.square(x - self.u[y]) / (2.0 * self.var_x[y]))) 23 | 24 | def evaluate(self, X, y): 25 | return (self.predict(X) == y).mean() -------------------------------------------------------------------------------- /gda.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | class GDABinaryClassifier: 4 | 5 | def fit(self, X, y): 6 | self.fi = y.mean() 7 | self.u = np.array([ X[y==k].mean(axis=0) for k in [0,1]]) 8 | X_u = X.copy() 9 | for k in [0,1]: X_u[y==k] -= self.u[k] 10 | self.E = X_u.T.dot(X_u) / len(y) 11 | self.invE = np.linalg.pinv(self.E) 12 | return self 13 | 14 | def predict(self, X): 15 | return np.argmax([self.compute_prob(X, i) for i in range(len(self.u))], axis=0) 16 | 17 | def compute_prob(self, X, i): 18 | u, phi = self.u[i], ((self.fi)**i * (1 - self.fi)**(1 - i)) 19 | return np.exp(-1.0 * np.sum((X-u).dot(self.invE)*(X-u), axis=1)) * phi 20 | 21 | def score(self, X, y): 22 | return (self.predict(X) == y).mean() -------------------------------------------------------------------------------- /kmeans.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | class KMeans: 3 | 4 | def __init__(self, n_clusters=4): 5 | self.K = n_clusters 6 | 7 | def fit(self, X): 8 | self.centroids = X[np.random.choice(len(X), self.K, replace=False)] 9 | self.intial_centroids = self.centroids 10 | self.prev_label, self.labels = None, np.zeros(len(X)) 11 | while not np.all(self.labels == self.prev_label) : 12 | self.prev_label = self.labels 13 | self.labels = self.predict(X) 14 | self.update_centroid(X) 15 | return self 16 | 17 | def predict(self, X): 18 | return np.apply_along_axis(self.compute_label, 1, X) 19 | 20 | def compute_label(self, x): 21 | return np.argmin(np.sqrt(np.sum((self.centroids - x)**2, axis=1))) 22 | 23 | def update_centroid(self, X): 24 | self.centroids = np.array([np.mean(X[self.labels == k], axis=0) for k in range(self.K)]) -------------------------------------------------------------------------------- /naive_bayes.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | class NaiveBayesBinaryClassifier: 4 | 5 | def fit(self, X, y): 6 | self.y_classes, y_counts = np.unique(y, return_counts=True) 7 | self.phi_y = 1.0 * y_counts/y_counts.sum() 8 | self.phi_x = [1.0 * X[y==k].mean(axis=0) for k in self.y_classes] 9 | return self 10 | 11 | def predict(self, X): 12 | return np.apply_along_axis(lambda x: self.compute_probs(x), 1, X) 13 | 14 | def compute_probs(self, x): 15 | probs = [self.compute_prob(x, y) for y in range(len(self.y_classes))] 16 | return self.y_classes[np.argmax(probs)] 17 | 18 | def compute_prob(self, x, y): 19 | res = 1 20 | for j in range(len(x)): 21 | Pxy = self.phi_x[y][j] # p(xj=1|y) 22 | res *= (Pxy**x[j])*((1-Pxy)**(1-x[j])) # p(xj=0|y) 23 | return res * self.phi_y[y] 24 | 25 | def evaluate(self, X, y): 26 | return (self.predict(X) == y).mean() -------------------------------------------------------------------------------- /spam filter.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 3, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "name": "stdout", 10 | "output_type": "stream", 11 | "text": [ 12 | " MAIL fil\n", 13 | "0 send send us send us us your password spam\n", 14 | "1 send us your review ham\n", 15 | "2 review your password ham\n", 16 | "3 review us spam\n", 17 | "4 send us your password spam\n", 18 | "5 send us your account spam\n", 19 | "{'send': 6, 'us': 7, 'your': 5, 'password': 3, 'review': 3, 'account': 1}\n", 20 | "{'spam': 4, 'ham': 2}\n", 21 | "Total words: 25\n", 22 | "{'spam': {'send': 0.75, 'us': 1.0, 'your': 0.75, 'password': 0.5, 'review': 0.25, 'account': 0.25}, 'ham': {'send': 0.5, 'us': 0.5, 'your': 1.0, 'review': 1.0, 'password': 0.5}}\n", 23 | "{'spam': 0.6666666666666666, 'ham': 0.3333333333333333}\n", 24 | "0.75\n", 25 | "0.3333333333333333\n" 26 | ] 27 | } 28 | ], 29 | "source": [ 30 | "import pandas as pd\n", 31 | "\n", 32 | "data = pd.read_csv(\"spam.csv\")\n", 33 | "print(data)\n", 34 | "\n", 35 | "bow = {}\n", 36 | "mails = data[\"MAIL\"]\n", 37 | "tot_word_count = 0\n", 38 | "\n", 39 | "for mail in mails:\n", 40 | " mail = mail.split()\n", 41 | " for word in mail:\n", 42 | " if word in bow:\n", 43 | " bow[word] += 1\n", 44 | " tot_word_count += 1\n", 45 | " else:\n", 46 | " bow[word] = 1\n", 47 | " tot_word_count += 1\n", 48 | " \n", 49 | "print(bow)\n", 50 | "\n", 51 | "fil = {'spam': 0, 'ham': 0}\n", 52 | "for fils in data[\"fil\"]:\n", 53 | " fil[fils] += 1\n", 54 | "print(fil)\n", 55 | "print(\"Total words: \", tot_word_count)\n", 56 | "\n", 57 | "prob = {'spam': {}, 'ham': {}}\n", 58 | "for index, row in data.iterrows():\n", 59 | " mail = row['MAIL']\n", 60 | " res = row['fil']\n", 61 | " done = []\n", 62 | " for word in mail.split():\n", 63 | " if word not in done:\n", 64 | " done.append(word)\n", 65 | " if word not in prob[res]:\n", 66 | " prob[res][word] = 1\n", 67 | " else:\n", 68 | " prob[res][word] += 1\n", 69 | "target_prob = {'spam': fil['spam']/len(data), 'ham': fil['ham']/len(data)}\n", 70 | "\n", 71 | "for res in prob:\n", 72 | " for word in prob[res]:\n", 73 | " prob[res][word] = prob[res][word] / fil[res]\n", 74 | "print(prob)\n", 75 | "print(target_prob)\n", 76 | "\n", 77 | "word = \"send\"\n", 78 | "\n", 79 | "p_ws = prob['spam'].get(word, 0)\n", 80 | "p_s = target_prob['spam']\n", 81 | "p_wh = prob['ham'].get(word, 0)\n", 82 | "p_h = target_prob['ham']\n", 83 | "\n", 84 | "p_sw = p_ws * p_s / (p_ws * p_s + p_wh * p_h)\n", 85 | "print(p_sw)\n", 86 | "\n", 87 | "word1 = \"review\"\n", 88 | "word2 = \"password\"\n", 89 | "\n", 90 | "p_w1s = prob['spam'].get(word1, 0)\n", 91 | "p_s = target_prob['spam']\n", 92 | "p_w1h = prob['ham'].get(word1, 0)\n", 93 | "p_h = target_prob['ham']\n", 94 | "\n", 95 | "p_w2s = prob['spam'].get(word2, 0)\n", 96 | "p_w2h = prob['ham'].get(word2, 0)\n", 97 | "\n", 98 | "p_sww = p_w1s * p_w2s * p_s / (p_w1s * p_w2s * p_s + p_w1h * p_w2h * p_h)\n", 99 | "print(p_sww)\n" 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "execution_count": null, 105 | "metadata": {}, 106 | "outputs": [], 107 | "source": [] 108 | } 109 | ], 110 | "metadata": { 111 | "kernelspec": { 112 | "display_name": "Python 3", 113 | "language": "python", 114 | "name": "python3" 115 | }, 116 | "language_info": { 117 | "codemirror_mode": { 118 | "name": "ipython", 119 | "version": 3 120 | }, 121 | "file_extension": ".py", 122 | "mimetype": "text/x-python", 123 | "name": "python", 124 | "nbconvert_exporter": "python", 125 | "pygments_lexer": "ipython3", 126 | "version": "3.7.3" 127 | } 128 | }, 129 | "nbformat": 4, 130 | "nbformat_minor": 2 131 | } 132 | -------------------------------------------------------------------------------- /svm.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 15, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# Muskan Pandey" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 10, 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "import matplotlib.pyplot as plt\n", 19 | "from matplotlib import style\n", 20 | "import numpy as np\n", 21 | "import pandas as pd\n", 22 | "style.use('ggplot')" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": 11, 28 | "metadata": {}, 29 | "outputs": [], 30 | "source": [ 31 | "class SVM(object):\n", 32 | " def __init__(self,visualization=True):\n", 33 | " self.visualization = visualization\n", 34 | " self.colors = {1:'r',-1:'b'}\n", 35 | " if self.visualization:\n", 36 | " self.fig = plt.figure()\n", 37 | " self.ax = self.fig.add_subplot(1,1,1)\n", 38 | " \n", 39 | " def fit(self,data):\n", 40 | " #train with data\n", 41 | " self.data = data\n", 42 | " # { |\\w\\|:{w,b}}\n", 43 | " opt_dict = {}\n", 44 | " \n", 45 | " transforms = [[1,1],[-1,1],[-1,-1],[1,-1]]\n", 46 | " \n", 47 | " all_data = np.array([])\n", 48 | " for yi in self.data:\n", 49 | " all_data = np.append(all_data,self.data[yi])\n", 50 | " \n", 51 | " self.max_feature_value = max(all_data) \n", 52 | " self.min_feature_value = min(all_data)\n", 53 | " all_data = None\n", 54 | " \n", 55 | " #with smaller steps our margins and db will be more precise\n", 56 | " step_sizes = [self.max_feature_value * 0.1,\n", 57 | " self.max_feature_value * 0.01,\n", 58 | " #point of expense\n", 59 | " self.max_feature_value * 0.001,]\n", 60 | " \n", 61 | " #extremly expensise\n", 62 | " b_range_multiple = 5\n", 63 | " #we dont need to take as small step as w\n", 64 | " b_multiple = 5\n", 65 | " \n", 66 | " latest_optimum = self.max_feature_value*10\n", 67 | " \n", 68 | " \"\"\"\n", 69 | " objective is to satisfy yi(x.w)+b>=1 for all training dataset such that ||w|| is minimum\n", 70 | " for this we will start with random w, and try to satisfy it with making b bigger and bigger\n", 71 | " \"\"\"\n", 72 | " #making step smaller and smaller to get precise value\n", 73 | " for step in step_sizes:\n", 74 | " w = np.array([latest_optimum,latest_optimum])\n", 75 | " \n", 76 | " #we can do this because convex\n", 77 | " optimized = False\n", 78 | " while not optimized:\n", 79 | " for b in np.arange(-1*self.max_feature_value*b_range_multiple,\n", 80 | " self.max_feature_value*b_range_multiple,\n", 81 | " step*b_multiple):\n", 82 | " for transformation in transforms:\n", 83 | " w_t = w*transformation\n", 84 | " found_option = True\n", 85 | " \n", 86 | " #weakest link in SVM fundamentally\n", 87 | " #SMO attempts to fix this a bit\n", 88 | " # ti(xi.w+b) >=1\n", 89 | " for i in self.data:\n", 90 | " for xi in self.data[i]:\n", 91 | " yi=i\n", 92 | " if not yi*(np.dot(w_t,xi)+b)>=1:\n", 93 | " found_option=False\n", 94 | " if found_option:\n", 95 | " \"\"\"\n", 96 | " all points in dataset satisfy y(w.x)+b>=1 for this cuurent w_t, b\n", 97 | " then put w,b in dict with ||w|| as key\n", 98 | " \"\"\"\n", 99 | " opt_dict[np.linalg.norm(w_t)]=[w_t,b]\n", 100 | " \n", 101 | " #after w[0] or w[1]<0 then values of w starts repeating itself because of transformation\n", 102 | " #Think about it, it is easy\n", 103 | " #print(w,len(opt_dict)) Try printing to understand\n", 104 | " if w[0]<0:\n", 105 | " optimized=True\n", 106 | " print(\"optimized a step\")\n", 107 | " else:\n", 108 | " w = w-step\n", 109 | " \n", 110 | " # sorting ||w|| to put the smallest ||w|| at poition 0 \n", 111 | " norms = sorted([n for n in opt_dict])\n", 112 | " #optimal values of w,b\n", 113 | " opt_choice = opt_dict[norms[0]]\n", 114 | "\n", 115 | " self.w=opt_choice[0]\n", 116 | " self.b=opt_choice[1]\n", 117 | " \n", 118 | " #start with new latest_optimum (initial values for w)\n", 119 | " latest_optimum = opt_choice[0][0]+step*2\n", 120 | " \n", 121 | " def predict(self,features):\n", 122 | " #sign(x.w+b)\n", 123 | " classification = np.sign(np.dot(np.array(features),self.w)+self.b)\n", 124 | " if classification!=0 and self.visualization:\n", 125 | " self.ax.scatter(features[0],features[1],s=200,marker='*',c=self.colors[classification])\n", 126 | " return (classification,np.dot(np.array(features),self.w)+self.b)\n", 127 | " \n", 128 | " def visualize(self):\n", 129 | " [[self.ax.scatter(x[0],x[1],s=100,c=self.colors[i]) for x in data_dict[i]] for i in data_dict]\n", 130 | " \n", 131 | " # hyperplane = x.w+b (actually its a line)\n", 132 | " # v = x0.w0+x1.w1+b -> x1 = (v-w[0].x[0]-b)/w1\n", 133 | " #psv = 1 psv line -> x.w+b = 1a small value of b we will increase it later\n", 134 | " #nsv = -1 nsv line -> x.w+b = -1\n", 135 | " # dec = 0 db line -> x.w+b = 0\n", 136 | " def hyperplane(x,w,b,v):\n", 137 | " #returns a x2 value on line when given x1\n", 138 | " return (-w[0]*x-b+v)/w[1]\n", 139 | " \n", 140 | " hyp_x_min= self.min_feature_value*0.9\n", 141 | " hyp_x_max = self.max_feature_value*1.1\n", 142 | " \n", 143 | " # (w.x+b)=1\n", 144 | " # positive support vector hyperplane\n", 145 | " pav1 = hyperplane(hyp_x_min,self.w,self.b,1)\n", 146 | " pav2 = hyperplane(hyp_x_max,self.w,self.b,1)\n", 147 | " self.ax.plot([hyp_x_min,hyp_x_max],[pav1,pav2],'k')\n", 148 | " \n", 149 | " # (w.x+b)=-1\n", 150 | " # negative support vector hyperplane\n", 151 | " nav1 = hyperplane(hyp_x_min,self.w,self.b,-1)\n", 152 | " nav2 = hyperplane(hyp_x_max,self.w,self.b,-1)\n", 153 | " self.ax.plot([hyp_x_min,hyp_x_max],[nav1,nav2],'k')\n", 154 | " \n", 155 | " # (w.x+b)=0\n", 156 | " # db support vector hyperplane\n", 157 | " db1 = hyperplane(hyp_x_min,self.w,self.b,0)\n", 158 | " db2 = hyperplane(hyp_x_max,self.w,self.b,0)\n", 159 | " self.ax.plot([hyp_x_min,hyp_x_max],[db1,db2],'y--')" 160 | ] 161 | }, 162 | { 163 | "cell_type": "code", 164 | "execution_count": 12, 165 | "metadata": {}, 166 | "outputs": [], 167 | "source": [ 168 | "#defining a basic data\n", 169 | "data_dict = {-1:np.array([[1,7],[2,8],[3,8]]),1:np.array([[5,1],[6,-1],[7,3]])}" 170 | ] 171 | }, 172 | { 173 | "cell_type": "code", 174 | "execution_count": 13, 175 | "metadata": {}, 176 | "outputs": [ 177 | { 178 | "name": "stdout", 179 | "output_type": "stream", 180 | "text": [ 181 | "optimized a step\n", 182 | "optimized a step\n", 183 | "optimized a step\n" 184 | ] 185 | }, 186 | { 187 | "data": { 188 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXhUZbrv/W+NqUpVpkqYwiSDqKiAyCRDmDKRZJWoNA6N3ajHAWjZ2LYtdjs2uje7d7u7X3vrq+cch916TrezrpUEApFZQGZBJgEBkTAlIaEyVCVVtc4faCnNIJhhVVL357q8LpKq1LofK/XLU0+t9dwmXdd1hBBCtHtmowsQQgjROiTwhRAiRkjgCyFEjJDAF0KIGCGBL4QQMUICXwghYoTV6AJ+TFlZmdElXJS0tDTKy8uNLsMQMvbYG3usjhuif+zp6ennvU1m+EIIESMk8IUQIkZI4AshRIyQwBdCiBghgS+EEDFCAl8IIWKEBL4QQsQICXwhhIgS9fX1FBYW8uKLL7bI40f9hVdCCNGe+f1+li5diqZpLF68mLq6OtLT07n33nux2+3NeiwJfCGEaGWBQIDly5ejaRqLFi2ipqaGlJQUbrrpJrxeLzfccAMWi6XZjyuBL4QQraChoYFVq1ahqiolJSWcOnWK5ORkFEVBURRGjhyJzWZr0Rok8IUQooUEg0FWr16NqqosWLCAqqoqEhMTycnJQVEUxowZ0+zLNhcigS+EEM0oFAqxZs0aNE2juLiYyspKXC5XJOTHjh1LXFycIbVJ4AshRBOFw2HWrVuHqqoUFxdz4sQJnE4nWVlZeL1exo0bh9PpNLpMCXwhhPgpwuEwGzduRNM0CgsLOXbsGA6Hg4kTJ6IoCpmZmVER8j8kgS+EEBdJ13XWrVvHm2++SWFhIWVlZcTFxTF+/Hi8Xi+ZmZm4XC6jyzwvCXwhhLgAXdfZtm0bmqahaRqHDh3CZrMxduxY5s6dS3Z2NgkJCUaXeVEk8IUQ4p/ous6OHTtQVZXCwkIOHDiA1WolIyODJ598klGjRpGUlGR0mZdMAl8IIb61e/duVFVF0zT27duHxWJh1KhRzJo1i9zcXDweT9S3OLyQiw78l156iU2bNpGUlMTzzz8PwJtvvsnGjRuxWq106tSJmTNnnnP9atasWTgcDsxmMxaLhfnz5zffCIQQogn27t2LpmmoqsqXX36J2WxmxIgR3HvvveTl5ZGammp0ic3mogN/3Lhx5ObmnrGpz4ABA7jjjjuwWCy89dZbfPjhh0ybNu2cP//UU0+RmJjY9IqFEKKJ9u/fH5nJ79y5E5PJxLBhw3juuefIy8ujY8eORpfYIi468Pv378/x48fP+N7AgQMj/+7Xrx9r165tvsqEEKIZff3115EPXrdt2wbAkCFDeOaZZ8jPz6dLly4GV9jymm0Nf8mSJYwcOfK8tz/33HMAZGVlkZmZed77lZaWUlpaCsD8+fNJS0trrhJblNVqbTO1NjcZe+yNva2M++uvv+b999/nvffeY8OGDQAMHTqUf//3f+fmm2+mR48el/yYbWXs59Isgf/BBx9gsVgYM2bMOW+fN28eHo+H6upqnn32WdLT0+nfv/8575uZmXnGH4S28uFIW/4gp6lk7LE39mge95EjRygqKkJVVTZu3AjAtddey+9//3sKCgrOCPmfMoZoHjtAenr6eW9rcuAvW7aMjRs38uSTT2Iymc55H4/HA0BSUhJDhw5l79695w18IYS4VMePH6eoqAhN01i3bh26rtO/f38effRRFEWhV69eRpcYFZoU+Fu2bOHjjz/mmWeeOe9mQH6/H13XcTqd+P1+tm7dypQpU5pyWCGEoKKiIjKTX7t2Lbquc8UVV/Dwww+jKAp9+/Y1usSoc9GB/5e//IUdO3bg8/l44IEHmDp1Kh9++CHBYJB58+YBcPnll3PfffdRWVnJK6+8wmOPPUZ1dTV/+tOfgNO7yI0ePZpBgwa1zGiEEO1aZWUlCxcuRFVVVq9eTSgUok+fPsyZMwdFUbjiiiuMLjGqmXRd140u4kLKysqMLuGiRPu6XkuSscfe2Ftz3FVVVZSUlKBpGitXriQYDHLZZZehKAper5errrrqvMvJLSHan/MWXcMXQojm5vP5IiG/fPlyGhsb6d69O/fddx9er5drrrmmVUO+vZDAF0JEhdraWhYvXoyqqixbtoxAIEB6ejp33303iqIwaNAgCfkmksAXQhimrq6OTz75BFVVWbJkCX6/n86dOzNt2jQUReH666/HbDYbXWa7IYEvhGhV9fX1LF26FE3TWLx4MfX19XTo0IHbbrsNr9fL0KFDJeRbiAS+EKLFBQIBli9fjqZplJSUUFtbi8fj4ZZbbsHr9TJixAgsFovRZbZ7EvhCiBbR0NDAypUrUVWVkpISfD4fycnJ3HjjjSiKwsiRI7FaJYJak/zfFkI0m8bGRlavXo2qqixcuJCqqioSExOZNGkSXq+X0aNHY7PZjC4zZkngCyGaJBgMsmbNGjRNo7i4mJMnT+J2u8nOzsbr9ZKRkXHeK/FF65LAF0JcslAoxJo1a1BVleLiYsrLy4mPjycrKwuv18u4ceNwOBxGlyn+iQS+EOKihMNhNm7ciKqqLFiwgCNHjuBwOMjMzERRFCZOnIjT6TS6THEBEvhCiPPSdZ1NmzahaRqFhYUcOXKEuLg4cnNzycnJITMz85xtTUV0ksAXQpxB13W2bt0a6Q71zTffYLfbGTt2LL/73e/IysqiV69eUb2fjDg3CXwhBLqus3379kjIHzx4EKvVSkZGBg8//DA5OTkkJSUZXaZoIgl8IWKUruvs3r070sz7q6++wmKxMHr0aB588EFyc3NJSUkxukzRjCTwhYgxe/bsQdM0VFVlz549mM1mbrjhBu6//34mTZpEamqq0SWKFiKBL0QM+Oqrr1BVlcLCQnbu3InJZGL48OFMnz6d/Px8OnToYHSJohVI4AvRTh08eDCyJv/FF18AMHToUP7whz+Qn59P586dDa5QtLZLCvyXXnqJTZs2kZSUxPPPPw9ATU0Nf/7znzlx4gQdOnTgoYcewu12n/Wzy5Yt44MPPgDg5ptvZty4cU2vXghxhm+++YbCwkJUVeXzzz8H4LrrruOpp54iPz+frl27GlyhMNIlBf64cePIzc3lxRdfjHzvo48+4tprr2Xy5Ml89NFHfPTRR0ybNu2Mn6upqeG9995j/vz5AMydO5chQ4ac8w+DEOLSlJWVRZp5b9q0CYABAwbw+OOPU1BQQPfu3Q2uUESLS9p0un///meF9Pr16xk7diwAY8eOZf369Wf93JYtWxgwYAButxu3282AAQPYsmVLE8oWIrYdO3aM1157jZtuuomhQ4fy9NNPEwgEmDt3Lp9++ikLFixgxowZEvbiDE1ew6+uro6cupWcnEx1dfVZ96msrDzjk3+Px0NlZWVTDy1ETCkvL6eoqAhN01i7di26rnPllVfym9/8BkVR6Nu3r9EliijXrB/amkymJvecLC0tpbS0FID58+eTlpbWHKW1OKvV2mZqbW4y9pYbe3l5OR9//DHvvvsuy5cvJxwOc8UVV/D73/+eKVOmcNVVV7XYsS9EnvO2OfYmB35SUhInT54kJSWFkydPkpiYeNZ9PB4PO3bsiHxdWVlJ//79z/l4mZmZZGZmRr5uK5dvp6WltZlam5uMvXnHXlVVxcKFC9E0jZUrVxIKhbjsssv41a9+hdfr5corr4xMrIz6/y7PefSOPT09/by3NTnwhwwZwvLly5k8eTLLly9n6NChZ91n0KBB/P3vf6empgaAzz//nDvuuKOphxai3Th16hQlJSVomsaKFStobGykR48ePPDAA3i9Xq6++uomv3sW4pIC/y9/+Qs7duzA5/PxwAMPMHXqVCZPnsyf//xnlixZEjktE2Dfvn0sXryYBx54ALfbzS233MJjjz0GwJQpU+QMHRHzampqWLx4MaqqsmzZMhoaGujatSv33HMPiqIwcOBACXnRrEy6rutGF3EhZWVlRpdwUaL9bV5LkrFf/Njr6uooLS1F0zSWLFmC3++nc+fOFBQUoCgKgwcPxmy+pJPnDCHPefSOvUWXdIQQF1ZfX8/SpUtRVZXS0lLq6+vp2LEjt99+O16vlyFDhrSJkBetQ9eDmEwtE80S+EK0gEAgwLJly9A0jUWLFlFbW0tqaipTpkzB6/UyfPhwLBaL0WWKKKHrQerqVuLzadTUlNCz50Jstua/hkICX4hm0tDQwIoVK9A0jZKSEnw+H8nJyUyePJmCggJGjhyJ1SovOXGargcJBo9js6UTDtdRVnY3JlMcbncOuh5skWPKb58QTdDY2MiyZctQVZWFCxdSXV1NUlISeXl5eL1eRo0ahc1mM7pMESV0PUR9/dpvZ/LF2Gw96dFDw2JJpFu394mLuxqzOa7Fji+BL8QlCgaDrFmzBk3TWLhwIRUVFbjdbnJycvB6vWRkZGC3240uU0SZqqo3qKj4/wiFjmMyOXG5MklIuDFyu9M5uMVrkMAX4iKEQiE+++wzNE2juLiY8vJyXC4XiqKQnZ3N2LFjcTgcRpcpooSuh/H7N+HzaXg8s7FaUzGZnDidQ0hIUHC5MjGb41u9Lgl8Ic4jHA6zYcMGVFWlqKiI48eP43Q6yczMRFEUJkyYQPfu3aP6FD3RenRdx+/fQk2Nhs+nEQyWYTLZcbnGYbWOJynpVpKSbjW0Rgl8IX5A13U2bdoU6Q519OhRHA4HEyZMQFEUMjMziY9v/ZmZiE66rqPrtZjNboLBwxw6VADYcLnGkpb2KC5XNhbL2dvNGEUCX8Q8Xdf5/PPPI92hDh8+jN1uZ9y4cTz++ONkZWXJleEiQtd1AoEd+HwqPp+G3X4lXbu+is3WjfT0V3E6R2CxJBtd5jlJ4IuYpOs627dvj4T8wYMHsdlsZGRk8Mgjj5CTk3POjQBFbKuqepNDh16nvn43YCY+fhRud07kdrc717jiLoIEvogZuq6za9cuVFVF0zT279+PxWJhzJgxzJ49m9zcXJKTo3NmJozR0LAXn6+IlJT7MZsdhELl2GydSUiYjtudh9XatrZJlsAXP0koBMXFDt59N55g0IrV6mHq1Dry8vy09C4BPzx2XZ2J+Hj9gsfes2dPJOT37NmD2Wxm5MiRzJgxg0mTJuHxeFq2YNGmNDQcwOdTqanRCAROb+vudA4jPv4GPJ45dOjQoc1+UC+BLy5ZebmZ6dM97NhhJRD4LmEdrFpl5+WXg7zxRiVpaeFWPDZnHXvfvn2R5Zpdu3ZhMpkYMWIEd911F/n5+W22gYVoGboewmSyEAhs5+DBbAAcjuvp0OFp3O4CbLYuAG1+91LZLbOZRPsOes0lHAavN43Nm89/YdF11zWgquXNPtP/8WN/RZcu/xeP5x9s374dgGHDhqEoCvn5+XTq1Kl5CyJ2nvd/1h7G3dh4mJqaQnw+jbi4a+jUaT66rlNd/d+4XFnYbF3P+XPRPnbZLVM0m+JiBzt2XPjXZscOKwsXOsjL87fCsQ8C73z73waOHAGncwhPP/00+fn5F/zlF7Gpuvptqqv/L37/BgDi4q4hLu50Bz6TyURy8nQDq2tZEvjikrzzTvwZSynnEgiY+cc/nM0e+N8f+xvgXeBt4LNvbx0C/AfwM3r16si9955s1mOLtisYPEFt7SckJt6KyWTC799EOFxLaupvSUhQsNt7G11iq5HAF5ekru7i1jDr65t3PefYsWPs2/c28D7w6bffvQ74N2Aq8P2Ltr4+0KzHFm1PMFhBTU0xPp9Gff0aIExc3DU4HNfQseM8TKbY3OtIAl9ckvj4i/vIx+ls+oe2J06coKioiMLCQtauXcvpj5uuBeZxOuT7tdixRdtVX7+eQ4duAULYbL3xeGaTkKAQF3clQMyGPTRD4JeVlfHnP/858vXx48eZOnUq+fn5ke9t376dP/7xj3Ts2BGA4cOHM2XKlKYeWhhg6tQ6Vq2yX3BZJy4uzG231f+kx6+srKS4uBhN01i9ejXhcJjLL7+cX//61yQk3MK//dvwFju2aHtCoWpqakrw+TSczqGkps4mLm4AHs+vcLvziIuT5u8/1OTAT09P5z/+4z+A05tN3X///QwbNuys+1111VXMnTu3qYcTBsvL8/Pyy8ELnqXTv3+Q3NyLX78/efIkJSUlqKrKqlWrCIVC9OrViwcffBCv18sVV1yByWQiHIaPP27eY4u2yefTOHXqferqlqPrDVit3XC5xgJgNseRlvZbgyuMTs26pLNt2zY6d+5Mhw4dmvNhRRQxm+GNNyrPeS58XFyY/v1Pnwv/Y6dknjp1KhLyK1eupLGxkZ49ezJjxgwUReHqq8+emTXXsUXbEw7XUl+/DpdrPACnTn1EIPAFycnTcbsVHI7rZCZ/EZr1PPyXXnqJ3r17k5t75n4S27dv5/nnnyc1NZWUlBTuvPNOunc/d7/G0tJSSktLAZg/fz4NDQ3NVV6LslqtBIMt05bsXEIh+OgjE3/7m4X6enA64Ze/DDF5st4qgRcO//D4JpxOnV/+MsSNN57/+D6fj8LCQt577z0WLVpEQ0MDPXv25JZbbmHKlCkMHjz4ol60Pzx2XR3Ex/Ojx24prf28R4vWGHcoVMfJkwupqHiXkycXEA7XM3jwLhyOXjQ2VmK1JmMytf5f92h/zi/UfKfZAj8YDHL//ffz/PPPn7UfSV1dHWazGYfDwaZNm3jjjTd44YUXLupx5cKrs53vatMfznJb6krXc7nQ2Ovq6li8eDGaprFkyRICgQCdO3dGURQURbnokI9W0X4RTktp6XHX1q6grOwedL0OiyUNtzufhAQFp3MYJpOxzd+j/TlvlQuvNm/eTK9evc65+dQP9w8fPHgwr776KqdOnZLdCH+CcBimT/eccx07EDCzebOd6dM9LXKl68Wqr69nyZIlqKpKaWkpfr+fjh078vOf/xyv18v111+PWdZdxLfC4QB1dSvw+VRcrgwSE3+Gw3ENiYk3kZDgxekcgckkJxQ2h2b7v/jpp58yatSoc95WVVVFUlISJpOJvXv3Eg6HSUhIaK5DxxQjr3S9EL/fz7Jly9A0jUWLFlFXV0dqaipTp07F6/UybNgwLBZjZ2YiutTWLv12k7KFhMOnMJuTiYu7GgCLxUOnTn80uML2p1kC3+/3s3XrVu67777I9xYtWgRAdnY2a9euZdGiRVgsFux2O3PmzGnTb+ONZOSVrv+soaGB5cuXs3jxYlRVxefzkZKSwk033YSiKNxwww1YrTIzE6fpepBAYBcOxzUAVFT8Jw0Ne3G7c3C7FVyuMTF9jnxraJZXo8Ph4LXXXjvje9nZ2ZF/5+bmnvVBrvhpjLrS9TuNjY2sWrUKVVUpKSmhurqa5ORk8vPz8Xq9jBw5EpvN1iLHFm2Proeor1/77Uy+mHC4ht69P8diSaRLlxexWDphNscZXWbMkOlXG9OaV7p+JxgMsnr1ajRNo7i4mKqqKhISEsjJycHr9XLTTTdx6tSpZjueaB9qa5dw9OivCYVOYDI5cbuzcLu9mEynA95m62FwhbFHAr+NaekrXb8TCoVYu3ZtJOQrKipwuVzk5OSgKApjx44lLu70C/dCp4GJ2KDrYfz+jfh8Gi5XJi5XBjZbD5zOYSQkKLhcmZjNTqPLjHkS+G1MS1zp+p1wOMz69etRVZWioiJOnDiB0+kkKysLRVEYP348Tqe8aMVpuq7j92+hpkbF5yskGCzDZIrDZuuGy5WB3d6X9PT/aXSZ4gck8NuY5r7aNBwOs2nTpkjIHz16FIfDwYQJE/B6vUycOPGM02pFbNN1nWDwCHC6Y9iRIzMIBo/ico0lLW0uLlc2FoucgRetJPDboLS0MKpazoIFDt5+20l9vRmn8/QyTm7uj/eU1XWdLVu2RFoAlpWVYbfbGT9+PE888QRZWVm4XK7WGYyIerqu09CwA59Pw+fTCIWq6dz5G0wmE+np/xObrScWS5LRZYqLIIHfRpnNkJ/vJz//4pZudF3niy++iIT8119/jc1mY+zYsTz66KNkZ2fLhXDiLDU1izlxYh6NjfsAC/Hxo/B4vOh6CACHY4CxBYpLIoHfjum6zs6dO1FVFU3TOHDgAFarlTFjxjBnzhxycnLOeWW0iF0NDXu//eA1C4fjGszmBKzWTqSk3IvbnYfVmgrw7amUPmOLFZdMAr8d+vLLLyMhv3fvXsxmM6NGjWLWrFnk5ubi8XiMLlFEkYaG/d8u16g0NOwETJjNiTgc1xAfP4L4+HeNLlE0Ewn8dmLv3r2R5Zrdu3djMpkYMWIE99xzD3l5eaSlpRldoogi4XAdZnM8ut7A11/nEQ6fwuEYQocOfyAhIR+rtbPRJYoWIIHfhh04cCAyk9+xYwcmk4lhw4bx7LPPkp+fH+kwJgRAY+NhfD6NmppCwuEaevZcislkp0uXF7Hbr8Bm62p0iaKFSeC3MYcOHYrM5Ldu3QrA9ddfzzPPPEN+fj5dunQxuEIRbWpqSqmsfAG/fyMAcXEDSEycCoQAKy7XBEPrE61HAr8NOHz4MIWFhWiaxubNmwEYNGgQTzzxBIqi0LWrzMzE94LB49TUFONyZWGzdSUcrkXX60lNfZSEBAW7vZfRJQqDSOBHqaNHj1JUVISqqmzYsAGAa6+9lt/97ncoikKPHrIPifheMFhBTU0RPp9Gff1aIEzHjhaSk+8kIcFLYuKNRpcoooAEfhQ5ceIERUVFaJrGZ599hq7rXHXVVfz2t79FURR69+5tdIkiiuh6GJPJTDhcw/79Q9H1ADZbHzyefyEhQSEu7goA2YpcREjgG6yiooLi4mI0TWPNmjWEw2H69evHww8/jKIo9O3b1+gSRRQJhaqpqVmIz6cBIbp1+ztms5uOHf8Vh2MAdvtVEvDivCTwDXDy5EkWLlyIqqp8+umnhEIhevfuzezZs/F6vVxxxRVGlyiiTG3tcqqqXqO2djnQiNXancTEG9F1HZPJRFLSbUaXKNoACfxWUl1dTUlJCZqmsWLFCoLBID179mTGjBl4vV769+8vMzMREQ7XUlNTiss1HoslkYaGLwkEdpCScjdut4LDMUh+X8Qla7bAnzVrFg6HA7PZjMViYf78+Wfcrus6r7/+Ops3byYuLo6ZM2e2+zVpn8/HokWL0DSN5cuX09DQQLdu3bjvvvtQFIVrr71WXrQiIhyup7a2FJ9PpbZ2Cbrup3Pnv5KYeDNJSb8gOfkeTCZp/i5+umad4T/11FPn3YBr8+bNHD16lBdeeIE9e/bwv//3/+Zf//Vfm/PwUaG2tpbS0lJUVWXp0qUEAgG6dOnC9OnTURSF6667TkJenCUYPM7+/SPR9Xoslg4kJd2O263gdA4FkDaAolm02pLOhg0byMjIwGQy0a9fP2prazl58iQpKSmtVUKLqa+v54MPPuCtt97ik08+we/306lTJ6ZNm4aiKFx//fWYL3aDetHuhcMB6uqW4/NpmM1OOnX6I1ZrRzyemTidw3E6R2AyWYwuU7RDzRr4zz33HABZWVlkZmaecVtlZeUZ+7mkpqZSWVnZZgPf7/ezdOlSNE1j8eLF1NXVkZaWxq233orX62XYsGES8uIMdXVrOHXqbWpqSgiHT2E2J5OYeEvk9tTUXxtYnYgFzRb48+bNw+PxUF1dzbPPPkt6ejr9+/e/5McpLS2ltLQUgPnz50fVpl+BQIDFixfz/vvvo2kaPp+PtLQ07rjjDm699VZGjRqFxRJ7MzOr1RpVz1NrutDYdT1IdfUyEhPHYjbbOHBgDbW1i0hNnUxa2hSSkiZgNttaueLmIc958489FAqxatUq3nvvPb755hs+/PDDZj9GswX+d1vuJiUlMXToUPbu3XtG4Hs8HsrLyyNfV1RUnHOb3szMzDPeHfzwZ4zQ0NDAqlWrUFWVkpISTp06RXJyMgUFBXi9XkaOHBn5BTC6VqPI2L8fu66HqK9fg8+n4vMVEw6fpGvX/4vLNRan8x569ZqF2RxHKASVldUGVt408pw3z9jD4TAbNmyItBg9fvw4TqeTzMxMjhw5gs126ROC9PT0897WLIHv9/vRdR2n04nf72fr1q1MmTLljPsMGTKEhQsXMmrUKPbs2UN8fHzULucEg0FWr16NqqosWLCAqqoqEhMTycnJwev1Mnr0aOz28zcRF7GpoeEAhw7dSChUjskUj9udTUKCF6dzOIC0ARTA6TMWv+sjXVhYeEYfaUVRyMzMbLE+0s0S+NXV1fzpT38CTr8tGT16NIMGDWLRokUAZGdnc91117Fp0yZmz56N3W5n5syZzXHoZhMKhVizZg2aplFcXExlZSVut5vs7GwURWHs2LHExcmZEuI0XQ/j929k//7FNDS4SE39F2y27rhcmbhc43G5JmI2O40uU0QJXdf5/PPPIzvdHj58GLvdzrhx43j88cfJysrC7Xa3eB0mXdf1Fj9KE5SVlbXYY4fDYdatW4eqqhQXF3PixAni4+PJyspCURTGjRuH03lxL1p5ixsbY/f7t3Lq1AfU1BQSDB7BbHaQkHArnTq1v1OMLySWnvN/drFj13Wd7du3R0L+4MGD2Gw2MjIyUBSFnJycFukj3eJLOm1JOBxm48aNaJpGYWEhx44dw+FwMHHiRLxeLxMnTrzokBftn67rNDTswG4/fSV0VdWr+Hwq8fFjSUv7HT173s7JkwGjyxRRQtd1du3aFWlMtH//fiwWC2PGjGH27Nnk5uYa2kc6JgJf13U2b94cCfmysjLi4uIYP348Xq+XzMxMXC6X0WWKKKHrOoHAdmpqNHy+QhobD9Cz52Li4vqTmvooHTr8IbIeb7EkABL4sW7Pnj2RkN+zZw9ms5mRI0cyY8YMJk2aFDV9pNtt4Ou6zrZt2yJvpw4dOoTNZmPcuHHMnTuX7OxsEhISjC5TRJlAYBdlZffS2PgVYCE+fjQez4NYrd0AsNnO/3ZZxJZ9+/ZF8mXXrl2RPtJ33XUX+fn5UXnaarsL/Pr6ev7yl79QWFjIgQMHsFqtZGRk8Otf/5qcnBySkuRMCfG9hoa9+HwqVms3kpKmYrN1x2brSUrK/SQk5GGxRMfMTNMC0wYAAB9jSURBVESHgwcP8tprr/GPf/yD7du3AzB06FDmzZtHfn4+nTp1MrjCC2t3gR8XF8fHH39Mr169+NWvfkVOTk7UvJ0S0aGh4St8Pg2fT6OhYSdgIilpGklJUzGbXXTr9pbRJYoo8s0330Rm8p9//jkAgwcP5qmnnqKgoOCCH5JGm3YX+GazmeXLl8splOIMweBxrNaOABw//jh1dctxOIbSocM8EhLysFo7G1yhiCZlZWUUFhaiqmqkj/TAgQN54oknuPPOO9vsZ37tLvABCXsBQGPj4chMPhDYSu/eG7BaO9Ghw5OYzQnYbNL8XXzv2LFjkT7S69evB+Dqq6/mscceo6CggMsuuwxo26ektsvAF7HN7/+C48cfw+/fBEBc3EDS0n6HyWT/9usrjSxPRJHv+kgXFhaydu3aSB/pRx55BEVR6NOnj9ElNisJfNHmBYPH8PmKsdt74XKNw2JJRdcDpKXNxe1WsNsvM7pEEUUqKysjfaRXr15NOBymb9++PPTQQyiKQr9+/YwuscVI4Is2KRgsp6amGJ9Ppb5+LaCTlHQnLtc4bLYu9Oy5yOgSRRQ5efIkJSUlqKrKqlWrCIVC9OrViwcffBBFUbjyyitjojGRBL5oM8Lh+sj+NIcPTyMQ2Ibd3heP5yESEhTi4trvzExculOnTkVCfuXKlTQ2NtKjRw9mzJiBoihcffXVMRHyPySBL6JaKFRFTU0JPp+G37+R3r03YjbH06HDU1gsSdjtV8XcizZmhUI4iouJf/ddTHV16PHx1E2dij8vD75tNlRTUxPpI71s2TIaGhro2rUr99xzD16vlwEDBsT074sEvohKfv82Kir+g9raFUAjNlsPkpJ+ga4HgHji428wukTRiszl5XimT8e6YwfmwPdbWdhXraL6xRd55+c/5+Nly1iyZAmBQIDOnTvzy1/+EkVRGDx4cEyH/A9J4IuoEA7XUFNTit3eG4djAGAiENhFSso9JCR4iYuL7ZlZTAuH8Uyfjv3b8+EB6oAFwNuBAIVbt1K/dSsdO3bk5z//OYqiMGTIEGkxeg4S+MIw4XAdtbWl+HwatbVL0HU/ycn34HAMIC7uanr1+kxCXuAoLsa6Ywd+YCHwDqACtUAHYDrwM5uNa+bNo7GgwLhC2wAJfNGqdD2MyWRG13UOHsymsXE/FktHkpLuICFBweEYAiBBL4DTLUbXvvQSHwUCfAycAlKBO4BbgbF8G2KNjdS/8w4nJfAvSAJftLhwOEBd3XJ8PhW/fyuXXbYMk8lMWtqjWCxpOJ3DMJlir/m7OLfGxsYz+khXV1eTDEwBpgITgHN1ejXX17dqnW2RBL5oMX7/F1RV/S9qakoIh32YzSkkJOQRDtdgsSSSkKAYXaKIEt/1kf6uxWhVVRUJCQnk5ORwx5495H/+OT/WRTosjYt+VJMDv7y8nBdffJGqqipMJhOZmZnk5eWdcZ/t27fzxz/+kY4dT29eNXz48LOanIu2T9cbqav7FJutF3Z7T0Kh49TULMbtzichQSE+fhQm07nmZiIWhUIh1q5dGwn5iooKXC4X2dnZeL1eMjIycDgcOAoLsc6eDYHzN5oJx8VRf9ttrVh929TkwLdYLNx555307t2b+vp65s6dy4ABA+jWrdsZ97vqqquYO3duUw8nooyuB6mqWsKxY2/h8xUTDp/E4/kX0tJ+S3x8Bn36bInsYSNEOBxm/fr1qKpKUVERJ06cwOl0kpmZidfrZfz48We1GPXn5RF8+eUzztL5Z8H+/fHn5rZ0+W1ekwM/JSWFlJQUAJxOJ127dqWysvKswBftj66HOHBgDI2NX2MyuXC7s0lI8BIfnwGAySQrhuJ0yG/atCkS8kePHsXhcDBhwgQURSEzM5P4+PjzP4DZTOUbb5zzPPxwXBzB/v2pfOONyMVX4vya9RV5/Phx9u/fT9++fc+67csvv+SRRx4hJSWFO++8k+7du5/zMUpLSyktLQVg/vz5Udkm7FysVmubqfWn0PUwPt8aysvfw+/fR//+KgCNjbOJj+9JQkIWFkvsraG29+f9fH5s3Lqus2HDBt577z3ef/99Dh06hN1uJycnh5/97Gfk5eVdWovRtDRYvZrQRx/B3/4GdXUQH0/ol7+EG2/E04ph35afc5Ou63pzPJDf7+epp57i5ptvZvjw4WfcVldXh9lsxuFwsGnTJt544w1eeOGFi3rcsrKy5iivxbXlPbIvJBDYTXX136mpKSQYPILJ5MDlmkDnzn/FbHYA7XfsFyNWx36uceu6zhdffBHpDvX1119js9nIyMjA6/WSnZ1NYmKiQRU3n2h/zi/UgatZZvjBYJDnn3+eMWPGnBX2wBlv1wYPHsyrr77KqVOn2sWT397ouk4gsBWbrTsWiwe/fwvV1f9NfPx40tIex+3OxGx2G12miBK6rrNz505UVUXTtEgf6TFjxjBnzhxycnJITk42ukzxrSYHvq7rvPzyy3Tt2pWC81z0UFVVRVJSEiaTib179xIOhy/t7ZxoUadDfjs+n0ZNjUZj40E6dPhDZFsDt3sSFov8cRbf27lzJ//93/+Npmns3bsXs9nMqFGjmDlzJpMmTZI+0lGqyYG/e/duVqxYQY8ePXjkkUcAuP322yNvebKzs1m7di2LFi3CYrFgt9uZM2eOXEkZJcJhP19/nUNDw17ASnz8GDyef8HtzgH4djvi2FubF2fbu3dvZLlm9+7dmEwmRowYwT333ENeXl6bXdeOJc22ht9SZA2/eQUCX+LzaYRCFXTq9K8AnDjxDDZbXxISJmGxXPrMrK2MvSW097EfOHAgslyzY8cOAIYNG8btt9/OuHHjItfWxJJof85bfA1fRLeGhoP4fB/i82k0NOwCTMTHj0HXQ5hMFjp0eMroEkUUOXToUGQmv3XrVgCuv/56nn76aQoKCujSpUvUh544Nwn8dqqh4QBWayfMZic1NSoVFf+B0zmMDh2eJSEhH6s19mZm4vwOHz5MYWEhmqax+dsLnAYNGsQTTzxBQUGBXFfTTkjgtyONjd/g82n4fCqBwFa6dPn/SUjwkpT0cxISpmCzdTG6RBFFjh49SlFREaqqsmHDBgCuueYafve731FQUEDPnj0NrlA0Nwn8diAUquLw4Wn4/adnZnFxg0hLewKncxgAFosHi2xGKYATJ05QVFSEpml89tln6LrOVVddxW9/+1sURaF3795GlyhakAR+GxQMHsXnK0LX/Xg8szCbk7Bau5CWNgm3uwC7XWZm4nsVFRUUFxejaRpr1qwhHA5z+eWX8+tf/xpFUbj88suNLlG0Egn8NiIYPEFNTRE+n0Z9/WeAjtN5AykpMzGZTKSn/y+jSxSt4SIaeQOcPHmShQsXoqoqn376KaFQiN69ezN79mwUReHKK680cBDCKBL4USwUqsRsTsZkMlNZ+Veqql7Fbr+c1NRf43YrxMXJzCyWXKiRd/Dll9n/wgss2LABTdNYsWIFwWCQnj17MmPGDLxeL/3795frX2KcBH6UCYVOUlOzEJ9Po65uFd26vUN8/AhSUv4HSUl3EBcnM7OYdI5G3nC65Z8aCPDO5s2UZGTQoOt069aNe++9F6/Xy7XXXishLyIk8KNEMHiMY8d+Q23tSqARm60nHs8MbLauANhsPYwtUBjqu0beADVAIfA2sAAIAN2AWWYzub/9LVfPmiUhL85JAt8goZCP2trFgE5i4i1YLB6CwROkpPwPEhK8xMXJzEz8wN//zgeBAG8DRUA90AW4n9N9Xm8AzKEQ9evWcfJXvzKwUBHNJPBbUThcR23tYnw+jdraJeh6AKdzBImJt2Ay2ejZc6HRJYoo4vf7Wbp0KZqmUbpiBbVAR+Au4FZgFPDPZ9tKI29xIRL4LSwc9kf2jT927Lf4fB9isXQiKWkaCQkKDsf1BlcookkgEGD58uVomsaiRYuoqakhJSWF2zp35o6yMsZydsj/kDTyFhcigd8CwmE/dXXL8PlUamoW07PnQuz2PqSk3E9S0jSczqGYTHIllDitoaGBVatWoaoqJSUlnDp1iuTkZBRFQVEURo4cSUJJCcmzZ59xds4/k0be4sdI4DejxsbDlJfPp7Z2EeFwDWZzComJN/HdnMzhuNbYAkXUCAaDrF69GlVVWbBgAVVVVSQmJpKTk4OiKIwZMwa7/fvm79LIWzQHCfwm0PVG6upWAlbS0m7GbHZRV7cKt1shIUEhPn4kJpPN6DJFlAiFQqxZswZN0yguLqayshKXyxUJ+bFjxxIXF3fuH5ZG3qIZSOBfIl0PUle3+tvuUMWEw1XEx2fQs+fNWCzJ9O69EZNJXnTitHA4zLp161BVleLiYk6cOIHT6SQrKwuv18u4ceNwXuS6ezgtjXJVxbFgAc6338ZcX0/Y6aT+tttOz+wl7MWPkMC/CLquR06RLCu7l9raRZhMLtzunG9n8mMj95WwF+FwmI0bN6JpGoWFhRw7dgyHw8HEiRNRFIXMzMyLDvmzmM348/Px5+c3b9EiJkjgn4euh6mvX4/Pp1JbW0LPnouwWDwkJ08nMXEqLte4b9v/CXF6UrB58+ZIyJeVlREXF8f48ePxer1kZmbicrmMLlPEuGYJ/C1btvD6668TDoeZOHEikydPPuP2xsZG/uu//ouvvvqKhIQE5syZE7Wt0RobD3Py5Cv4fEWEQkcxmRy4XBMJhXxYLB5crrE//iAiJui6zqZNm3jzzTfRNI1Dhw5hs9kYO3Ysc+fOJTs7m4SEBKPLFCKiyYEfDod59dVXefzxx0lNTeWxxx5jyJAhZ3TIWbJkCS6Xi7/+9a98+umn/J//83946KGHmnroZqHrOn7/FkwmGw7HNeh6kOrqt4iPH09Cghe3OxOzWWZm4jRd19mxYweqqlJYWMiBAwewWq1kZGTw0EMPkZubS1JSktFlCnFOTQ78vXv30rlzZzp16gTAyJEjWb9+/RmBv2HDBn72s58BMGLECF577bUz1sVbm67rBAJf4POp+HwaweAh3O580tP/J3Z7T/r0+QKzOd6Q2kR02r17d6SZ9759+7BYLIwaNYpHH32U0aNH4/FcevN3IVpbkwO/srKS1NTUyNepqans2bPnvPexWCzEx8fj8/lITEw86/FKS0spLS0FYP78+aSlpTW1xLPs2OGlqqoEk8lKUtIE0tKexOPxYrUm/+THtFqtLVJrW9Bex75r1y7ef/993n33XXbu3InZbI7M5CdPnkyHDh2wWq0Eg0GjS2117fU5vxhteexR96FtZmYmmZmZka/Ly8ub9HiBwO5vtxpeRvfuH2Ay2XE48ujUKRO3OxeL5fTMrKoqCPz0Y6WlpTW51raqPY19//79kZn8zp07MZlMDBs2jOeee468vLwzPnsqLy9vV2O/FLE6boj+saenp5/3tiYHvsfjoaKiIvJ1RUXFWW9vv7tPamoqoVCIurq6Fv0wq7HxCKdO/QOfT6OhYTdgxukcQTB4AputK4mJU1rs2KLt+frrr9E0DU3T2LZtGwBDhgzhmWeeIT8/ny5dpPm7aB+aHPh9+vThyJEjHD9+HI/Hw+rVq5k9e/YZ97n++utZtmwZ/fr1Y+3atVx99dUtun4fDB6mouJ5nM5hdOz4HG53HlZrdJ4VJIxx+PDhSMhv2bIFgOuuu44nn3ySgoICunbtanCFQjS/Jge+xWLh7rvv5rnnniMcDjN+/Hi6d+/O22+/TZ8+fRgyZAgTJkzgv/7rv3jwwQdxu93MmTOnOWo/L4djML16rcdmk5mZ+N6RI0coKipCVVU2btwIwLXXXsvvf/97CgoK6NFDmsyI9s2k67pudBEXUlZWZnQJFyXa1/VaUjSP/fjx4xQVFaFpGuvWrUPXdfr37x/ZibJXr15NevxoHnuz+0EDdXswSIPVes4G6u1dtD/nLbqGL0S0qaioiMzk165di67rXHHFFTz88MMoikLfvn2NLrHNOVcDdQffN1CvfOMNwm30zJVYIoEv2oXKykoWLlyIqqqsXr2aUChEnz59mDNnDoqicMUVVxhdYtt1ngbqAOZAAPvmzXimT6dcVWNqpt8WSeCLNquqqoqSkhI0TWPlypUEg0Euu+wyZs6cidfr5aqrrpK+wM3ghw3Uz8e6YweOhQtPL++IqCWBL9oUn88XCfnly5fT2NhI9+7due+++/B6vVxzzTUS8s0s/p13LthpC07P9J3/+IcEfpSTwBdRr7a2lsWLF6OqKsuWLSMQCJCens7dd9+NoigMGjRIQr4FmerqLup+0kA9+kngi6hUV1fHJ598gqqqLFmyBL/fT+fOnZk2bRqKonD99ddjlvXiVqHHX9y+UtJAPfpJ4IuoUV9fz9KlS9E0jcWLF1NfX0+HDh247bbb8Hq9DB06VELeAHVTp2JftUoaqLcDEvjCUIFAgOXLl6NpGiUlJdTW1uLxeLjlllvwer2MGDECi8VidJkxTRqotx8S+KLVNTQ0sHLlSlRVpaSkBJ/PR3JyMjfeeCOKojBy5EisVvnVjBrSQL3dkFeVaBWNjY2sXr0aVVVZuHAhVVVVJCYmMmnSJLxeL6NHj8ZmsxldpjiPf26gHhcMErBapYF6GyOBL1pMMBhkzZo1aJpGcXExJ0+exO12k52djdfrJSMjg7i4OKPLFBfrBw3U09LSOBnF2wuIc5PAF80qFAqxbt06VFWluLiY8vJy4uPjycrKwuv1Mm7cOBwOh9FlChGTJPBFk4XDYdavX4+qqhQVFXHs2DEcDgeZmZkoisLEiRNxyil7QhhOAl/8JLqus2nTJjRNY8GCBXzzzTfExcUxYcIEFEUhMzMTl0uavwsRTSTwxUXTdZ2tW7dGGod888032O12srKyePTRR8nKymrRTmZCiKaRwBcXpOs627dvj4T8wYMHsVqtZGRk8PDDD5OTk0OfPn2ien9wIcRpEvjiLLqus3v37kgz76+++gqLxcLo0aN58MEHyc3NJSUlxegyhRCXqEmB/+abb7Jx40asViudOnVi5syZ51y3nTVrFg6HA7PZjMViYf78+U05rGghe/bsQdM0VFVlz549mM1mbrjhBu6//34mTZpEamqq0SUKIZqgSYE/YMAA7rjjDiwWC2+99RYffvgh06ZNO+d9n3rqKRITE5tyONECvvrqK1RVpbCwkJ07d2IymRg+fDjTp08nPz+fDh06GF2iEKKZNCnwBw4cGPl3v379WLt2bZMLEi3v4MGDkTX5L774AoChQ4fyhz/8gfz8fDp37mxwhUKIltBsTcznz5/PyJEjycjIOOu2WbNm4Xa7AcjKyiIzM/O8j1NaWkppaWnkMRsaGpqjvBZntVoJBoNGl3FeBw8e5P333+e9995j48aNAAwbNowpU6Zw8803071795/82NE+9pYUq2OP1XFD9I/dbref97YfDfx58+ZRVVV11vdvu+02hg4dCsAHH3zAvn37+M1vfnPORhSVlZV4PB6qq6t59tlnueuuu+jfv/9FFV9WVnZR9zNaNHayLysrizTz3rRpE3B6Gc7r9VJQUNCkkP+haBx7a4nVscfquCH6x56enn7e2350SeeJJ5644O3Lli1j48aNPPnkk+ftOuTxeABISkpi6NCh7N2796IDX1yaY8eOUVRUhKZprFu3DoCrr76auXPnoigKl112mbEFCiEM06Q1/C1btvDxxx/zzDPPnHcTLL/fj67rOJ1O/H4/W7duZcqUKU05rPgn5eXlkZBfu3Ytuq5z5ZVX8pvf/AZFUejbt6/RJQohokCTAv/VV18lGAwyb948AC6//HLuu+8+KisreeWVV3jssceorq7mT3/6E3B6Y63Ro0czaNCgplce4yorK1mwYAGapvHpp58SDofp27cvDz30EIqi0K9fP6NLFEJEmWb70LalyBr+96qqqigpKUFVVVauXEkoFOKyyy7D6/Xi9Xq58sorDWnmHe1rmi0pVsceq+OG6B97k9bwhbFOnTrFokWLUFWVFStW0NjYSI8ePXjggQfwer1cffXVhoS8EKLtkcCPQjU1NSxevBhN01i6dCkNDQ107dqVe+65B0VRGDhwoIS8EOKSSeBHibq6OkpLS9E0jSVLluD3++ncuTO/+MUvUBSFwYMHY5Y2ckKIJpDAN1B9fT1Lly5FVVVKS0upr6+nY8eO3H777Xi9XoYMGSIhL4RoNhL4rSwQCLBs2TI0TWPRokXU1taSmprKlClT8Hq9DB8+HIvFYnSZQoh2SAK/FTQ0NLBixQo0TaOkpASfz0dycjKTJ0+moKCAkSNHYrXKUyGEaFmSMi2ksbGRTz/9FFVVWbhwIdXV1SQlJZGXl4fX62XUqFHYbDajyxRCxBAJ/GYUDAZZs2YNmqZRXFzMyZMncbvd5OTk4PV6ycjIuODGRkII0ZIk8JsoFArx2WefsXjxYj744APKy8txuVxkZ2ejKApjx47F4XAYXaYQQkjg/xThcJgNGzagqipFRUUcP36c+Ph4Jk6ciKIoTJgwAafTaXSZQghxBgn8i6TrOps2bYp0hzp69CgOh4MJEyagKAq33nor9fX1RpcphBDnJYF/Abqu8/nnn0e6Qx0+fBi73c64ceN4/PHHycrKijR2cblcEvhCiKgmgf9PdF1n+/btkZA/ePAgNpuNjIwMHnnkEXJycqQ3rxCiTZLA53TI79q1C1VV0TSN/fv3Y7FYGDNmDLNnzyY3N5fk5GSjyxRCiCaJ6cDfs2dPJOT37NmD2Wxm5MiRzJgxg0mTJkU6dQkhRHsQc4G/b9++yHLNrl27MJlMjBgxgrvuuov8/HzS0tKMLlEIIVpETAT+wYMHIzP57du3AzBs2DDmzZtHfn4+nTp1MrhCIYRoee028L/55pvITP7zzz8HYPDgwTz99NPk5+dfsCuMEEK0R00K/HfeeYdPPvkkctbK7bffzuDBg8+635YtW3j99dcJh8NMnDiRyZMnN+WwF1RXV8fUqVPZvHkzAAMHDuSJJ56goKCAbt26tdhxhRAi2jV5hp+fn4/X6z3v7eFwmFdffZXHH3+c1NRUHnvsMYYMGdJi4RsfH0+vXr3Izc1FURR69uzZIscRQoi2psWXdPbu3Uvnzp0j6+QjR45k/fr1LTrb/utf/9pijy2EEG1VkwO/pKSEFStW0Lt3b37xi19Erjz9TmVlJampqZGvU1NT2bNnz3kfr7S0lNLSUgDmz5/fZs6asVqtbabW5iZjj72xx+q4oW2P/UcDf968eVRVVZ31/dtuu43s7GymTJkCwNtvv83f/vY3Zs6c2aSCMjMzyczMjHxdXl7epMdrLWlpaW2m1uYmY4+9scfquCH6x36hE1J+NPCfeOKJizrIxIkT+fd///ezvu/xeKioqIh8XVFRIRc0CSGEAZrUIfvkyZORf69bt47u3bufdZ8+ffpw5MgRjh8/TjAYZPXq1QwZMqQphxVCCPETNGkN/6233uLAgQOYTCY6dOjAfffdB5xet3/llVd47LHHsFgs3H333Tz33HOEw2HGjx9/zj8MQgghWpZJ13Xd6CIupKyszOgSLkq0r+u1JBl77I09VscN0T/2C63hN2lJRwghRNshgS+EEDEi6pd0hBBCNA+Z4TeTuXPnGl2CYWTssSdWxw1te+wS+EIIESMk8IUQIkZI4DeTH24HEWtk7LEnVscNbXvs8qGtEELECJnhCyFEjJDAF0KIGNFue9q2ptZs4RgtysvLefHFF6mqqsJkMpGZmUleXp7RZbWqcDjM3Llz8Xg8bfpUvUtVW1vLyy+/zKFDhzCZTMyYMYN+/foZXVarKCwsZMmSJZhMJrp3787MmTOx2+1Gl3XRJPCbqLVbOEYLi8XCnXfeSe/evamvr2fu3LkMGDCg3Y/7h4qLi+natSv19fVGl9KqXn/9dQYNGsTDDz9MMBgkEAgYXVKrqKysZMGCBfz5z3/Gbrfzn//5n6xevZpx48YZXdpFkyWdJvphC0er1Rpp4djepaSk0Lt3bwCcTiddu3alsrLS4KpaT0VFBZs2bWLixIlGl9Kq6urq2LlzJxMmTABOd39yuVwGV9V6wuEwDQ0NhEIhGhoaSElJMbqkSyIz/Ca61BaO7dHx48fZv38/ffv2NbqUVvPGG28wbdq0mJvdHz9+nMTERF566SUOHjxI7969mT59Og6Hw+jSWpzH40FRFGbMmIHdbmfgwIEMHDjQ6LIuiczwRZP4/X6ef/55pk+fTnx8vNHltIqNGzeSlJQUeYcTS0KhEPv37yc7O5s//vGPxMXF8dFHHxldVquoqalh/fr1vPjii7zyyiv4/X5WrFhhdFmXRAK/iWK5hWMwGOT5559nzJgxDB8+3OhyWs3u3bvZsGEDs2bN4i9/+QtffPEFL7zwgtFltYrU1FRSU1O5/PLLARgxYgT79+83uKrWsW3bNjp27EhiYiJWq5Xhw4fz5ZdfGl3WJZElnSb6YQtHj8fD6tWrmT17ttFltThd13n55Zfp2rUrBQUFRpfTqu644w7uuOMOALZv346maTHxnAMkJyeTmppKWVkZ6enpbNu2LWY+qE9LS2PPnj0EAgHsdjvbtm2jT58+Rpd1SSTwmyhWWzju3r2bFStW0KNHDx555BEAbr/9dgYPHmxwZaKl3X333bzwwgsEg0E6duzIzJkzjS6pVVx++eWMGDGCRx99FIvFwmWXXdbmtlmQrRWEECJGyBq+EELECAl8IYSIERL4QggRIyTwhRAiRkjgCyFEjJDAF0KIGCGBL4QQMeL/AbBn0ebPGXFRAAAAAElFTkSuQmCC\n", 189 | "text/plain": [ 190 | "
" 191 | ] 192 | }, 193 | "metadata": {}, 194 | "output_type": "display_data" 195 | } 196 | ], 197 | "source": [ 198 | "\n", 199 | "svm = SVM() # Linear Kernel\n", 200 | "svm.fit(data=data_dict)\n", 201 | "svm.visualize()" 202 | ] 203 | }, 204 | { 205 | "cell_type": "code", 206 | "execution_count": 14, 207 | "metadata": {}, 208 | "outputs": [ 209 | { 210 | "data": { 211 | "text/plain": [ 212 | "(-1.0, -1.000000000000098)" 213 | ] 214 | }, 215 | "execution_count": 14, 216 | "metadata": {}, 217 | "output_type": "execute_result" 218 | } 219 | ], 220 | "source": [ 221 | "svm.predict([3,8])" 222 | ] 223 | } 224 | ], 225 | "metadata": { 226 | "kernelspec": { 227 | "display_name": "Python 3", 228 | "language": "python", 229 | "name": "python3" 230 | }, 231 | "language_info": { 232 | "codemirror_mode": { 233 | "name": "ipython", 234 | "version": 3 235 | }, 236 | "file_extension": ".py", 237 | "mimetype": "text/x-python", 238 | "name": "python", 239 | "nbconvert_exporter": "python", 240 | "pygments_lexer": "ipython3", 241 | "version": "3.7.3" 242 | } 243 | }, 244 | "nbformat": 4, 245 | "nbformat_minor": 2 246 | } 247 | -------------------------------------------------------------------------------- /train_test_split.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/train_test_split.pyc -------------------------------------------------------------------------------- /utils.py: -------------------------------------------------------------------------------- 1 | import random 2 | import math 3 | 4 | def Kfold(X, folds=5, shuffle=False): 5 | X = list(range(X)) if isinstance(X, int) else list(range(len(X))) 6 | if shuffle: random.shuffle(X) 7 | i = 0 8 | fold_size = int(math.ceil(1.0 * len(X)/folds)) 9 | while i < folds: 10 | yield X[0:fold_size*i] + X[fold_size*(i + 1):], X[fold_size*i:fold_size*(i + 1)] 11 | i += 1 12 | 13 | def train_test_split(X, y, test_size=0.2, shuffle=True): 14 | idx = range(len(y)) 15 | if shuffle: random.shuffle(idx) 16 | test_idx = int(test_size * len(y)) 17 | return X[idx[test_idx:]], X[idx[:test_idx]], y[idx[test_idx:]], y[idx[:test_idx]] 18 | 19 | def accuracy_score(pre, y): 20 | return 1 - sum(1.0 * (pre - y)**2)/len(y) -------------------------------------------------------------------------------- /utils.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bamtak/machine-learning-implemetation-python/1d36b0d152a23c2f1cf7186e4f8a208b9c0768fc/utils.pyc --------------------------------------------------------------------------------