├── logistic.png ├── README.md ├── linear.py ├── 01. Linear regression.ipynb ├── 05. A note on gradient descent and convergence.ipynb ├── 14. SVM part 2.ipynb └── 13. SVM part 1.ipynb /logistic.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/felperez/machine-learning/HEAD/logistic.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # **Machine learning for the working mathematician** 2 | ## A comprehensive, mathematically detailed revision 3 | 4 | In this repository, I will upload some projects I have been working on. These include the building algorithms from scratch, as well as some applications of these. These notebooks follow the perspective of a theoretical mathematician, so they provide details of the algorithms which are normally skipped. The suggested order of the notebooks is 5 | 6 | 1. Linear regression 7 | 2. Multiple and polynomial regression 8 | 3. Logistic regression 9 | 4. Neural networks 10 | 5. A note on gradient descent and convergence 11 | 6. Stochastic gradient descent 12 | 7. K-nearest neighbor 13 | 8. K-means 14 | 9. Evaluation 15 | 10. Linear discriminant analysis 16 | 11. Quadratic discriminant analysis 17 | 12. Naive Bayes 18 | 13. Support vector machines, part 1 19 | 14. Support vector machines, part 2 20 | 15. Support vector machines, part 3 21 | 22 | Most of the examples are done using basic datasets in order to illustrate each concept. 23 | 24 | Sources: my sources are mainly my notes from Andrew Ng's course and The Elements of Statistical learning by Hastie et al. 25 | 26 | Disclaimer: I am not a programmer, so my code is not the most elegant possible. In contrast, as a mathematician, my notebooks will tend to be full of math and equations, so beware. 27 | -------------------------------------------------------------------------------- /linear.py: -------------------------------------------------------------------------------- 1 | # Linear regression algorithm 2 | # optimization method: gradient descent 3 | 4 | import numpy as np 5 | import matplotlib.pyplot as plt 6 | 7 | def cost_function(X, y, theta): 8 | m = len(X) 9 | M = np.hstack(((np.ones(len(X))).reshape(-1,1),X.reshape(-1,1))) 10 | return (1/(2*m))*np.dot((np.dot(M,theta)-y).T,(np.dot(M,theta)-y)) 11 | 12 | def gradient(X,y,theta): 13 | m = len(X) 14 | M = np.hstack(((np.ones(len(X))).reshape(-1,1),X.reshape(-1,1))) 15 | return (1/m)*(np.dot(np.dot(M.T,M),theta)-np.dot(M.T,y)) 16 | 17 | class linear_regression: 18 | 19 | def __init__(self): 20 | self.theta = np.array([0,0]) 21 | self.cost = 0 22 | self.theta_history = [] 23 | 24 | def train(self,X,y,num_steps,learn_rate): 25 | self.theta = 2*np.random.rand(2)-1 26 | self.theta_history.append(self.theta) 27 | for i in range(0,num_steps): 28 | self.theta = self.theta - learn_rate*gradient(X,y,self.theta) 29 | self.theta_history.append(self.theta) 30 | self.cost = cost_function(X,y,self.theta) 31 | 32 | def get_parameters(self): 33 | return self.theta 34 | 35 | def get_score(self,X,y): 36 | return cost_function(X,y,self.theta) 37 | 38 | def get_cost_history(self,X,y): 39 | return [cost_function(X,y,theta) for theta in self.theta_history] 40 | 41 | def predict(self,z): 42 | return self.theta[0]+self.theta[1]*z 43 | -------------------------------------------------------------------------------- /01. Linear regression.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Machine learning from scratch: linear regression\n", 8 | "\n", 9 | "## Introduction\n", 10 | "\n", 11 | "In this notebook we will investigate the method of linear regression. We will follow a statistical learning approach. We start by importing some useful scientifc libraries." 12 | ] 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": 1, 17 | "metadata": {}, 18 | "outputs": [], 19 | "source": [ 20 | "import numpy as np\n", 21 | "import matplotlib.pyplot as plt" 22 | ] 23 | }, 24 | { 25 | "cell_type": "markdown", 26 | "metadata": {}, 27 | "source": [ 28 | "We generate (pseudo) random data following a linear trend. For this, we will start by generating linear data and then adding random noise. " 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "execution_count": 2, 34 | "metadata": {}, 35 | "outputs": [ 36 | { 37 | "data": { 38 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAHRlJREFUeJztnWuMJcdVx/9nx7vE4xjs3N0IFHtmsJSgWFaAeBSMIkFCUGQtUfwlQkZjYxSLlReELAEfQCvx1AqBRJCRgmAETjbeSQhEPCxrEa84OESxw2zs2E5MkDE7i2WDX2BircGO5/Ch7/X29vSjqrue3f+fdDUzd/p2n+q+/a/Tp06dElUFIYSQfNgX2wBCCCF2ULgJISQzKNyEEJIZFG5CCMkMCjchhGQGhZsQQjKDwk0IIZlB4SaEkMygcBNCSGZc5GOnBw8e1LW1NR+7JoSQUXL69OnnVPWQybZehHttbQ3b29s+dk0IIaNERHZMt2WohBBCMoPCTQghmUHhJoSQzKBwE0JIZlC4CSEkMyjchBAylK0tYG0N2Lev+Lm15fVwXtIBCSFkMmxtAUeOAOfOFX/v7BR/A8DGhpdD0uMmhJAhHDt2XrQXnDtXvO8JCjchhAzh7Fm79x1A4SaEkCGsrNi97wAKNyGEDOH4cWB5+cL3lpeL9z1B4SaETA+XWSAbG8DmJrC6CogUPzc3vQ1MAswqIYRMDR9ZIBsbXoW6Cj1uQsi0iJAF4hoKNyEkbVxPbnGdBRJ48g3AUAkhJGV8hDVWVor91L2fgn0G0OMmhKSLj7CGyyyQSGEXCjchJF18TG5xmQUSYfINwFAJISRlXIY1yrjKAvFlXwf0uAlJiQgDXUkTYXKLEYvrtLNTeO1lAthH4SYkFRYDXTs7gOr5ga4pi3eEyS2dlK8TUFyrhXgHsk9U1flO19fXlau8E2LJwoOrsroKnDkT2hrShKfrJCKnVXXdZFt63ISkQqSBLmJIOTxSR8DrROEmJBUiVJkLSs7x+2p4pI6A14nCTUgqpDoQ54Lc4/d1+dplAl8nCjchqZDiQJwrcqkP0vRU0BYGiXCdmMdNSEoErjIXjBzi923T15vytSMNHNPjJoT4J1b83iau3vZUkFgYi8JNCPFPKOErC/XBg8CHP2weV297KkgsjMU8bkJIGLa2Cu/17NnC0z5+3K3wVUMdTTSFNyLn0TOPmxCSHhsbhQDu7hY/XXurXZkfC5o868TCIW0YCbeInBGRR0TkIRGhK02Ib3LOeY6F6UBnOa5ePs/HjgG33JJMOKQNm6yS96rqc94sIYQURCrOnz1NmR9lyh503Xk+cSJZsS7DUAkhqZFLznNsqk8lhw/vDXXs3w/MZvUedMbn2VS4FcDfiMhpETni0yBCJk8OOc+xqZuJeeLE3lDHxz4GPPdcfVw94/NsGip5t6o+JSJvBvC3IvLPqnpfeYO5oB8BgJWx1FYgJAaRivNnRZO3fOqUeQZIxufZyONW1afmP58B8OcA3lWzzaaqrqvq+qFDh9xaSciUyCi7wRtdg7MuvOWMz3OncIvIJSJy6eJ3AO8H8KhvwwiZLIlN9giOSUEqFzMxMz7PnRNwROQqFF42UIRWPqmqrV0SJ+AQQnpjMhGmbrLN8nI2wluHzQSczhi3qj4B4LsHW0UIIV1sbZktVFDODPE1EzNhWB2QEJIGCy+6iWoYZKyVFA1gHjchJA3apqzHHDRMcBYrhZsQ4h8T8WvLCLn4Yl+WtZPoyj0UbkKIX9rEryzo+1rk6Pnn7QXThaec6OxKlnUlZMz4LqVqQlOWyGwGvPyyWUW/BaYlVl1lnezbV3Q2VUSK2ZgOYVlXQsiwx3yXcd2mEMjzz9eL9tKS/b6quPKUY63c0wGFm5Cx0le8XMd1bUVud7fwrIfsy1UdkrbZlREHLSnchIyVvuLlOq7bJH6zWf32i5DOkOnoTQKv2iyydULcNLsSiDtoqarOX9dee60SQgJz8qTq6qqqSPFzNlMtZOXC1+pq+35E6j8n0t8mQHVp6fzxT54sXsvLFx5jebl4v649i/dNj1vdd9NxmravblNm0Sbbc9sCgG011FgKNyFjoE549u9XPXDAXIwWuBIlEzEcIs4mx29qS7U9XdtV7XLZuc2hcBMyNZqEZzazF0Zb79PWpj5eaVngZ7PiZdomE5Ft2qap/fS4CSGDce0BuvCE22yy2b9t2KNKl8iePHk+jNP2Kouyq86tBIWbkKnhwQP0ZtNs5iaebNrONpHt6hSqr+p+HYZ5KNyETA0PHmDrsUwEq8mmpkHTprBOVxjD5MmiyWaTTqF8jKNHvcXkKdyETBFfA33V+LLNgGedTSZCXN6vrcdtcx5MbSmLt6fOkcJN8sBnRkFK5NxO01CCTUjGxstdnK8uGxaDlbYdi40tLtreAoWbpE/IR/uY1LVz4bXlIOKmwmYzCGoTV17s13QA0VZcbcM5Q9veAoWbpE+Kg2k+6BK+1Dsr01BCnxxv28lCbZ3gEHGteyKyORY9bjIZPExg8MaQUIeJsKTcWZl43C46H9MnsOq1GOpxd9lUPtbRo16fEincJH1y8biHhnRMxKUcDkgtFt40I7M6AcaF7X32EapjGWKjIRRukhamj6Iphg2GdjAm8dymAbhQ56NLjEz+bxPHdyl+ph1LBlC4STp0TX5IzcOs0hXSMWlDOaWtKZ0s1hOIiw7DJo7vo4PK4XtkAIWbpEMuIZEm2uzvI0JNIhMr5u/i+tjE8XP/PnjERrhZj5v4xVVB+y58FbVvqwvdp271xkax9NbubvFzsYxWrJVWXFwfExsX+wv1fRg5FG5iRl9hDCFIPlfibiqkv7HhVoSGLhzQFxfXp872pv0luhRYdpi65jYvhkpGxpC4ZIhBt9CP3zZ1nvvsd2isdmjlvT7XxySO7/J4IwSMcROnuMis8Dl4FDI+PLTEqG9cxt2H2DAkS2WiULiJW1KfLBPS4+7ytEOLUJ8ZiDb7K3vKFFuvULiJW1LPBAj5+O1qcQAX9Kn5Ybu/5WXvMwZJAYWbuCWHuGQo0WzqxGwXB/BpS99O1rZSXiod90iwEW5mlZBu2jIrYlLOdDl2rMhu2N0tft5+e2GrCHDwoP/0QMA+NXAoptkrptkpttkwKaXw+UoHTRVThbd50eMm3mma6nzJJfXe4YEDfutVxBgHaPP++zx95Opx5/BEaAAYKiGjp08BfJ9CE2McwIVglTuhukUIml4mx4kdvkqlYzGEwk3Gj+2SU76931he3xBxbCvQ1NUBmoh2CgPGGeFFuAEsAXgQwD1d21K4iXdS87hV80uZc12HxXTfIduRETbCbTM4eTuAx1zH2Anphck06zIHDvifPt5UhyRV2qbsDx2QDlmTJFa5gIgYCbeIXAHgRwD8oV9zCDGkKiyzWSHOdcxmwJ13piekfTIhXGZPdNUNMemImuwJWZMk1awnn5i45QA+A+BaAO8BQyUkVVIKVZRrdywWua3OROwzNd1l3Hjo/rpqrY8g0yMkcBnjBvABAL83/71RuAEcAbANYHtlZSVcawlJjbYZjUMWTvARyx3S2XXZk1JHWiZRu1wL928AeBLAGQD/AeAcgJNtn6HHTZIkdnpaWdj6ZEKklj2Rmj0mJPwkYCPcnTFuVf1FVb1CVdcA3Ajgs6p6k5M4DSGh8Fmzu0rXANzZs/1iwKnVsk7Nni62toBbbgk/w9UDnPJOpkGf1Wr60iVcKyv9MiFSy55IzZ42Fh33a6/V/z+l6fsmmLrmNi+GSsggfIQ0UqnZXV1UwLRu9Wx2fmJM3WBnLBKNF+/BJHwVGXDmJMkWXzHImKvkNGWVdIl2ygs25EbbTNtEziWFm+SLL4H10SH09TZNbDGZGdp2TnLxhEPRdD6XlpI5NxRuki8+QxouxWxIR2DSOZnUYmk6JwlnTkQjg3NC4Sb5Yutxm4hxXax4qHgPeTIw6ZyGeNx9bRu7l554+yjcJF+6ZuOVbzyTJbV8xYqHPBmYCKvpsmR1AtTHtr4zORMWwtygcJOw2NzAth7yYps6YWkSqLIADo0VNzHE4zYVyZMnzw9str1s4uNN57zPk07ioYfcoHCTcN6QzQ085GbvqhHd5FkOiRW7anfT502uj6nnXV71pmtBhDo7bb30kZRSTQkK99QJ4Q211dsA6kfrh8ReTUW7r8fdp6Nr+ozrTrO8P9Nz0LUgQvWc216bHKe7Jw6Fe+r49oZMvcBqZ9H3ZrdZNKFPjNskVj7k3LjsNG3OhU1NFFu76XE7h8I9dXx7Q7bi0fW5rpu9y9Nc/L9vVolLEUql01ycFxt7bMcqGON2CoV76vgWD5tH9nJn0fdm9zXA2NWePh1diBBCVWCbQiIL8fUlsMwqcQqFe+qk9rheta1PLLnLyxwijDl53HV0XW8KbBZQuInfm7VOKPbvr89kmM3cenc+PG6XHV2sEILrlEwSHAo38U9TrnXdY7tL4fIljK6nw6cqjIxNJwuFe6ykLAgLQky3TvE8pGhTHcwGSRYK9xjJ4RFcNdx0a1/29z1GLl4s86+ThcI9RlIc9HJlp6+2hRLUWF5sn06JHneyULjHSAxPqc9NblOHo2s24NC2hRKpGNemb6cUaqo+sYbCPUZieEp9Banr5jadRDK0bW32uxSgGNdmaJGrPm3PKSSUIRTuMRLjpmkSh3JBoz6iZ5IH7nO5stnM7bmMcW1yeQIjxlC4x0rox1TTfG1bkeoKj7hqW5Ogts00HHKskNcmpycwYgSFm7jDZnq1KSFFp05QbQovpRrPTekJjB63EyjcxB8uvK7YsVITAYptowlDOhZXpQdSOycZQ+Em/nDldcX0Zk0EaMze5RABTvkpJHMo3KbwS2jPWLyurmvvI56byvfNZ958Cu3LFAq3CaEFaAxf6nKhp8VaiLm2pQvXGTUpdXi+OqVU2pcpFG4TXHodffKWY3+pbTuSFNvgE9cZNW3ftzFkpIw5tBQICrcJrryOHOOloaay547LjJq2FMjQHaKPTpipgoOhcJvgSohM9pPal7pP20O1IXZIqe34Q85B0zlfhJxCd4iuz/MUO3bHULhNcOV1mNzMqX2p+whQiDbEDsd0HX/oNPO6fTd54bl5qrGv3QigcJviwuvIMSfYZ/EoH3YtbIsd+/VRoCn2ZKSc9j9yKNwhMb2ZTb/UIb78fQXIt21dixCnUP/D9TkI1amn5jyQPVC4Q+PqZg55c6XoHZkUn/IZXooV0gpxLVIL15E9ULhzJWbKWN/9u7SrruMKGfsds1ea2gA52YNT4QbwBgBfAvAVAF8F8Ktdn6Fw9yRWytiQ0Ilru9rivjG93xSfUGygx508roVbALxx/vt+AA8AuK7tMxTunsRKGes7KOhTDFLyflOypS9jaMPI8RYqAbAM4MsAvq9tOwp3T2KljPUdFPT9+J2KlzsWbzWV80lqcS7cAJYAPATgJQC/2bDNEQDbALZXVlbCtXZs+EgZ67ph+w4KjkXQumB82C/sUFTVr8d9GYB7AVzTth097h60fXmHPOaafLbvoOBUHr+n0kHFYCrfIQO8ZpUA+GUAP9+2DYW7gosiVH29ElPR6TsoOAZvycX1If1gp/g6rgcnDwG4bP77xQA+D+ADbZ8JKtyxhcPFTW/75bVps+1j/tREyvUEKmIHw1Cv41q43wHgQQAPA3gUwC91fSaYcMcWmaGi3OXlughP9J3ePhWRoscXF57/15nOBJzYF93k+F0ZG20vFwOCbUI/JYFugh5fXGI7XwlhI9z7kDNnz9q9H+P4Kyv99r28DBw/3u+YZTY2gM1NYHUVECl+bm4W/ztyBNjZKW6XnZ3i762tfvbmStP16XvdiB1N38+NjdiWpY2pwtu8RuNxVz3So0cv/NuksL5JxkZTGMVnm2M/raQCPT6SCJhMqMTnTWciuKZLWZU7gD7hkS67Fvu1CXcwRHAehoxIAkxHuFX93XQmk1IA+8Vj2/Zrms1QHtSsCrCLNRAJIcGZlnD3wUTsTQcVXaxRuegAbFMJh4hvjBBBap5tavaQSUPhbsNUsEw97j4eqqlgdAnz0HBHSOFKLZacmj1k8oxXuF0Ijc1Mwq4Yt+8bvcvLzynckZqtqdlDJo+NcOeTDri15SZ9zTSdrpymBBSpSlX27QNuvhlYW3OfRre1VX9M4Hyq2vHjRdpgmaY0wsU+19YKu33Y3Ebs1E3T48ayhxAbTBXe5jXY4x5SIa9vJbw2T8skbOJ6YYOmGtwi/WqYhA4NVO0ySZ0MicmMVsa+SUCQdaikLd2tK57btxJel4CZDlT2FaGyUMxme1MMqy/TfZVFJ2RooO4cm6ZOhqLpe3D0KGPfJAp5C7fpoGCd8Ljyyvva1CcH2naCTleNkSbRCZm33XS+bFMnF23y5f0OebIjxDF5C3eXd93mCfkSJ1Nx7XNz23RUXZ5fm+iEFCRX1yFG5gcnJpFI2Ah3eoOTbTUiVNtrGviqO1GtpzCbAQcOXLhN26BgG6aDYUtL3TUc2gbcbAcyh+DqOhw7Bpw7d+F7584V7/uCtUtIDpgqvM1rcIy7yevpMx3cl4fm6hHe5cBnl1cdatDNZpJRGzG8X+Z3k0gg61CJajFA1Hcqd19xipVJ0DSQN5v1a0MqonPyZH0miY09seLNzCohEchfuFXzndXXx26Xba1mqPTpAFwxdLA4pY6IEM+MQ7hD4sqzS0loUrDFJNTRZSe9XzIRbIRbiu3dsr6+rtvb28736419+wrJqCIC7O6a72dtrZjRWWV1FThzpq91/bCxZWurGPA7e7YYhDt+3E0hexMbUjpnhERERE6r6rrJtulllcTAVSZB7GnU5SntdWJYZ4urUgJ1mGSyxD5nhGQIhRtwlyoXM5WsKsBNVG3xmXJnsiwV0+8IsYbCDbhb9y5krnS1YNTtt+8V4Cp1tvj2eDc2ipDH7m7xs3pOQ54zQsaCaTDc5pXd4KRLQgym2U6Tb7MlhSneHIAkhIOTSeNiILBpQK+OrkG+RYil7K0vL3OlbUICw8FJl7isYe27pngVk5CDqzARISQcpq65zct5qCSlWY2L0p+mNbB91KQ2LUwVagIRIWQwGNUEnJgTSZoE0mQ6vk0c2tWCwyadQZswpzBph5CJMi7h9j141iZkpgso1NljU651yILDph3J4jO+Vo0nhAxiXMLts0JcXyEzscdU9F14tKbhDd+rxhNCemMj3OkMTjYNAvqcoNE1+aQux7hrAd8u+2Yz9wOBXbnSC7pytjkZhpAsSEO427ItTCZo9M386BKyuoyL225rtqdsx0sv1S+2cMcdZiLrgy5h5mQYQvLA1DW3eVmHSoYsADBkQK1vTLfOHpd1tX1huphyKmVhCZkQyC7GPSS2OmRAzWUWRS4De6bxcGaYEBIUG+FOY+bkkNKebSVZ77pr7yxF4ML3Dh8GTp0aXtLUVWnYVGC5VUKCYjNzslO4ReRKAJ8A8O0AdgFsquodbZ+xFu4h066bBGY2A15++cJ97t9fCOkrr9gfp4uxCd3YOiJCEsf1lPdvAvg5VX07gOsA/LSIXD3EwD0MmXbdNKAG7M0YefXVC0V7sY2LEqZjG9hjhgkhydIp3Kr6tKp+ef77NwA8BuAtzi0xTWmrZpAA9aL/wgvmx3ZRwnRsNT/G1hERMiKsYtwisgbgPgDXqOr/NG3nrTqgTUjFZQW9qeJrSTNCyB6cxrhLO30jgH8AcFxV/6zm/0cAHAGAlZWVa3dMRdOGrjhyWWje9CbgG9/YGxqpwhKmhJAEcC7cIrIfwD0A/lpVP9K1vTePuyuDpOqN798PvPZa82Da0hJw4gRFmxASHaeDkyIiAP4IwGMmou2VtgGzuunrr74KXH558/52dynahJDsMMkqeTeAmwH8kIg8NH8d9mxXPW0DZk0DjC+8UKQG1sEMCUJIhphklfyjqoqqvkNVv2f+OhXCuD20ZW60eeN33MEMCULIaEijyJQNTWmDbd742FL1CCGTJi3hHrK+Y5c4m+aJE0JI4lwU24DXqeZoL0q7AuYiu7FBQSaEjJ50PO6uRQ1yw+Xq8IQQUiIdj7trUYOccPH0QAghDaTjcY+pqNHYnh4IIUmRjnCPqajRmJ4eCCHJkY5wjyllb0xPD4SQ5EhHuIH4KXuuBhTH9PRACEmOtIQ7Jm0rzdsypqcHQkhypC/codLqXA8oxn56IISMlnTSAesImVbHAUVCSCak7XGHTKvjgCIhJBPSFu6QXjAHFAkhmZC2cIf0gjmgSAjJhLSFO7QXzAFFQkgGpC3c9IIJIWQPaWeVACzVSgghFdL2uAkhhOyBwk0IIZlB4SaEkMygcBNCSGZQuAkhJDPyFW6u6UgImSjppwPWwTUdCSETJk+P23fxKXrzhJCEydPj9ll8it48ISRx8vS4fRaf4grthJDEyVO4fRaf4oIKhJDEyVO4fRaf4oIKhJDEyVO4AX8lWLmgAiEkcfIVbl+wlCwhJHHyzCrxDUvJEkIShh43IYRkRqdwi8idIvKMiDwawiBCCCHtmHjcHwdwvWc73MPZj4SQkdIZ41bV+0Rkzb8pDuHsR0LIiBlnjJuzHwkhI8aZcIvIERHZFpHtZ5991tVu+8HZj4SQEeNMuFV1U1XXVXX90KFDrnbbD85+JISMmHGGSjj7kRAyYkzSAT8F4IsAvktEnhSRW/2bNRDOfiSEjBhRVec7XV9f1+3tbef7JYSQsSIip1V13WTbcYZKCCFkxFC4CSEkMyjchBCSGRRuQgjJDAo3IYRkhpesEhF5FsBOz48fBPCcQ3NygG2eBlNsMzDNdvdp86qqGs1e9CLcQxCRbdOUmLHANk+DKbYZmGa7fbeZoRJCCMkMCjchhGRGisK9GduACLDN02CKbQam2W6vbU4uxk0IIaSdFD1uQgghLUQTbhG5XkS+LiKPi8gv1Pz/W0Tk0/P/P5Dd8mk1GLT5Z0XkayLysIj8vYisxrDTJV1tLm33IRFREck++8CkzSLyo/Nr/VUR+WRoG11j8N1eEZF7ReTB+ff7cAw7XdK1kLoU/O78nDwsIu90dnBVDf4CsATgXwFcBeAAgK8AuLqyzU8B+P357zcC+HQMWwO3+b0Alue/H51Cm+fbXQrgPgD3A1iPbXeA6/xWAA8CuHz+95tj2x2gzZsAjs5/vxrAmdh2O2j3DwB4J4BHG/5/GMBfARAA1wF4wNWxY3nc7wLwuKo+oaqvAPhjADdUtrkBwIn5758B8D4RkYA2uqazzap6r6ouFsu8H8AVgW10jcl1BoBfB/BbAP43pHGeMGnzTwL4qKr+FwCo6jOBbXSNSZsVwLfOf/82AE8FtM8LqnofgBdaNrkBwCe04H4Al4nId7g4dizhfguAfy/9/eT8vdptVPWbAF4EMAtinR9M2lzmVhS9dc50tllEvhfAlap6T0jDPGJynd8G4G0i8gURuV9Erg9mnR9M2vwrAG4SkScBnALwM2FMi4rtPW/MRS520oM6z7ma3mKyTU4Yt0dEbgKwDuAHvVrkn9Y2i8g+AL8D4CdCGRQAk+t8EYpwyXtQPFV9XkSuUdX/9mybL0za/GMAPq6qvy0i3w/grnmbd/2bFw1vGhbL434SwJWlv6/A3ken17cRkYtQPF61PZakjkmbISI/DOAYgA+q6v8Fss0XXW2+FMA1AD4nImdQxAHvznyA0vS7/Zeq+qqq/huAr6MQ8lwxafOtAP4EAFT1iwDegKKex5gxuuf7EEu4/wnAW0XkO0XkAIrBx7sr29wN4Jb57x8C8FmdR/wzpbPN87DBH6AQ7dzjnkBHm1X1RVU9qKprqrqGIq7/QVXNed07k+/2X6AYiIaIHEQROnkiqJVuMWnzWQDvAwAReTsK4X42qJXhuRvAj8+zS64D8KKqPu1kzxFHZA8D+BcUo9HH5u/9GoobFygu7J8CeBzAlwBcFXsUOUCb/w7AfwJ4aP66O7bNvttc2fZzyDyrxPA6C4CPAPgagEcA3Bjb5gBtvhrAF1BknDwE4P2xbXbQ5k8BeBrAqyi861sB3AbgttJ1/uj8nDzi8rvNmZOEEJIZnDlJCCGZQeEmhJDMoHATQkhmULgJISQzKNyEEJIZFG5CCMkMCjchhGQGhZsQQjLj/wH4hdyLrg21IgAAAABJRU5ErkJggg==\n", 39 | "text/plain": [ 40 | "" 41 | ] 42 | }, 43 | "metadata": {}, 44 | "output_type": "display_data" 45 | } 46 | ], 47 | "source": [ 48 | "#set the number of observations\n", 49 | "num_obs = 200\n", 50 | "\n", 51 | "#We generate random data that follows approximately a straight line with\n", 52 | "#known parameters and known fluctuations\n", 53 | "X = np.arange(0,1,1/num_obs)\n", 54 | "y = 3*X + 2\n", 55 | "\n", 56 | "#Now we add the random fluctuations\n", 57 | "noise = np.random.normal(0,0.5,num_obs)\n", 58 | "y = y + noise\n", 59 | "\n", 60 | "#Now we plot the data\n", 61 | "plt.plot(X,y,'ro')\n", 62 | "plt.show()" 63 | ] 64 | }, 65 | { 66 | "cell_type": "markdown", 67 | "metadata": {}, 68 | "source": [ 69 | "The model we want to fit to this data is linear: that means that it will be defined by parameters $\\alpha,\\beta,\\gamma$ and the predicting line is then given by $\\ell = \\{ (x_1,x_2):\\ \\alpha +x_1\\beta +x_2\\gamma =0 \\}$. If we assume that $\\gamma\\neq 0$ then we can reparametrize our model as $\\ell=\\{ (x_1,x_2): x_2 = f(x_1)=\\theta_0+x_1\\theta_1\\}$.\n", 70 | "\n", 71 | "## The model\n", 72 | "\n", 73 | "As mentioned above, we will try to find the line $\\ell$ that fits the data *the best possible way*. This is rather vague, so we need a precise notion of how well a line fits the data. Suppose we count with a collection of examples $X=\\{x^{(1)},\\dots,x^{(m)} \\}$ labeled with values $y=\\{ y^{(i)}\\}$ which give rise to a prediction function $f(z)=\\theta_0+z\\theta_1$. In particular we can compare the predicted value $f(x^{i})$ to the **actual** value of the observed data $y^{(i)}$. This is, we assess how well our predicting function $f$ predicts on the set we used to construct it. We stress that this is to fit the model, not to asses its performance on future predictions. This idea is central to many models, as estimating the accuracy of predictions by assessing the model with the training data does not yield precise information about the generalization of such predictions to previously unseen data.\n", 74 | "\n", 75 | "In order to assess the performance of the model for a given choice of parameters, we compute the mean square error (see details [here](https://github.com/felperez/ML-miscellanea/blob/master/Unbiased%20estimators%20and%20linear%20regression.ipynb)). We could make different choices for the error function, but this one behaves particularly well as it makes some computations relatively easy (such as the gradient computation). With the notation introduced before, the mean square error is defined as\n", 76 | "\n", 77 | "$$\n", 78 | "\\text{MSE}= J(\\theta_0,\\theta_1)=\\dfrac{1}{2m}\\displaystyle\\sum_{i=1}^m (y^{(i)}-f(x^{(i)}))^2 = \\dfrac{1}{2m}\\displaystyle\\sum_{i=1}^m (y^{(i)}- (\\theta_0+ x_1^{(i)}\\theta_1) ))^2.\n", 79 | "$$\n" 80 | ] 81 | }, 82 | { 83 | "cell_type": "markdown", 84 | "metadata": {}, 85 | "source": [ 86 | "## Finding the appropriate parameters\n", 87 | "\n", 88 | "This can be done in several ways, but we describe two: the normal equations and the gradient descent method.\n", 89 | "\n", 90 | "#### Normal equations\n", 91 | "\n", 92 | "For this approach, we first vectorize our dataset and the MSE. First, construct a matrix $X$ by stacking all the training points $\\{x^{(1)},\\dots,x^{(m)} \\}$ as rows of this matrix, as well as a column of $1$'s. Hence, each row of $X$ is of the form $[1,x^{(i)}]$. We also stack all the labels $\\{y^{(1)},\\dots,y^{(m)} \\}$ of these points in a vector $y$. Then we can write the MSE in matrix form as\n", 93 | "\n", 94 | "$$\n", 95 | "J = \\dfrac{1}{2m}(X\\theta-y)^T(X\\theta-y).\n", 96 | "$$\n", 97 | "\n", 98 | "In order to find a minimum, a necessesary condition is that the gradient of our function vanishes. Proceding componentwise, we can show that the gradient of $J$ is given by\n", 99 | "\n", 100 | "$$\n", 101 | "\\nabla J = \\dfrac{1}{m}(X^TX\\theta- X^Ty).\n", 102 | "$$\n", 103 | "\n", 104 | "If we equate this to zero, we obtain the necessary equations to have a minimum for the MSE with respect to the parameters $\\theta$:\n", 105 | "\n", 106 | "$$\n", 107 | "\\theta = (X^TX)^{-1}X^Ty.\n", 108 | "$$\n", 109 | "\n", 110 | "Since these equations involve computing the inverse of a matrix, this approach can be computationally expensive.\n", 111 | "\n", 112 | "\n", 113 | "\n", 114 | "\n", 115 | "\n" 116 | ] 117 | }, 118 | { 119 | "cell_type": "markdown", 120 | "metadata": {}, 121 | "source": [ 122 | "#### Training the model: Gradient descent\n", 123 | "\n", 124 | "Another approach to find a minimum is via statistical learning, particularly by using the method of gradient descent. This method relies on the simple fact that for a function $J=J(\\theta)$, its gradient $\\nabla J$ points in the direction of maximal/minimal growth. Thus, if we follow the direction of the gradient over time, we should end up at a local minimum. Throughout the whole process we compute $J$ and $\\nabla J$ with the data set $X,y$ as a fixed parameter, so we omit their dependence. We can summarize the algorithm by\n", 125 | "\n", 126 | "\n", 127 | "\\begin{align*}\n", 128 | "& 1. \\ \\text{intialize } \\theta \\text{ to a random value} \\\\\n", 129 | "& 2. \\ \\text{for } i = 1\\dots \\text{ num_steps do} \\\\\n", 130 | "& \\qquad 2a. \\ \\text{compute the gradient }\\nabla J(\\theta) \\\\\n", 131 | "& \\qquad 2b. \\ \\text{update the parameters }\\theta \\text{ by } \\theta \\leftarrow \\theta - \\alpha\\nabla J \\\\\n", 132 | "& 3. \\ \\text{return } \\theta\n", 133 | "\\end{align*}\n", 134 | "\n", 135 | "Here $\\alpha$ is the learning rate, which controls the weight with which we update the paramters $\\theta$. This is a [hyperparameter](https://en.wikipedia.org/wiki/Hyperparameter) of the model and has to be optimized via cross validation methods. With this method, we use **all** training examples in every iteration of the algorithm. This can be computationally expensive, and there are alternatives such as mini-batch gradient descent or stochastic gradient descent. We will explore such algorithms in future notebooks." 136 | ] 137 | }, 138 | { 139 | "cell_type": "markdown", 140 | "metadata": {}, 141 | "source": [ 142 | "## Building the algorithm\n", 143 | "\n", 144 | "We already defined our training data set $X,y$. We start by initializing the parameters $\\theta$:" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": 3, 150 | "metadata": {}, 151 | "outputs": [], 152 | "source": [ 153 | "theta = 2*np.random.rand(2)-1" 154 | ] 155 | }, 156 | { 157 | "cell_type": "markdown", 158 | "metadata": {}, 159 | "source": [ 160 | "Now we build the cost function and the gradient of the cost function:" 161 | ] 162 | }, 163 | { 164 | "cell_type": "code", 165 | "execution_count": 4, 166 | "metadata": {}, 167 | "outputs": [], 168 | "source": [ 169 | "def cost_function(X, y, theta):\n", 170 | " m = len(X)\n", 171 | " M = np.hstack(((np.ones(len(X))).reshape(-1,1),X.reshape(-1,1)))\n", 172 | " return (1/(2*m))*np.dot((np.dot(M,theta)-y).T,(np.dot(M,theta)-y))\n", 173 | "\n", 174 | "def gradient(X,y,theta):\n", 175 | " m = len(X)\n", 176 | " M = np.hstack(((np.ones(len(X))).reshape(-1,1),X.reshape(-1,1)))\n", 177 | " return (1/m)*(np.dot(np.dot(M.T,M),theta)-np.dot(M.T,y))\n" 178 | ] 179 | }, 180 | { 181 | "cell_type": "markdown", 182 | "metadata": {}, 183 | "source": [ 184 | "Finally, we define the iterative process. This function will return the learned parameters of the system. We include the initialization process in this function, so we do not need of a starting parameter." 185 | ] 186 | }, 187 | { 188 | "cell_type": "code", 189 | "execution_count": 5, 190 | "metadata": {}, 191 | "outputs": [], 192 | "source": [ 193 | "def linear_regression(X, y, num_steps, learn_rate):\n", 194 | " theta = 2*np.random.rand(2)-1\n", 195 | " for i in range(0,num_steps):\n", 196 | " theta = theta - learn_rate*gradient(X,y,theta)\n", 197 | " return theta" 198 | ] 199 | }, 200 | { 201 | "cell_type": "markdown", 202 | "metadata": {}, 203 | "source": [ 204 | "Now we can run our algorithm and learn the parameters for the linear regression. We can plot the corresponding line and see how it fits the data." 205 | ] 206 | }, 207 | { 208 | "cell_type": "code", 209 | "execution_count": 6, 210 | "metadata": {}, 211 | "outputs": [ 212 | { 213 | "data": { 214 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJztnXl8VNXZx38nYdGAuASqbTEXNxTFioKgpSpaV+ryWtcaWvvWvlS0trX17WBjtdYX26rVulZxqUKGxQUVR1CrIuBCNMguZSdh35Q1gSzzvH9MIsNk7p1zZ+468/t+PvNJMrnLc+6d+d3nPOc5z1EiAkIIIeGhyG8DCCGE2IPCTQghIYPCTQghIYPCTQghIYPCTQghIYPCTQghIYPCTQghIYPCTQghIYPCTQghIaOdGwft2rWr9OjRw41DE0JIXjJz5szNItJNZ1tXhLtHjx6orq5249CEEJKXKKVqdLdlqIQQQkIGhZsQQkIGhZsQQkIGhZsQQkIGhZsQQkIGhZsQQnIlGgV69ACKihI/o1FXT+dKOiAhhBQM0SgwdChQV5f4u6Ym8TcAlJe7ckp63IQQkgsVFXtFu5W6usT7LkHhJoSQXKittfe+A1C4CSEkF8rK7L3vABRuQgjJhREjgJKSfd8rKUm87xIUbkJI4eFkFkh5OTByJGAYgFKJnyNHujYwCTCrhBBSaLiRBVJe7qpQp0KPmxBSWPiQBeI0FG5CSLBxenKL01kgHk++ARgqIYQEGTfCGmVlieOkez8I9mlAj5sQElzcCGs4mQXiU9iFwk0ICS5uTG5xMgvEh8k3AEMlhJAg42RYIxmnskDcsi8D9LgJCRI+DHQFGh8mt2jRep9qahJeezIe2EfhJiQotA501dQAInsHugpZvH2Y3JKR5PsEJO5Vq3h7ZJ8SEccP2q9fP+Eq74TYpNWDS8UwgJUrvbaGmOHSfVJKzRSRfjrb0uMmJCj4NNBFNEkOj6TDw/tE4SYkKPhQZc5Twhy/Tw2PpMPD+0ThJiQoBHUgzgnCHr9Pl6+djMf3icJNSFAI4kCcU4SlPohZr8AqDOLDfWIeNyFBwuMqc54Rhvi91fR1s3xtnwaO6XETQtzHr/i9nbi6Va8gYGEsCjchxH28Er5koe7aFfjZz/Tj6la9goCFsZjHTQjxhmg04b3W1iY87REjnBW+1FCHGWbhDZ/z6JnHTQgJHuXlCQGMxxM/nfZWM2V+tGLmWQcsHGKFlnArpVYqpeYppWYrpehKE+I2Yc559gvdgc7kuHryda6oAK6/PjDhECvsZJWcLSKbXbOEEJLAp+L8occs8yOZZA863XV+4YXAinUyDJUQEjTCkvPsN6m9ksGD24Y62rcHSkvTe9Ahvs66wi0A3lFKzVRKDXXTIEIKnjDkPPtNupmYL7zQNtTxr38Bmzenj6uH+DrrhkoGishapdQ3APxbKfUfEZmWvEGLoA8FgLJ8qa1AiB/4VJw/VJh5y5Mm6WeAOHydRQQbtu/BYQful9X+dtDyuEVkbcvPjQBeBdA/zTYjRaSfiPTr1q2bs1YSUkiEKLvBNTINzjrhLTt0nbfVNeK5D1fg3Aen4sonP0Zz3PkU61QyetxKqU4AikRkR8vv5wP4s+uWEVKoJMdg3cp5DjI6g7NOeMs5XGcRwexVWxGtqsUbc9ZiT1McfQ4/CDcOKENcBMVQGY+RCxkn4CiljkTCywYSQj9GRCwfSZyAQwjJGp2JMOkm25SUuJ4RsnNPE16btQZjqmrxxbrt6NShGJed/G1c178Mvb99YE7HtjMBJ6PHLSLLAZyUk0WEEKJDNKq3UIHHvZIFa7chWlWL12etwa6GZvT6ZheMuLw3LuvzbXTu6H2tPlYHJIQEg1Yv2ozUMIjLlRTrG5oRm7sW0apazF61FR3bFeGSk76F8gFl6HP4QVCpiwR7CIWbEBIMrKasezg4u2TDDkSrajHh89XYvrsJR3dsxl0zJ+CHH76CAw8tDcR4A4WbEOI+OgWmrDJC9t/fVfP2NDXjrfnrEa2qxacrvkT7YoWLen8T5TsWof+tN0B9PVC6MxCzWFkdkBDiLlYDicBeQS8qApqbzY9jd/BR42GxcvMujP20Fi/NXI0vdzXAKC3Bj/qX4cq+3dG1c0dPKwbaGZykcBOSz7hdSlUHM/ErLQXq6/Uq+rWiK5gWD4vGa3+E9xZuQLSqFtOXbEZxkcJ5vQ5F+WllGHhUVxQVJcWui4oSMzNTUSoxG9NBHM0qIYSElFyKVTkp+GYhkC1b0r9fXGzueetOsEkTL19T3Anjx36McSsPxcYde/DNA/fDb8/riWtOPRyHdjGZ7RjQWaz0uAnJV7Lt5judI21mhxlK5b7GY4un3KyKMPWIUxA9eTCmHNkXohQGHXcoygcYGHRsN7QrzjB5XDfM40Bvxo7HDRFx/NW3b18hhPiMUiKJjv6+L6Ws9zOM9PsZRnZ2VFaKlJTse6ySEpHSUvPzmO1TWal1yg3HniiPnn61fPfG58SIxKTvzaPl/jOGSG2Xb+w9fjo7DSNxfZK3Sfd+jvalA0C1aGoshZuQfCFVYKyE0YpsBd/KJkCkuHhfYc4kfmZCakJzc1ymL94kwyqr5ajIG2JEYnLdNf8nbx47UBqKiq1F1q4QO/1wEwo3IYVHOuFp316kQwf7XqFToqQjhjbFOR1bdu6Rp6YulbPue1+MSEz63P22jHjzC1n+7BjztqS2J9N2qXY5+XBrwY5wM8ZNSD5glbnRubO9OKxTMW4nU+mSB0sPOQQCoHr/wxAdeAUmHXEqGkTh1B4Ho3yAgQt7H4b92hfv3VcnM8Rsm1ZS2+9CmiBj3IQUGk57gA54wpY22Tl+kue+tWMnef6Ui+W8nz0uRiQmvX8zXu668GZZ9MwY8/0z9SAqK/eGcaxeyR66zzFuetyE5AMeThTRxk7+toVHLz16YO6eDoiefBEm9joTu9vvh5PWLkb57Mm4+D/TUNK4x7qdmTJDUv9nRbJeOpwjT4+bkELDBQ/Q8lw63rLdbJLS0n2Ou/OFSonOqJEfXP8PMSIx6XXrSzL8gl/K3EOPst+zMLPZKrad7hzDhuXeEzEBHJwkpABxIryR6bilpfYGPNPZZBZCaXl90a2HVJw3TE649UUxIjG54BdPyag+F8n2DvvrhzF0r0MGW9KKt0sPRwo3CQduCU3QCHM703nNuWacpPFy69t1kJdPOEcuH3K/GJGYHPO7CXLr4Ful+pRBEh+tYUNpaXYPFjsetxNtt4DCTYKPl117P0nXzlavLQwiritsdgZBk67JkkO6y93n/Fy+86uxYkRicvbPn5Sn+10mX+3Xed/j6g4g2hVXu+GcXNtugR3h5uAk8YcgDqa5Qabp3h4st5UTmdLkWrFx3xqa4nj7qZcRrV6DGYf2RPvmJlywshrlVa/jtFXz9l2tMdNyZUrp2WdVFCrdICOgfy6HPrMcnCTBx4UJDK6RS6hDJ4bqUFfbFXQ8bs2eUs3mXfLXyQul7z3viBGJycC/viePT1kiG7fv1u+Bpd4LN8MZqecaNszVXiIYKiGBx4Upw66Qa0hHR1ySwwFBi4WbzchsjSknT19PY3tjU7O8NX+d/PjZKukxPCZHDI/J/7zwmXywaKM0N8fbnstu+x18sGhfD2aVkILAoyI9rpDrA0ZncM+Boko5kUmMdP6fYvvaA7rKgwOvkwG3jBYjEpMBI96Vh/69SNZurXNW/HQfLCGAwk2Cg5UgBdHDTCVTSEenDcmFlszSyfzqgTjxwGixvUkVyftH9pWfX14hR/zv69Lj9xPlJ1f9Sd4+4UxpHJ1UOMrpB1QYPkcaULhJcAhLSMQMK/uzESEzkfEr5u/A/dnY6WB57LSrZOAvnkmUUP3laPnbmT+R2gMPbXu8sH8eXMSOcDOrhLiLV0s/ubVEl9V06YoK5zJj/MqyyfL+iAg+Wb4F0apavD17NZqKinF6zRyUz56M8xfPQId4U/rjebgUWNiwk1WSYfkHQlqIRhPiUlSU+BmN6u1ntsSTk0s/tYprTU1CFFqX6NK10Yry8oRIG0ZCXAxjb/qe2TJaustrJTNiROKBkExJyd7UNLeweX++2tWAZ6Yvx/f/PhXXPV2FD5dsxvXdGvFu5W8wdlwFLv7Ph21FO/l4XnweCgFd19zOi6GSPCOXuKQXg25ed7+tYtK5nNepWG2WlffM7k88HpfqlVvk1nGz5JiKSWJEYvLDJz6SV2aukvqGpn3PaRXH1zxfoQLGuImjOJFZ4ebgkZfx4UxZIn6LkINx9231DTLq4xVywUNTxYjE5IQ735I7Xp0nX6zdltmGXLJUChQ7ws0YN8lM0OOSXsaHrWZCGoZzsXVdUmP7O3emXz1d91pEo5j3wJOIHnYyXj9hEOrbdUTvb3fBEKzHJQ/fgU7Llzg7hkC+hjFu4ixBj0t6GR82i18rlThfRYX9cYBsSRfbTyfaQMa4e11DE8Y/+hIufXczLrlgOF7vdRYuXfABJo4fjtjCsbh2+E/Radli58cQSFbQ4yaZcWopKzdxK6skFYcWB3DVlnSYeNz/Wb8dY6pq8erna7BjTxN6bqrBkFmT8F8LpqBLQ4bFBfKtrozPsFYJcZ4gxiXNbKqs3Le6W2mps1Oe7VSTczM/WbeWdEqMu76hSSZ8vkqueOKjRAnViknym3Gz5LPux0tc53he5ZjbIYifT5uAg5Mk7zGb6typU3qR6dDB3XoVfkygMRs0TllJprXdyzbukHveWCAn3f22GJGYDLp/ioycuky27NxjfTyns2ecJk8yVSjcJP/JpgC+m0Ljx4xADcHa09gssTlr5UcjPxEjEpOjbn9TbqqcKR8u2ZQo8pRpdZtcsme88oLzZDYmhZvkP3aXnHLb+/XL6zMRx9otu+S+txZK33v+LUYkJt/9y3vy2PtLZMP2emubWws0ZXoA6oi2V9cjTCWCLbAj3NqDk0qpYgDVANaIyMVW23JwkriOnYG5VtweTPNqgNSEpuY4pizahGhVDaYu3gQF4JzjvoHyAQbO7NkNxUVq3x2s0ihHjMhtQDoIKZohGzy1MzjZzsZxfw1gIYAuWVlFiJOkExYrOnRwf/p4ebkvWTbrt+3G+M9WYdxntVi3bTe+cUBH3HLOMbj21MPxrYP2N9/Rasp+azuyfRA5WQ4gE2YPGbfvt5/ouOUAugN4D8A5AGKZtmeohHiCbnzWyawSJ8kmBtyyT7MqkqkDLpChIybIkbe/KUYkJkOemSGT562ThqZmvfM7ERs2a4NfZQiYVbKPcL8MoC+AQRRuEliC9OVNrt3RushtaspiFlPTN5UeJk8MuELOGPq0GJGYnHJLVP7ywMuycvPO7GzMJQ6dqdZ6HmR6eImjwg3gYgBPtPxuKtwAhiIRA68uKyvzrrWEBA2reiZZLJwQj8flk2Wb5ZfX/kmOvu1VMSIxufpHf5HXe50pu4vb5ebF5vKwy9SGID1IkwmoXXaEO+PgpFLqLwB+DKAJwH5IxLgniMgQs304OEkCid+zK1sxjIQN6b57SfVfttY14JXP12BMVQ2WbdqFLrt34or576F89mQcvWV12n08Jeg1bNIR4FnAdgYnbU15V0oNAnCbMKuEhA0vv7BmgtaKUokHRxpxF8PArGmzEJ1Ri9jctdjTFMfJZQehfICBH1x9NvZfvrTt8fzKnghbNkc0Clx/PdDc3PZ/AbCZRaYISaWiom0GSl1d4n2nyVR8q9XbTyqMtbPD/qjsfxkG//QR/PCJj/HW/HW4sm93vPmr7+HVmwbiyr7dsf+f/+TPYgtm+LX4Qza0PrjTiTbgTraLm+jGVOy8ODhJcsKNGGRQananLCow76SBMvyCX0qv374iRiQmF/1jmlTOWCk7dje2zZppnRiTbrDTLwIaL25Dppm2AZhlCc6cJKHFrWwEP1fJSRHauj1NMv6RF+XSnz8mRiQmx/5ugtz211dkVu1XEo/H9+4f5AUbwobVTNuAXEsKNwkvbgmsGw8Em97m4vXb5a7X50vv298QIxKT79/whDzX9xLZ2rFTW1t0arFYXZOweMJeYXY9i4sDc20o3CS8uBnScFLMNB8Euxub5LVZq+Wqf36cKKH6h0nyq2vukqruJ7QtoZosxDq1WMyuCXOo2xKCa0LhJuHFrsetI8bpYsW5incGO1ds2in3vvmFnPznd8SIxOTM+96XJz9YKpt37NZ7OOXicWfba8l3Lz3g7aNwk/CSaTZe8hdv2LDMXpRbseI04ttQVCyTjh0o5U/PECMSkyNvf1NuHF0t0xZvTJRQbUVHWDPZnbxPqv3Z9FocXGSYZAeFm3iLnS+wXQ+5dZt0wmImUMkCmGus2Iyk464+oJs8cMYQ6XfzKDEiMTn93nflkXcXy/pt9en31RXJysq9A5tWLzvxcbNrnk1PJ+Chh7BB4SbeeUN2vsC5fNkz1Yg28yxziRVb0DS6Ut49/nvysyvulCP+93Xp8fuJ8t9X3y3vPjFOmpK9azN074+u55286k2mBRHSXXO7XnqeLF4QJCjchY4X3pBVvQ0g/Wh9LrFXXdHO1uPWFNIN2+rlkXcXy3f/8p4YkZj0+1VUHjhjiKzqdfK+BaScfGgmH0/3GmRaECH1mtu9N3myeEGQoHAXOm57Q7peYOrDItsvu51lyrKJcWeIlTc3x2X64k1y4+hqOaqlhGr50zNk0ty1bUuouv3QtHMtrMQ+9ZrbtZset+NQuAsdt70hu+KRab9MX/ZMnmbr/7PNKjGxa3PPE+TJD5bKWfe9L0YkJn3ufltGvPmFLN9kUUI1KA/N1utixx67YxWMcTsKhbvQcVs87HTZkx8W2X7Z3RpgTNOeOCBV3U+QX118mxzzu0QJ1av++bG8Nmu11Dc0ZX9tnAwhpAqsWUikVXzdElhmlTgKhbvQCVp3PdU2u192HS8zF2E0DNnasZP865SL5dwbHhcjEpPevx4nd11+myxav932sTwPIWS63xTYUEDhJu5+WdMJRfv26TMZnFo2LNNgaBbCGI/HZVbtV3LbX1+RY1uKPF3647/L+BPPlbouB2Vnt18hBKdTMonnULiJ+5jlWqfrtjspXA4I447djRKdUSODH54mRiQmvf44WYb/7RWZ1+d7zk2HD6owMjYdWCjc+UqQBaEVL6ZbZ3kdFqzZJn+YMFeO/+NkMSIxueChqTLqk5Wyvb5Ba3/H7PcTZoMEFgp3PhKGLriId9OtNakfVSkvnXW1/NeQB8SIxKTn8Dfkt+Nny8yaL/eWUM2VMHmxzL8OLBTufCSIg15O2elC25Zs2CF33/eynPjrcWJEYnL2z/8pz/S7VL46uJvzguqXF5uNl0+PO7BQuPMRPzylbL7kdupwZJoNaLNtuxub5PXZa+TqJxMlVI++7TW5+dLfy8eHn7hvCVWnRcqPe5Otl59r7yAsIaEQQuHOR/zwlLIVpExfbt1JJJptq9m8S/4yaaGc0lJC9Xt/e0+emLJUNnU6yNx+JwXIj3uTyzmzbXuYQkIhhMKdj/jxpTETh+SCRtmInk4eeIa2NTY1y1vz18mPn636uoTq0FGfyQeLkkqoWtnv5LX0496EpQdGtKFw5yted1N187XtilSm8IhF29Z8VSd/f2eR9B/xbzEiMTnt3nflH/9eLOu2pimhaiaoVjMNs8XrexOmHhjRgsJNnMPO9GpdbIpOU3Nc3l+4QW54/jM5YnhMegyPyfXPVck7C9ZLY2qRp0z2V1baK7wU1HhukHpg9LgdgcJN3MMJr0tTdDZsr5fH3l/ydQnVvve8I/e9tVBqt+zKrQ06AhSGeG4uDxanSg8E7ZqEGAo3cQ+nvC4T4YjH4/LRkk1yU+XMr0uo/mjkJxKbs1b2NGbwru2cO5MA5bN3mYsAB7kXEnIo3LrwQ2gfl7yuL3fukZFTl8nZ908RIxKTk+5+W+55Y4Es3bjDIcNTyHTv3YjnBuXz5tZDKSjtCykUbh287vblw4c6udBT61qIObQlHo/LZyu2yG/GzZJjKiaJEYnJFU98JBM+X6VXQtVNnM6oCVKYwa2HUlDaF1Io3Do46XVkk7fs94fa7oPEwTZsq2+QFz5eIec/OFWMSExOuPMt+eNr82Thum1ZNcUVnM6osfq85UNGSj6HljyCwq2DU15HGOOlXk1lT2HOqq/k9y/NkePuSBR5uuTR6TK2qkZ27m7MuUmu4GRGjVUKpNcPdTccCaYK5gyFWwenxFTnOEH7UGfT9izbsGtPo4ytqpGLH5kuRiQmx90xWSIvz5E5q75Kv4PfISWr8+dyH82ueWvIyeuHutPXOWjOSQihcOvglNeh82UO2oc6GwGy2YaF67bJH1+bJ73vfEuMSEzOf3CqvPDxCtlmVULV75BSpvPnOs083bHNvPCweap+37s8gMKtixNeRxhzgl0qHlXf0CSvzFwlP3ziIzEiMTmmYpLcOm6WVK/coldC1WoqfBBiv24UaPLyoe52b8bv3lLIoXB7ie6XWfdD7cWHP1sBMrFt6cYdcs8bC+Sku98WIxKTQfdPkaenLZMvd+6xZ1emRYiDUP/D6fvj1UM9aM4DaQOF22uc+jJ7+eXK0eY9jc3yxpw1cu1Tn4gRiclRt78pN1XOlI+WbMp+gQKd4lNuhpf8Cml58bAOWriOtIHCHVb8TBnTPH7tll3yt8kLpe89iRKq371jojx24f/Ihs4H525XugeXl7HffPZKgzZATtrgqHAD2A/ApwDmAFgA4O5M+1C4s8SvlLEMgtXY1CzvLFgv1z9XJT2Gx+SI4TG54fnP5P1/jpOmTp2ctcsq7uun9xv2+C097sDjtHArAJ1bfm8PoArAaVb7ULizxK+UMZPzrutcKg8NHiYDKiaKEYlJ/xH/lr+/s0jWfFVnba8TdgXJ+w2SLdmSD23Ic+wIt0psr4dSqgTAhwCGiUiV2Xb9+vWT6upq7eOSFqJRYOhQoK5u73slJfv+nYxSQDye+3mLihJfZQBxKEw/4mRE+1yE947uj+aiYpxZMxvlZxyD7//iSrQrLkq7nyt2RaNARQVQWwuUlQEjRgDl5bkf1y49egA1NW3fNwxg5UqvrcmeoFxPkhal1EwR6ae1sY66AygGMBvATgB/M9lmKIBqANVlZWUePaPyEDdSxjJ18w1DNpUcKE8MuEK+94tnxIjE5JRfVspfz7xeag481PxchdL9ZnzYXcIehnIIuDU4CeAgAFMA9LbajqGSLLD68ObSzbXYNx6Py8dLN8vN906Qo297TYxITK659l6ZeNwZsru4XWaRKpTud6E8oPygUD5DGrgm3Ilj4y4At1ltQ+FOwYkiVNl6JWlE56v9Ossz5/1Uzn5gihiRmJx411ty930vy5Le/dMLlJVI5YO3FMYiYfkCH4pf46hwA+gG4KCW3/cHMB3AxVb7eCrcfguHE196ux9eO21u6ebHAan+1nFy6+BbpedvXxEjEpPLH/9QXqpOKaFaaCLl9AQqYg+Gob7GaeH+DoBZAOYCmA/gzkz7eCbcfotMrqKcKfXNgfDE9qOOlVF9LpIL/vtRMSIxOf43L0rFecNkwUnftW5XoYgUPT5/4fX/GjvCbSurRBfPskr8Hu3XOb9Z5oUO6dqh2eb5a7YhWlWD16trURdXOGH9UpTPnoxLF05D53YKGDkysWGhZxm4nRlDrDHLpBo5suA+i3ayStq5bYyr1Nbae9+P85eVpRfaTJSUJITUxjnrGpoQm7MO0aoazFm9Dfu1L8KlpxyO675aiJPGPQSVLNDAvl+YmprE30BhfWHM7k9Zmfe2FCKtn7VCdyDsouua23l5Fipxu5uVGjIYNmzfv3UK6+tM4zYLo2i2eVHXMrnr8tuk912JEqrn/v0D+deHy2VrnUUJVXZRE/gdbiOkBRRMrRI3v3Q6gqu7lFXyA0BHtDXsqi9uL68eP0iuuu6viRKqv3tVfn3NnfLpU2P1ijxxUGgvhRTTJ4GlcIRbxL0vnU6lOsD+4rFWx9XIZlj+7BgZcf/L0uc348SIxOSsoSPlqf6Xy5b9u9h7cNHjJiRQFJZwZ4OO2Ot4x9l4qGaefGmpaSphQ1GxTOr5XSm/5p5ECdXIG3Lj6GqZ3v98aUYaO3XE148QQdA826DZQwoaCrcVuoKl63Fn46FqCsaqXifL/WcMkX43j0qUUL3xOXn09Ktlw7EnJjbINdzhpXAFLZYcNHtIwWNHuMOVDuhEkRzdFMJ0aUqpuJC21BwXfLBoI6JVtZiycD0A4Jxl1SifNQlnrfgcxRLfm6rmdzqkHYJma9DsIQWP40Wm7L5c8bid8pDseKnJE2TS7de5s2Pe6vpt9fLwu4vl9HvfFSMSk1MrJsoDZwyRVV26mXv5dq+Jn6GBoA2GBs0eUvAg9KGSXCrkaVTCsx3y0AmbZPEQaW6Oy7TFG+UXo6rlyNvfFCMSkyHPzJDJj4+XhvYd0p9HqexqmHgdGki1Syd10kv8XG2IkDSEW7jTCYzVQGHqQq46xZrsCpjuQKWmCG3esVue/GCpnHnf+2JEYtLnDxPl3kt/LSsO/lZC4FJTDFNfma5fOtHxMosk3TXWTZ30CrPPwbBhjH0TXwi3cOsOCqYTHqe88mxtsuhmx+NxmbFss9wy5nM55g+TxIjE5KonP5bXHhsvuzt3ya69qVg9lLwMDZhdL7upk61tcsv7zaVnR4jD2BHu4A1OWtX2UGrf/6UODrpVd0JnoBJIO7C1ra4RE2atRrSqFks37sQB+7XDFad0R/mAMhxz6AHmg2TpyDQYajXgBng3GOfUffCjjgVrlxCfCPfgZCbv1un4tS7J3lm6cEZSdzoej8vnNV/KbS/OlmPvSHjXlz32oYz/rFbq9jTte1zdMExxcWZv08qr9jLG7dR98MP7pcdNfAKhDpVYdes1p4N7Ik5putk7djdK5YyVctE/pokRiUmvP06W2yfMlXmrt5ofx8mBz0yi49Wgm51JRlb4kfnB/G7iE+EWbpHEAFHql9bOUl3ZiFMOojZ/zVb5w4S5cvwfJ4sRicmF/5gmoz9ZKdvrLYo8JZ833UBeaWl2bQiK6FRWps8ksWOPX94vs0qID4RfuEUCP6uvbk+TvPhZrVz22IdiRGLSs2KS/O7F2TJz5FhXqX11AAAMdklEQVSJ+zkAlxrSyeYB4BS5DhYH6UFEiMvkh3B7iQ3PbsmG7fKnifPlxJYSquc8MEWenb5ctu5qCJbQBMEWnVBHJjvp/ZICwY5wBy+rxA8yZBLsaWrGW/PXY0xVLapWfIn2xQoX9v4mygeUYcARh0Apldg+SNOo7djiRCmBbG0I0jUjxEcKZwUcpzBZBaXm+FMwZvJCvFS9Gl/uakDZISUYftFxuLJvd3Tt3LHtcfxekSdZgM0eyKm2pKbcObkSzogR6dP5klf28fuaERJCKNzAPgLTWFSM947qj2jfH2C60QfF01fg3F7fQPkAA987uiuKipT5cfxcBks31zzVloqKtvvU1SXez1W4dZal4tJhhNiGwg0A5eVY26Aw7pWPMN4YgA0HlOKb7eP47aCeuObUw3Fol/30jqPjYTpFanhj587Mop3OFrc93vJy6weAl9eMkDyhoIW7OS6YtngTolU1eH/xgZATB2NQz24YMcDAoGO7oV1xkb0DerXwabrwhhVKmdvit8fLxWIJsU1BDk5u3LEbL362CmM/XYU1W+vRtXNHXHNqd1x7ahkOP6TE3ZO7WVM8HZkG+fyYVk4IaQMHJ9MQjws+Wb4F0aoavLNgA5rigoFHl6LiB71wbq9D0aGdiXftZMaFUwOBumEMnZADPV5Cwodu3qCdl+N53Dnk8m7ZuUdGTl0mg+6fIkYkJifd/bb8X2yBLNu4Q++8ZqU/dWtgu1GTWrdaYTY5z8ybJsQXkFcTcLKYSBKPx+XTFVvk12P3llC94omPZMLnq6S+ocl0vzaYCaTOdHyzeh1O1N7QObbZw8BKmIMwaYeQAiW/hNvGrMZt9Q3y/Ecr5LwHPxAjEpPed74ld742T/6zbrv58a2ETLdyXzp77NQVz2XBYd0HSes+VsLMyniE+IYd4Q7+4KRGfeS5q7ciOqMWE+esRX1jM77T/UCUDyjDJSd9CyUdLML4mQbm7AwCptZrtqornowTA4G6cfhMsxRZi5oQ3whnPW6bS27tPKqnjKmqkR88kiihetwdk2X4K3Nk7iqLEqqp6JRB1V1GTdfjzmYVGKfIVDuEHjchvoHQhUqsuvAp//uiWw+546Jb5ITb3xAjEpMLHpoqox5+UbYdfax9MdQtgpQstFZrEtpYbMEXsnlQ+W0zIQVC+IQ7g6DUj6qUV866Sn5Yfp8YkZgcM/wNuXX8LKleuUXio3MQm2w9zHS9AyfraruF7mLKQSkLS0gBET7hNvF8l5Z2lz+/sUC+86e3xYjE5Oz7p8jT05bJlzv37N03l+69kx5mWMIMuul+9L4J8RQ7wh2MwcmkQbOGonZ4u+fpGNPnQnxinIR2RQoX9D4M5QPKcPqRpXtLqLZiNaA2enTbQTtg3/cGDwYmTcp98km+Deyx3CohnmJncDKjcCulDgcwCsBhAOIARorIw1b72BbuaBR1N92CR/tcipe+cy42dzoY3bdtxHU9D8BVP78E3Q5IU0K1FTOBKS0F6uv3zRhp3z4hpA0Ne99zanp3vgldvj2ICAk4doRbp4pSE4DfiUgvAKcBuFkpdXwuBrahvBwdH38Uk3oPwslrF+H5Dx7HtIEdcNOtV1qLNpDwkEtS6ou0/p1aLa+xcV/Rbt2moiI3+63sCGuVO7MiUyy3Soj/6MZUWl8AXgdwntU22U7A0ZrVaDYwmPqenckzTq0ank/TxRnjJsRT4FaMWynVA8A0AL1FZLvZdq5VB7RTyc7JCnqFiltLmhFC2uBojDvpoJ0BTAUwQkQmpPn/UABDAaCsrKxvja5o2iFTHDlZaA45BNixo21oJBWWMCWEBADHhVsp1R5ADMDbIvJgpu1d87gzZZCkeuPt2wPNzeaDacXFwAsvULQJIb7j6OCkSuTfPQtgoY5ou4rVgFm6tRMbG4GDDzY/XjxO0SaEhA6drJKBAH4M4Byl1OyW12CX7UqPVeaG2eICX36ZSA1MBzMkCCEhJKNwi8iHIqJE5Dsi0qflNckL49pQXp6IRxtGIjxiGHvj01be+MMP51eqHiGkoLG5Gm4AKC9PDETG44mfraEOK2/cSvAJISRkBEu4o9FE5khRUeJnNKq/byZxNhN8QggJGcFZLNiJhXTLyynIhJC8Jzged7qsEKemo/tBLr0HQgixIDget1lWiNn7QcaJ3gMhhJgQHI87n4oa5VvvgRASKIIj3PlUXS+feg+EkMARHOHOp5S9fOo9EEICR3CEG/A/Zc+pAcV86j0QQgJHsITbT1oHFGtqEoWsWgcUsxHvfOo9EEICR/CF26u0OqcHFP3uPRBC8pbgpAOmw8u0Og4oEkJCQrA9bi/T6jigSAgJCcEWbi+9YA4oEkJCQrCF20svmAOKhJCQEGzh9toL5oAiISQEBFu46QUTQkgbgp1VArBUKyGEpBBsj5sQQkgbKNyEEBIyKNyEEBIyKNyEEBIyKNyEEBIywivcXNOREFKgBD8dMB1c05EQUsCE0+N2u/gUvXlCSIAJp8ftZvEpevOEkIATTo/bzeJTXKGdEBJwwincbhaf4oIKhJCAE07hdrP4FBdUIIQEnHAKN+BeCVYuqEAICTjhFW63YClZQkjACWdWiduwlCwhJMDQ4yaEkJCRUbiVUs8ppTYqpeZ7YRAhhBBrdDzu5wFc6LIdzsPZj4SQPCVjjFtEpimlerhvioNw9iMhJI/Jzxg3Zz8SQvIYx4RbKTVUKVWtlKretGmTU4fNDs5+JITkMY4Jt4iMFJF+ItKvW7duTh02Ozj7kRCSx+RnqISzHwkheYxOOuBYAJ8AOFYptVopdYP7ZuUIZz8SQvIYJSKOH7Rfv35SXV3t+HEJISRfUUrNFJF+OtvmZ6iEEELyGAo3IYSEDAo3IYSEDAo3IYSEDAo3IYSEDFeySpRSmwDUZLl7VwCbHTQnDLDNhUEhthkozHZn02ZDRLRmL7oi3LmglKrWTYnJF9jmwqAQ2wwUZrvdbjNDJYQQEjIo3IQQEjKCKNwj/TbAB9jmwqAQ2wwUZrtdbXPgYtyEEEKsCaLHTQghxALfhFspdaFSapFSaqlSania/3dUSo1v+X9V6JZPS4NGm3+rlPpCKTVXKfWeUsrww04nydTmpO2uVEqJUir02Qc6bVZKXd1yrxcopcZ4baPTaHy2y5RSU5RSs1o+34P9sNNJMi2krhI80nJN5iqlTnHs5CLi+QtAMYBlAI4E0AHAHADHp2xzE4AnW36/FsB4P2z1uM1nAyhp+X1YIbS5ZbsDAEwDMANAP7/t9uA+HwNgFoCDW/7+ht92e9DmkQCGtfx+PICVftvtQLvPBHAKgPkm/x8MYDIABeA0AFVOndsvj7s/gKUislxEGgCMA3BZyjaXAXih5feXAXxfKaU8tNFpMrZZRKaISOtimTMAdPfYRqfRuc8AcA+A+wDs9tI4l9Bp8/8AeFxEvgIAEdnosY1Oo9NmAdCl5fcDAaz10D5XEJFpAL602OQyAKMkwQwABymlvunEuf0S7m8DWJX09+qW99JuIyJNALYBKPXEOnfQaXMyNyDxtA4zGduslDoZwOEiEvPSMBfRuc89AfRUSn2klJqhlLrQM+vcQafNfwIwRCm1GsAkALd4Y5qv2P3Oa9POiYNkQTrPOTW9RWebMKHdHqXUEAD9AJzlqkXuY9lmpVQRgIcA/NQrgzxA5z63QyJcMgiJXtV0pVRvEdnqsm1uodPmHwF4XkT+rpQ6HcDoljbH3TfPN1zTML887tUADk/6uzvadp2+3kYp1Q6J7pVVtyTo6LQZSqlzAVQAuFRE9nhkm1tkavMBAHoD+EAptRKJOODEkA9Q6n62XxeRRhFZAWAREkIeVnTafAOAFwFARD4BsB8S9TzyGa3vfDb4JdyfAThGKXWEUqoDEoOPE1O2mQjg+pbfrwTwvrRE/ENKxja3hA2eQkK0wx73BDK0WUS2iUhXEekhIj2QiOtfKiJhXvdO57P9GhID0VBKdUUidLLcUyudRafNtQC+DwBKqV5ICPcmT630nokAftKSXXIagG0iss6RI/s4IjsYwGIkRqMrWt77MxJfXCBxY18CsBTApwCO9HsU2YM2vwtgA4DZLa+JftvsdptTtv0AIc8q0bzPCsCDAL4AMA/AtX7b7EGbjwfwERIZJ7MBnO+3zQ60eSyAdQAakfCubwBwI4Abk+7z4y3XZJ6Tn23OnCSEkJDBmZOEEBIyKNyEEBIyKNyEEBIyKNyEEBIyKNyEEBIyKNyEEBIyKNyEEBIyKNyEEBIy/h8zOc8hujGfOgAAAABJRU5ErkJggg==\n", 215 | "text/plain": [ 216 | "" 217 | ] 218 | }, 219 | "metadata": {}, 220 | "output_type": "display_data" 221 | } 222 | ], 223 | "source": [ 224 | "[a,b]=linear_regression(X, y, 5000 , 1)\n", 225 | "linear_fit = b*X+a\n", 226 | "plt.plot(X,y,'ro')\n", 227 | "plt.plot(X,linear_fit)\n", 228 | "plt.show()\n" 229 | ] 230 | }, 231 | { 232 | "cell_type": "markdown", 233 | "metadata": {}, 234 | "source": [ 235 | "We can also use this parameters to perform future predictions:" 236 | ] 237 | }, 238 | { 239 | "cell_type": "code", 240 | "execution_count": 7, 241 | "metadata": {}, 242 | "outputs": [ 243 | { 244 | "data": { 245 | "text/plain": [ 246 | "4.975218592849181" 247 | ] 248 | }, 249 | "execution_count": 7, 250 | "metadata": {}, 251 | "output_type": "execute_result" 252 | } 253 | ], 254 | "source": [ 255 | "def predict(z,theta):\n", 256 | " return theta[0]+theta[1]*z\n", 257 | "\n", 258 | "predict(1,[a,b])" 259 | ] 260 | }, 261 | { 262 | "cell_type": "markdown", 263 | "metadata": {}, 264 | "source": [ 265 | "In future notebooks we will investigate how to evaluate the performance of such predictors. We also point out that even though this algorithm was implemented using only one dimensional features, the idea generalizes trivially to higher dimensional data." 266 | ] 267 | } 268 | ], 269 | "metadata": { 270 | "kernelspec": { 271 | "display_name": "Python 3", 272 | "language": "python", 273 | "name": "python3" 274 | }, 275 | "language_info": { 276 | "codemirror_mode": { 277 | "name": "ipython", 278 | "version": 3 279 | }, 280 | "file_extension": ".py", 281 | "mimetype": "text/x-python", 282 | "name": "python", 283 | "nbconvert_exporter": "python", 284 | "pygments_lexer": "ipython3", 285 | "version": "3.6.4" 286 | } 287 | }, 288 | "nbformat": 4, 289 | "nbformat_minor": 2 290 | } 291 | -------------------------------------------------------------------------------- /05. A note on gradient descent and convergence.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# A note on gradient descent and convergence\n", 8 | "\n", 9 | "We have seen in previous notebooks that the gradient descent algorithm is a very useful tool to find minima of functions with sufficiently nice convexity propertys. Although, it is important to remark that under the strongest convexity properties, the algorithm can fail to reach a minimum. We summarize the algorithm as follows. If $f$ is the function to optimize\n", 10 | "\n", 11 | "\\begin{align*}\n", 12 | "& \\ \\text{intialize } x \\text{ to a random value } x_0 \\\\\n", 13 | "& \\ \\text{for } i = 1\\dots \\text{ num iterations do} \\\\\n", 14 | "& \\qquad \\ \\text{compute the gradient }\\nabla J = \\nabla J(x) \\\\\n", 15 | "& \\qquad \\ \\text{update the parameters }\\theta \\text{ by } \\theta \\leftarrow \\theta - \\alpha\\nabla f \\\\\n", 16 | "& \\ \\text{return } x\n", 17 | "\\end{align*}\n", 18 | "\n", 19 | "Here $\\alpha$ is the learning rate and represents the weight with by which we multiply the gradient to substract it from the current value of the parameters.\n", 20 | "\n", 21 | "Consider the function to optimize given by $J(x)=x^4$ so then $\\nabla J(x)=4x^3$. Then $f$ is strictly convex, smooth and has a global minimum at zero. When we start the algorithm, we have to make a random guess $x_0$ for the parameter $x$, which we think is close to the optimum $x_\\min$. We also have to choose the learning rate $\\alpha$ before we initialize the algorithm. A poor choice of these two parameters can lead not only to the algorithm not to converge, but to diverge and escape from the minimum. \n", 22 | "\n" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": 2, 28 | "metadata": {}, 29 | "outputs": [ 30 | { 31 | "data": { 32 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xl8nGW5//HPlb3Zm61pm7RpukPpRqCh7C0oexWVRVZFKvhT8ehPxcM5Hs/x5wFcQPS4UIUDKCKIYEEWaVlk6ULT0r3pQrokTdNsbZq22ef6/TETCCFNp0lm7lmu9+s1ZpYnc399SOea57mf+75FVTHGGBO9YlwHMMYY45YVAmOMiXJWCIwxJspZITDGmChnhcAYY6KcFQJjjIlyVgiMMSbKWSEwxpgoZ4XAGGOiXJzrAP7IycnRoqIi1zGMMSasrF69ul5Vc4+3XVgUgqKiIsrKylzHMMaYsCIiu/3Zzk4NGWNMlLNCYIwxUc4KgTHGRDkrBMYYE+WsEBhjTJQLWCEQkYdFpFZENvZ6/msislVENonIjwPVvjHGGP8E8ojgEeCink+IyPnAAmC6qp4M/DSA7RtjjPFDwAqBqr4JNPZ6+nbgHlVt821TG6j2AV7fWsuv39gRyCaMMSbsBbuPYBJwtoisFJF/ishpx9pQRBaKSJmIlNXV1Q2osWU76vn50u20d3oGmtcYYyJesAtBHDAcKAW+DTwlItLXhqq6SFVLVLUkN/e4I6T7NKMwk/ZOD+U1hwYc2BhjIl2wC0EV8Ix6vQt4gJxANTazMBOAdZUHA9WEMcaEvWAXgr8B8wBEZBKQANQHqrHRmcPISU3gPSsExhhzTAGbdE5EngDOA3JEpAr4D+Bh4GHfJaXtwE2qqgHMwMzCTDsiMMaYfgSsEKjqtcd46fpAtdmXGQWZLN1SS1NLBxnD4oPZtDHGhIWIH1k8c4y3n2BDVZPjJMYYE5oivhBML/B1GFfZ6SFjjOlLxBeCjGHxFOem8N4eKwTGGNOXiC8EADMLMllbeZAA9ksbY0zYio5CMCaT+sNtVDe1uo5ijDEhJyoKwYwCG1hmjDHHEhWFYOrIdBJiY1hrhcAYYz4mKgpBQlwMJ41Kt0JgjDF9iIpCAN55hzZUNdHZZTORGmNMT1FVCFo6uthee9h1FGOMCSlRVQgAOz1kjDG9RE0hGJudTGZyvF05ZIwxvURNIRARZvgGlhljjPlQ1BQC8K5Ytm1/M0faOl1HMcaYkBFVhWBWYSYehQ17bSZSY4zpFrBCICIPi0itbxGa3q/9XxFREQnYMpV9mV6QAdgIY2OM6SmQRwSPABf1flJECoELgT0BbLtP2amJjMlKtn4CY4zpIWCFQFXfBBr7eOl+4DuAk6lAZ9jSlcYY8xFB7SMQkSuAvaq6Lpjt9jSzMJPqplZqD9lMpMYYA0EsBCKSDNwFfN/P7ReKSJmIlNXV1Q1ZjpmF3n4COz1kjDFewTwiGA+MA9aJyC6gAFgjIvl9bayqi1S1RFVLcnNzhyzEyaMyiIsRKwTGGOMTF6yGVHUDkNf92FcMSlS1PlgZAJLiY5kyMs3WMDbGGJ9AXj76BLAcmCwiVSJyS6DaOlEzCzNZX9mEx2NLVxpjTCCvGrpWVUeqaryqFqjqQ71eLwr20UC3GQWZNLd1UlFvM5EaY0xUjSzuNmuMdybS9/bY6SFjjInKQlCck0paYpz1ExhjDFFaCGJihOmFGXblkDHGEKWFALz9BOX7mmnt6HIdxRhjnIraQjCzMJNOj7Kp2mYiNcZEt6guBABrK60QGGOiW9QWgrz0JEZlJFk/gTEmJO2qP8KZ97zGsh2Bv8o+agsBwOyxw1m1sxFVG1hmjAktKyoa2Huwhbz0pIC3FdWFoLQ4m5pDrexuOOo6ijHGfMSKigZyUhMZn5sS8LaivhCAd4cbY0yoUFVWVDQypzgLEQl4e1FdCMbnppCTmsjKnX2tn2OMMW7sbjhKzaHWD76sBlpUFwIRYU5xFisqGqyfwBgTMlbu9J6lOKM4KyjtRXUhAO/poX1NrexptH4CY0xoWFHRSE5qAuNzU4PSXtQXgu6Ka/0ExphQ4O0faGBOcXZQ+gfACgHjc1PJSU1gRYX1Exhj3NvTeJR9Ta2UjgvOaSGwQuDrJ8hmpfUTGGNCQPfZiWB1FENgVyh7WERqRWRjj+d+IiLlIrJeRJ4VkcxAtX8iSsdlUd3USmVji+soxpgot7KikeyUBCbkBad/AAJ7RPAIcFGv55YA01R1OrAN+F4A2/ebjScwxoSC7v6B0iD2D0Bgl6p8E2js9dwrqtrpe7gCKAhU+ydiQl4q2SkJVgiMMU5VNrZQ3dRKaZAuG+3mso/gi8BLDtv/gIhQWpxt4wmMMU51fxmdE8T+AXBUCETkLqATeLyfbRaKSJmIlNXV1QU805xibz9B1QHrJzDGuLGiooGslAQmBrF/ABwUAhG5CbgMuE77+fqtqotUtURVS3JzcwOeq7ufYLmdHjLGOKCqrNzZSGmQ5hfqKaiFQEQuAr4LXKGqITWUd2JeKlnWT2CMcaTqQAt7D7YE9bLRboG8fPQJYDkwWUSqROQW4H+ANGCJiKwVkd8Gqv0T5e0nyGJlha1PYIwJvuUOxg90iwvUG6vqtX08/VCg2hsKc8Zl8+KGGqoOtFCYlew6jjEmirjqHwAbWfwRNp7AGOPKyopG5owLfv8AWCH4iA/7CWzeIWNM8FQ2HnXWPwBWCD4iJkaYMy7LjgiMMUHlYn6hnqwQ9FJanM3egy1U2voExpggWVHRyPDkeCf9A2CF4GPm2PoExpggW1HRwJxx2cTEBL9/AKwQfMykvDSGJ8fbOsbGmKD4sH8guPML9WSFoBdvP0G2HREYY4Ki+0tn6Xg3/QNghaBPpcVZVB2wfgJjTOCtqGhgeHI8k/LSnGWwQtCH7pn/7PSQMSbQVlQ0cPq4LGf9A2CFoE+TR6SRmRxvp4eMMQFVdeAoVQfcjR/oZoWgD93jCVbutEJgjAmclb7Bq1YIQlRpcTaVjS1UHbB+AmNMYKyoaCAzOZ7JI9z1D4Cfk86JyGhgbM/tfUtRRqzuCr2yopGCU20COmPM0Fuxs4E5jvsHwI9CICL3AlcDm4Eu39MKRHQhmDwijayUBN7aXsdnTg2JpZWNMRFkV/0RKhtbuOXMca6j+HVE8Clgsqq2BTpMKImJEc6dlMsbW2vp8iixjiu2MSayvFZeC8D5U/IcJ/Gvj6ACiA90kFA0b0oeB452sLbygOsoxpgI8/rWWsbnpjA2O8V1FL8KwVFgrYg8KCK/6L4d75dE5GERqRWRjT2eyxKRJSKy3fdz+GDCB9o5k3KJjRFe3VLrOooxJoIcbutkRUUD80LgaAD8KwTPAT8ElgGre9yO5xHgol7P3Qm8qqoTgVd9j0NWxrB4SsYO/+AQzhhjhsLb2+vp6FLmTRnhOgrgRyFQ1UeBJ/iwAPzJ99zxfu9NoPfQ3AVA9+8+irf/IaTNm5JHeU0z1QdbXEcxxkSI18trSUuKo6QoNE6KHLcQiMh5wHbgV8CvgW0ics4A2xuhqvsAfD+PeVwkIgtFpExEyurq6gbY3OB1H7rZUYExZih4PMprW2s5Z2Iu8bGhMZTLnxQ/Az6hqueq6jnAJ4H7AxsLVHWRqpaoaklubm6gmzumCXmpFGYN43UrBMaYIbCp+hB1zW0h0z8A/hWCeFXd2v1AVbcx8KuI9ovISADfz5D/dBUR5k3O453362nt6Dr+LxhjTD9eK69FBM6b7O4Lbm/+FIIyEXlIRM7z3X6Hf53FfXkOuMl3/yZg8QDfJ6jmTR1Ba4eH5e/b3EPGmMF5rXw/MwszyU5NdB3lA/4UgtuBTcDXgTvwjjC+7Xi/JCJPAMuBySJSJSK3APcAF4rIduBC3+OQN2dcFsPiY62fwBgzKHXNbayramLe5NA5LQR+jCz2jSi+z3fzm6pee4yX5p/I+4SCpPhYzpyQw2vltfyXKiI2ytgYc+Le2Bo6o4l7OuYRgYg85fu5QUTW974FL2JomD81j70HW9i2/7DrKMaYMPVaeS0j0hM5eVS66ygf0d8RwR2+n5cFI0ioO3/yh5eRTs53O2WsMSb8tHd6eGt7PZfPGBlyZxWOeUTQfb0/8BVV3d3zBnwlOPFCR35GEiePSrfLSI0xA1K2q5HDbZ0hM5q4J386iy/s47mLhzpIOJg3JY+y3Y0cPNruOooxJsy8Wl5LQlwMZ05wuxpZX/rrI7hdRDYAU3r1D+wENgQvYug4f0oeHoV/bnM30tkYE55eL6+ltDib5AS/1gMLqv6OCP4EXI73Wv/Le9xOVdXrgpAt5MwoyCQ7JcEuIzXGnJCd9UeoqD/C/BC7Wqhbf30ETaq6C3gAaOzRP9AhInOCFTCUxMYI507O5Z/b6ujyqOs4xpgw0f3lMZSmlejJnz6C3wA9r5k84nsuKs2fMoKDRzt4b48tVmOM8c/r5bVMzEulMCs01z/3pxCIqn7w9VdVPfi56H0kOntSDnExwqt2esgY44fDbZ2s3Bk6i9D0xa+lKkXk6yIS77vdgXf5yqiUnhRPSdFwu4zUGOOXt7fX0dGlITeauCd/CsFtwFxgL1AFzAEWBjJUqJs/ZQTlNc3stcVqjDHH8Vp5LelJcZw6NjQWoemLPyuU1arqNaqap6ojVPXzqhrVX4fPt8VqjDF+8HiU18rrOGdS6CxC05fjnusXkVzgVqCo5/aq+sXAxQpt43NTGJudzOvltdxQOtZ1HGNMiNpY3UT94TbmTw3d00LgX6fvYuAtYClgK7PgXazm/Ml5PPHuHlrauxiWEOs6kjEmBHUvQnPupPAvBMmq+t2AJwkz86bk8ciyXbyzo54LTgq9uUOMMe69uqWWWYWZZKUkuI7SL39OWv1dRC4ZykZF5F9EZJOIbBSRJ0QkaSjfPxhKi7PJTI7n7+urXUcxxoSgnfVH2LC3iUtOGek6ynH5UwjuwFsMWkTkkIg0i8ihgTYoIqPxrnZWoqrTgFjgmoG+nysJcTFccspIXtm8n6Ptna7jGGNCzHNrqxGBy6aPch3luPy5aihNVWNUdZiqpvseD3ZVhThgmIjEAclAWH6tXjBjFEfbu1i6xa4eMsZ8SFVZvG4vpeOyyc8I/RMe/lw1dE5fz6vqmwNpUFX3ishPgT1AC/CKqr4ykPdy7bSiLEZmJPHc2r1cMSP0q74xJjg2VR+iou4IC88udh3FL/50Fn+7x/0k4HRgNTBvIA2KyHBgATAOOAj8RUSuV9U/9tpuIb6Ba2PGjBlIUwEXEyNcMWMUD729kwNH2hke4h1CxpjgWLx2L/GxwsXTQr9/APw7NXR5j9uFwDRg/yDavADYqap1qtoBPIN35HLvdhepaomqluTm5g6iucC6fMYoOj3KSxtrXEcxxoSALo/y3Lpqzp2UR0ZyvOs4fhnIULcqvMVgoPYApSKSLN6FO+cDWwbxfk6dPCqd8bkpLF6713UUY0wIeHdnI/sPtbFgZvicLvanj+CXQPfsozHATGDdQBtU1ZUi8jSwBugE3gMWDfT9XBMRFswczf1Lt1F9sIVRmcNcRzLGOPTcur0kJ8RywdTwGV/kzxFBGd4+gdXAcuC7qnr9YBpV1f9Q1SmqOk1Vb1DVtsG8n2tXzBiFKjamwJgo19bZxYsbavjkyflhNeNAf2sWv+q7e5KqPuq7Pa6q7wQpW9goyklhRmEmi9daITAmmr25rZ6mlg6uCKPTQtD/EcFIETkXuEJEZonI7J63YAUMFwtmjGJT9SF21Da7jmKMcWTx2r1kpSRw1oQc11FOSH+F4PvAnUABcB/wsx63nwY+Wni5bPpIYsQ7mtAYE30Ot3WydMt+Lj1lZEhPOd2X/havf1pVLwZ+rKrn97oNaAxBJMtLT2Lu+BwWr6umx8qexpgosWRzDa0dnrC6WqibP+MIfhiMIJHgipmj2N1wlHVVTa6jGGOCbPHaakZnDmP2mNBdiexYwuv4JcRdNC2fhLgYG1NgTJRpONzGW9vruWLmKGJixHWcE2aFYAilJ8Vz/uRcnl+3jy6PnR4yJlq8uMH7bz5c5xw7biEQkT/485zxWjBzNPWH21j+foPrKMaYIFm8tppJI1KZkp/mOsqA+HNEcHLPByISC5wamDjhb96UPFIT4+z0kDFRorLxKGW7D7Bg5mi8s+aEn/4GlH1PRJqB6b4FaQ75HtfiXcfY9CEpPpZPnpzPyxtraO2wJZ6NiXTP+2YUCNfTQtD/5aN3q2oa8BPfgjTdi9Jkq+r3gpgx7CyYOYrmtk7e2FrnOooxJsCeW1vN7DGZFGYlu44yYP6uWZwCICLXi8h9IjI2wLnC2tzx2eSkJvDcOjs9ZEwk21rTTHlNMwtmjnYdZVD8KQS/AY6KyAzgO8Bu4LGApgpzcbExXDZ9FEu31NLU0uE6jjEmQBav3UtsjITFAvX98acQdKp3qOwC4AFVfQAIz67xIPrM7ALaOz38dXWV6yjGmABo6+ziqbJKzpuUS25aous4g+JPIWgWke8BNwAv+K4aCo9ldxw6pSCDWWMy+cOK3XhsTIExEeelDTXUH27nxrlFrqMMmj+F4GqgDfiiqtYAo4GfBDRVhLh5bhE764/w5nbrNDYm0jyybBfFOSmcHWYzjfbFn7mGaoDHgQwRuQxoVdVB9RGISKaIPC0i5SKyRUTOGMz7haqLp40kJzWRx5bvdh3FGDOE1lUeZG3lQW44Y2xYTinRmz8ji68C3gU+B1wFrBSRzw6y3QeAl1V1CjCDMF6zuD8JcTF8fs4YXt9ay+6GI67jGGOGyGPLd5OSEMtnTy1wHWVI+HNq6C7gNFW9SVVvBE4H/n2gDYpIOnAO8BCAqrar6sGBvl+ou27OGGJF+IMdFRgTERoOt/H8+mqunF1AWlJkdJf6UwhiVLW2x+MGP3/vWIqBOuB/ReQ9Efl99ziFSDQiPYmLpuXzVFklR9s7XccxxgzSn1dV0t7p4cYzImc4lT8f6C+LyD9E5GYRuRl4AXhpEG3GAbOB36jqLOAI3pXQPkJEFopImYiU1dWFd2frTXOLONTayd/es9XLjAlnnV0eHl+xmzMnZDNxRORcRe9PZ/G3gQeB6XjP5y9S1e8Mos0qoEpVV/oeP423MPRud5GqlqhqSW5u7iCac69k7HBOGpnOY8t32eplxoSxpVv2U93Uyo1nFLmOMqT6m3RugoicCaCqz6jqN1X1X4AGERk/0AZ9VyFVishk31Pzgc0Dfb9wICLcNHcs5TXNrNzZ6DqOMWaAHlm2i9GZw7hg6gjXUYZUf0cEPwea+3j+qO+1wfga8LiIrAdmAv89yPcLeQtmjiYzOZ7Hlu9yHcUYMwBba5pZUdHI9aVjiY2AS0Z7iuvntSJVXd/7SVUtE5GiwTSqqmuBksG8R7hJio/l6pJCfv/2TvY1tTAyY5jrSMaYE/DY8l0kxsVwzWmFrqMMuf6OCJL6ec0+xQbg+tKxeFR5fMUe11GMMSegqaWDZ9bs5YoZoxiekuA6zpDrrxCsEpFbez8pIrcAqwMXKXIVZiUzf8oInnh3D22dtmiNMeHi6dVVtHR0cVMEzCvUl/5ODX0DeFZEruPDD/4SIAH4dKCDRaqb5xaxdMt+Xli/jytnR8aoRGMimcej/GH5Lk4dO5xpozNcxwmI/lYo26+qc4H/BHb5bv+pqmf4rvwxA3DmhGzG56bwqI00NiYs/HN7HbsajkbUALLe/BlH8Lqq/tJ3ey0YoSKZ91LSog8mrTLGhLZHl+0iNy2Ri6eF9+Iz/RnMVBFmgK6cXUBqYhyPLdvlOooxph+76o/wxtY6Pn/6GBLiIvfjMnL/n4Ww1MQ4PntqAc+vr6ay8ajrOMaYY3jwzQriY4Xr5oxxHSWgrBA48uVzixERfvHqdtdRjDF92N1whL+UVXLt6WPIS+/vavrwZ4XAkZEZw7ihdCx/XVNFRd1h13GMMb08sHQ7sTHCV8+f4DpKwFkhcOj288aTFB/L/UvtqMCYULJ9fzPPrt3LTXOLIv5oAKwQOJWTmsgXzizi+XXVbNl3yHUcY4zP/Uu3kRwfy23nDnh+zbBihcCxhWePJy0pjvuWbHMdxRgDbNzbxIsbarjlrHFkReB0En2xQuBYRnI8C88uZsnm/ayzcQXGOHffkm1kDIvnlrOLXUcJGisEIeALZ41jeHI8P31lq+soxkS11bsP8Fp5LQvPKSZjWGSsR+wPKwQhIDUxjtvPG89b2+tZWdHgOo4xUetnr2wlJzWBL5xZ5DpKUFkhCBE3lBaRl5bIz17ZZstZGuPAsh31LHu/gdvPm0ByQn/zcUYeZ4VARGJF5D0R+burDKFkWEIsX503gXd3NfLW9nrXcYyJKqrKT1/ZSn56UsSPIu6LyyOCO4AtDtsPOVefVsjozGH89JWtdlRgTBC9vrWWNXsO8rX5E0iKj3UdJ+icFAIRKQAuBX7vov1QlRgXyx3zJ7K+qoklm/e7jmNMVPB4lJ+9so0xWclcVRJ5y1D6w9URwc+B7wCeY20gIgtFpExEyurq6oKXzLErZ49mXE4K9y3ZhsdjRwXGBNrLm2rYVH2IO+ZPJD42OrtNg/7/WkQuA2pVtd/lLlV1kaqWqGpJbm5ukNK5FxcbwzcumEh5TTN/37DPdRxjIlqXR7lvyTbG56bwqVmjXcdxxkX5OxO4QkR2AX8G5onIHx3kCFmXTx/FlPw07n2pnCNtna7jGBOx/rRyNztqD/PNCycTGyOu4zgT9EKgqt9T1QJVLQKuAV5T1euDnSOUxcQIP/zUNPYebLGpJ4wJkOqDLdz78lbOnpjDJafku47jVHSeEAsDpxVlcX3pGP73nZ22pKUxQ0xV+fe/baTLo/z3p09BJHqPBsBxIVDVN1T1MpcZQtl3LppCXloSd/51PR1dx+xXN8acoBc27OPV8lq+9YlJFGYlu47jnB0RhLD0pHh++KlplNc0s+jNCtdxjIkIB4+284PnNjG9IIOb5xa5jhMSrBCEuAtPGsGlp4zkgVe3876tZGbMoP3ohS0cONrBPVdOJy5KLxftzfZCGPiPK04iKS6G7z2zwcYWGDMIb2+v5y+rq/jyOcWcNCrddZyQYYUgDOSlJfFvl57Euzsb+fOqStdxjAlLLe1d/OuzGxiXk8LX5090HSekWCEIE58rKWDu+GzufnEL+w+1uo5jTNj5+dJt7Gk8yt1XnhKV8wn1xwpBmBAR/vvTp9De5eH7ize6jmNMWNm4t4nfvVXBtacXUlqc7TpOyLFCEEaKclL4lwsn8Y9N+3l5o00/YYw/Oro8fOfp9WSnJnLnxVNdxwlJVgjCzJfOGsfJo9L598WbaGrpcB3HmJD30Ns72bzvED9ccHJULT95IqwQhJm42Bju/cx0Go+086/PbrB1C4zpx8a9Tdy/ZBufPHkEF00b6TpOyLJCEIamjc7g/35iMi+s38eDNtDMmD7VH25j4WNlZKck8KNPn+I6TkizQhCmbju3mMumj+Tel8t5Y2ut6zjGhJSOLg9feXwNDUfaefCGEnJSE11HCmlWCMKUiPDjz05nSn46X3viPXbWH3EdyZiQ8cO/b+bdnY3c+5npnFKQ4TpOyLNCEMaSE+JYdMOpxMUItz5WRnOrdR4b8+SqPTy2fDe3nj0uqhebORFWCMJcYVYyv7puNjvrj/DNp9bZFBQmqq3efYB//9smzp6Yw3cvmuI6TtiwQhAB5o7P4d8uncqSzft54NXtruMY48T+Q63c/sfV5Gck8ctrZ9mEcifA9lSEuHluEZ+ZXcADr27n5Y01ruMYE1StHV18+Q+rOdzWye9uLCEzOcF1pLDiYvH6QhF5XUS2iMgmEbkj2BkikYjwo09PY0ZhJt96ai3b9je7jmRMUKgq31+8kbWVB7nvqhlMzk9zHSnsuDgi6AS+papTgVLg/4jISQ5yRJyk+FgevP5UhiXEcetjZRw40u46kjEB9+iyXTxVVsXX502wQWMD5GLx+n2qusZ3vxnYAljX/hDJz0jiwRtms+9gK5///UoaDre5jmRMwDy+cjc/eH4zF0wdwTcumOQ6Tthy2kcgIkXALGBlH68tFJEyESmrq6sLdrSwdurYLH5/Uwk76w9z9aIV1Nq01SYCPfz2Tu56diPzpuTxP5+fRUxMdC9APxjOCoGIpAJ/Bb6hqod6v66qi1S1RFVLcnNzgx8wzJ0zKZdHvnA61QdbuOrB5VQfbHEdyZgh8+s3dvBff9/MxdPy+e31p9r6AoPkpBCISDzeIvC4qj7jIkM0KC3O5g+3zKHhcDtXPbicysajriMZMyiqyv1LtvHjl7eyYOYofnntLBLi7OLHwXJx1ZAADwFbVPW+YLcfbU4dO5w/3VrK4bZOPvfb5VTUHXYdyZgBUVXuebmcB17dzlUlBdx31UwbKzBEXOzFM4EbgHkistZ3u8RBjqhxSkEGT9xaSkeXh6seXMHWGru01IQXj0f5z+c38+A/K7i+dAz3XDmdWOsTGDIurhp6W1VFVaer6kzf7cVg54g2U0em8+SXzyA2Bq5ZtJyNe5tcRzLGLx6PctffNvDIsl186axx/HDBNOsYHmJ2XBVFJuSl8tSXzyA5IY5rf7eCJZv3u45kTL+aWjr4yuNreOLdSr42bwJ3XToV79llM5SsEESZsdkpPPnlUsZkJXPrY2X84LlNtHV2uY5lzMes2XOASx54i6Vb9vNvl07lW5+YbEUgQKwQRKGC4ck885W53Dy3iEeW7eLKXy+zTmQTMjwe5TdvvM9Vv12OCDx12xl86exi17EimhWCKJUYF8sPrjiZ391Ywt6DLVz2y7d5Zk2V61gmytU1t3HT/77LvS+X84mTR/DC189m9pjhrmNFPCsEUe7Ck0bw0h1nM21UBt98ah3ffGotR9o6XccyUeit7XVc/MBbvLuzkR99ehq/+vxsMobFu44VFawQGEZmDONPt87h6/Mn8ux7e7n8l2/bVUUmaDq6PNz7cjk3PvwumcnxLP7qmVw3Z6z1BwSRFQIDQFxsDN+8cBJ/+lIpR9o7WfCrd/jXZzfYPEUmYFSVlzfu45P3v8lv3nifq0sKef6rZzElP911tKgjqqG/tGFJSYmWlZW5jhFr7NwtAAALFklEQVQ1Go+084tXt/PHFbuJj43h1nOKWXhOMamJca6jmQixalcjd7+4hTV7DjIxL5U7L57C/KkjXMeKOCKyWlVLjrudFQJzLLvqj/CTV7bywvp95KQmcMf8iVxz+hjibVi/GaAdtc3c+/JWlmzez4j0RL554SQ+M7vApooIECsEZsisrTzI3S9uYeXORsblpPDtT07m4mn5dg7X+K32UCv3L93Ok6v2kJwQx+3njeeLZ45jWILNGhpIVgjMkFJVXt9ayz0vlbNt/2Gmjkzn2tMLWTBjNBnJdmWH+ThVZc2egzy5ag/Pr9tHp8fDdXPG8rV5E8hOTXQdLypYITAB0eVR/rqmikeX7WJT9SES42K45JSRXH1aIXPGZdlRgqHxSDvPrKniyVWVbK89THJCLJdPH8VXzh/P2OwU1/GiihUCE3Ab9zbx5KpK/rZ2L82tnRRlJ3PVaYV8dnYBeelJruOZIPJ4lHfer+fPqyp5ZVMNHV3KrDGZXHNaIZdOH2UXGjhihcAETUt7Fy9t3MefV1Xy7s5GYmOE04uyOGtiDmdNyGHa6AybMjgCNR5p550d9byzo55/bqtjX1MrmcnxXDmrgKtPK2RyfprriFHPCoFx4v26wzy9uorXy2sp9617kJkcz9zx2Zw1IZezJ+ZQmJXsOKUZiNaOLsp2HeDtHfW8vaOOTdWHUIW0xDjOGJ/N5TNG8YmTR5AYZx3AoSKkC4GIXAQ8AMQCv1fVe/rb3gpBeKprbmPZ+/W8tb2et7fXU+MbnDYmK5lTCjKYMiKNSflpTMlPo3B4ss0xH0KaWzvYtv8wW2ua2ba/mS37DrG28iBtnR7iYoTZY4dz1oQczpqYw/TRGXb5Z4gK2UIgIrHANuBCoApYBVyrqpuP9TtWCMKfqvJ+3RHe3l7H8ooGtuxrZk+PNZSHxccyaUQqk/PTmDQijVGZwxiRnkR+RhJ5aYk2dmGIqSqHWjvZf6iVfU2t7G9qZVfDEbbWNFNe08zegy0fbJuSEMuk/DRmFmZy9sQc5ozLJsXO+YeFUC4EZwA/UNVP+h5/D0BV7z7W71ghiExH2jrZtr+ZrTXNbO3+WdNMw5H2j2wnAjmpieT7CkNuWiJpSXGkJ8WTlhTnvSV2348nOSGWhLiYD2+xMSTGxUTUFU2dXR7auzy0d3pvbb5bc2sHza2dvpvvfpv3/sGjHdQ0tX7w4d/S8dF1KOJihOLcFCbnpzMlP43JI9KYnJ/G6MxhdrQWpvwtBC7K+migssfjKmCOgxzGsZTEOGaNGc6sXtMMHzjS7v2WeqiVmkOtH/nwqmw8ynt7DnCotZP2Ts8JtRcfKyTGxRIbI8TGCDECMdJ9X4iJgViRDwrGBx998pEfgyoo3V+89IP/+eAHqopHwaOKx6N0dT/23e/yKB2+D3/PCXx/E4HUxDgyhsWTn57E1FHpnD8lj/z0JEZkJJGfnsTIjCRGpCeREGdHXtHIRSHo61/Rx/6sRWQhsBBgzJgxgc5kQsjwlASGpyRw0qj+Jx9r6+z6+Lff1k6OtnuLRPc35jbft+b2Lg9tHR66PN4P0i7fB65HlS4Pvp89Pqjp9cH9kTt9UxTp80+8hx6FpXfRiRGI8RWmWBHffT4oVt1HON1HO4k9j3riY0lLjPvgyKj7aCklIc6+0Zt+uSgEVUBhj8cFQHXvjVR1EbAIvKeGghPNhJPEuFgSU2PJsVGqxgyKi+PAVcBEERknIgnANcBzDnIYY4zBwRGBqnaKyFeBf+C9fPRhVd0U7BzGGGO8nFwDpqovAi+6aNsYY8xH2SUCxhgT5awQGGNMlLNCYIwxUc4KgTHGRDkrBMYYE+XCYhpqEakDdg/w13OA+iGMM5Qs28BYtoGxbAMTztnGqmru8d4kLArBYIhImT+TLrlg2QbGsg2MZRuYaMhmp4aMMSbKWSEwxpgoFw2FYJHrAP2wbANj2QbGsg1MxGeL+D4CY4wx/YuGIwJjjDH9iLhCICI/EZFyEVkvIs+KSOYxtrtIRLaKyA4RuTNI2T4nIptExCMix+zpF5FdIrJBRNaKSFDW6DyBbC72W5aILBGR7b6fw4+xXZdvn60VkYBObX68/SAiiSLypO/1lSJSFMg8J5jtZhGp67GvvhSkXA+LSK2IbDzG6yIiv/DlXi8is4ORy89s54lIU4999v0gZisUkddFZIvv3+gdfWwzuH2nqhF1Az4BxPnu3wvc28c2scD7QDGQAKwDTgpCtqnAZOANoKSf7XYBOUHeb8fN5nC//Ri403f/zr7+m/peOxykfXXc/QB8Bfit7/41wJMhlO1m4H+C+ffla/ccYDaw8RivXwK8hHfBtlJgZQhlOw/4e7D3ma/tkcBs3/00YFsf/00Hte8i7ohAVV9R1U7fwxV4V0Dr7XRgh6pWqGo78GdgQRCybVHVrYFuZyD8zOZkv/naeNR3/1HgU0Fosz/+7IeemZ8G5stgFjse2mxOqOqbQGM/mywAHlOvFUCmiIwMkWzOqOo+VV3ju98MbMG79ntPg9p3EVcIevki3irZ22igssfjKj6+Y11S4BURWe1buzlUuNpvI1R1H3j/UQB5x9guSUTKRGSFiASyWPizHz7YxvfFpAnIDmCmE8kG8BnfKYSnRaSwj9ddCPV/l2eIyDoReUlETnYRwHeKcRawstdLg9p3ThamGSwRWQrk9/HSXaq62LfNXUAn8Hhfb9HHc0Ny+ZQ/2fxwpqpWi0gesEREyn3fWFxnc7LfTuBtxvj2WzHwmohsUNX3hyJfL/7sh4Dtq+Pwp93ngSdUtU1EbsN75DIv4MmOz9U+88cavNM1HBaRS4C/ARODGUBEUoG/At9Q1UO9X+7jV/zed2FZCFT1gv5eF5GbgMuA+eo7gdZLFdDzW1ABUB2MbH6+R7XvZ62IPIv3cH/QhWAIsjnZbyKyX0RGquo+3+Fu7THeo3u/VYjIG3i/OQWiEPizH7q3qRKROCCD4Jx6OG42VW3o8fB3ePvSQkHA/r4Gq+cHr6q+KCK/FpEcVQ3KHEQiEo+3CDyuqs/0scmg9l3EnRoSkYuA7wJXqOrRY2y2CpgoIuNEJAFvZ15ArzLxl4ikiEha9328nd99XsnggKv99hxwk+/+TcDHjl5EZLiIJPru5wBnApsDlMef/dAz82eB147xpSTo2XqdO74C7znnUPAccKPvCphSoKn7lKBrIpLf3ccjIqfj/exs6P+3hqxtAR4CtqjqfcfYbHD7zkUveCBvwA6858rW+m7dV26MAl7ssd0leHvf38d7aiQY2T6Nt3K3AfuBf/TOhvdqj3W+26ZQyuZwv2UDrwLbfT+zfM+XAL/33Z8LbPDttw3ALQHO9LH9APwX3i8gAEnAX3x/j+8CxcHYV35mu9v3t7UOeB2YEqRcTwD7gA7f39otwG3Abb7XBfiVL/cG+rmyzkG2r/bYZyuAuUHMdhbe0zzre3yuXTKU+85GFhtjTJSLuFNDxhhjTowVAmOMiXJWCIwxJspZITDGmChnhcAYY6KcFQJjjIlyVgiMMSbKWSEwZgBE5DTfpG1JvtHgm0RkmutcxgyEDSgzZoBE5P/hHUE8DKhS1bsdRzJmQKwQGDNAvrl8VgGteKcc6HIcyZgBsVNDxgxcFpCKd9WoJMdZjBkwOyIwZoDEuy7yn4FxwEhV/arjSMYMSFiuR2CMayJyI9Cpqn8SkVhgmYjMU9XXXGcz5kTZEYExxkQ56yMwxpgoZ4XAGGOinBUCY4yJclYIjDEmylkhMMaYKGeFwBhjopwVAmOMiXJWCIwxJsr9fwPBUAGQStP+AAAAAElFTkSuQmCC\n", 33 | "text/plain": [ 34 | "" 35 | ] 36 | }, 37 | "metadata": {}, 38 | "output_type": "display_data" 39 | } 40 | ], 41 | "source": [ 42 | "import numpy as np\n", 43 | "import matplotlib.pyplot as plt\n", 44 | "import math\n", 45 | "\n", 46 | "L = np.arange(-2.0, 2.0, 0.1)\n", 47 | "plt.plot(L,L**4)\n", 48 | "plt.xlabel('x')\n", 49 | "plt.ylabel('Cost function')\n", 50 | "plt.show()" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": {}, 56 | "source": [ 57 | "Suppose we make an initial guess of $x_0 = 1$ (in fact, any number larger in modulus than $1$ give the same result), and suppose we take $\\alpha = 1$. Then, we get the following sequence of updates and costs:\n", 58 | "\n", 59 | "\\begin{align*}\n", 60 | "&1) \\quad x_0 = 0 \\ , \\ J(x_0) = 1 \\\\\n", 61 | "&2) \\nabla J (x_0) = 4, \\quad x_0 \\leftarrow x_0 - \\alpha\\nabla J(x_0)=1-1\\cdot4 = -3 \\ , \\ J(x_0) = 81 \\\\\n", 62 | "&3) \\nabla J (x_0) = -108, \\quad x_0 \\leftarrow x_0 - \\alpha\\nabla J(x_0)=-3-1\\cdot (-108) = 105 \\ , \\ J(x_0) = 121550625 \\\\\n", 63 | "&\\text{and so on.}\n", 64 | "\\end{align*}" 65 | ] 66 | }, 67 | { 68 | "cell_type": "markdown", 69 | "metadata": {}, 70 | "source": [ 71 | "Let us quickly implement the gradient descent algorithm for this cost function with these parameters:" 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": 3, 77 | "metadata": {}, 78 | "outputs": [], 79 | "source": [ 80 | "def cost_function(x):\n", 81 | " return float(x**4)\n", 82 | "\n", 83 | "def gradient(x):\n", 84 | " return 4*x**3\n", 85 | "\n", 86 | "def gradient_descent(x0,learn_rate,num_iter):\n", 87 | " x = x0\n", 88 | " C = []\n", 89 | " for i in range(num_iter):\n", 90 | " x = x - learn_rate*gradient(x)\n", 91 | " C.append(cost_function(x))\n", 92 | " return C" 93 | ] 94 | }, 95 | { 96 | "cell_type": "markdown", 97 | "metadata": {}, 98 | "source": [ 99 | "We run the algorithm and plot the logarithm of the cost as a function of the number of iterations. Note that with only $5$ iterations we reach astronomical numbers." 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "execution_count": 4, 105 | "metadata": {}, 106 | "outputs": [ 107 | { 108 | "data": { 109 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xl8FfX1//HXIYGEJQmyCgEMyGIAFRRwrV9wKS5Y+7Xuda0t7c9u39qva6u21ra29au1tlWptlpbt4pWRFyKFVHrBooKCbsIYV8kCUsgy/n9MRMI15twA7l3bpL38/G4j9yZ+czMuQN3zp2ZM58xd0dERCRWm6gDEBGR9KQEISIicSlBiIhIXEoQIiISlxKEiIjEpQQhIiJxKUFIq2Nmy8zs5PD9jWb2QERxjDWzkijWLZIIJQhJK2Z2gZm9Y2ZbzWxd+P4qM7NkrM/df+HuX9/f5ZhZgZm5mWU2RVxRM7OHzOy2qOOQaClBSNowsx8CdwO/AQ4EegLfAo4D2tUzT0bKAhRpZZQgJC2YWR5wK3CVuz/l7uUe+MDdv+ruO8J2D5nZvWY2zcy2AuPM7Awz+8DMysxshZn9JGbZl5jZp2a20cx+FDPtJ2b2tzrDR5vZf8xss5l9aGZj60ybYWY/M7M3zazczF42s27h5Jnh381mtsXMjonzGduH8X9mZkXA6Jjpvc1sspmtN7NPzOx7daaNMbNZ4Wdca2Z31pl2fJ2YV5jZ5eH4LDO7w8yWh/PcZ2btw2ljzazEzH4YHqmtNrMrwmkTga8C14af5bkE/gmlJXJ3vfSK/AWcClQBmXtp9xBQSnBU0QbIBsYCh4bDhwFrgS+H7YcCW4ATgCzgznA9J4fTfwL8LXyfD2wETg+XdUo43D2cPgNYAgwG2ofDt4fTCgBvKH7gduB1oAvQF5gLlITT2gCzgZsJjpYGAEuB8eH0t4BLwvedgKPD9/2AcuBCoC3QFRgRTvstMCVcXw7wHPDLcNrYcDvcGs53OrANOKDOdr4t6v8XekX70hGEpItuwAZ3r6odUedX8XYzO6FO22fd/U13r3H3Cnef4e4fh8MfAY8B/xW2PQeY6u4zPTgKuQmoqSeGi4Fp7j4tXNa/gFkEO89af3H3he6+HXgSGNGIz3ge8HN33+TuK4Df1Zk2miAR3eruO919KfAn4IJweiUw0My6ufsWd387HP9VYLq7P+bule6+0d3nhNdsvgH8IFxfOfCLOsurXeat4XzTCBLpkEZ8HmnhWsQFNWkRNgLdzCyzNkm4+7EAYaVP3R8zK+rOaGZHEfw6H07w6zsL+Ec4uXfd9u6+1cw21hPDQcC5ZnZmnXFtgVfrDK+p834bwa/5RO0RC/BpzLp7m9nmOuMyCI44AK4k+LU/38w+AX7q7lMJjkSWxFlXd6ADMLvO9X0Ll1lrY92EvA+fR1o4JQhJF28BO4CzgMl7aRvbBfGjwO+B09y9wsx+S3BEArAaKKxtaGYdCE7DxLMCeMTdv9HI2OPFFM9qgh36vHC4X8y6P3H3QXEX7r4IuNDM2gBnA0+ZWddwvjFxZtkAbAeGufvKxD7Cnqvch3mkhdEpJkkL7r4Z+CnwRzM7x8w6mVkbMxsBdNzL7DnApjA5jAEuqjPtKWBCeCG3HcGv8Pr+3/8NONPMxptZhpllhxdz+yTwEdYTnLoa0ECbJ4EbzOyAcJnfrTPtXaDMzK4LL2ZnmNlwMxsNYGYXm1l3d68Bao8yqoG/Ayeb2XlmlmlmXc1sRNjuT8BdZtYjXEa+mY1P4LNAcB2noc8irYAShKQNd/81cDVwLbCOYCd1P3Ad8J8GZr0KuNXMygku8j5ZZ5nzgG8THGWsBj4D4t6cFl4XOAu4kWCHvwK4hgS+J+6+Dfg58GZ43eToOM1+SnBa6RPgZeCROvNXA2cSXNP4hOAI4AEgL2xyKjDPzLYQlAJfEF5/WU5wjeSHwCZgDnB4OM91wGLgbTMrA6aT+DWGB4Gh4Wf5Z4LzSAtj7jqSFBGRz9MRhIiIxKUEISIicSU1QZhZZzN7yszmm1mxmR1jZl3M7F9mtij8e0DY1szsd2a22Mw+MrMjkhmbiIg0LNlHEHcDL7r7IQQXzoqB64FXwnK+V8JhgNOAQeFrInBvkmMTEZEGJO0itZnlAh8CA7zOSsxsATDW3VebWS9ghrsPMbP7w/ePxbarbx3dunXzgoKCpMQvItJSzZ49e4O7d99bu2TeKDeAoFTwL2Z2OEE/M98Hetbu9MMk0SNsn8+ed5mWhOP2SBBhR2ITAfr168esWbOS+BFERFoeM/t0762Se4opEzgCuNfdRwJb2X06KZ54/f1/7vDG3Se5+yh3H9W9+14ToIiI7KNkJogSgp4q3wmHnyJIGGvDU0uEf9fVad+3zvx9gFVJjE9ERBqQtATh7muAFWZWe+fmSUARQffDl4XjLgOeDd9PAS4Nq5mOBkobuv4gIiLJlezO+r4L/D3sA2cpcAVBUnrSzK4ElgPnhm2nEXQZsJigV8kr9mWFlZWVlJSUUFFRsb+xS4Kys7Pp06cPbdu2jToUEWlCSU0Q7j4HGBVn0klx2jpBnzn7paSkhJycHAoKCrDkPMZY6nB3Nm7cSElJCf379486HBFpQi3uTuqKigq6du2q5JAiZkbXrl11xCbSArW4BAEoOaSYtrdIy9QiE4SISEtVU+PcNrWIxeu2JH1dShBJ0KlTap7aOGfOHKZNm5bUdfzjH/9g2LBhtGnTRjcliqSBZz9cyQNvfMKcFZv33ng/KUGkSHV19T7NV1VVVe+0VCSI4cOH8/TTT3PCCSckdT0isnfbdlbxqxcWcGh+HmePzE/6+pQgkmjGjBmMGzeOiy66iEMPPZRly5ZRWFjIN77xDYYNG8YXv/hFtm/f/rn5Lr/8cq6++mrGjRvHddddx7vvvsuxxx7LyJEjOfbYY1mwYAE7d+7k5ptv5oknnmDEiBE88cQTbN26la997WuMHj2akSNH8uyzz8aJqnEKCwsZMiTRh5CJSDJNmrmUNWUV3HzmUNq0Sf61v2TfBxGpnz43j6JVZU26zKG9c7nlzGEJt3/33XeZO3cu/fv3Z9myZSxatIjHHnuMP/3pT5x33nlMnjyZiy+++HPzLVy4kOnTp5ORkUFZWRkzZ84kMzOT6dOnc+ONNzJ58mRuvfVWZs2axe9//3sAbrzxRk488UT+/Oc/s3nzZsaMGcPJJ59Mx467H+lcXl7OF77whbixPvroowwdOrSRW0REUmF16Xbue20JZxzWi9EFXVKyzhadINLBmDFj9rg/oH///owYMQKAI488kmXLlsWd79xzzyUjIwOA0tJSLrvsMhYtWoSZUVlZGXeel19+mSlTpnDHHXcAQcnv8uXLKSws3NUmJyeHOXPmNMVHE5EU+vWLC6hxuP7UQ1K2zhadIBrzSz9Z6v56B8jKytr1PiMjI+4pptj5brrpJsaNG8czzzzDsmXLGDt2bNx53J3Jkyc3eEpIRxAizc8Hyz/jmQ9WctXYg+nbpUPK1tuiE0RLUVpaSn5+cEHqoYce2jU+JyeH8vLyXcPjx4/nnnvu4Z577sHM+OCDDxg5cuQey9IRhEjz4u7cOrWIbp2yuGrcwJSuWxepm4Frr72WG264geOOO26Paqhx48ZRVFS06yL1TTfdRGVlJYcddhjDhw/npptu2u91P/PMM/Tp04e33nqLM844g/Hjx+/3MkUkcVM+XMUHyzdz7fghdMpK7W/6pD1RLhVGjRrlsbX5xcXFe5xzl9TQdhdpett3VnPS/83ggI7tmPKd48loosolM5vt7vH6yduDjiBERNLUn15fyqrSCm6aMLTJkkNjKEGIiKShNaUV3DtjCacNP5CjB3SNJIYWmSCa82mz5kjbW6Tp/fql+VTXODecFt2p2xaXILKzs9m4caN2WilS+zyI7OzsqEMRaTE+XLGZp99fydeO70+/rqkra43V4spc+/TpQ0lJCevXr486lFaj9olyIrL/dpe1tuPb4w6ONJYWlyDatm2rJ5uJSLM19aPVzP70M24/+1BysqN9jG+LO8UkItJcVVRWc/sL8ynslcu5o/pGHY4ShIhIunjg9aWs3LydmyYURlLWGksJQkQkDawtq+CPM5YwflhPjj24W9ThAEoQIiJp4TcvLaCyuoYbT0+fHgmUIEREIvZxSSlPzS7ha8f156CuHfc+Q4ooQYiIRCgoa51H147t+PaJqe2tdW+UIEREIjTt4zW8t+wzfvjFIeRGXNYaSwlCRCQiFZXV/GJaMYccmMP5o6Mva42V1ARhZsvM7GMzm2Nms8JxXczsX2a2KPx7QDjezOx3ZrbYzD4ysyOSGZuISNQefOMTVm7ezs0R9da6N6k4ghjn7iPq9D1+PfCKuw8CXgmHAU4DBoWvicC9KYhNRCQS68oq+OOrizllaE+OHZgeZa2xojjFdBbwcPj+YeDLdcb/1QNvA53NrFcE8YmIJN0dLy9gZ5qVtcZKdoJw4GUzm21mE8NxPd19NUD4t0c4Ph9YUWfeknDcHsxsopnNMrNZ6pBPRJqjuStL+cfsEi4/toD+3dKnrDVWsjvrO87dV5lZD+BfZja/gbbxTsB9rs9ud58ETILgkaNNE6aISGrU9tZ6QId2fOfEQVGH06CkHkG4+6rw7zrgGWAMsLb21FH4d13YvASoexm/D7AqmfGJiKTai3PX8O4nm7j6lMHktU+vstZYSUsQZtbRzHJq3wNfBOYCU4DLwmaXAc+G76cAl4bVTEcDpbWnokREWoKKymp+8UIxQ3rmcEEalrXGSuYppp7AM2ZWu55H3f1FM3sPeNLMrgSWA+eG7acBpwOLgW3AFUmMTUQk5f7y5jJWbNrOI1eOITMj/W9DS1qCcPelwOFxxm8ETooz3oFvJyseEZEorSuv4A+vLubkwh58YVD3qMNJSPqnMBGRFuDOlxdSUVmd1mWtsZQgRESSbN6qUp6YtYLLji1gQPdOUYeTMCUIEZEkcnd+NrWIzu3b8r00L2uNpQQhIpJELxet5e2lYVlrh/Qua42lBCEikiQ7qoLeWgf16MSFY/pFHU6jKUGIiCTJw/9Zxqcbt3HThKHNoqw1VvOLWESkGdiwZQf3vLKYEw/pwQmDm0dZaywlCBGRJLjzXwvZ3szKWmMpQYiINLHi1WU8/u5yLjnmIAb2aD5lrbGUIEREmpC7c9vzReS2b8v3T2peZa2xlCBERJrQ9OJ1vLl4Iz84eTCdO7SLOpz9ogQhItJEdlbV8PPnixjYoxMXHdX8ylpjKUGIiDSRv761jGUbt/HjMwpp2wzLWmM1/08gIpIGNm7Zwd2vLGLskO6MHdJj7zM0A0oQIiJN4K7pC9m2s5ofn9F8y1pjKUGIiOynBWvKefSd5Vxy9EEM7JETdThNRglCRGQ/1PbWmpPd/MtaYylBiIjsh3/PX8cbizfw/ZMGcUDH5l3WGksJQkRkHwVlrcUM6N6RS445KOpwmpwShIjIPnrk7U9ZumFriylrjdXyPpGISAps2rqTu6cv5AuDujGuhZS1xlKCEBHZB7+dvpAtO6q4acJQzCzqcJJCCUJEpJEWri3n7+8s56tHHcTgni2nrDWWEoSISCPUlrV2aJfBD04ZHHU4SZW5twZm1gY4HOgNbAfmufvaZAcmIpKOZixYz+uLNvDjMwrp0sLKWmPVmyDM7GDgOuBkYBGwHsgGBpvZNuB+4GF3r0lFoCIiUausruFnzxfRv1tHLj2mIOpwkq6hU0y3AX8DDnb38e5+sbuf4+6HAV8C8oBL9rYCM8swsw/MbGo43N/M3jGzRWb2hJm1C8dnhcOLw+kF+/vhRESa0t/e/pSl67fyo9MLaZfZ8s/Q1/sJ3f1Cd5/p7h5n2jp3/627P5zAOr4PFNcZ/hVwl7sPAj4DrgzHXwl85u4DgbvCdiIiaeGzrTv57fRFHD+wGycVtsyy1lgJpUAzO9bMLjKzS2tfCc7XBzgDeCAcNuBE4KmwycPAl8P3Z4XDhNNPspZaOyYizc7dryyivKKSH08obLFlrbESuUj9CHAwMAeoDkc78NcElv9b4Fqgtg6sK7DZ3avC4RIgP3yfD6wAcPcqMysN22+IiWciMBGgX7/m/8QmEUl/i9eV88jbn3LhmH4ccmBu1OGkzF4TBDAKGBrvVFNDzGwCsM7dZ5vZ2NrRcZp6AtN2j3CfBEwCGDVqVKNiEhHZF7c9X0yHthlc3cLLWmMlkiDmAgcCqxu57OOAL5nZ6QTVT7kERxSdzSwzPIroA6wK25cAfYESM8skuAi+qZHrFBFpUq8uWMeMBev50emFdO2UFXU4KZXINYhuQJGZvWRmU2pfe5vJ3W9w9z7uXgBcAPzb3b8KvAqcEza7DHg2fD8lHCac/u/GHrWIiDSlyuqgt9aCrh247NiCqMNJuUSOIH7SxOu8DnjczG4DPgAeDMc/CDxiZosJjhwuaOL1iog0yqPvLGfxui1MuuTIVlHWGmuvCcLdXzOznsDocNS77r6uMStx9xnAjPD9UmBMnDYVwLmNWa6ISLJs3raTu6Yv5NiDu3LK0J5RhxOJvaZEMzsPeJdg530e8I6ZndPwXCIizdvdryyibHtli+6tdW8SOcX0I2B07VGDmXUHprP7XgYRkRZlyfotPPLWp5w/uh+FvVpPWWusRE6qtYk5pbQxwflERJqlXzxfTHbbDH74xdZV1horkSOIF83sJeCxcPh8YFryQhIRic7Mhet5Zf46bjjtELq1srLWWIlcpL7GzL5CcF+DAZPc/ZmkRyYikmJV1TXc9nwR/bp04PLjCqIOJ3KJHEHg7pOByUmORUQkUo+9t4KFa7dw38VHkpWZEXU4kWvoeRBvuPvxZlbOnl1eGODu3nqv3IhIi1O6vZI7X17A0QO6MH5Y6yxrjVVvgnD348O/LfeBqyIioXteWcTmVl7WGiuR+yAeSWSciEhztXT9Fh76zzLOH9WXYb3zog4nbSRSrjqs7kDYkd6RyQlHRCT1fjFtfljWOiTqUNJKvQnCzG4Irz8cZmZl4ascWMvuDvZERJq1NxZtYHrxWr49biDdc1p3WWushh45+svw+sNv3D03fOW4e1d3vyGFMYqIJEVVdQ0/m1pE3y7tuUJlrZ+TyCmmd81s10k5M+tsZl9uaAYRkebgiVkrWLC2nBtPKyS7rcpaYyWSIG5x99LaAXffDNySvJBERJKvrKKS/3t5IWP6d+HU4QdGHU5aSqgvpjjjErrBTkQkXf3+34v5bNtOblZZa70SSRCzzOxOMzvYzAaY2V3A7GQHJiKSLMs2bOUvb37CuUf2YXi+ylrrk0iC+C6wE3gC+AdQAXw7mUGJiCTTL6YV0y6jDf+rstYGJdJZ31bg+hTEIiKSdP9ZvIGXi9Zyzfgh9MjNjjqctLbXBGFmg4H/BQrqtnf3E5MXlohI06uucW6dWkR+5/ZceXz/qMNJe4lcbP4HcB/wAFCd3HBERJLnyVkrmL+mnD9cdITKWhOQSIKocvd7kx6JiEgSlVVUcsdLCxhdcACnH6qy1kQkkiCeM7OrgGeAHbUj3X1T0qISEWlif3h1MZu27eShCWNU1pqgRBLEZeHfa+qMc2BA04cjItL0Pt24lb+8sYyvHNGHQ/uorDVRiVQx6UqOiDRrv5w2n8wM45rxKmttjESqmC6NN97d/9r04YiINK23lmzkxXlr+OEpg+mpstZGSeQU0+g677OBk4D3gQYThJllAzOBrHA9T7n7LWbWH3gc6BIu5xJ332lmWeEyjwQ2Aue7+7LGfRwRkd2qa5yfTS2id1423zhBZ8UbK5FTTN+tOxz27JrIE+V2ACe6+xYzawu8YWYvAFcDd7n742Z2H3AlcG/49zN3H2hmFwC/As5v3McREdntqdkrKFpdxu8uHKmy1n2QSFcbsbYBg/bWyANbwsG24cuBE4GnwvEPA7Vdh58VDhNOP8lUaiAi+6i8opLfvLSQI/p15szDekUdTrOUyDWI5wh27BAklKHAk4ks3MwyCDr2Gwj8AVgCbHb3qrBJCZAfvs8HVgC4e5WZlQJdgQ0xy5wITATo169fImGISCv0xxlL2LBlBw9cNkplrfuo3gRhZlnuvgO4o87oKuBTdy9JZOHuXg2MMLPOBPdRFMZrVrvKBqbVXeYkYBLAqFGjPjddRGTFpm08+PonnD0ynxF9O0cdTrPV0BHEW8ARwNfd/ZL9WYm7bzazGcDRQGczywyPIvoAq8JmJUBfoMTMMoE8QDfjiUij/fKFYjLaGNecqrLW/dFQgmhnZpcBx5rZ2bET3f3phhZsZt2ByjA5tAdOJrjw/CpwDkEl02XAs+EsU8Lht8Lp/3Z3HSGISKO8s3Qj0z5eww9OHkyvvPZRh9OsNZQgvgV8FegMnBkzzYEGEwTQC3g4vA7RBnjS3aeaWRHwuJndBnwAPBi2fxB4xMwWExw5XNCoTyIirV5tb6298rKZqLLW/VZvgnD3NwhKU2e5+4P1tWtg/o+AkXHGLwXGxBlfAZzb2PWIiNSa/H4J81aVcfcFI2jfTmWt+2uvZa77khxERFJty44qfvPSAkb268yXDu8ddTgtwr7cByEiknbum7GE9eU7uGnCUJW1NhElCBFp9ko+28ak15fy5RG9OaLfAVGH02LsNUGY2SuJjBMRicrtL8ynjcG1px4SdSgtSkM3ymUDHYBuZnYAu29kywV0gk9E0sKsZZuY+tFqvn/SIHp3VllrU2qozPWbwP8QJIPZ7E4QZQTdZoiIRKomLGs9MDebb/6XylqbWkNlrncDd5vZd939nhTGJCKSkGc+WMlHJaXcdf7hdGiXyNMLpDESuUi9xsxyAMzsx2b2tJkdkeS4REQatHVHFb9+aT6H9+3MWYfn730GabREEsRN7l5uZscD4wm65L43uWGJiDTs/teWsLZsBzdPGEqbNiprTYZEEkR1+PcM4F53fxZol7yQREQatnLzdu6fuZQvHd6bIw9SWWuyJJIgVprZ/cB5wLTw0aC6f0JEIvOrF+YDcN1pKmtNpkR29OcBLwGnuvtmgmdJX5PUqERE6jH7001M+XAV3zxhAPkqa02qRPpi2kbwJLjxZvYdoIe7v5z0yEREYgRlrcX0zM3im/91cNThtHiJ3En9feDvQI/w9Tcz+26yAxMRifXshyv5cMVmrh1/CB2zVNaabIls4SuBo9x9K4CZ/YrgoT66N0JEUmbbzip+9cICDuuTx3+PVFlrKiRyDcLYXclE+F41ZSKSUve/tpQ1ZRUqa02hRI4g/gK8Y2bPhMNfZvdT4EREkm7V5u3cP3MJEw7rxaiCLlGH02rsNUG4+51mNgM4nuDI4Qp3/yDZgYmI1Pr1i/OpcbheZa0p1VBvrqOBbu7+gru/D7wfjv+SmbVx99mpClJEWq/3l3/GP+es4jvjBtLngA5Rh9OqNHQN4jdAcZzxReE0EZGkcndufa6I7jlZ/L+xKmtNtYYSRFd3XxY70t0XA12TFpGISGjKh6uYs2Iz144forLWCDSUIBq6RbFjUwciIlLX9p3V3P7CfIbn5/KVI/pEHU6r1FCCmG5mP7eYp3+b2U+Bfyc3LBFp7SbNXMrq0gpunjBMZa0RaeiY7YfAA8BiM5sTjjscmAV8PdmBiUjrtbp0O/e9toQzDu3FmP4qa41KQ0+U2wpcaGYDgGHh6HnuvjQlkYlIq/WbFxdQ7a6y1og1VOZa4O7LwoTwuaQQnnrKd/eSZAYoIq3LnBWbefqDlVw19mD6dlFZa5QaLHM1s8lmdqmZDTOzHmbWz8xONLOfAW8ChfXNbGZ9zexVMys2s3lhp3+YWRcz+5eZLQr/HhCONzP7nZktNrOP9FhTkdYnKGudR7dOWVw1bmDU4bR69SYIdz8XuAkYAvwBeB2YAnwDWACc6O7/amDZVcAP3b0QOBr4tpkNBa4HXnH3QcAr4TDAacCg8DURPdZUpNV57qPVvL88KGvtpLLWyDX4L+DuRcCP9mXB7r4aWB2+LzezYiAfOAsYGzZ7GJgBXBeO/6u7O/C2mXU2s17hckSkhauorOb2acUM653LV45UWWs62GuKNrOz44wuBT5293WJrMTMCoCRwDtAz9qdvruvNrMeYbN8YEWd2UrCcXskCDObSHCEQb9+/RJZvYg0A3+auZRVpRXcef4IMlTWmhYSfR7EMcCr4fBY4G1gsJnd6u6PNDSzmXUCJgP/4+5lMbdV7NE0zjj/3Aj3ScAkgFGjRn1uuog0P2vLKvjjjCWcNvxAjh6gjhrSRSLPg6gBCt39K+7+FWAosAM4iuDUUL3MrC1Bcvi7uz8djl5rZr3C6b2A2qOQEqBvndn7AKsS/SAi0nz9+sUFVNc4N5xWb92LRCCRBFHg7mvrDK8DBrv7JqCyvpnCMtgHgWJ3v7POpCnAZeH7y4Bn64y/NKxmOhoo1fUHkZatdFslP5kyj8nvl/C14/vTr6vKWtNJIqeYXjezqcA/wuFzgJlm1hHY3MB8xwGXAB/XuRP7RuB24EkzuxJYDpwbTpsGnA4sBrYBVzTmg4hI81Fd4zzx3grueHkBm7ft5KtH9eP7Jw2KOiyJYUHRUAMNgiOBs9n9wKA3gMm+txlTYNSoUT5r1qyowxCRRpi1bBO3TJnHvFVljOnfhZ+cOYyhvXOjDqtVMbPZ7j5qb+0SeaKcm9kbwE6Ci8bvpkNyEJHmZU1pBbe/UMw/56yiV14291w4kgmH9aKBwhWJWCJlrucRPCBoBsERxD1mdo27P5Xk2ESkBaiorObBNz7hD68upqrG+d6JA/nW2IPp0E43wqW7RP6FfgSMrr3nwcy6A9MBJQgRqZe7M714Hbc9X8SnG7dx6rAD+dEZhepfqRlJJEG0ibkhbiOJVT+JSCu1eN0Wbp1axMyF6xnUoxN/u/Iojh/ULeqwpJESSRAvmtlLwGPh8PkEFUciInsoq6jkd9MX8dB/ltG+XQY3TxjKJcccRNsM/aZsjhK5SH2NmX2FoGzVgEnu/kzSIxORZqOmxnlqdgm/fmk+G7fu5ILRffnfLw6ha6esqEOT/ZDQVSJ3n0xwR7SIyB7eX/4ZP50yjw9LSjnyoAP4y+VjOLRPXtRhSRNo6IFB5cTpC4ljggFpAAAQx0lEQVTgKMLdXYXLIq3YurIKbn9xPk+/v5KeuVn89vwRnDWit8pWW5CGHjmak8pARKR52FFVzV/eXMY9ryyistq5auzBfHvcQDrq+Q0tjv5FRSRhr85fx61Ti/hkw1ZOLuzJj88opKBbx6jDkiRRghCRvVq6fgs/m1rEqwvWM6B7Rx66YjRjh/TY+4zSrClBiEi9tuyo4p5/L+LPb3xCVmYGPz6jkEuPKaBdpspWWwMlCBH5nJoa55kPVnL7i/NZX76D80b14Zrxh9A9R2WrrYkShIjs4cMVm7llyjzmrNjMiL6d+dOloxjRt3PUYUkElCBEBID15Tv4zUvzeXJWCd1zsvi/cw/nv0fm00bPh261lCBEWrmdVTX89a1l3D19ERVV1XzzhAF858SB5GS3jTo0iZgShEgr9trC9dz63DyWrN/KuCHduWnCUAZ07xR1WJImlCBEWqFPN27lZ1OLmV68lv7dOvLny0dx4iE9ow5L0owShEgrsnVHFX94dTEPvP4JbTOM6087hCuOKyArMyPq0CQNKUGItALuzrNzVvHLF4pZW7aDs4/I5/pTD6FHbnbUoUkaU4IQaeHmrizlJ1PmMevTzzisTx73XnwkR/Q7IOqwpBlQghBpoTZu2cEdLy/g8fdW0KVDO379lcM458g+KluVhClBiLQwldU1PPLWp9w1fSHbd1bzteP6872TBpHXXmWr0jhKECItyJuLN/CTKfNYtG4LXxjUjVvOHMrAHuq5X/aNEoRIC7Bi0zZ+/nwxL85bQ78uHZh0yZGcMrSnHt4j+0UJQqQZ276zmntnLOb+mUtpY8Y144dw5fH9yW6rslXZf0lLEGb2Z2ACsM7dh4fjugBPAAXAMuA8d//Mgp85dwOnA9uAy939/WTFJtLcuTtTP1rNL6cVs6q0grNG9Ob60w6hV177qEOTFiSZnbo/BJwaM+564BV3HwS8Eg4DnAYMCl8TgXuTGJdIs1a0qozzJ73Ndx/7gAM6tuMf3zqGuy8YqeQgTS5pRxDuPtPMCmJGnwWMDd8/DMwArgvH/9XdHXjbzDqbWS93X52s+ESam8+27uT//rWAR99ZTl77tvzivw/l/NF9yVDZqiRJqq9B9Kzd6bv7ajOrfWZhPrCiTruScNznEoSZTSQ4yqBfv37JjVYkDVRV1/DYu8u54+WFbNlRxaXHFPCDkweT10Flq5Jc6XKROt5PII/X0N0nAZMARo0aFbeNSEvx1pKN/PS5ecxfU86xB3flljOHMeRAla1KaqQ6QaytPXVkZr2AdeH4EqBvnXZ9gFUpjk0kbazcvJ1fPF/M8x+vJr9ze+67+AjGDztQZauSUqlOEFOAy4Dbw7/P1hn/HTN7HDgKKNX1B2mNKiqruf+1pdz72mIArj5lMBNPGKCyVYlEMstcHyO4IN3NzEqAWwgSw5NmdiWwHDg3bD6NoMR1MUGZ6xXJikskHbk7L85dw23PF7Ny83bOOKwXN55eSH5nVSZJdJJZxXRhPZNOitPWgW8nKxaRdLZgTTk/fW4e/1mykUMOzOHxiUdz9ICuUYclkjYXqUVandJtldw1fSGPvP0pOdmZ/OzLw7lwdF8yM5J5e5JI4pQgRFKsusZ5/L3l3PHSAkq3V/LVow7i6lMGc0DHdlGHJrIHJQiRFHpv2SZueXYeRavLOKp/F37ypWEU9sqNOiyRuJQgRFJgdel2fjltPlM+XEXvvGx+f9FIzji0l8pWJa0pQYgkUUVlNQ+8vpQ/vLqEane+d9Ig/t9/HUz7dipblfSnBCGSBO7Oy0Vrue35IlZs2s6pww7kR2cU0rdLh6hDE0mYEoRIE9m+s5riNWXMW1nKi/PW8ObijQzu2Ym/f/0ojhvYLerwRBpNCUJkH5RVVFK0qoy5K0uZF/5dsn4LNWHvYN06ZXHLmUO5+OiDaKuyVWmmlCBE9mLT1p3MXVnK3FVBMpi3spRlG7ftmt4zN4vhvfM47dBeDO+dy/D8PHrlZesCtDR7ShAiIXdnXfmOIBmsLAsSwspSVpVW7GrTt0t7hvfO49xRfRnWO5dhvfPonpMVYdQiyaMEIa2Su1Py2fZdRwZzV5Yxb1UpG7bsBMAMBnTryKiCLgzPz2V47zyG9c7TMxikVVGCkBavusb5ZMNW5q3afb1g7spSyiqqAMhoYwzq0YmxQ3rsOkVU2CuXjln6ekjrpm+AtCiV1TUsXrdlj4vHRavL2LazGoB2mW0oPDCHCYf3Do8KchlyYI660xaJQwlCmq2KymoWrCnfdYqoaFUpxWvK2VlVA0CHdhkM7ZXLeeH1guH5eQzs0UlVRSIJUoKQZmHrjiqKV4enh8Ijg0XrtlAd1pXmZmcyPD+Py48t2JUMCrp2JKONKolE9pUShKSd0m2VzFu1++Lx3FWlfLJhK77rHoN2DM/P4+TCnruSQZ8D2qusVKSJKUFIpNaX79jz4vGqUlZs2r5reu+8bIbl53HW4flBNVF+Hj1yspQMRFJACUJSwt1ZXVqx6xTRvDAZrC3bsatNQdcOHNanMxeNOYjh+cE9Bl30jASRyChBSJOrqXGWb9q2x/0F81aVsWlrcI9BG4OBPTpx3MHdGBqeIhraO5fcbN1jIJJOlCBkv1TXOEvXb9l9vWBlKUWryijfEdxj0DbDGNwzh1MKewZHBfl5FB6Yq+6uRZoBJQhJ2M6qGhauLQ8uIIcXj4tXl1FRGZSVZrdtQ2GvXL48Mn/XKaLBPXNol6myUpHmSAmilaqpccp3VFG2vZLS7ZWUba+krKL2fRWltePDcevKdrBoXTmV1UEpUaesTIb1zt11vWB4fh4DunUkU/cYiLQYShDN2I6q6l07890798rdO/2KKkq37Z62q922Ssp3VO0qG40no42Rm51Jbvu25LVvS/ecLE4Y3H1Xv0T9unSgje4xEGnRlCAi5O5s2VG1x6/2enf0dYZr29We2qlPdts25LVvS252sJPvmZvN4J454bhg51+bAHa16xBM65SVqVJSkVZOCWI/VVbXxOy499yZl1XUeR8nCdQ08CveDHKyMsOddrATH9ij0x478rxwJ59bdyffvi257TPJytSFYBHZd2mVIMzsVOBuIAN4wN1vT/Y63Z1tO6v3OP3SmB19bSdw9WmX0SbceQe/2Lt2aseA7h332JHX/QW/a0ffvi05WZk6jSMikUmbBGFmGcAfgFOAEuA9M5vi7kVNva4n3lvOfa8t3ZUAqhr6GU/wK373r/RMDuraYddOfNfOPdzR1+74a6erl1ARaa7SJkEAY4DF7r4UwMweB84CmjxBdOmYxbDeuZ/fycf5Vd8pK1OVOSLSKqVTgsgHVtQZLgGOim1kZhOBiQD9+vXbpxWdMrQnpwztuU/zioi0Fun00zjeyfbPnftx90nuPsrdR3Xv3j0FYYmItE7plCBKgL51hvsAqyKKRUSk1UunBPEeMMjM+ptZO+ACYErEMYmItFppcw3C3avM7DvASwRlrn9293kRhyUi0mqlTYIAcPdpwLSo4xARkfQ6xSQiImlECUJEROJSghARkbjMG+rzOc2Z2Xrg032cvRuwoQnDaSqKq3EUV+Ola2yKq3H2J66D3H2vN5I16wSxP8xslruPijqOWIqrcRRX46VrbIqrcVIRl04xiYhIXEoQIiISV2tOEJOiDqAeiqtxFFfjpWtsiqtxkh5Xq70GISIiDWvNRxAiItIAJQgREYmrxScIMzvVzBaY2WIzuz7O9CwzeyKc/o6ZFaRJXJeb2XozmxO+vp6iuP5sZuvMbG49083MfhfG/ZGZHZEmcY01s9I62+vmFMTU18xeNbNiM5tnZt+P0ybl2yvBuKLYXtlm9q6ZfRjG9dM4bVL+fUwwrki+j+G6M8zsAzObGmdacreXu7fYF0GvsEuAAUA74ENgaEybq4D7wvcXAE+kSVyXA7+PYJudABwBzK1n+unACwQPeDoaeCdN4hoLTE3xtuoFHBG+zwEWxvl3TPn2SjCuKLaXAZ3C922Bd4CjY9pE8X1MJK5Ivo/huq8GHo3375Xs7dXSjyB2Pefa3XcCtc+5russ4OHw/VPASWYW7+l2qY4rEu4+E9jUQJOzgL964G2gs5n1SoO4Us7dV7v7++H7cqCY4NG5daV8eyUYV8qF22BLONg2fMVWyaT8+5hgXJEwsz7AGcAD9TRJ6vZq6Qki3nOuY78ou9q4exVQCnRNg7gAvhKelnjKzPrGmR6FRGOPwjHhaYIXzGxYKlccHtqPJPj1WVek26uBuCCC7RWeLpkDrAP+5e71bq8Ufh8TiQui+T7+FrgWqKlnelK3V0tPEIk85zqhZ2E3sUTW+RxQ4O6HAdPZ/SshalFsr0S8T9C/zOHAPcA/U7ViM+sETAb+x93LYifHmSUl22svcUWyvdy92t1HEDxSeIyZDY9pEsn2SiCulH8fzWwCsM7dZzfULM64JtteLT1BJPKc611tzCwTyCP5pzL2Gpe7b3T3HeHgn4AjkxxTotLy2eHuXlZ7msCDB0+1NbNuyV6vmbUl2An/3d2fjtMkku21t7ii2l511r8ZmAGcGjMpiu/jXuOK6Pt4HPAlM1tGcBr6RDP7W0ybpG6vlp4gEnnO9RTgsvD9OcC/PbziE2VcMeepv0RwHjkdTAEuDatzjgZK3X111EGZ2YG1517NbAzB/+2NSV6nAQ8Cxe5+Zz3NUr69Eokrou3V3cw6h+/bAycD82Oapfz7mEhcUXwf3f0Gd+/j7gUE+4h/u/vFMc2Sur3S6pGjTc3rec61md0KzHL3KQRfpEfMbDFB5r0gTeL6npl9CagK47o82XEBmNljBBUu3cysBLiF4KId7n4fwSNhTwcWA9uAK9IkrnOA/2dmVcB24IIUJPrjgEuAj8Pz1wA3Av3qxBXF9kokrii2Vy/gYTPLIEhIT7r71Ki/jwnGFcn3MZ5Ubi91tSEiInG19FNMIiKyj5QgREQkLiUIERGJSwlCRETiUoIQEZG4lCCkVTOzLeHfAjO7qImXfWPM8H+acvkiyaYEIRIoABqVIMK6+YbskSDc/dhGxiQSKSUIkcDtwBfCvv5/EHbe9hszey/soO2bsOs5Cq+a2aPAx+G4f5rZbAueJTAxHHc70D5c3t/DcbVHKxYue66ZfWxm59dZ9oywM7j5Zvb3Onc7325mRWEsd6R860ir1KLvpBZphOuB/3X3CQDhjr7U3UebWRbwppm9HLYdAwx390/C4a+5+6awm4b3zGyyu19vZt8JO4CLdTYwAjgc6BbOMzOcNhIYRtBf05vAcWZWBPw3cIi7e223ECLJpiMIkfi+SNCH0hyCrrK7AoPCae/WSQ4QdMPwIfA2Qcdpg2jY8cBjYQ+ia4HXgNF1ll3i7jXAHIJTX2VABfCAmZ1N0GWHSNIpQYjEZ8B33X1E+Orv7rVHEFt3NTIbS9C52zFh19kfANkJLLs+O+q8rwYyw37+xxD0zvpl4MVGfRKRfaQEIRIoJ3g8Z62XCDqzawtgZoPNrGOc+fKAz9x9m5kdQvBY0VqVtfPHmAmcH17n6E7wONV36wvMguc65IXdcv8PwekpkaTTNQiRwEdAVXiq6CHgboLTO++HF4rXE/x6j/Ui8C0z+whYQHCaqdYk4CMze9/dv1pn/DPAMQTPInfgWndfEyaYeHKAZ80sm+Do4wf79hFFGke9uYqISFw6xSQiInEpQYiISFxKECIiEpcShIiIxKUEISIicSlBiIhIXEoQIiIS1/8HpCwS9/HBW/kAAAAASUVORK5CYII=\n", 110 | "text/plain": [ 111 | "" 112 | ] 113 | }, 114 | "metadata": {}, 115 | "output_type": "display_data" 116 | } 117 | ], 118 | "source": [ 119 | "GD_cost = gradient_descent(1,1,5)\n", 120 | "plt.plot(range(len(GD_cost)),np.log(GD_cost))\n", 121 | "plt.title('Gradient descent')\n", 122 | "plt.xlabel('Iterations')\n", 123 | "plt.ylabel('log(Cost function)')\n", 124 | "plt.legend(['lrn rate = 1'])\n", 125 | "plt.show()" 126 | ] 127 | }, 128 | { 129 | "cell_type": "markdown", 130 | "metadata": {}, 131 | "source": [ 132 | "There are many ways to solve this issue. Randomizing the starting point is always recommended, and shrinking the learning rate as we iterate the algorithm also helps avoiding this kind of situation." 133 | ] 134 | }, 135 | { 136 | "cell_type": "code", 137 | "execution_count": 5, 138 | "metadata": {}, 139 | "outputs": [], 140 | "source": [ 141 | "def shrinking_rate(num_iter):\n", 142 | " x = np.random.rand(1)\n", 143 | " C = []\n", 144 | " learn_rate = 1\n", 145 | " for i in range(num_iter):\n", 146 | " x = x - learn_rate*gradient(x)\n", 147 | " learn_rate = learn_rate/2\n", 148 | " C.append(cost_function(x))\n", 149 | " return C" 150 | ] 151 | }, 152 | { 153 | "cell_type": "code", 154 | "execution_count": 6, 155 | "metadata": {}, 156 | "outputs": [ 157 | { 158 | "data": { 159 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ4AAAEWCAYAAABWn/G6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xt8VNW5//HPNxcCQW4JeIEQQMEqWEUJ1EttvbSC2kqrqGhb8VI9bdWe2p7WS6u1/Nrf0dP+alu19tDq0XpaQaVqbKlaaq2XKhAUL6BAFJCIF+7hTi7P74+9AsMwk0xCZibJPO/Xa16zZ+211n72kPBk7732XjIznHPOuUzJy3YAzjnncosnHueccxnlicc551xGeeJxzjmXUZ54nHPOZZQnHueccxnlice5diRpuaTPhOUbJP0uS3GcJKkmG9t2riWeeFzOkDRZ0hxJWyR9FJa/IUnp2J6Z/V8z++q+9iNpqCSTVNAecWWbpHsl/Tjbcbjs8cTjcoKk7wC/BH4KHAgcAHwNOAHolqRNfsYCdC6HeOJxXZ6kPsBU4Btm9rCZbbLIK2b2JTPbEerdK+kuSbMkbQFOlnSmpFck1UpaKenmuL6/ImmFpLWSvh+37mZJ/xvz+VhJ/5K0QdKrkk6KWfeMpP8j6QVJmyQ9Jal/WP1seN8gabOk4xLsY48Q/3pJi4CxcesHSpopabWkZZK+GbNunKSqsI8fSvp5zLpPxsS8UtLFobxI0s8kvRva/EZSj7DuJEk1kr4Tjizfl3RJWHcF8CXge2FfHk/hn9B1MZ54XC44DigCHkuh7oXAT4BewPPAFuAioC9wJvB1SV8AkDQSuAv4CjAQKAXKEnUqaRDwF+DHQAnwH8BMSQPitn0JsD/RUdh/hPJPhfe+Zrafmb2YYBM/BA4Jr/HAlJht5wGPA68Cg4BTgW9JGh+q/BL4pZn1Du0fDO3Kgb8CtwMDgNHAgtDmVuDQUDY89HtTTDwHAn1C+WXAnZL6mdk04A/Af4V9+Xyi78t1bZ54XC7oD6wxs/qmgpi/4rdJ+lRM3cfM7AUzazSz7Wb2jJm9Hj6/BjwAfDrUnQT82cyeDUdNNwKNSWL4MjDLzGaFvv4GVAFnxNT5HzNbYmbbiP7zH92KfTwP+ImZrTOzlcCvYtaNBQaY2VQz22lm7wC/BSaH9XXAcEn9zWyzmb0Uyr8EzDazB8yszszWmtmCcE3scuCasL1NwP+N6a+pz6mh3SxgM/CxVuyP68K6xMVK51qwFugvqaAp+ZjZ8QBh5FfsH2ArYxtK+gRwC3AE0VFIEfBQWD0wtr6ZbZG0NkkMQ4BzJcX+hV8I/CPm8wcxy1uB/VLauwSxACvitj1Q0oaYsnzgubB8GdGpyLckLQN+ZGZ/BgYDbyfY1gCgGJgfMy5Doc8ma2MTfRv2x3VhnnhcLngR2AFMBGa2UDf+ce1/BO4ATjez7ZJ+QXQEBfA+cHhTRUnFRKfbElkJ3G9ml7cy9kQxJfI+UaJYGD6Xx217mZmNSNi52VLggnBK7mzgYUmlod24BE3WANuAUWb2Xmq7sOcm29DGdSF+qs11eWa2AfgR8GtJkyTtJylP0migZwvNewHrQtIZR3QdpsnDwOfCBfhuREcNyX6n/hf4vKTxkvIldQ8X4RNeE4qzmugU3sHN1HkQuF5Sv9Dn1THr5gK1kq4NgxDyJR0haSyApC9LGmBmjUDTUVED0bWYz0g6T1KBpFJJo0O93wK3Sdo/9DEo5ppRSz5sYV9cF+eJx+UEM/sv4NvA94CPiP7z+2/gWuBfzTT9BjBV0iaii+cPxvS5ELiS6KjofWA9kPCmzXDdZSJwA1EiWQl8lxR+B81sK9GAhxfCdaljE1T7EdHptWXAU8D9Me0bgM8TXTNaRnTE8juii/8AE4CFkjYTDTSYHK5vvUt0Deo7wDqigQVHhTbXAtXAS5Jqgdmkfg3nbmBk2JdHU2zjuhD5RHDOOecyyY94nHPOZZQnHueccxnlicc551xGeeJxzjmXUX4fTwL9+/e3oUOHZjsM55zrVObPn7/GzAa0VM8TTwJDhw6lqqoq22E451ynImlFy7X8VJtzzrkM88TjnHMuozzxOOecyyi/xuOc69Tq6uqoqalh+/bt2Q4lZ3Tv3p2ysjIKCwvb1N4Tj3OuU6upqaFXr14MHTqUmGkaXJqYGWvXrqWmpoZhw4a1qQ8/1eac69S2b99OaWmpJ50MkURpaek+HWF64nHOdXqedDJrX79vTzzt6K0PavmvJ95i47a6bIfinHMdlieedvTu2q38+pm3Wb5mS7ZDcc5l0H77ZWZW7wULFjBr1qy0buOhhx5i1KhR5OXlpe1G+rQmHkkTJC2WVC3pugTriyTNCOvnSBoas+76UL44dmbDZH1KOkXSy5LekHSfpIJQfpKkjZIWhNdN6drfIaXRZJYr1m1N1yacc51EQ0NDm9rV19cnXZeJxHPEEUfwpz/9iU996lNp20baEo+kfOBO4HRgJNGc7iPjql0GrDez4cBtwK2h7UhgMjCKaHbEX4fpehP2GeaKv49o5sQjiGZinBKznefMbHR4TU3TLjO4pAcAKz3xOJeTnnnmGU4++WQuvPBCPv7xj7N8+XIOP/xwLr/8ckaNGsVpp53Gtm3b9mp38cUX8+1vf5uTTz6Za6+9lrlz53L88cdz9NFHc/zxx7N48WJ27tzJTTfdxIwZMxg9ejQzZsxgy5YtXHrppYwdO5ajjz6axx57bJ/34fDDD+djH0t1Mtm2Sedw6nFAtZm9AyBpOtHUv4ti6kwEbg7LDwN3KLpqNRGYbmY7gGWSqkN/JOlzNbDDzJaEOn8DrieaYjdjirsV0H+/Ilas9VNtzmXDjx5fyKJVte3a58iBvfnh50elXH/u3Lm88cYbDBs2jOXLl7N06VIeeOABfvvb33Leeecxc+ZMvvzlL+/VbsmSJcyePZv8/Hxqa2t59tlnKSgoYPbs2dxwww3MnDmTqVOnUlVVxR133AHADTfcwCmnnMI999zDhg0bGDduHJ/5zGfo2bPnrn43bdrEiSeemDDWP/7xj4wcGX88kH7pTDyDiOaVb1IDfCJZHTOrl7QRKA3lL8W1HRSWE/W5BiiUVGFmVcAkYHBMveMkvQqsAv7DzBbGByvpCuAKgPLy8lbs5p6GlBbzrh/xOJezxo0bt8f9LcOGDWP06NEAjBkzhuXLlydsd+6555Kfnw/Axo0bmTJlCkuXLkUSdXWJByw99dRTVFZW8rOf/QyIhpa/++67HH744bvq9OrViwULFrTHrrWbdCaeROPtLMU6ycoTnRo0MzNJk4HbJBUBTwFNJ0pfBoaY2WZJZwCPAiMSdDINmAZQUVERH2fKykuKmbtsXVubO+f2QWuOTNIl9mgDoKioaNdyfn5+wlNt8e1uvPFGTj75ZB555BGWL1/OSSedlLCNmTFz5sxmT43l2hFPDXsedZQRHXEkqlMTBgP0Ada10DZhuZm9CJwIIOk04NBQvuu428xmSfq1pP5mtmaf9i6JwSXFPLrgPXbUN1BUkJ+OTTjnuriNGzcyaFB0kufee+/dVd6rVy82bdq06/P48eO5/fbbuf3225HEK6+8wtFHH71HXx3xiCedo9rmASMkDZPUjWiwQGVcnUp2DwKYBDxtZhbKJ4dRb8OIjlDmNtenpP3DexFwLfCb8PnAcN0ISeOI9nltmvaZISXFmMF76xP/VeOccy353ve+x/XXX88JJ5ywx+i4k08+mUWLFu0aXHDjjTdSV1fHkUceyRFHHMGNN964z9t+5JFHKCsr48UXX+TMM89k/PjxLTdqLTNL2ws4A1gCvA18P5RNBc4Ky92Bh4BqosRycEzb74d2i4HTm+szlP8UeDPU/1ZM+VXAQuBVoutGx7cU95gxY6yt5i5ba0Ou/bP9460P29yHcy51ixYtynYIOSnR9w5UWQq5Ia0PCTWzWcCsuLKbYpa3A+cmafsT4Cep9BnKvwt8N0H5HcAdrY29rYaUFAP4AAPnnEvCn1zQzgb0KqKoII9313ricc65RDzxtDNJlJf4kGrnMik6y+MyZV+/b088aeD38jiXOd27d2ft2rWefDLEwnw83bt3b3MfPhFcGgwuKeZfb0e/CP64dufSq6ysjJqaGlavXp3tUHJG0wykbeWJJw3KS4rZurOBNZt3MqBXUcsNnHNtVlhY2OaZMF12+Km2NBhS6iPbnHMuGU88aVAehlT7U6qdc25vnnjSoKxflHhW+JBq55zbiyeeNOhemM+Bvbv7qTbnnEvAE0+alJcU+6k255xLwBNPmpSXFrNinU8I55xz8TzxpEl5STEf1u5ge13b5l13zrmuyhNPmjSNbKtZ76fbnHMulieeNCkv9ZFtzjmXiCeeNCn36RGccy4hTzxpUtqzGz275fsRj3POxfHEkyaSGOxDqp1zbi+eeNLI5+Vxzrm9eeJJo6Z5eRobfZ4Q55xrktbEI2mCpMWSqiVdl2B9kaQZYf0cSUNj1l0fyhdLGt9Sn5JOkfSypDck3SepIJRL0q9C/dckHZPOfY5VXlLMjvpGVm/ekalNOudch5e2xCMpH7gTOB0YCVwgaWRctcuA9WY2HLgNuDW0HQlMBkYBE4BfS8pP1qekPOA+YLKZHQGsAKaEbZwOjAivK4C70rTLexnsI9ucc24v6TziGQdUm9k7ZrYTmA5MjKszkShhADwMnKpoys6JwHQz22Fmy4Dq0F+yPkuBHWa2JPT1N+CcmG383iIvAX0lHZSOHY43pLQn4PfyOOdcrHQmnkHAypjPNaEsYR0zqwc2EiWRZG2Tla8BCiVVhPJJwOBWxIGkKyRVSapqryl0B/XtQZ78iMc552KlM/EoQVn8VfZkdVpVbmZGdGruNklzgU1AfSviwMymmVmFmVUMGDAgQZPW61aQx0F9eviQaueci1GQxr5r2H3UAVAGrEpSpyYMBugDrGuhbcJyM3sROBFA0mnAoa2II23KS4pZsdafUu2cc03SecQzDxghaZikbkRHJJVxdSrZPQhgEvB0OHqpBCaHUW/DiAYGzG2uT0n7h/ci4FrgNzHbuCiMbjsW2Ghm76dnl/cW3cuzLVObc865Di9tRzxmVi/pKuBJIB+4x8wWSpoKVJlZJXA3cL+kaqIjncmh7UJJDwKLiE6ZXWlmDQCJ+gyb/K6kzxEl07vM7OlQPgs4g2iAwlbgknTtcyLlpcWs2byDLTvq6VmUzgNM55zrHBQdYLhYFRUVVlVV1S59Pf7qKq5+4BWe+NaJHHZg73bp0znnOiJJ882soqV6/uSCNNv1lGofUu2cc4AnnrQbUuo3kTrnXCxPPGnWp0chvboXeOJxzrnAE0+aSdr1sFDnnHOeeDKivKTYr/E451zgiScDBpcUU7N+Gw0+PYJzznniyYQhJT3Z2dDIB7Xbsx2Kc85lnSeeDPAh1c45t5snngxoSjz+sFDnnPPEkxED+3YnP0+sWOcPC3XOOU88GVCQn8egvj38YaHOOYcnnozxe3mccy7iiSdDBpcU867Py+Occ554MqW8pJj1W+uo3V6X7VCccy6rPPFkyBAf2eacc4AnnowZ7PfyOOcc4IknY8p9egTnnAM88WRM7+6F9CsuZIUnHudcjmsx8UjKk3S0pDMlnSLpgFQ7lzRB0mJJ1ZKuS7C+SNKMsH6OpKEx664P5YsljW+pT0mnSnpZ0gJJz0saHsovlrQ6lC+Q9NVU429v5SXFfo3HOZfzCpKtkHQIcC3wGWApsBroDhwqaSvw38B9ZtaYpH0+cCfwWaAGmCep0swWxVS7DFhvZsMlTQZuBc6XNBKYDIwCBgKzJR0a2iTr8y5gopm9KekbwA+Ai0ObGWZ2VWu+mHQoL+3JazUbsh2Gc85lVXNHPD8G/hc4xMzGm9mXzWySmR0JnAX0Ab7STPtxQLWZvWNmO4HpwMS4OhOB+8Lyw8CpkhTKp5vZDjNbBlSH/prr04DeYbkPsKqlnc+08pIevLd+G/UNCXO1c87lhKRHPGZ2QTPrPgJ+0ULfg4CVMZ9rgE8kq2Nm9ZI2AqWh/KW4toPCcrI+vwrMkrQNqAWOjal3jqRPAUuAa8wsto+MKS8ppr7ReH/j9l2j3JxzLtekNLhA0vGSLpR0UdMrlWYJyuJnQktWp7XlANcAZ5hZGfA/wM9D+ePA0HCkNpvdR1h7BiJdIalKUtXq1asTVdln5SU9AR/Z5pzLbakMLrgf+BnwSWBseFWk0HcNMDjmcxl7n/7aVUdSAdEpsnXNtE1YLmkAcJSZzQnlM4DjAcxsrZntCOW/BcYkCtbMpplZhZlVDBgwIIXda72mIdUr/F4e51wOS3qqLUYFMNLMWjtv8zxghKRhwHtEgwUujKtTCUwBXgQmAU+bmUmqBP4o6edEgwtGAHOJjngS9bke6CPpUDNbQjT44E0ASQeZ2fthe2c1lWfDgb270y0/z494nHM5LZXE8wZwIPB+SxVjhWs2VwFPAvnAPWa2UNJUoMrMKoG7gfslVRMd6UwObRdKehBYBNQDV5pZA0CiPkP55cBMSY1EiejSEMo3JZ0V+lnH7pFuGZefJ8r69fAh1c65nKaWDmQk/QMYTXTE0XTKCjM7K72hZU9FRYVVVVWlpe8p98xl7ZYd/PnqE9PSv3POZYuk+WbW4qWYVI54bt73cFyTIaXFvPLu+myH4ZxzWdPi4AIz+yfwFtArvN4MZa4NykuKqd1ez4atO7MdinPOZUUqo9rOIzrNdi5wHjBH0qR0B9ZV7XpKtV/ncc7lqFROtX0fGBtuGiUMXZ5N9KQB10pDYp5SfWRZ3yxH45xzmZfKDaR5TUknWJtiO5fA4H5+L49zLrelcsTzhKQngQfC5/OBWekLqWvrWVRA//2KfEi1cy5ntZh4zOy7ks4BTiC6gXOamT2S9si6sPKSHn6NxzmXs1I54sHMZgIz0xxLzigvKWbech9S7ZzLTUmv1Uh6PrxvklQb89okqTZzIXY95aU9eX/jNnbW+/QIzrnc09y0CJ8M770yF05uKC8pptHgvQ3bGNa/Z7bDcc65jEr16dQtlrnUlfu9PM65HJbKsOhRsR/C9AUJpxZwqYm9l8c553JNc9d4rpe0CTgy9voO8CHwWMYi7IIG7FdEUUEe767dku1QnHMu45ImHjP7z3B956dm1ju8eplZqZldn8EYu5y8PFFeUuxHPM65nJTKqba5kvo0fZDUV9IX0hhTTogSz7Zsh+GccxmXSuL5oZltbPpgZhuAH6YvpNwwuKSYd9duofUTuzrnXOeW0rPaEpSldOOpS25IaTFbdjawbotPj+Ccyy2pJJ4qST+XdIikgyXdBsxPd2BdnQ+pds7lqlQSz9XATmAG8BCwHbgynUHlAk88zrlclcoMpFvM7DozqzCzMWZ2vZmlNA5Y0gRJiyVVS7ouwfoiSTPC+jmShsasuz6UL5Y0vqU+JZ0q6WVJCyQ9L2l4S9vIpl0Twvn0CM65HJPKkwsOlTRN0lOSnm56pdAuH7gTOB0YCVwgaWRctcuA9WY2HLgNuDW0HQlMJrp5dQLwa0n5LfR5F/AlMxsN/BH4QXPbyLbuhfkc0LuIFX7E45zLMakMEngI+A3wO6ChFX2PA6rN7B0ASdOBicCimDoTgZvD8sPAHZIUyqeb2Q5gmaTq0B/N9GlA71CnD7CquW1YBxhONqSkp59qc87lnFQST72Z3dWGvgcBK2M+1wCfSFbHzOolbQRKQ/lLcW0HheVkfX4VmCVpG1ALHNvCNtbEBiLpCuAKgPLy8tbsZ5sNLinmX2+vabmic851IakMLnhc0jckHSSppOmVQjslKIs/ykhWp7XlANcAZ5hZGfA/wM9bEQdmNi1cx6oYMGBAgibtr7ykmA9qt7O9rjUHks4517mlcsQzJbx/N6bMgINbaFcDDI75XMbu01/xdWrCw0f7AOtaaLtXuaQBwFFmNieUzwCeaGEbWTektBgzqFm/jeH775ftcJxzLiNSGdU2LMGrpaQDMA8YIWmYpG5EgwUq4+pUsjuxTQKeDtdeKoHJYUTaMGAEMLeZPtcDfSQdGvr6LPBmC9vIuqaRbSv9Oo9zLoe0eMQj6aJE5Wb2++bahespVwFPAvnAPWa2UNJUoMrMKoG7gfvD4IF1RImEUO9BokED9cCVZtYQ4tmrz1B+OTBTUiNRIro0hJJwGx1B0708K/wp1c65HKKW/viXdHvMx+7AqcDLZjYpnYFlU0VFhVVVVaV9O2bGqB8+yeSx5dz0+fiR5s4517lImm9mFS3Va/GIx8yujuu4D+AzkLYDyadHcM7lnlRGtcXbSnTNxbWDKPH4qTbnXO5I5RrP4+wefpxH9MSAB9MZVC4pLynm2aWrMTOie2edc65rS5p4JBWFJwf8LKa4HlhhZjVpjyxHlJcWs72ukdWbdrB/7+7ZDsc559KuuSOeF4FjgK+a2VcyFE/OiX1KtSce51wuaC7xdJM0BThe0tnxK83sT+kLK3fEJp6Koak8EMI55zq35hLP14AvAX2Bz8etM8ATTzsY1K8HEqzw6RGcczkiaeIxs+eB5yVVmdndGYwppxQV5DOwTw9/eoFzLmek8sgcTzppNrikh9/L45zLGW25j8e1syElPX1COOdczvDE0wGUlxazetMOtu306RGcc11fKlNf/z2VMtd2u55Svd6PepxzXV/SxCOpe5jwrb+kfjGTwA0FBmYqwFwwZNdTqj3xOOe6vuaGU/8b8C2iJDOf3TN51gJ3pjmunBJ7L49zznV1zQ2n/iXwS0lXm9ntyeq5fde3uJBeRQW86/PyOOdyQCqDCz6Q1AtA0g8k/UnSMWmOK6dIorzUp0dwzuWGVBLPjWa2SdIngfHAfcBd6Q0r9/i8PM65XJFK4mka43smcJeZPQZ0S19Iuam8tJiV67fR2Nj8jLDOOdfZpZJ43pP038B5wCxJRSm2c61QXlLMzvpGPty0PduhOOdcWqWSQM4DngQmmNkGoAT4biqdS5ogabGkaknXJVhfJGlGWD8nDNVuWnd9KF8saXxLfUp6TtKC8Fol6dFQfpKkjTHrbkol9kzbNbLNh1Q757q4FmcgNbOtkt4GxocE8JyZPdVSO0n5RMOuPwvUAPMkVZrZophqlwHrzWy4pMnArcD5kkYCk4FRRMO5Z0s6NLRJ2KeZnRiz7ZnAYzHbec7MPtdSzNk0pKQnACvWbeUTB5dmORrnnEufVJ5c8O/AH4D9w+t/JV2dQt/jgGoze8fMdgLTgYlxdSYSDVYAeBg4VdH8zxOB6Wa2w8yWAdWhvxb7DCPwTgEeTSHGDuOgvt3Jz5M/pdo51+WlcqrtMuATZnaTmd0EHAtcnkK7QcDKmM81oSxhHTOrBzYCpc20TaXPLwJ/N7PamLLjJL0q6a+SRiUKVtIVkqokVa1evTqF3Wtfhfl5DOzb3Z9e4Jzr8lJJPGL3yDbCspLUjW8XL37IVrI6rS2PdQHwQMznl4EhZnYUcDtJjoTMbJqZVZhZxYABAxJVSbshJT19SLVzrstLJfH8DzBH0s2SbgZeAlKZo6cGGBzzuQxYlayOpAKgD7CumbbN9implOh03F+aysys1sw2h+VZQKGk/inEn3GDS4r9VJtzrstLZSK4nwOXECWE9cAlZvaLFPqeB4yQNExSN6LBApVxdSqBKWF5EvC0mVkonxxGvQ0DRgBzU+jzXODPZrZrTLKkA8N1IySNC/u8NoX4M25IaTFrt+xk8476bIfinHNpk3RUm6SxQH8z+6uZvUx0ygpJZ0nKM7P5zXVsZvWSriIaip0P3GNmCyVNBarMrJLoyOl+SdVEiW1yaLtQ0oPAIqAeuNLMGsL29+ozZrOTgVviQpkEfF1SPbANmBySW4cztDQaUv3W+7VUDC3JcjTOOZceSvZ/sKRngIvNbHlc+XBgmpmdkvbosqSiosKqqqoyvt1N2+sY+5PZTBpTxo+/8PGMb9855/aFpPlmVtFSveZOtZXGJx0AM6smGnnm2lmv7oVMGHUglQtWsb3OZyN1znVNzSWeHs2s69negbjIpDGDqd1ez+w3P8x2KM45lxbNJZ7Zkn7SdGG+iaQfAU+nN6zcddwhpQzs052H59dkOxTnnEuL5hLPd4CDgWpJM8OrGvgY8O2MRJeD8vPEF48ZxLNLVvNRrT8w1DnX9SRNPGa2xcwuIHou2r3hdZqZTW66L8alxznHlNFo8Mgr72U7FOeca3dJE0/Tk6LDc9EeD693YtZLUln6Q8w9Bw/YjzFD+vHw/Bo66Mhv55xrs+ZOtf00nF67SNIoSftLKpd0iqT/A7wAHJ6hOHPOOceUsfSjzbz+3sZsh+Kcc+2quVNt5wI3El3TuRN4jugpAZcDi4FTzOxvmQgyF5155EEUFeT5IAPnXJfT7Hw8Ye6c72coFhejT49Cxo86kMcWrOL7Zx5OUUF+tkNyzrl20eJEcJLOTlC8EXjdzD5q/5Bck0ljyqh8dRV/f/Mjzvj4QdkOxznn2kWLiYdoPp7jgH+EzycRPaH6UElTzez+NMWW804Y3p8De3dn5vwaTzzOuS4jlWkRGoHDzewcMzsHGAnsAD4BXJvO4HJd0z09zyxZzUeb/J4e51zXkEriGWpmsc9v+Qg41MzWAXXpCcs1OeeYMhoajcdeiZ/KyDnnOqdUEs9zkv4saYqkKUQj256V1BPYkN7w3PD992P04L5+T49zrstIJfFcSTQL6WjgaOA+ovlxtpjZyekMzkUmjSlj8YebWLiqNtuhOOfcPktlBlIDnid6MOhs4NmOOpFaV/X5IwfSze/pcc51ES0mHknnEU07PQk4D5gjaVK6A3O79Sku5LMjD+CxBe+xs74x2+E459w+SeVU2/eBsWY2xcwuAsYRPdHAZdCkMWWs31rH02/5rVPOuc4tlcSTF3ej6NoU2yFpgqTFkqolXZdgfZGkGWH9nKYHk4Z114fyxZLGt9SnpOckLQivVZIeDeWS9KtQ/zVJx6QSe0dz4vD+7N+ryE+3Oec6vVQSyBOSnpR0saSLgb8As1pqJCmf6BlvpxPd+3OBpJFx1S4D1pvZcOA24NbQdiQwGRgFTAB+LSm/uT7N7EQzG21mo4EXgT+FbZwOjAjF2g+nAAAU8ElEQVSvK4C7UtjnDqcgP48vHjOIfyz+iNWbdmQ7HOeca7NUBhd8F5gGHAkcBUwzs1RuHB0HVIdpFXYC04GJcXUmEo2SA3gYODXMeDoRmG5mO8xsGVAd+muxT0m9gFOAR2O28XuLvAT0ldQpHwMwqemengU+T49zrvNK6ZSZmc00s2+b2TVm9kiKfQ8CVsZ8rgllCeuYWT3RM+BKm2mbSp9fBP5uZk1jj1Npg6QrJFVJqlq9enWLO5cNIw7oxVFlfZj5sice51zn1dxEcJsk1SZ4bZKUyg0lSlAWPww7WZ3Wlse6AHiglXFgZtPMrMLMKgYMGJCgSccwaUwZb75fy8JVPk+Pc65zam4+nl5m1jvBq5eZ9U6h7xpgcMznMiD+uS+76kgqAPoA65pp22yfkkqJTsf9pZVxdBqfP2og3fL9nh7nXOeV0qm2NpoHjJA0TFI3osEClXF1KoEpYXkS8HS4ObUSmBxGvQ0jGhgwN4U+zwX+bGaxT9SsBC4Ko9uOBTaa2fvtu6uZ07e4G58ZuT+PLVjl9/Q45zqltCWecM3mKuBJ4E3gQTNbKGmqpLNCtbuBUknVwLeB60LbhcCDwCLgCaJH9DQk6zNms5PZ8zQbRCPw3iEaoPBb4BvtvrMZNmlMGeu27OSZxX5Pj3Ou85E//WZvFRUVVlVVle0wkqpvaOTY/3yaY8r7Mu2iimyH45xzAEiab2Yt/qeUzlNtLk0K8vP44tEDefqtj1i72e/pcc51Lp54OqlzxpRR32g8tqDTjpNwzuUoTzyd1GEH9ubjg/ow82Uf3eac61w88XRi5xwziIWralnk8/Q45zoRTzyd2FmjB1GYLz/qcc51Kp54OrGSnt049bBonp66Br+nxznXOXji6eQmjSljzead/HNxx3y+nHPOxfPE08l9+mMDKO3ZzR+h45zrNDzxdHKF+Xl84ehB/P2tD1m/ZWe2w3HOuRZ54ukCJo0po67BqHzV7+lxznV8nni6gMMP6s3Ig3r76TbnXKfgiaeLmDSmjNff28hbH/g9Pc65js0TTxcxcfRACvLETD/qcc51cJ54uojS/Yo45bD9eeSVVdT7PT3OuQ7ME08Xcv7YwazZvIOfPbUk26E451xSnni6kFMO258vfaKc3/zzbR6Y+262w3HOuYQKsh2Aaz+S+NFZo3hvwzZ+8OgbDOzbg08fOiDbYTnn3B78iKeLKcjP444Lj+HQA3px5R9e5s33fZSbc65j8cTTBe1XVMA9F1fQsyifS++dx4e127MdknPO7ZLWxCNpgqTFkqolXZdgfZGkGWH9HElDY9ZdH8oXSxrfUp+K/ETSEklvSvpmKD9J0kZJC8LrpnTuc0dxUJ8e3HPxWGq31XHpvfPYsqM+2yE55xyQxsQjKR+4EzgdGAlcIGlkXLXLgPVmNhy4Dbg1tB0JTAZGAROAX0vKb6HPi4HBwGFmdjgwPWY7z5nZ6PCa2v572zGNGtiHOy48hjffr+WbD7xCQ6NlOyTnnEvrEc84oNrM3jGznUSJYGJcnYnAfWH5YeBUSQrl081sh5ktA6pDf831+XVgqpk1ApjZR2nct07j5MP250cTj+Dvb33E1McXYubJxzmXXelMPIOAlTGfa0JZwjpmVg9sBEqbadtcn4cA50uqkvRXSSNi6h0n6dVQPipRsJKuCG2rVq/uWnPbfOXYIVx+4jDue3EF97ywPNvhOOdyXDoTjxKUxf+5naxOa8sBioDtZlYB/Ba4J5S/DAwxs6OA24FHEwVrZtPMrMLMKgYM6HpDkK8//XAmjDqQH/9lEU+88UG2w3HO5bB0Jp4aomsuTcqA+Of276ojqQDoA6xrpm1zfdYAM8PyI8CRAGZWa2abw/IsoFBS/33Zsc4oL0/cdv5ojizry7dmvMKClRuyHZJzLkelM/HMA0ZIGiapG9Fggcq4OpXAlLA8CXjaoosQlcDkMOptGDACmNtCn48Cp4TlTwNLACQdGK4bIWkc0T6vbfe97QR6dMvndxdV0H+/Ir563zxWrtua7ZCcczkobYknXLO5CngSeBN40MwWSpoq6axQ7W6gVFI18G3gutB2IfAgsAh4ArjSzBqS9Rn6ugU4R9LrwH8CXw3lk4A3JL0K/AqYbDl8hX1AryLuvWQsO+sbueTeeWzcVpftkJxzOUY5/H9wUhUVFVZVVZXtMNLqX2+vYco9cxk7tIR7LxlHtwK/l9g5t28kzQ/X2Zvl/9vkqOMP6c9/nn0k/3p7LTc88roPs3bOZYw/JDSHTRpTxrvrtvKrvy9lSEkxV586ouVGzjm3jzzx5LhrPjOCleu28v/+toTy0mImjo6/1co559qXJ54cJ4lbzvk4723Yxncfeo2D+vRg3LCSbIflnOvC/BqPo6ggn2lfGUNZSQ8u/30Vjy14j0Z/rptzLk088TgA+hZ3496LxzGwbw/+ffoCzrrzeZ5fuibbYTnnuiBPPG6X8tJi/nL1J/n5eUexfksdX757Dl+5ew5vvLcx26E557oQTzxuD3l54uxjyvj7dz7ND848nNff28jnbn+eb01/xZ904JxrF34DaQK5cANpqjZuq+M3/3ybe55fhhl8+dghXHXKcEp6dst2aM65DibVG0g98STgiWdvH2zczm1/W8JD81fSs1sBXzvpEC45YSjF3XxgpHMu4olnH3jiSW7ph5u49YnFzH7zQ/bvVcQ1nz2Uc8eUUZDvZ22dy3X+yByXFiMO6MXvplTw0NeOo6xfD67/0+uM/8WzPLnwA3/sjnMuJZ54XJuMHVrCzK8fz39/ZQwG/Nv985n0mxeZu2ydJyDnXLP8VFsCfqqtdeobGnmwqoZfzF7CR5t2cEDvIk44pD/HD+/PCcNLOahPj2yH6JzLAL/Gsw888bTN1p31VC5YxXPVa3jx7bWs27ITgIP79+T44aWccEh/jjuklL7FPiLOua7IE88+8MSz7xobjbc+2MS/3l7DC9VrmLNsHVt3NiDBqIG9dx0RjRtaQo9u+dkO1znXDjzx7ANPPO2vrqGRV1du4IXqtbzw9hpeeXc9dQ1GYb44urwfnwyn5Y4s60uhj5BzrlPyxLMPPPGk39ad9cxbvp5/Va/hhbfXsHBVLWbQozCfIaXFDC4pZnC/YgaX9GBwv2LKwnvPIr9vyLmOKtXE47/FLiuKuxXw6UMH8OlDBwCwfstOXnpnLXOXr+PdtVtZsXYLzy9dw7a6hj3alfTsxuB+PSiLS0yDS4oZ2Lc7RQV+2s65ji6tRzySJgC/BPKB35nZLXHri4DfA2OAtcD5ZrY8rLseuAxoAL5pZk8216ckAT8Gzg1t7jKzX4XyXwJnAFuBi83s5ebi9iOejsHMWLtlJyvXbWXl+m2sXLeVmvVbWbluGyvXb2XVhm3UNez++ZXggF7d6VtcSO8ehfTpUUjv7oX07lEQs9xUXrB7uUchPbvlE/2oOOfaKutHPJLygTuBzwI1wDxJlWa2KKbaZcB6MxsuaTJwK3C+pJHAZGAUMBCYLenQ0CZZnxcDg4HDzKxR0v6h/unAiPD6BHBXeHcdnCT671dE//2KOLq8317rGxqND2u375GY3tuwjQ1b66jdXsfKdVup3VZH7fZ6Nu+ob3Zb+Xmid/cCenUvpEdhPkWFeRQV5FFUkE/3wui9qCAvlOfvfi8I9Qp3Lxfm55GfJwryFN6jz4X5e34u2PV5z/I8QZ5EnoTympajd8WsyxOeLF2nlM5TbeOAajN7B0DSdGAiEJt4JgI3h+WHgTvCEcpEYLqZ7QCWSaoO/dFMn18HLjSzRgAz+yhmG7+36NDuJUl9JR1kZu+nY6dd5uTniYF9ezCwb48W/5Kob2hk0/Z6arfXUbutno3b6sJy9L5xW1Reu72O7XUN7KhvZEddI1t31rN+ayM76htjysN7fWNG9rMleyQqRUd+IiwTJScBxH6OW6dQoamcXWW7+wpdhHV7Jrym7TbVj1+3x+e4+Pfqq6UdbqHCvqbizp7M9zX688cO5qsnHtwusSSTzsQzCFgZ87mGvY80dtUxs3pJG4HSUP5SXNtBYTlZn4cQHS19EVhNdHpuaZI4BgF7JB5JVwBXAJSXl6e8k65zKMjPo1/PbvRrx6dqmxk7GxrZXtfIjvoGdtRFyaih0ahvjN7rGmyPz/WNRkNDeI8vbzTqGxoxouHojQaNZlh43/1593KjET6HskbDQmxmhGUwos9NcceXN32OloipG9XZowziPtuuwvgT9/Gn8vdeT7PrE33nza5voX2LOvlYK2uHHei/X1E7RNK8dCaeRIk3/ltJVidZeaJxtk19FgHbzaxC0tnAPcCJKcaBmU0DpkF0jSdBG+f2ICmcbssHCrMdjnOdRjpvmKghuubSpAxYlayOpAKgD7CumbbN9VkDzAzLjwBHtiIO55xzGZLOxDMPGCFpmKRuRIMFKuPqVAJTwvIk4OlwLaYSmCypSNIwooEBc1vo81HglLD8aWBJzDYuUuRYYKNf33HOuexJ26m2cM3mKuBJoqHP95jZQklTgSozqwTuBu4PgwfWESUSQr0HiQYN1ANXmlkDQKI+wyZvAf4g6RpgM/DVUD6LaCh1NdFw6kvStc/OOeda5k8uSMDv43HOudbzieCcc851SJ54nHPOZZQnHueccxnlicc551xG+eCCBCStBla0sXl/YE07htPeOnp80PFj9Pj2jce3bzpyfEPMbEBLlTzxtDNJVamM6siWjh4fdPwYPb594/Htm44eXyr8VJtzzrmM8sTjnHMuozzxtL9p2Q6gBR09Puj4MXp8+8bj2zcdPb4W+TUe55xzGeVHPM455zLKE49zzrmM8sTTRpImSFosqVrSdQnWF0maEdbPkTQ0g7ENlvQPSW9KWijp3xPUOUnSRkkLwuumTMUXtr9c0uth23s9kTVMY/Gr8P29JumYDMb2sZjvZYGkWknfiquT8e9P0j2SPpL0RkxZiaS/SVoa3vslaTsl1FkqaUqiOmmK76eS3gr/ho9I6pukbbM/D2mM72ZJ78X8O56RpG2zv+9pjG9GTGzLJS1I0jbt31+7sjCVrr9SfxFNyfA2cDDQDXgVGBlX5xvAb8LyZGBGBuM7CDgmLPcimpsoPr6TgD9n8TtcDvRvZv0ZwF+JZpA9FpiTxX/rD4hujMvq9wd8CjgGeCOm7L+A68LydcCtCdqVAO+E935huV+G4jsNKAjLtyaKL5WfhzTGdzPwHyn8DDT7+56u+OLW/z/gpmx9f+358iOethkHVJvZO2a2E5gOTIyrMxG4Lyw/DJwqKdE03O3OzN43s5fD8ibgTWBQJrbdjiYCv7fIS0BfSQdlIY5TgbfNrK1Psmg3ZvYs0bxVsWJ/zu4DvpCg6Xjgb2a2zszWA38DJmQiPjN7yszqw8eXiGYAzook318qUvl932fNxRf+7zgPeKC9t5sNnnjaZhCwMuZzDXv/x76rTvjF2wiUZiS6GOEU39HAnASrj5P0qqS/ShqV0cDAgKckzZd0RYL1qXzHmTCZ5L/s2fz+mhxgYUbd8L5/gjod5bu8lOgoNpGWfh7S6apwKvCeJKcqO8L3dyLwoZktTbI+m99fq3niaZtERy7x49JTqZNWkvYDZgLfMrPauNUvE50+Ogq4nWjq8Ew6wcyOAU4HrpT0qbj1HeH76wacBTyUYHW2v7/W6Ajf5feJZhP+Q5IqLf08pMtdwCHAaOB9otNZ8bL+/QEX0PzRTra+vzbxxNM2NcDgmM9lwKpkdSQVAH1o22F+m0gqJEo6fzCzP8WvN7NaM9sclmcBhZL6Zyo+M1sV3j8CHiE6nRErle843U4HXjazD+NXZPv7i/Fh0ynI8P5RgjpZ/S7DYIbPAV+ycEEiXgo/D2lhZh+aWYOZNQK/TbLdbH9/BcDZwIxkdbL1/bWVJ562mQeMkDQs/FU8GaiMq1MJNI0emgQ8neyXrr2F88F3A2+a2c+T1Dmw6ZqTpHFEPwtrMxRfT0m9mpaJLkC/EVetErgojG47FtjYdEopg5L+lZnN7y9O7M/ZFOCxBHWeBE6T1C+cSjotlKWdpAnAtcBZZrY1SZ1Ufh7SFV/sdcMvJtluKr/v6fQZ4C0zq0m0MpvfX5tle3RDZ30RjbpaQjTa5fuhbCrRLxhAd6JTNNXAXODgDMb2SaJTAa8BC8LrDOBrwNdCnauAhUQjdF4Cjs9gfAeH7b4aYmj6/mLjE3Bn+H5fByoy/O9bTJRI+sSUZfX7I0qC7wN1RH+FX0Z03fDvwNLwXhLqVgC/i2l7afhZrAYuyWB81UTXR5p+DptGeg4EZjX385Ch+O4PP1+vESWTg+LjC5/3+n3PRHyh/N6mn7uYuhn//trz5Y/Mcc45l1F+qs0551xGeeJxzjmXUZ54nHPOZZQnHueccxnlicc551xGeeJxLg0kbQ7vQyVd2M593xD3+V/t2b9z6eaJx7n0Ggq0KvFIym+hyh6Jx8yOb2VMzmWVJx7n0usW4MQwT8o1kvLDHDXzwoMp/w12ze/zD0l/JLqhEUmPhoc+Lmx68KOkW4Aeob8/hLKmoyuFvt8Ic7OcH9P3M5IeVjQ3zh9inrpwi6RFIZafZfzbcTmpINsBONfFXUc038vnAEIC2WhmYyUVAS9IeirUHQccYWbLwudLzWydpB7APEkzzew6SVeZ2egE2zqb6GGXRwH9Q5tnw7qjgVFEzxh7AThB0iKix8QcZmamJJO0Odfe/IjHucw6jegZdAuIpqooBUaEdXNjkg7ANyU1PZJncEy9ZD4JPGDRQy8/BP4JjI3pu8aih2EuIDoFWAtsB34n6Wwg4bPUnGtvnnicyywBV5vZ6PAaZmZNRzxbdlWSTiJ6OORxFk298ArR8/9a6juZHTHLDUSzgtYTHWXNJJpA7olW7YlzbeSJx7n02kQ0/XiTJ4Gvh2krkHRoeKJwvD7AejPbKukwoum/m9Q1tY/zLHB+uI40gGgq5bnJAgvzNfWxaFqHbxGdpnMu7fwaj3Pp9RpQH06Z3Qv8kug018vhAv9qEk9X/QTwNUmvAYuJTrc1mQa8JullM/tSTPkjwHFETyk24Htm9kFIXIn0Ah6T1J3oaOmatu2ic63jT6d2zjmXUX6qzTnnXEZ54nHOOZdRnnicc85llCce55xzGeWJxznnXEZ54nHOOZdRnnicc85l1P8HISom2CQ8+WoAAAAASUVORK5CYII=\n", 160 | "text/plain": [ 161 | "" 162 | ] 163 | }, 164 | "metadata": {}, 165 | "output_type": "display_data" 166 | } 167 | ], 168 | "source": [ 169 | "GD_cost = shrinking_rate(20)\n", 170 | "plt.plot(range(len(GD_cost)),(GD_cost))\n", 171 | "plt.title('Gradient descent')\n", 172 | "plt.xlabel('Iterations')\n", 173 | "plt.ylabel('log(Cost function)')\n", 174 | "plt.legend(['lrn rate = 1'])\n", 175 | "plt.show()" 176 | ] 177 | }, 178 | { 179 | "cell_type": "markdown", 180 | "metadata": {}, 181 | "source": [ 182 | "We conclude remarking that even though this example might seen artificial, the more general situation is analogue, and this is a problem that can raise if not enough care is taken when tuning the parameters of the model." 183 | ] 184 | }, 185 | { 186 | "cell_type": "code", 187 | "execution_count": null, 188 | "metadata": {}, 189 | "outputs": [], 190 | "source": [] 191 | } 192 | ], 193 | "metadata": { 194 | "kernelspec": { 195 | "display_name": "Python 3", 196 | "language": "python", 197 | "name": "python3" 198 | }, 199 | "language_info": { 200 | "codemirror_mode": { 201 | "name": "ipython", 202 | "version": 3 203 | }, 204 | "file_extension": ".py", 205 | "mimetype": "text/x-python", 206 | "name": "python", 207 | "nbconvert_exporter": "python", 208 | "pygments_lexer": "ipython3", 209 | "version": "3.6.4" 210 | } 211 | }, 212 | "nbformat": 4, 213 | "nbformat_minor": 2 214 | } 215 | -------------------------------------------------------------------------------- /14. SVM part 2.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "metadata": { 5 | "trusted": true, 6 | "_uuid": "ea9ab27b57fbc7617ef9a747c0bec9b083adb44b" 7 | }, 8 | "cell_type": "markdown", 9 | "source": "# Support vector machine, part 2: soft margins\n\n## Introduction \n\nIn the previous notebook we studied and implemented the Support vector machine algorithm for linearly separable datasets. In this notebook we study the case of non linearly separable data which can be reasonably separated by linear decision boundaries. \n\n## Non-linearly separable datasets\n\nSuppose we have a dataset $S = \\{(x^1,y^1),\\dots,(x^m,y^m) \\}\\subset \\mathbb{R}^n\\times\\{-1,1\\}$ consisting of pairs (features,class). This time we do not assume the data can be separated by a hyperplane, and thus, solutions to the optimization problem posed in the previous notebook may not exist. We recall the optimization problem in its primal form:\n\n$$\n\\begin{align*}\n\\min_{w,b}\\ &\\dfrac{1}{2}\\|w\\|^2\\\\\n\\text{subject to } &y^j(w\\cdot x^j + b) > 1 \\text{ for all } j. \\\\\n\\end{align*}\n$$\n\nHere $w$ represents the normal vector to the separating hyperplane, while $b$ represents a position vector for it. The condition $y^j(w\\cdot x^j + b) > 1 $ can be thought as forcing each point $x^j$ to be in the correct side of the hyperplane. If this condition cannot be met for all points, we will relax it by requiring $y^j(w\\cdot x^j + b) > 1 -\\epsilon_j $ each $j$, where $\\epsilon_j\\geq 0$. This can be thought that we allow points to cross the hyperplane and be in the wrong side of it, up to a certain margin, controlled individually for each point by $\\epsilon_i$. We want to allow the smallest margin of error as possible, so we penalize the size of the of the coefficients $\\epsilon_i$. Thus, we reformulate the optimization problem as \n\n$$\n\\begin{align*}\n\\min_{w,b,\\epsilon}\\ &\\dfrac{1}{2}\\|w\\|^2 + C\\sum_{i=1}^m \\epsilon_i \\\\\n\\text{subject to } &y^j(w\\cdot x^j + b) > 1 - \\epsilon_i ,\\\\\n&\\epsilon_i \\geq 0\\text{ for all } j.\n\\end{align*}\n$$\n\nHere $C$ is a constant that gauges the interaction between the two terms to optimize. It constitutes a hyperparameter of the model and can be optimized using cross-validation methods. The Lagrangian to minimize for the dual problem is given by\n\n$$\n\\mathcal L (w,b,\\alpha,\\beta,\\epsilon) = -(\\dfrac{1}{2}\\|w\\|^2 + C\\sum_{i=1}^m \\epsilon_i - \\sum_{i=1}^m \\alpha_i (y^i(w\\cdot x^i + b) - (1 -\\epsilon_i)) -\\sum_{i=1}^m \\beta_i \\epsilon_i).\n$$\n\nThe [KKT](https://en.wikipedia.org/wiki/Karush–Kuhn–Tucker_conditions) for this problem are then:\n\n$$\n\\begin{align*}\n\\nabla_w \\mathcal{L} = -(w - \\sum_{i=1}^m \\alpha_i y^i x^i) &= 0, \\\\\n\\nabla_b \\mathcal{L} = -\\sum_{i=1}^m \\alpha_i y^i &= 0 , \\\\\n\\nabla_\\epsilon \\mathcal{L} = - (C-\\alpha_i - \\beta_i) &=0, \\\\\n\\alpha_i (y^i(w\\cdot x^i + b) - (1-\\epsilon_i) ) & = 0,\\\\\n\\beta_i \\epsilon_i &=0 , \\\\\n\\alpha_i & \\geq 0, \\\\\n\\beta_i & \\geq 0, \\\\\n\\epsilon_i & \\geq 0,\n\\end{align*}\n$$\n\nfor all $i$. Substituting the gradient equations in the Lagrangian, we obtain a new optimization problem given by\n\n$$\n\\begin{align*}\n\\min_{\\alpha} \\dfrac{1}{2} \\sum_{i,j=1}^m \\alpha_i \\alpha_ j y^i y^j x^i\\cdot x^j -\\sum_{i=1}^m \\alpha_i \n\\end{align*}\n$$\n\nsubject to \n\n$$\n\\begin{align*}\n\\sum_{i=1}^m \\alpha_i y^i &= 0 \\\\\nC\\geq \\alpha_i & \\geq 0. \\\\\n\\end{align*}\n$$\n\nIf we solve this problem, we can recover $w$ from\n\n$$\nw = \\sum_{i=1}^m \\alpha_i y^i x^i\n$$\n\nand to recover $b$, we fix a support vector $x^s$ (that is, $0<\\alpha_s < C$) and $\\epsilon_s = 0$ (since $\\beta_s\\neq 0$) and from the condition $\\alpha_i (y^i(w\\cdot x^i + b) - (1-\\epsilon_i) ) = 0$ it follows that \n\n$$\nb = y_s^{-1} - w\\cdot x^s.\n$$\n\nThe predictions are then given by\n\n$$\n\\widehat y = \\mathrm{sign}(w\\cdot x + b).\n$$\n\nWe proceed to implement the algorithm.\n" 10 | }, 11 | { 12 | "metadata": { 13 | "trusted": true, 14 | "_uuid": "bad345e3bbb070073830b35876e66e8606756d9f" 15 | }, 16 | "cell_type": "markdown", 17 | "source": "## Implementation\n\nIn this section we implement the algorithm described above. We follow the same outline of the previous notebook." 18 | }, 19 | { 20 | "metadata": { 21 | "trusted": true, 22 | "_uuid": "f8a0d13fd5d6c5ad71612970ce8490fa47f02e30" 23 | }, 24 | "cell_type": "code", 25 | "source": "import numpy as np\nimport matplotlib.pyplot as plt\nfrom scipy.optimize import minimize", 26 | "execution_count": 1, 27 | "outputs": [] 28 | }, 29 | { 30 | "metadata": { 31 | "trusted": true, 32 | "_uuid": "617875a12555609e9edfe89819d68a503fc87247" 33 | }, 34 | "cell_type": "markdown", 35 | "source": "We generate two classes of points which are not linearly separable. We fix a seed of random numbers so we always get the same dataset:" 36 | }, 37 | { 38 | "metadata": { 39 | "trusted": true, 40 | "_uuid": "1527939ac76db262eeebe28cf3092d647075ed7e" 41 | }, 42 | "cell_type": "code", 43 | "source": "np.random.seed(6)\nclass_0 = 25 \nclass_1 = 25\n\nx0 = np.random.multivariate_normal([-2, 0], [[1, .15],[.15, 1]], class_0)\nx1 = np.random.multivariate_normal([0, 2], [[1, .25],[.25, 1]], class_1)\n\nplt.plot(x0[:,0],x0[:,1],'x',color='red')\nplt.plot(x1[:,0],x1[:,1],'x',color='blue')\nplt.figtext(0.5, 0.01, 'Figure 1: two Gaussian clouds of points. Red points represent class 1 and blue class 0', \n wrap=True, horizontalalignment='center', fontsize=12)\nplt.axis('equal')\nplt.show()\n\ny0 = -np.ones(len(x0))\ny1 = np.ones(len(x1))\n\nX_train = np.vstack((x0,x1))\ny_train = np.append(y0,y1)", 44 | "execution_count": 2, 45 | "outputs": [ 46 | { 47 | "output_type": "display_data", 48 | "data": { 49 | "text/plain": "
", 50 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhUAAAEPCAYAAAD1QTYeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XmcXFWZ//HvAwRBQFDSihAgKuAIyCIdxG2kWTRBkGUUUIEfo77QzDjCAEaidn4h6CCtoDI6IKP+HEHFHlkUFFkmBf4cAbvDpmEbRCBs0oBIEBTRZ/44p+jb1bV2bt1Ty+f9etWr+1bduve5+1PnnHuPubsAAADW1FqpAwAAAL2BpAIAAOSCpAIAAOSCpAIAAOSCpAIAAOSCpAIAAOSCpAIAAOSCpAIAAOSCpAIAAORindQBoLHZs2f73LlzU4cBAF1jxYoVj7r7QOo4+g1JRReYO3euxsfHU4cBAF3DzO5NHUM/ovoDAADkgqQCAADkgqQCAADkgqQCAADkgqQCAADkgqQCADrEyIhUKk19r1QK7wPdgKQCADrEvHnSoYdOJhalUhieNy9tXECzeE4FAHSIoSFpdDQkEgsXSmedFYaHhlJHBjSHkgoA6CBDQyGhOOWU8JeEAt2EpCIRM1vbzG40s0tTxwKgc5RKoYRieDj8rWxjAXQykop0jpV0W+ogAHSOchuK0VFp2bLJqhASC3QLkooEzGyOpHdI+lrqWAB0jrGxqW0oym0sxsbSxgU0i4aaaXxR0iJJG6UOBEDnWLRo+ntDQ7SrQPegpKJgZra/pEfcfUWD8Y4xs3EzG5+YmCgoOgAAZo6konhvkvROM7tH0vmS9jKz8ypHcvdz3H3Q3QcHBgaKjhEAgJaRVBTM3Re7+xx3nyvpcEnL3f2IxGEBALDGSCoAtA2PnQb6C0lFQu5+tbvvnzoOoF147HR7kbSh05BUAGib7GOnlyyZfAYDdzPkg6QNnYakAkBb8djp9iFpQ6chqQDQVjx2ur1I2tBJSCoAtA2PnW4/kjZ0EpIKAG3DY6fbi6QNncbcPXUMaGBwcNDHx8dThwGgw4yMhEaZ2SqPUikkbdUe+d1PzGyFuw+mjqPfkFR0AZIKAGgNSUUaVH8AAIBckFQAAIBckFQAAIBckFQAAIBckFQAAIBckFQAAIBckFQAAIBckFQAAIBckFQAAIBckFQAAIBckFQAAIBckFQAAIBckFQAAIBckFQAAIBckFQAAIBckFQAAIBckFQAAIBckFQUzMzWM7NfmNnNZrbSzE5OHRPQbiMjUqk09b1SKbwPoHeQVBTvT5L2cvedJe0iab6Z7ZE4JqCt5s2TDj10MrEolcLwvHlp4wKQr3VSB9Bv3N0lPRUHZ8WXp4sIaL+hIWl0NCQSCxdKZ50VhoeGUkcGIE+UVCRgZmub2U2SHpF0pbtfnzomoN2GhkJCccop4S8JBdB7SCoScPe/uPsukuZI2t3Mdqwcx8yOMbNxMxufmJgoPkggZ6VSKKEYHg5/K9tYtBNtOoBikFQk5O5PSCpJml/ls3PcfdDdBwcGBooPDshRuQ3F6Ki0bNlkVUhRiQVtOoBikFQUzMwGzGyT+P/6kvaVdHvaqID2Ghub2oai3MZibKyY+WfbdCxZMpngUAUD5IuGmsV7uaT/MLO1FZK6UXe/NHFMQFstWjT9vaGhYi/q2TYdw8MkFEA7kFQUzN1vkbRr6jiAflPZpqPopAboB1R/AOh5qdt0AP2CpAJAz0vdpgPoFxaexYRONjg46OPj46nDAICuYWYr3H0wdRz9hpIKAACQC5IKAACQC5IKAACQC5IKAF2NR3ADnYOkAkBSa5oUtPoIbpIQoH1IKgAktab9crT6CG76AQHah6QCQFJTkoJ9fq5DD352alLQRDFCK92q0w8I0D4kFQCSez4p+K83auGzZ2pIrRUjtNqteqMkhCoSYGZIKgAkNyUpWPejKh30paaLEWbyCO5GSQhVJMAMuTuvDn/ttttuDvSq5cvdZ88Of58ffuFTvlx7ug8PN/z+aadNfjc7zdNOa2F+s6tPY/bsEEK1z9HZJI17B5y/++1FSQWAwlSrVjj/fOmQQzL9cqik0XXep7G9FzdVl7Fo0fSCjKGh6t2tS833A9JKOw0AAUkFgMJUq1a48ELp8MM1+cahh2ro4mO16Kq3taU70WaTkFbbaQAgqQBQoIZ3XnRId6J0lQ7MDL2UdgF6KUWvWbIkVCsMD4eLdqcZGQmlKtkSjVIp5Da1qlXQWeilNA2Sii5AUoFeUi4FWLgwVCvwjAi0A0lFGlR/ACgM1QpAbyOpAFCYDmkyAaBNSCoAFIJ2CkDvI6kAUAieUgn0vnVSBwCgP2RvJ6WRJtCbKKkAUBieUgn0NpKKgpnZlmZWMrNbzWylmR2bOiagKDylEuhtVH8U7zlJJ7j7DWa2kaQVZnalu9+aOjCgnbK3kw4NhVcTnZAC6CKUVBTM3R9y9xvi/6sl3SZpi7RRAe3XSbeTVuvYrFQK7wOYOZKKhMxsrqRdJV2fNhKg/VrtTbSduBMFaA+SikTMbENJF0g6zt2frPL5MWY2bmbjExMTxQeIrsIv79Y07NgMwIyQVCRgZrMUEopvu/uF1cZx93PcfdDdBwcGBooNEF2HX96t404UIH8kFQUzM5P0dUm3ufsZqeNBb+CXd+u4EwXIH0lF8d4k6UhJe5nZTfG1X+qg0P345d08OjYD2oOkomDu/jN3N3ffyd13ia8fp44L3Y9f3s3L3olSbneSvROF9ijAzJBUAD2AX96tyd6JUm6PUn6f9ijAzJFUAD2gk54B0W1ojwLkx9w9dQxoYHBw0MfHx1OHAfS0JUtCe5Th4VDag+5mZivcfTB1HP2GkgoAfY/2KEA+SCoA9DXaowD5IakA0NdojwLkhzYVXYA2FQDQGtpUpEFJBQAAyAVJBYC2o8MzoD+QVABoOzo8A/rDOqkDAND7sg+YWrgw3LbJA6aA3kNJBYBC0OEZ0PtIKgAUggdMAb2PpAJA2/GAKaA/kFQAaLtOfMAUd6QA+SOpANB22a7Gy4aGwvupcEcKkD/u/gDQl7gjBcgfJRUA+lard6RQZQLUR1IB9DKugpJqr4bXv14688ypd6TUWz1UmQD1kVQAvYyroKTqq+Hgg6WVKyX3yaqQgw+WDjqo9urJVpksWTJ5RwtVJkBAUgH0Mq6CkqqvhsMOky65RLr44smEw106/PD6q2cmD/GiwAj9gqQC6HU8ylLS9NXw1a+G98bGpAULwvvHHhver3fBn8lDvCgwQt9wd14d/tptt90cmLHly91nz3YfHg5/ly9PHVEStVbD6ae7m7kfeWR4//TTa6+m8jTKn1UOz2T+aA9J494B5+9+eyUPgFfjF0lFnzrttOlXnuXLw/vNWpOrYA+ptRpOP9199gZ/8NMX3umzZ4fEwsz99IV3Vl3Pa7pJhofDWXd4eA0XCA2RVKR5Uf1RMDP7hpk9Yma/Sh0LOlweZead+CjLBGqthquukkaX3aHj//ONWrjgHp17rrTP6x7Tc//xnefXc7YqZE0e4kXfJ+gLqbOafntJ+ltJr5P0q2a/Q0lFH6PMvBDLT7/BZ9uEH/naG930l1BS4fkV7FBgVDxRUpHkRUlFwdz9p5IeTx0HugSNLNuuVJIOPXVXjR5xib71y131+X0v14lnb6ujjsrvZhkKjNAvSCqATkaZeduNjUmji2/U0GWLpOFhHX/jUTpin4d07rn55XGd2PcJ0A4kFR3KzI4xs3EzG5+YmEgdTn9L9ZCBNvQXzvMSpls0r6ShU9/2/HouLb5Cl101S8NH3kMeB7SIpKJDufs57j7o7oMDAwOpw+lvqR4y0IYyc56XUEVmPT9fFfL5VVq242geeRzQVyy0Z0GRzGyupEvdfcdmxh8cHPTx8fG2xoQGylffHujOsocWZZqRkZAgZZenVAp5QzNVDWv6fXQOM1vh7oOp4+g3lFQUzMy+K+laSa82s/vN7AOpY0ITeqjBZA8tyjRrWhJD2wdgzZBUFMzd3+PuL3f3We4+x92/njomNKGHGkz20KJMQ1cnQFokFUAjbWgwmUoPLUpNvVwSA3Q6kgqgkU5/yEALt3R0+qJIWuNbVHq5JAboeKmfvsWr8YsnaqKuXntc4xosT6+tCsyceKJmkhclFUC367WGBGuwPF1REgP0MG4p7QLcUoqmLFkSGhIMD4cGE92u15YHheKW0jQoqQB6Qa81JOi15QH6BEkF0O167ZaOXlseoI+QVADdrtcaEvTa8gB9hDYVXYA2FWgbnkuNHkWbijQoqQB6wUyf7UAPYwByRFIB9IKZJgeJb0elK3agt5BUAL1gTZKDhM+1pqAE6C0kFUCvmGlykPD2zV57bhfQ70gqgF4xk+SgA27fpAMwoHeQVAC9YKbJQQfcvslzroDewS2lXYBbStFQl94ams2FhoamDwMzxS2laZBUdAGSCvSqLs2F0AVIKtIgqegCJBUA0BqSijRoUwEAAHJBUgEAAHJBUgEAAHJBUgEAAHJBUgEAAHJBUgG0Ez1mAegjJBUJmNl8M7vDzO4ys5NSx4M2oscsAH2EpKJgZra2pK9IWiBpe0nvMbPt00aFtim6xyxKRgAkRFJRvN0l3eXud7v7s5LOl3Rg4pgwU81cxIvsMYuSEQAJkVQUbwtJqzLD98f30I2auYgX2WMWfYkDSIikokOZ2TFmNm5m4xMTE6nDQS2NLuIpuhbPlowsWDC9Yw2qQgC0CUlF8R6QtGVmeE58bwp3P8fdB919cGBgoLDgMAP1qjdSdC1eLhk58kjpvPOkM86YfJ+qEABttE7qAPrQmKRtzewVCsnE4ZLemzYkrJHK6o2hockkolpXm9nP2xFLtrRkl12kE0+UbrpJuvDCUFpCl6AA2oSSioK5+3OSPiLpckm3SRp195Vpo8KMpajeqKeyZOT446UjjpDOPVc65BDp1FNpxAmgbSipSMDdfyzpx6njQA7qVW+kaBxZWeJQKkmXXTZZirJ4cUgkFi4MwzTiBJAjc/fUMaCBwcFBHx8fTx0Guk1lVUh5eMGCUHIxPBxKV1IYGQklJFTFoE3MbIW7D6aOo99Q/QH0qmqlKIsXSxdcUMztrfXwPA2gJ1FS0QUoqUAuapVcpKoCKc+fqhi0ASUVaVBSAfSLFLe31lPkk0YBFIKSii5ASQV6EiUVaCNKKtKgpAJA8TrtVlwAuSCpAFC8TquKAZALqj+6ANUfANAaqj/SoKQCAADkgqQCQOcaGZnezoKeVoGORVIBoHPxkCygq9D3B4DOVW7Aya2nQFegpAJAZ+MhWUDXIKkA0NlKpVBCkbq/EgANkVQA6Fw8JAvoKiQVADoXD8kCugoPv+oCPPyqICMj4a6CbJ19qRQuYIsWpYsLQMt4+FUalFQAZdy+OBXPiADQIpIKoCx7++KSJZN1+f16t0GeSRYJCtAXSCqALG5fnJRnkkUpENAXSCqALG5fnCqvJItSIKAvkFQAZdy+OF2eSRalQEDPI6kAyrh9caq8kyxKgYCexy2lXYBbSpFEnrfYZhOUoaHpw0DOuKU0DZKKApnZuyUtlfQaSbu7e1OZAkkFuh7PAEHBSCrSoJfSYv1K0iGSvpo6EKBQ1RKHoSFKKYAeQ1JRIHe/TZLMLHUoAADkjoaaAAAgF5RU5MzMrpK0WZWPPunuP2hhOsdIOkaSttpqq5yiAwCgfUgqcubu++Q0nXMknSOFhpp5TBMAgHai+gMAAOSCpKJAZnawmd0v6Q2SfmRml6eOCQCAvFD9USB3v0jSRanjAACgHSipAAAAuSCpAAAAuSCpAAAAuSCpAAAAuSCpAAAAuSCpADrByEjotTOrVArv98L8APQFkgqgE8ybJx166OSFvlQKw/Pm9cb8APQFnlMBdIKhIWl0NFzYFy6UzjorDLera/Ci5wegL1BSAXSKoaFwgT/llPC31gU+r6qLZucHAE0iqQCK0EwiUCqFEoPh4fC3cvyyvKoump0fADSJpAIoQqNEoDw8OiotWzZZNVHtQp+tuliyZPJ7rZQ0tDI/AGgSSQVQhEaJwNjY1OHy+GNjtae3JlUXrc4PAJpg7p46BjQwODjo4+PjqcNIa2Qk/KrPXjxLpXARXLQoXVytWrIkJALDw6GEYKbKJQ00sgSqMrMV7j6YOo5+Q0kFukMv3AKZVxsGqi4AdCiSCnSHPNoRpJRnIkDVBYAORfVHF6D6IyOv6oOi9Ur1DdAlqP5Ig6SiC5BURLQjANAkkoo0qP5Ad6AdAQB0PJIKdAfaEQBAx6P6owtQ/QEAraH6Iw1KKoB+RNfnANqApALoR73w3A8AHYeuz4F+RNfnANqAkgqgX9H1OYCckVQUyMw+Z2a3m9ktZnaRmW2SOib0Mbo+B5AzkopiXSlpR3ffSdKdkhYnjgf9iud+AGgDkooCufsV7v5cHLxO0pyU8aCP8dwPAG1AQ8103i/pe6mDQJ+q1t/I0BDtKgCsEZKKnJnZVZI2q/LRJ939B3GcT0p6TtK360znGEnHSNJWW23VhkgBAMgXSUXO3H2fep+b2dGS9pe0t9d5nKm7nyPpHCk8UTPPGAEAaAeSigKZ2XxJiyS91d2fTh0PAAB5oqFmsb4saSNJV5rZTWZ2duqAAADICyUVBXL3bVLHAABAu9BLaRcwswlJ987w67MlPZpjON2K9RCwHgLWw6ReXRdbu/tA6iD6DUlFjzOzcbr/ZT2UsR4C1sMk1gXyRJsKAACQC5IKAACQC5KK3ndO6gA6BOshYD0ErIdJrAvkhjYVAAAgF5RUAACAXJBU9Akz+yczu93MVprZSOp4UjKzE8zMzWx26lhSMLPPxX3hFjO7yMw2SR1TkcxsvpndYWZ3mdlJqeNJwcy2NLOSmd0azwnHpo4JvYGkog+Y2ZCkAyXt7O47SPp84pCSMbMtJb1N0n2pY0noSkk7uvtOku6UtDhxPIUxs7UlfUXSAknbS3qPmW2fNqoknpN0grtvL2kPSf/Yp+sBOSOp6A8LJX3W3f8kSe7+SOJ4UvqCQv8rfduYyN2vcPfn4uB1kuakjKdgu0u6y93vdvdnJZ2vkHD3FXd/yN1viP+vlnSbpC3SRoVeQFLRH7aT9BYzu97MrjGzeakDSsHMDpT0gLvfnDqWDvJ+SZelDqJAW0halRm+X31+MTWzuZJ2lXR92kjQC+j7o0eY2VWSNqvy0ScVtvNLFIo550kaNbNX1ut6vVs1WA+fUKj66Hn11oO7/yCO80mFYvBvFxkbOoeZbSjpAknHufuTqeNB9yOp6BHuvk+tz8xsoaQLYxLxCzP7q8Lz/ieKiq8otdaDmb1W0isk3WxmUijyv8HMdnf3hwsMsRD19gdJMrOjJe0vae9eTC7reEDSlpnhOfG9vmNmsxQSim+7+4Wp40FvoPqjP1wsaUiSzGw7SeuqNzsQqsndf+nuL3X3ue4+V6HY+3W9mFA0YmbzFdqVvNPdn04dT8HGJG1rZq8ws3UlHS7ph4ljKpyFzPrrkm5z9zNSx4PeQVLRH74h6ZVm9iuFhmn/p89+nWKqL0vaSNKVZnaTmZ2dOqCixAaqH5F0uULjxFF3X5k2qiTeJOlISXvFfeAmM9svdVDofjxREwAA5IKSCgAAkAuSCgAAkAuSCgAAkAuSCgAAkAuSCgAAkAuSCgAAkAuSCgAAkIsZJxVmtpWZPRW7EkaXMrOVZrZnwfN0M9umDdNdambn5TzN9c3sEjP7vZn9Z87TPtvMhvOcZgpt3J6F75sI2rVNO42ZfdPMPl3js6PN7Gd1vnu1mX2wfdFVnWfHb5eGSYWZ3WNmz8QEovza3N3vc/cN3f0vRQTaIMZ1zez7MVZv9URU1M5hwUfM7BYze9rMHo7zPrzd867F3Xdw96tTzb8LvEvSyyRt6u7vznPC7v5hdz+lmXHrnfxaVXFMPxynvWEe085TK/tmXKa6/Z30EzObG8+FHd+/k5ntaGaXm9mjZsbTGDuImf1zPEc8aWbfMLMXNPpOsyUVB8QEovx6cA1jrWuGB8LPJB0hqZP7cjhT0nGSTpC0qUKXy5+SND9lUKhra0l3xsc795ID3H1DSbsodHu9OHE8HSHvi3A3XNQ7wJ8ljUr6QOpAMMnM3i7pJEl7K5wHXynp5IZfdPe6L0n3SNqnyvtzJbmkdeLwKyT9VNJqSVdJ+oqk8+Jne0q6v9Z0JS2V9H1J50l6UtIHFRKekyT9WtJjCjvdS5qI935JezYaLzP+ZyT9RdIfJT2l0C/CyZL+NX4+S9IfJH0uDq8fx31JHH6npJWSnpB0taTX1JjPdnE+gw3i+XuFPglWS7pb0ocynx0t6WcV47ukbeL/+0m6NX73AUknxvdnS7o0xvi4pP8vaa0q22F3SdfG8R6K62Ldinl9WNL/xHG+ovio9yrLsbZCV+O/jvGskLRllZg3lvQthR5T71VIssqxLS3vQ3X2uWvi9K+M8Zb3ufXi/vRYjHVM0stqxPqauO2eiNvynfH9kyU9q3DSe0rSB6p8d6nCvvu9GMcNknZuNO342TclfTp7jCgknI/E9f/38bNjYgzPxjguie9/PG7n1ZLuUOhxtOVjWtKIpB9lhl8g6fOS7pP0W0lnS1o/8/nHYnwPSnp/dntWmdfVkk6V9AuFY/sHyhzHqnP8aPo5YjTuK6vjdwbjZ+dK+qukZ+L6WdTi9r8nrstbJP1JoffmzRV68JyQ9BtJH21hm7c6vd0ljcf181tJZ2Q+20PSz+My3KzMuS2ur1Mk/XeM4wpJs+Nn98Xt8lR8vWENjtF3SLoxxrdK0tLMNGquZ4Xz1d1x2r+R9L4G++U2kryJ/fdLMY4nY8xvqdg2VfeT+PmucXutjtvvfMVjsMp8jo7r9suSfi/pdmWOsbj+P9jkuWpjhQ7cHlI4Zj8tae01PHe2dbtI+o6kf8kM7y3p4Ybbp9UTUJ2Vdq3CiWhdSW+OC9pKUvFnSQcpJBPrSzpW0nUKXRO/QNJXJX23iXinJRWS3ivpljrfeX7niMN7Sfpl/P+NceNen/ns5vj/dgoJx74KycciSXcpcyHOTPPDku5pIv53SHqVJJP0VklPK/SmWd4Z6iUVDykeYJJenPneqQoXhlnx9RZN9vuS3Q67KZzE1onb9zZJx1XM61JJm0jaSuEEOb/GcnxM0i8lvTouy84KVQiVMX9L4UKzUZznnYoXbzU+UK+VdEbcP/5W4SAp73MfknSJpBcqHKS7SXpRlThnxW32CYV9d684nVdXi6HK95cq7LvvitM6UeFAndXEtL+pqUnFc5KWxe/tF7f9iyvHjcOvVjiRbJ5ZN69qtH9V2eZz4nb6UubzLyj03PmSuF0ukXRq/Gy+woVvR0kbKJx4GiUVD2TGvyCzjeoeP5p+jvhjXC9rK+zT19U6TzW7/TPfvUmhS/T1Fc5BKyQtidvtlQon4bc32uYznN61ko6M/28oaY/4/xYKF4X94jT2jcMDmXX767ge14/Dn612rKzhMbqnpNfGGHaK2/+geus5busnNbmvv1zSDg32y2aTiiMUSnrXUUjCH5a0XqP9JK77eyX9c9xu74rbsV5S8Vxm/MMUkovyD8qr1XxScZHCNWwDSS9VSLI/VGO+HbFdFJLYwzLDs+P8N627fZo8AT2lkO08IeniypWmcIF5TtILM987T60lFT+t+Pw2Tc0KXx53gJoHSRyvpZKKyp0jDpdLIzZVKC35RJzuhgq/Xs+M4w0r9HJY/t5aCifQafNX+AV+XZVYn4jz2rpGbBdLOjazk9dLKu6LO9OLKsZZpnDhnnbiV42kMX52nKSLKub15szwqKSTanz3DkkH1vjMFU4gayv8+t4+89mHJF3d6EDN7HMbZD7/Tmafe7/CL7ydGmz7tyiclNbKvPddxay/MoYq31+qqRe3tRSTuyam/U1NTSqeye7fCiUWe1SOG4e3iZ/vo3gxa2F/v0fhmF4d1+d/SdokfmYKF/pXZcZ/g6TfxP+/oXjhisPbqXFSkR1/+7jN11aD40fTzxFXVUznmVr7cbPbP/Pd92eGXy/pvopxFkv6f422+Qyn91OF88rsinE+LuncivcuV+hluLxuP5X57B8k/aTyWKmz3A2P0RqffVHSF+qtZ4WL1xOS/k6ZUq4G26GppKLK936nWFJUbz9R+OHxoDKlqzH2eklF5fi/0GQCeLWaSCoU2mT9SVNL+94jqdTJ20UhYZ2fGZ4V5z+33veabVNxkLtvEl8HVfl8c0mPu/vTmfdWNTntWuNvLekiM3vCzJ5QSDL+orCB2srdn1Eojnyrwo54jcIGelN875o46uYKmW/5e39VWI4tqkz2MYXEKDufOQrZ3wsUTuYyswVmdp2ZPR6Xe784TjP+Lo5/r5ldY2ZviO9/TuEX4BVmdreZnVTty2a2nZldWm6YI+lfqsw722blaYVEq5otFXbKemYr7Kj3Zt67V9XXX6XNJf3O3f9Q8d2ycxVOwOeb2YNmNmJms2pMZ1Xcdq3GUPb8vhunc3+cbqvTfsyntt2ouX7d/S6FpG+ppEfM7Hwz27yFmA9y940Ukpm/0eR2HlD4dbMic+z9JL6v8jJVLE8jlePPivNr5fiRpu9769Vps9Ds9q8W49aSNi8vf1wHn9DUc0+tbT6T6X1AITm73czGzGz/zPfeXfG9N2vqeaTZ47GaZo5RmdnrzaxkZhNm9nuFUtfy/lJ1Pcfj8rA47kNm9iMz+5sWYqsXz4lmdlu8I+sJhaqF7Hmq1n6yuaQHPF4ho0b7b7XxWznOpLAdZymsh/J2/KpCiUU1nbJdnlIo3Sgr/7+6Xlx5PafiIUkvMbMXZt7bMvP/HxROVJIkC7ehDmgqrxheJWlBJpnZxN3Xc/cHcoq53rylkDjspVAHNxaH365Q//nTOM6DCjuMpHB3h8JyV4txuaQ5ZjZYK4jYsvYChWqkl7n7JpJ+rJhwaPp63GzKQriPufuBCjvrxQolCXL31e5+gru/UqEO+3gz27tKCGcp1Btu6+4vUjjxWZXxmrFKoRqnnkcVSp+2zry3lSbX35TllZRd3ockvdjMNqj4riTJ3f/s7ie7+/YKVVj7SzqqSgwPStrSzLLHQjaGZjy/r8cWuJCHAAAFuklEQVTpzInTzWPaZdP2UXf/jru/WWH9uaTTWp6o+zUKpSCfj289qlBiskPmuNvYQ6NOKaz37LG9lRqrHP/PcT6tHD8NF2XKQPPbv9r3VymUzGTPPRu5+37Vlqlim7c8PXf/H3d/j8Jxe5qk78f9epVCSUX2exu4+2dbXR81NHOMSqEE8IcK9fobK1SlWoy95np298vdfV+FJOh2Sf/exLzqMrO3KFSTHapQNbiJQpVEM+ephyRtEfezskb7b7Xxq92oUO9ctUqhpGJ2Zju+yN13qDHPTtkuKxWqXsp2lvRbd3+sXlC5JBXufq/CL/ul8fbON0g6IDPKnQrZ4jvir4VPKfw6r+dsSZ8xs60lycwGzOzAWiOb2QvMbL04uK6ZrVexM9TzW4V6zqxrFDbCre7+rGJRl8LJYSKOMyrpHWa2d1yuExR2np9XzsDd71DITs83s30tPP9gbYUNXrauwnqZkPScmS2Q9LbM5zdL2sHMdonLujSz/Oua2fvMbGN3/7NCvdlf42f7m9k2cX38XqHEJ/vruWyj+L2nYva6sPYqa+hrkk4xs20t2MnMNs2O4OF25FGF7bxR3NbHK1SdSaFe+m8tPBNlY2XuUMjscyfHZX+zMvucmQ2Z2WvjOn5S4UJWbZmvV/g1s8jMZlm4HfkAhQZczdrNzA6Jv4aOU9gHrstp2mVT9lEze7WZ7RUT0T8qJALVlq8ZX5S0r5ntHH91/7ukL5jZS+O8trDQElwK2+toM9s+/oj4v01M/4jM+MskfT+z7Zs6fppQuX6a3f7V/ELSajP7ePk4tXDb47zMOLW2ecvTM7MjzGwgrvsn4nf+qnAcHGBmb4/fWc/M9jSzOU0sw0ScRuV5LavhMRptpFAS/Ucz212hjZpi7FXXs5m9zMwOjMnRnxR+9VZd/3He6ymc/xSXs9b1YSOFas8JSeuY2RJN/TVdz7Xxux+Nx+MhCj8S63lpZvx3KzS8/nGV8eqdqx5SaER7upm9yMzWMrNXmdlba8yzI7aLQnu3D8RjdxOF6/Y3a4z7vDyfqPk+hbrXxxRatn5PIWi5++8V6vu+pvAr5A8KxYX1fEkhC7vCzFYrHLCvrzP+HQon1i0Uin2eUfwVFC+2KxvM611m9jszOzO+93OFthXlUolbFU7e5eFyonCEpH9V+OV1gMKtes/WmM8/KtxWeobCXRj3K7TePkyhznW1pI8qnGx/p7CT/DAzvzsVTspXKdyBUflgliMl3WOh6uLDCttEkraN33lK4cD6N3cvVYnvxDjP1QoXlu/VWI5mnBGX4wqFnfrrCuuz0j8p7A93x+X5jkK9vdz9yhjDLQoN3S6t+O57FfaJxxUubt/KfLaZQgv9JxWqzq5RKBKcIm6rAyQtUNiG/ybpKHe/vYVl/YHCNvydwjY4JP5SyGPaZV+XtL2F4tOLFZLPz8bpPqxw8lssNbW/TxGT5G8pNCSUQl3+XZKui/vSVQqNxuTulykkIcvjOMubmMW5CiejhxVapX80TqvV46eeUyV9Kq6fE9Xk9q8mJjz7K9xu+5sY29cUitnLqm7zGU5vvqSVZvaUwrnocHd/xt1XSTpQocRwQuEX7MfUxHk7VkV/RtJ/x3WyR5XRmj1G/0HSsngeXhK/U1ZrPa+l8APhQYXj862q/SNla4XzdXmffUbhfF7N5QrVcXcqVEX8UU1Wtcf96hCFthKPK2y/Cxt87XqF8+ejCuvzXdV+qTdxrjpKIWm6VWGf+b4qqsMzOmK7uPtPFO4MKym017tXTfyIKN8BkDsz+56k2929mV8yQFcys6UKDaeOSB1LJzKzqxUasH0tdSx5YZsDteVWUmFm82KRzlpmNl8hw744r+kDAIDOlufT3jZTKEraVKFYf6G735jj9AEAQAdrW/UHAADoL3R9DgAAckFSAQAAckFSAQAAckFSAQAAckFSAQAAckFSAQAAcvG/bIlsrx5J8qAAAAAASUVORK5CYII=\n" 51 | }, 52 | "metadata": {} 53 | } 54 | ] 55 | }, 56 | { 57 | "metadata": { 58 | "trusted": true, 59 | "_uuid": "96404aa419f0afcb1b393dcd175c9ffabdc243c1" 60 | }, 61 | "cell_type": "markdown", 62 | "source": "We compute the matrix containing the inner products of the points of the dataset." 63 | }, 64 | { 65 | "metadata": { 66 | "trusted": true, 67 | "_uuid": "a5165bdaa07a0f94d387c2d3b6fb2f5754ca7510" 68 | }, 69 | "cell_type": "code", 70 | "source": "def inner_prods(X_train):\n return np.dot(X_train,X_train.T)", 71 | "execution_count": 3, 72 | "outputs": [] 73 | }, 74 | { 75 | "metadata": { 76 | "_uuid": "15af46d41d392d59fc5988e6213a6dc93034a8e7" 77 | }, 78 | "cell_type": "markdown", 79 | "source": "With this, we can write the cost function as:" 80 | }, 81 | { 82 | "metadata": { 83 | "trusted": true, 84 | "_uuid": "6a83a7499ebe162cc407b85b26493c0d1ef9abbd" 85 | }, 86 | "cell_type": "code", 87 | "source": "def cost(alpha): \n return -(np.sum(alpha) - (1/2)*np.dot(np.multiply(alpha,y_train),\n np.dot(inner_prods(X_train),np.multiply(alpha,y_train))))", 88 | "execution_count": 4, 89 | "outputs": [] 90 | }, 91 | { 92 | "metadata": { 93 | "_uuid": "f8c74947cc7f78563b2a6cfd2ac24b10cad9b3ac" 94 | }, 95 | "cell_type": "markdown", 96 | "source": "We define the constraint given by the KKT conditions. This is of the form $\\sum_{i=1}^m \\alpha_i y^i = 0$." 97 | }, 98 | { 99 | "metadata": { 100 | "trusted": true, 101 | "_uuid": "565e648ae4ec0623dfa1c79171af121bc753e0aa" 102 | }, 103 | "cell_type": "code", 104 | "source": "def cons1(alpha):\n return np.dot(alpha,y_train)\n\ncons = ({'type': 'eq', 'fun': cons1})", 105 | "execution_count": 5, 106 | "outputs": [] 107 | }, 108 | { 109 | "metadata": { 110 | "_uuid": "c329f3b3b2067cc037b09d059618db2a5831cbed" 111 | }, 112 | "cell_type": "markdown", 113 | "source": "We also set the conditions for the Lagrange multipliers: $C\\geq\\alpha_i \\geq 0$. We set $C = 1000$ for the example:" 114 | }, 115 | { 116 | "metadata": { 117 | "trusted": true, 118 | "_uuid": "ec4c8ee6382e84eb3d572a1302eb925d04d23407" 119 | }, 120 | "cell_type": "code", 121 | "source": "C = 1000\nbds = [(0,C) for _ in range(len(X_train))]", 122 | "execution_count": 6, 123 | "outputs": [] 124 | }, 125 | { 126 | "metadata": { 127 | "_uuid": "0fee47d9ba66f8bc42dec14848557508a9435ae5" 128 | }, 129 | "cell_type": "markdown", 130 | "source": "Now, we need an initial guess for the value of $\\alpha_i$ for the optimizer:" 131 | }, 132 | { 133 | "metadata": { 134 | "trusted": true, 135 | "_uuid": "23cb29a47e86344313477bf65fd09dcd746791a5" 136 | }, 137 | "cell_type": "code", 138 | "source": "alpha = np.random.random(len(X_train))", 139 | "execution_count": 7, 140 | "outputs": [] 141 | }, 142 | { 143 | "metadata": { 144 | "_uuid": "196bbeb5cd0b41db7f4f6059040056ae9a2ac6a4" 145 | }, 146 | "cell_type": "markdown", 147 | "source": "Finally, we optimize the cost function " 148 | }, 149 | { 150 | "metadata": { 151 | "trusted": true, 152 | "_uuid": "58420d8706d87d2695faac8d82f59da1f6352099" 153 | }, 154 | "cell_type": "code", 155 | "source": "res = minimize(cost, alpha , bounds=bds, constraints=cons)", 156 | "execution_count": 8, 157 | "outputs": [] 158 | }, 159 | { 160 | "metadata": { 161 | "_uuid": "692ac353c202e4de221de9388e6e9261a492ba48" 162 | }, 163 | "cell_type": "markdown", 164 | "source": "We recover the values of $\\alpha_i$ that optimize the Lagrangian:" 165 | }, 166 | { 167 | "metadata": { 168 | "trusted": true, 169 | "_uuid": "5c6343f0598677e5eaccaa893d203cb309117a9a" 170 | }, 171 | "cell_type": "code", 172 | "source": "alpha = res.x", 173 | "execution_count": 9, 174 | "outputs": [] 175 | }, 176 | { 177 | "metadata": { 178 | "_uuid": "25f31cfc13796b8a5006daa8531c1d93868a79e7" 179 | }, 180 | "cell_type": "markdown", 181 | "source": "We will set to zero the values $\\alpha_i$ which are smaller than $10^{-7}$:" 182 | }, 183 | { 184 | "metadata": { 185 | "trusted": true, 186 | "scrolled": true, 187 | "_uuid": "5001faa00b97641f2bd3c05792d5bd06cac7a4b1" 188 | }, 189 | "cell_type": "code", 190 | "source": "alpha = alpha*(1*(alpha > 10**(-7)))", 191 | "execution_count": 10, 192 | "outputs": [] 193 | }, 194 | { 195 | "metadata": { 196 | "_uuid": "073f23d46de099555abd7e1d6a2a652dcfe25e2f" 197 | }, 198 | "cell_type": "markdown", 199 | "source": "We can now construct the parameters $w,b$:" 200 | }, 201 | { 202 | "metadata": { 203 | "trusted": true, 204 | "_uuid": "ed71dd719b6a20e6eb2cae497d2e2d5e17cb8e81" 205 | }, 206 | "cell_type": "code", 207 | "source": "w = 0\nfor i in range(len(X_train)):\n w += y_train[i]*alpha[i]*X_train[i,:]\n\nb = y_train[(alpha > 0) & (alpha < C)][0] - np.dot(w,X_train[(alpha > 0) & (alpha < C)][0])", 208 | "execution_count": 11, 209 | "outputs": [] 210 | }, 211 | { 212 | "metadata": { 213 | "_uuid": "0d1d4676674ebce1c8abd1da0ebc30b5e9dc7f34" 214 | }, 215 | "cell_type": "markdown", 216 | "source": "With this, we can build the prediction function:" 217 | }, 218 | { 219 | "metadata": { 220 | "trusted": true, 221 | "_uuid": "1440c5e3027132f81c60c51c40a92db7a6a5da22" 222 | }, 223 | "cell_type": "code", 224 | "source": "def predict(x,w,b):\n return np.sign(np.dot(x,w)+b)", 225 | "execution_count": 12, 226 | "outputs": [] 227 | }, 228 | { 229 | "metadata": { 230 | "_uuid": "4642ebb91cbb509cc2077fe26ad2c86968e4ef5c" 231 | }, 232 | "cell_type": "markdown", 233 | "source": "Finally, we can plot the decision boundary given by the SVM as well as the regions predicted for each class. We can see how the separating hyperplane maximizes the margin to both classes." 234 | }, 235 | { 236 | "metadata": { 237 | "trusted": true, 238 | "_uuid": "98a44a029aeef73ae3b5434bcde4338df1f66e23" 239 | }, 240 | "cell_type": "code", 241 | "source": "plt.plot(x0[:,0],x0[:,1],'x',color='red')\nplt.plot(x1[:,0],x1[:,1],'x',color='blue')\nplt.figtext(0.5, 0.01, 'Figure 2: SVM decision boundary', \n wrap=True, horizontalalignment='center', fontsize=12)\n\nx = np.linspace(-5, 5, 200)\ny = np.linspace(-5, 5, 200)\nX, Y = np.meshgrid(x, y)\nz = np.zeros(X.shape)\nZ = np.array(z)\nfor i in range(Z.shape[0]):\n for j in range(Z.shape[1]):\n Z[i,j] = predict(np.array([x[j],y[i]]).reshape(1,2),w,b)\nplt.contourf(X, Y, Z, alpha=.5, cmap='jet_r')\nC = plt.contour(X, Y, Z, colors='black',zorder=4)\nplt.axis('equal')\nplt.show()", 242 | "execution_count": 13, 243 | "outputs": [ 244 | { 245 | "output_type": "display_data", 246 | "data": { 247 | "text/plain": "
", 248 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEPCAYAAABWc+9sAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VPW9//HXBwik7CBUCKAJArZIETBugGC4aF1xS2pxuxY3LO7WDb2Wti6IWqGV1qptrXVDUrW1xZ/obSoXcItWUav2cg2KRCtKAVkMQr6/P87JOJlsk8yZnJkz7+fjkccjM3My85kQ3uc7n/M932POOUREJDo6hF2AiIgES8EuIhIxCnYRkYhRsIuIRIyCXUQkYhTsIiIRo2AXEYkYBbuISMQo2EVEIqZTGC/ar18/V1hYGMZLt6vq6h1hlyCSEwoKOoddQrt45ZVXPnXO9W9pu1CCvbCwkMrKyjBeul3NmfNB2CWI5IQ5c/YIu4R2YWbvJ7OdWjEiIhGjYBcRiRgFu4hIxCjYRUQiRsEuIhIxCnYRkYhRsIuIRIyCXUQkYhTsIiIRo2AXEYkYBbuISMQo2EVEIkbBLiISMQr2NJr/4F384heH45wLuxQRySEK9jTatPpm1q9/lx//uJDbb98f52rDLklCtmJFD6qqutS7r6qqCytW9AipIokiBXsa7TdoUOz7LVs+Yd68sQr3HFdQsIPy8n6xcK+q6kJ5eT8KCnRRFglOYMFuZh3N7O9m9uegnjPbPXH22fyqtJTRAwYA8MUXG7n1VoV7LisqqqG09FPKy/tRUdGL8vJ+lJZ+SlFRTdilSYQEOWK/GHg7wOeLhKP22Ye/nHce00aOBGD79o389KcHKtxzWFFRDcXFW1i2rBfFxVsU6hK4QILdzAYDRwP3BvF8UbSwrIynzz0X8NoyGrnnrqqqLlRWdmfSpE1UVnZv0HMXSVVQI/b5wJWAkqoZIwcO5LlZswBv5K6ee+6p66mXln5KScmmWFtG4S5BSjnYzewY4BPn3CstbHeumVWaWeX69etTfdmsNbRfv1hb5osvFO65prq6c72eel3Pvbq6c8iVSZQEMWKfAEwzszXAI8AUM3sgcSPn3N3OuWLnXHH//v0DeNnstbCsTOHeSlGZJjhhwucNeupFRTVMmPB5SBVJFKUc7M65a5xzg51zhcB3gb86505LubKISwx3HVBtXq5ME4zKDkzCpXnsIYoP9y1bPlG4NyNXpgnmyg5M0ivQYHfO/c05d0yQzxl1C8vKYgdUdRJT83JhmmCu7MAkvTRizwBD+/XjJ0ceCajn3pxcmSaYCzswSS8Fe4Y484ADdEC1Gbk0TTBXdmCSPgr2DLKwrIzj4sL9xz8u4ua5+7Nzp0ZsuTJNMJd2YJI+CvYMc2fcAVWAHTWfMHfu2JwP91yZJpgrOzBJLwV7BlpYVsa8Y47h96ecAh06sGvXVm66+Vs5H+65IFd2YJJeCvYMNX2//Th0+HBu/Pa3AXC1Ndx08yiFu4i0SMGe4c444ABmTZgAgKvdoZG7iLRIwZ4Frp46NS7caxTuItIsBXuWSAx3HVAVkaYo2LPI1VOn8ux550FH74DqzTfvq3AXkQYU7Flm7wEDeO3SywCord2utoyINKBgz0K7devG+ePHA5otIyINKdiz1OzDDkuYLaNwFxGPgj2LXT11Kt+Pjdx3cPPcMZEPd61XLtIyBXuWuyZu5F67a1vkZ8tovXKRlinYI+DqqVN57Qc/iM2WifIBVa1XLtIyBXtE7NatGxXnzQS8A6o33rg3CxYcwvbtm0KuLHhBrleu1o5EkYI9Qob17x9ry4Bj48YPmDdvDNu3bwy1rqAFuV65WjsSRQr2iLl66lSuP+wweubn+/fUMu/W/SIzcg96vXK1diSKFOwRdM748bx11VXMPfporKOB2xmZcE/HeuW6FJ1EjYI9wk4tLuadq64h72sdwX0ZibZMOtYr16XoJGoU7BHXNS+Pv/znOf6tWubNG5v14R4kXYpOokjBngO+ufvuzD36aP+Wwj2eLkUnUaRgzxGnFhc3Eu7Z33NPVlPTGgFdik4iR8GeQxqEewQOqCY7Dz2ZaY2a0y5RoWDPMacWF/PEjBnk5fsHVG8dl9VtmbrA7vbkUgqrVtYL7MKqlUxYcReQ3LRGzWmXqFCw56D9hgzhzcuu+moqZBb33OsC+8dvncY7D63lsUW9vAOhVFBWPot1BaPrbdvctMbGwv8b39jW4DU1ipdMp2DPUV3z8ph7xFH+rezquSe2TIqKahi6905u2XkFs3b9nBlrbqKsfBaLSxeypmh8bLtkpjUmhv+oUds0ipeso2DPYac0OKA6JivCPbFl8vzz3Vm1qhujR2/hLjuf2mWrqCw+rUGoJzOtMTH8AZ2ZKllHwZ7jGs6W2Ze77z6WTZuqQ62rOfEtk8cf78vSpX04/PB/c+mYP/NQh1M5Ie9Jtr64hsKqlbGfSWZaY1PhD+jMVMkqCnZJCHfHRx+tYv788Rkf7sXFW1i1qjujR29l+oCllJXP4pOTp3Pc9K3ct8+PKCufFQv3ZM5YbSr833yzq85MlayiYBfAC/f/nT2b6w4spEMHAMf8+QdnbLjHt0xWr/4a69/cEuupFxXVMPLYviwuXcig6lVJP2dj4Q/wzjtddWaqZBUFu8Tk5+Vx3hH/yX0nT4/dN3/+wWzcuC7EqupbsaIHzz/fvV7LZOLETVz7xgwqKKm37Zqi8ayYMDOl19OZqZKNFOzSQMmIEdw//atwX7Agc9oyBQU7qKjozcSJmygqqqGqqgvLl/eipGRjWsI2HYuOiaSbgl0alRjumdJzLyqqYfr09Sxf3qveLJWDD96isBXxpRzsZjbEzCrM7B9m9paZXRxEYRK+unD/quc+IWPCXbNURJoWxIh9J3C5c24kcBAwy8xGBvC8kgFKRozg/2ZfR69eHYHajOi5p2v9dK0VI1GRcrA75z5yzr3qf/858DYwKNXnlczRqWNHXvn+lbHbCxaMrxfu7RmI6Vw/XWvFSFQE2mM3s0JgLPBiI4+da2aVZla5fv36IF9W2kGXzp2bPKDanoGYzlkquv6pREVgwW5m3YE/AJc45zYnPu6cu9s5V+ycK+7fv39QLyvtqOEBVa8t056BmO5ZKurfSxQEEuxmlocX6g865x4L4jklM5WMGMEDp5wSu71gwcRYuEchEKuquvD88z0YPXpLvf69eu2STYKYFWPAr4G3nXM/Tb0kyXSThw/nkdNOo2fPDkAtCxZM5I031mf9afd1LaSSko2sXv01Jk7cRHl5v9gJUeq1S7YIYsQ+ATgdmGJmr/lfR7X0Q5LdJuy1F69fNJvu3bxwf+yxYo444rWsPu2+rn9/8MFbKC39lOXLezFs2HYqKnqr1y5ZJYhZMcudc+acG+2cG+N/LQmiOMlsnTp25J4TT43dfuyxYp5++if07bsmK0+7j+/fxy8ydvDBja8hI5KpdOappGTi0KH1eu4vvHAv8+cfRJculVl9Jmi65sqLtAcFu6Rs8vDh9WbLANxzzzFUV78RUkWpSedceZH2oGCXQJSMGMHaH/6QFy66iMGDvRZM0OHeXidCaUVHyXYKdgnUoD59WDnjanr37ggEG+7tdSKUVnSUbKdgl8CZGYu+c1bsdlDhrjNDRZKjYJe0GDlwIE+fe27sdpDhHoUToUTSScEuaZOOcE/3bBWt8ChRoGCXtKoLdzPvdirh3h6zVbTCo0SBgl3SbuTAgbz/X9czZLAXlm0N93TMVkkcoRcV1TBx4iYefrh/o318jeglGyjYJXDdV6ygS1VVvfvy16zh9b0npjRyT8dslcQR+pNP9uG553rzzW9ui/XxgVhwa0Qv2UDBLoH7sqCAvuXlsXDvUlVF3/Jydg4axP87J/gDqqlInGnz1lvdqK11vPtuVyZN2sSLL/Zg0aKvgjuZmTka1UvYFOwSuJqiIjaUltK3vJyeFRX0LS9nQ2kpNUVFaZstk4r4mTZ7770N54zaWuc/6gDj44/zYsHc0swcjeolbAp2SUpj7ZUuVVV0X7Gi0e1riorYWlxMz2XL2FpcTE1RUeyxTAv3+Jk2b7/dlSN2fwmrdSxb1osDD9zC5Mkb+dt/9+DwDX9osH1jM3M0317CpmCXpDTVXvmyoKDR7btUVdGtspLNkybRrbKywU6hLtx32+2rM1Tfeedpamq2pveNJEicaTN9+nr++5NxdNr1BR1tF8uX9+B/KrrzeKdS+o/qzvPPd+fhh/u3ODNH8+0lTAp2SUpz7ZVEdaG/obSUzSUlsZ9rLNz/PutaBg/yZrUsWnQuc+eO5N13n22X9wQNZ9oA1HboyP7DP6bMFlNb24FdX9by3OSLqaCEiorelJRsbHFmjlaHlDAp2COqta2TZDTXXomXV13NhtJS5lefxsqqwthOIa+6mpVVhfxyxYTYtmbG4pNm1vv5Rx45i3fffabNdbZG4kyb6urOnHzyp+x5cA+e6ngMp3M/O+jC/asmUF7ej+nT13PwwVvqPUfizBytDilhU7BHVGtbJ8loqb1SZ8uECdQUFbFvwTrOLy+LhfvSglM5v7yMfQvW1dt+cJ8+vHvNNdwft677I4+cnZZwb2nGSl1AP7aoFw91OJWzJr3Cdzs+yscfd2HYsO1JtVS0OqSETcEeUa1pnSQj2fZKvPFFa/hl6WLOLy/jtooSzi8v45elixlftKbBtl07d6Zk+HBeuvTS2H2PPHI2//xnsG2ZZGasbH/zIxZxMp+cPJ3fFM7mz52O47sdF/HOP/KTGnVrdUgJm4I9wpJtnSSjrr1S9xzx7ZXmjC9aw+nFlSxYNpnTiysbDfV4A3v25KVLL2XMHnkAPPzwWYH23JOZsXJO30f55OTpVFBCeXk/Tjx5E98+1XHD6PvUUpGsoGCPsGRbJ8moa6/EqykqYsuECc3281dWFfL7ymIunvQcv68sZmVVYYuvNbBnT5783mz2HeKFe7I99wkr7qKwamW9+wqrVjJhxV317mtpxsqKCTNZUzS+XktlTdF4th57uFoqkhUU7BHVltZJWzXVz6/oMCXWfvlBSUWsLZNMuAP8uuzC2PfJ9NzXFYymrHxWLNwLq1ZSVj6LdQWj622X7IwVtVQkWynYI6qtrZO2aKqfX1lbXK+nXtdzf716UFLPu3uPHg167s2F+5qi8SwuXUhZ+SxKKm6nrHwWi0sXsqZofGwbzViRXGDOuZa3ClhxcbGrrKxs99dtbx/OmRN2Ce2qZ0UFPZctY/OkSWwuKQnseT/avJkD7rgjdnv69F8zYsTUJrcvqbidyct+xnOTLqKi5PJ6j61Y0YOCgh31RuJVVV2oru6skXgWmzNnj7BLaBdm9opzrril7TRil0AE2c9PVHdA1V8YstkDqoVVKymufIDnJl1EceUDDXruaq9ILlCwS8p6P/kkfRctqt/PX7SI3k8+Gdsm1ROmBvbsyfvXX8+BezV9QLWup764dCEVJZfH2jKJ4S4SdQp2CYS1cDuIE6bMjPLTZtPNn5TyyCNn89xzf4s9Pqh6FT+euJgHq78NfNVzH1S9qpXvRiS7qceeRrnUY68L6q3FxXSrrGz0ZKhktklG9ebNHBjXc5806V4mTy7h/fe7aSXFHKUee30asUsgkjkZKqgTpgr8nnudZcvO5ic/2YuHHipXqIugYJeAJHPwNMgDrAMTwh1g584L+PTTe9v8nCJRoWCXlCVzMlQ6Tpga2LMn9333l3wtbyHDeo8DYMmS63jppd+n/J5EspmCXVKWzMlQ6ThhamVVIZf96XTum96ViouPJb9jJwCeeuo6Xnrp/hTekUh2U7BLyppbR6Y127TW69WD6p3Z+vcrfhB77Kmn/kvhLjlLwS5Z6/wJK+qtFtm9Sxf+cdVVsT9qL9zVlpHco2CXSOmRn8+b9cL9Ol5+WeEuuSWQYDezI8zsXTNbbWZXB/Gc0j7ScQm9sNWFe/Hu3QEdUJXck3Kwm1lHYCFwJDASmG5mI1N9XmkfQV1CL9N2ED3y83nsvMuYNnQ3QAdUJbcEMWI/AFjtnHvPObcDeAQ4LoDnlQA1Fbx1s1VSvYReOq6xmioz447pMxN67gp3ib4ggn0QsDbu9of+fZJBmgveIM4IrZu+uNvDD9Pn8cfr7SDCHLl37tQpoeeuA6oSfe128NTMzjWzSjOrXL9+fXu9rPiau7h1UGeE1hQVsf2b36TbqlV8MWwYedXVdH/++Xoj9zBCXgdUJdcEEezrgCFxtwf799XjnLvbOVfsnCvu379/AC8rrdXYyDzIM0K7VFWRv3o1W0ePpuuqVeS/9x69li7l84kT671WGO2ZunDv6N9esuQ6Xn5ZbRmJpiCC/WVguJkVmVln4LvAnwJ4XglYYyPzoM4Ijd9B/PuEE9g2ejT5771HzdCh9Fi+PKX+fWPacrC2R34+q6+7LnZAdckStWUkmlIOdufcTuAC4GngbeBR59xbqT6vBKupkXldjz1eW84Ijd9BxI/cO69dyxfDhqW8omOith6s7dSxI3eeNothvfMBry3zox8V8eyztwRSl0gm6BTEkzjnlgBLgnguSY/mRuZBhG3djiB+B1JTVMSXAwbQa+lSto4eTbfKSmoKCwN5vfhjBq1d393M+NN5FzPqlluoBaCWFSt+QW3tlxx++HUp1yYSNp15miPSsVZLYxJH7j2WL2fT4Yfz5de/HsiKjvFSmc3TIz+ft66+msOHD4/13Z9//h6eeeamQGoTCVMgI3aROvE7isRPCUCgnxISjxm09tNA9y5d+PUpp7Bj505G33ITW3c6Vq78FWAcdtg1KdcnEhaN2CVt0vkpIcjZPJ07deKda6/ngjHe6RcrV97F0qU3pFyjSFgU7JKVmjtm0NblDa467mz27v01wGvLKNwlWynYJSs192kgleUNlsy6jC7+/wqFu2QrBbtETnNn2bakc6dO/OOaaxXuktUU7BJJqcyYaSzcn3nm5jRVKhI8BbtEUqrr39SF+/7+mu4rV96lcJesoWCPiExbDz0Z6ao5qBkznTt14rGZl2u2jGQdBXtEZOJ66C1pbc3J7giCWv+mzlXHnc3XOhqgnrtkBwV7RKRywDAsra052R1BOubPv3n1bB1QlayhYI+QIC6Y0d5aU3OYO6/GDqjeeOPeLFlyfdpfW6S1FOwREtQFM9pTa2sOc+eVGO47d37Byy//jkcf/X671SCSDAV7RAR5in17aUvNYe+86sJ92axZnD9+PABvv/0XHn10ZrvWIdIcBXtEBH3AsD20tuZM2Xl17tSJon79mH3YYfT9mrcEwdtvP6Vwl4yhYI+I9lqWN0itrTkTd15/v+KKhHA/P7RaROoo2CVrZOLOq4NZQrgvobz8wtDqEQEFu0jKEsP9rbf+pHCXUCnYRQLQwYzXrriCH37724AX7uq5S1gU7CIBMTPOPuggvjNmDKCeu4RHwS4SsFunTavXc9fIXdqbgl0kYA0PqGoqpLQvBbs0KsiVF7Nx5clUNRbuOqAq7UXBLo0KcrXIbFx5Mgh14T5qwADAO6D6hz9cFHJVkgsU7NKolhbcas0oPBtXngxKBzOeOu885vizZd58849qy0jaKdilyZDOq65ucsGt1o7Cs3HlySCdddBBDOzRA/DaMrfffgDPPjs35KokqhTs0mRI06FDkwtutXYUHvbiXZnghUsvjfXct2z5FytW/JJ77jku5KokihTsGSLMA4yNhfTnEyfSY/nyZhfcSnYUnimLd4Wtruc+rG/f2H3V1a9x331lIVYlUaRgzxBhH2BMDGlqa1tccCvZUXgmLt4Vlg5mVFx4IWt/+EOmjx0LwPvvv8RvflMacmUSJQr2DBH2AcbEkP6yoKDZBbdaMwrPxMW7MsG8adP4VZk3Wl+79mXuuWdayBVJVCjYM0hYBxjb0irRKDwYR40cyXC/NVNd/bp67hIIBXsGCesAY1tCWqPw4Pz1wgsZ2qcP4PXcFe6SKgV7hgjzAGO6QzoXzzxtrecuuqheuN977wkhVyTZTMGeIaLc2gj7wHC2iA/3dete5be/1WwZaRsFe4aIcmsj7APD2eS5iy7iF6XeDJkPPnhJbRlpk5SC3cxuNbN3zGyVmT1uZr2DKkyiJdfPPG2NY/fZh9uPPRZQz13aJtUR+zPAKOfcaOCfwDWplyRRpDNPW+c748YlHFDVVEhJXkrB7pxb6pzb6d98ARicekkSNTrztG3qH1B9XeEuSQuyxz4DeKqpB83sXDOrNLPK9evXB/iykumifGA43RLD/Wc/O4QVK36Jcy7kyiSTWUt/IGb2LDCgkYeudc790d/mWqAYONEl8RdXXFzsKisr21BudvlwzpywS5CIOPquu1j1r3/FbvfsWcAll6zEzEKsKnPMmbNH2CW0CzN7xTlX3NJ2LY7YnXNTnXOjGvmqC/UzgWOAU5MJdRFpvb/MnEn5mWcybs88ADZvrmb+/PEauUujUp0VcwRwJTDNObctmJJEpDEH7rknfzxzNgcOGQJ44b5gwQSFuzSQao/9TqAH8IyZvWZmdwVQk4g0o3zGDA7yw33TpnUKd2kg1Vkxw5xzQ5xzY/wvXfNLpB0sTgj3hQunKNwlRmeeimSpxTNmcMKoUQB89tl73Hmnwl08CnaRLPazk07ij2edBcCGDe+pLSOAgl0k640bPJipw4cD6rmLR8EuEgG/PeUUHVCVGAW7SEQsnjEjNhVS4Z7bFOwiEVKeEO6aLZObFOwiEVM+YwYPnnYa4M2WWbBgosI9xyjYRSJo0l57xWbLbNr0oZYfyDEKdpGIGjd4cL3lB3760wNZt+51BXwOULCLRFh8z33Lln9x773TmDdvX2prd4VcmaSTgl0k4uLXlgH44otN3H77/gr3CFOwi+SAxTNm8PoVV7DknHPAjG3bPmPevH1xrjbs0iQNFOwiOaJv1658q6CA584/H4Cams+55ZbRGrlHkIJdJMcM7d+f0tGjAS/c1XOPHgW7SA6644QTFO4RpmAXyVGJ4a4DqtGhYBfJYXeccALTx44Fg23bPlO4R4SCXSTHzZs2jeUXXFhvtozCPbsp2EWEPfv25QeTJwPquUeBgl1EALh48mQdUI0IBbuIxGi2TDQo2EWknjtOOIHSffcF6mbLFLN166daPCyLKNhFpIE7jj+eR04/3T+guoHbbtuPG274Bl9++UXYpUkSFOwi0qgJQ4ey/IIL6NDJAKit/YKbbx6lcM8CCnYRadKefftSefFlfMdvzTj3pcI9CyjYRaRZ/bt35/bjj+fSQw4BFO7ZQMEuIkm5bMqUeuE+95axCvcMpWAXkaRdNmUKL15yCdapI7W7tnHz3G8p3DOQgl1EWqWgVy9ev/QyAFztDrVlMpCCXURarU/Xruq5ZzAFu4i0SWLPXeGeORTsItJmDQ6ozh2jcM8ACnYRSUks3M2ord3O3FsU7mFTsItIyi6bMoVVP/iBP1tmu9oyIQsk2M3scjNzZtYviOcTkezTp2tXHj/9DKCu566pkGFJOdjNbAhwOPBB6uWISDbbb4894nrumgoZliBG7HcAVwJa01NEGhxQvemmvbnhxlFs3boh5MpyR0rBbmbHAeucc68HVI+IREB8uAPs2vk5t92+P9u2/TvEqnJHp5Y2MLNngQGNPHQtMBuvDdMiMzsXOBdgjz32aEWJIpKNLpsyhW8NGsSuXbv4/l8e58ttO7n11nFcccWrdO3aJ+zyIq3FEbtzbqpzblTiF/AeUAS8bmZrgMHAq2bW2E4A59zdzrli51xx//79g3wPIpKhDtt7b44YOZKHyk7176nl1lvHaeSeZm1uxTjn3nDOfd05V+icKwQ+BMY55z4OrDoRiYSDCgu5fdo0/5bCPd00j11E2sV3xo5VuLeTwILdH7l/GtTziUj0fGfsWOYfd5x/q5ZbbyvWbJk00IhdRNrVSWPG8MIll9C5aydwO7nttmKN3AOmYBeRdjeoVy/euuRKrIMBu9SWCZiCXURCkZ+Xx23HHOvfUs89SAp2EQlNg577rePUcw+Agl1EQnXSmDH1wv02HVBNmYJdREJ30pgx/HTaNDCAXdx2+/4K9xQo2EUkI5SNHcv/XjM7brbMWG64cR82bHg/7NKyjoJdRDJGfl4eqy68PHZ7184t/PznkxTuraRgF5GM0i0/n/umT2don68WClO4t46CXUQyzn+MGMFzF13EojPOiN33859PVrgnScEuIhlrfFERi844g7w8AKdwT5KCXUQy2viiIlZfcz19+3bEC3e1ZVqiYBeRjNfBjJXnXhG7rXBvnoJdRLJCty5dEnruCvemKNhFJGvU9dzrKNwbp2AXkaySGO53/XZWiNVkJgW7iGSd8UVFPDNzJlOm9GTOG4eGXU7GMedc+7+o2Xog0z4/9QOy5QpQ2VQrZFe92VQrZFe92VQrZGa9ezrn+re0USjBnonMrNI5Vxx2HcnIplohu+rNplohu+rNploh++qNp1aMiEjEKNhFRCJGwf6Vu8MuoBWyqVbIrnqzqVbIrnqzqVbIvnpj1GMXEYkYjdhFRCJGwZ7AzC40s3fM7C0zmxd2PS0xs8vNzJlZv7BraY6Z3er/XleZ2eNm1jvsmhKZ2RFm9q6ZrTazq8OupylmNsTMKszsH/7f6cVh15QMM+toZn83sz+HXUtzzKy3mZX7f69vm9nBYdfUWgr2OGZWAhwH7Ouc2we4LeSSmmVmQ4DDgQ/CriUJzwCjnHOjgX8C14RcTz1m1hFYCBwJjASmm9nIcKtq0k7gcufcSOAgYFYG1xrvYuDtsItIwgLg/znnvgHsS3bUXI+Cvb7zgbnOuRoA59wnIdfTkjuAK4GMP1DinFvqnNvp33wBGBxmPY04AFjtnHvPObcDeARvJ59xnHMfOede9b//HC94BoVbVfPMbDBwNHBv2LU0x8x6AZOAXwM453Y45zaGW1XrKdjrGwEcYmYvmtlzZrZ/2AU1xcyOA9Y5514Pu5Y2mAE8FXYRCQYBa+Nuf0iGhyWAmRUCY4EXw62kRfPxBiG1YRfSgiJgPfBbv210r5l1C7uo1uoUdgHtzcyeBQY08tC1eL+Pvngfb/cHHjWzoS6kqUMt1Dobrw2TMZqr1zmFBak+AAAKBElEQVT3R3+ba/FaCQ+2Z21RZGbdgT8AlzjnNoddT1PM7BjgE+fcK2Z2aNj1tKATMA640Dn3opktAK4G/ivcslon54LdOTe1qcfM7HzgMT/IXzKzWrz1Ita3V33xmqrVzL6FN7J43czAa2u8amYHOOc+bscS62nudwtgZmcCxwD/EdbOshnrgCFxtwf792UkM8vDC/UHnXOPhV1PCyYA08zsKCAf6GlmDzjnTgu5rsZ8CHzonKv7BFSOF+xZRa2Y+p4ASgDMbATQmcxbBAjn3BvOua875wqdc4V4f4zjwgz1lpjZEXgfxac557aFXU8jXgaGm1mRmXUGvgv8KeSaGmXe3vzXwNvOuZ+GXU9LnHPXOOcG+3+r3wX+mqGhjv9/aK2Z7e3f9R/AP0IsqU1ybsTegt8AvzGzN4EdwH9m4MgyW90JdAGe8T9lvOCcmxluSV9xzu00swuAp4GOwG+cc2+FXFZTJgCnA2+Y2Wv+fbOdc0tCrClKLgQe9Hfw7wHfC7meVtOZpyIiEaNWjIhIxCjYRUQiRsEuIhIxCnYRkYhRsIuIRIyCXUQkYhTsIiIRo2DPIGa2h5lt8ZeQlQCY2X1mdkOKz3GqmS1NYru7zCzwNUXMbI6ZPRD087ahjkJ/7X+d2JjhFOwhMLM1ZrbdD/G6rwLn3AfOue7OuV0ZUONBZvaMmW0ws/VmttjMBrbi5/cxs6X+z280s1fM7CgzG2RmO81sr0Z+5nEzu83/3pnZJ/EhYmZ5/n3teladc+5B51yLC64552Y6537SHjWJNEfBHp5j/RCv+6pO54u1YZTVB+9ivoXAnsDnwG9b8fNP4l1cYwDwdeAiYLNzbh3w33inxMfX1xc4Cvhd3N3/xrvwRZ0j/fsky2iU374U7Bkk8aOuvyDVMjP73MyeNbOFdR/JzexQM/sw4efXmNlU//s5/uW9HjCzzcCZZtbBzK42s/8zs8/M7FE/UBtwzj3lnFvsnNvsL9p1J94aJcm8j354q0/e41+oYIdzboVzbrm/ye9ICHa8xaH+4Zx7I+6+3wNnxN0+A7i/hdcea2av+r+zRXirCcY/foyZveZ/ilhpZqPjHhtiZo/5n1A+M7M7/fvPNLPl/vdmZnf4nxw2m9kbZjbKf6xe28fMzjHvMnsbzOxPZlYQ95gzs5lm9r9+LQv9xb2akm9mi/z39aqZ7Rv3XN80s7/5z/OWmU2Le+xvZnZ23O3Ye2mpDvMuZXebmX1qZu/hXSgj/nf5PfMuHfe5mb1nZufFPXaomX1oZleZ2cd465u/aWbHxm2T5z/32Gbet7SBgj2zPQS8BOwGzKFhGLbkOLxlR3vjrX9+IXA8MBkowBv9LkzyuSYBsUWxzOwUM1vVxLafAauBB8zseDPbPeHxx4F+ZjYx7r7TqT9aB2+1zUnmXYOyD3AI8MemCjRv0aYn8HYIfYHFwElxj4/FW+jtPLzf6a+AP5lZF/OOa/wZeB/vU8ogvKsoJToc73cxAugFfMd/v4m1TAFu9h8f6D9v4vMdg7fu/2h/u2839d7w/i0X++/rIeAJPxjz8D4dLcX7ZFS3gNXeTT5TQ03VcY7/2FigGChN+LlP/Md74i2UdYeZjYt7fIBf757AuXg75fhVHY8CPnLO/b0VtUoynHP6aucvYA2wBdjofz3h31+Id5m7TsAeeBek6Br3cw8AD/jfH4q3bnTi8071v58DLEt4/G28tdDrbg8EvgQ6tVDvaGADcEgr3uNgvFH+/+FdNWcZMDzu8XuBu/3vh+Otpvn1uMcdMMzf7jxgJnCPf59r4jUnAdX4i9v5960EbvC//yXwk4SfeRdvR3cw3rr7DX4XwJnAcv/7KXjXbD0I6JCw3X1xr/VrYF7cY93933Vh3PubGPf4o8DVTbyvOXirYdbd7gB8hLejOwT4OL4W4GFgjv/934CzG3svLdUB/BWYGffY4f72jf694O1UL477+9wB5Mc9XoDX0uvp3y4Hrgz7/2MUvzRiD8/xzrne/tfxjTxeAGxw9dcuX9vIds1J3H5P4HH/I/dGvKDfBSSOqGPMbBjeZewuds79T7Iv7Jz70Dl3gXNuL/91t1K/jfI7oMzM8vFG60+7xq8xez9eC6bFNgze72yd81PD937c93sCl9e9f/93MMT/uSHA++6r67I29b7+irfDWgh8YmZ3m1nPJmp5P+7ntuCN7OMvtxe/fv42vPBvSuzf0jlXi7cGf4H/tda/r877tO6yfk3VUUD9v6H43yVmdqSZveC3mjbijcD7xW2y3jn3RVzd1cAK4CQz6413zERX0koDBXvm+gjoa2Zd4+6Lv8LPViD2mN9K6J/wHImzR9YCR8btUHo75/Kdd0CzATPbE3gWb5T7+7a+EefcWrwgHBV393K8TwHH4X08T2zD1PkfvE8Wu/s/05yPgEEJveo94r5fC9yY8P67Ouce9h/bw5I4yOec+5lzbj9gJF5L5opGNqvG25EAYN51M3ej7Vdliv3bm1kHvE9E1f7XEP++OnvEvU69vxMav3RhUz6i/t9c7HdpZl3wruB0G7C7c643sASI/903Nnvpd3j/3mXA80397UlqFOwZyjn3PlAJzDGzzmZ2MHBs3Cb/xDugdrTfZ70O70IWzbkLuNEPbMysv3kXxW7AzAbhfRS/0zl3V2tqN7M+ZvYjMxtm3gHbfngXsH4h7v05vBH4LXjHAJ5s7Ln87Y7Fu/JSS9Mcn8drX13k959PBA6Ie/weYKaZHegfBO3m//564B3L+AiY69+fb2YNDhab2f7+z+fhheYXNH6B5oeB75nZGD8EbwJedM6taeE9NGU/MzvR3/FcAtTg/T5fxBtlX+m/50Pxfl91/fzXgBPNrKv/6eusVrzmo3i/y8H+MY74S8R1xvt7Ww/sNLMjSe4avE/gXVP0Ylr+BCZtpGDPbKfi9X4/A24AFuH9h8Y5twn4Pl4Peh1eyHzY+NPELMC73NtSM/scLxgObGLbs4GheDuW2Hz7ugfNO2mnqSsM7cA7XvAssBl406/7zITt7scbBS5yztU0VbRz7i2XxNWMnHM7gBP919kAnAw8Fvd4Jd4BwTvxDhyvrqvJeecOHIvXw/8A73d5ciMv0xNvB/FvvNbEZ8CtjdTyLN4FkP+At8PYC2/mT1v90a/n33itqxOdc1/67/lYvLbGp8AvgDOcc+/4P3cH3r/Hv/BGy61pfdyDd0Wp14FXqf+7/BxvCuujfk2nkMSlBJ1z2/F+J0XxzyfB0hWUsoh50/fecc79MOxaRNrKzK4HRrgMve5pFGjEnsH8j/17+e2MI/D60U+EXZdIW5l33sRZeCe/SZoo2DPbALzpaluAnwHnO835lSxlZufgHaR+yjm3LOx6okytGBGRiNGIXUQkYhTsIiIRo2AXEYkYBbuISMQo2EVEIkbBLiISMf8fn9N7aG81H90AAAAASUVORK5CYII=\n" 249 | }, 250 | "metadata": {} 251 | } 252 | ] 253 | }, 254 | { 255 | "metadata": { 256 | "_uuid": "835d5be8da0bf460e03d0b95a95c6b06fa6727d0" 257 | }, 258 | "cell_type": "markdown", 259 | "source": "We can check the support vectors and plot them together with the separating hyperplane" 260 | }, 261 | { 262 | "metadata": { 263 | "trusted": true, 264 | "_uuid": "282b83ac5feca00c626be63acdb19d2735fbf967" 265 | }, 266 | "cell_type": "code", 267 | "source": "sup_vect = X_train[alpha > 0]\nsup_vect", 268 | "execution_count": 14, 269 | "outputs": [ 270 | { 271 | "output_type": "execute_result", 272 | "execution_count": 14, 273 | "data": { 274 | "text/plain": "array([[-0.70967236, 2.48106195],\n [-2.55221151, 1.0616012 ],\n [-0.66915023, 0.72634764],\n [-0.58912856, -0.52122884],\n [ 0.15695433, 1.0690201 ],\n [-1.79937151, 0.50534982],\n [-2.03140158, 1.21456088],\n [-0.90047109, -0.24801617],\n [-0.82341316, 0.4473006 ],\n [ 0.12726628, 1.19871734],\n [ 0.17361615, 1.12422702],\n [ 0.26801452, 1.0253296 ],\n [-1.0225964 , 1.51861516],\n [-2.23271581, 1.90999537],\n [-0.2382763 , 0.8559366 ],\n [-0.05097576, 1.22412656],\n [-1.93091128, 0.91924923]])" 275 | }, 276 | "metadata": {} 277 | } 278 | ] 279 | }, 280 | { 281 | "metadata": { 282 | "trusted": true, 283 | "_uuid": "dc5a969d0d5e50c1b52abd8c6acbf06eb9e83bbd" 284 | }, 285 | "cell_type": "code", 286 | "source": "plt.plot(x0[:,0],x0[:,1],'x',color='red')\nplt.plot(x1[:,0],x1[:,1],'x',color='blue')\nplt.figtext(0.5, 0.01, 'Figure 3: support vectors defining the separating hyperplane', \n wrap=True, horizontalalignment='center', fontsize=12)\n\nx = np.linspace(-5, 5, 200)\ny = np.linspace(-5, 5, 200)\nX, Y = np.meshgrid(x, y)\nplt.contourf(X, Y, Z, alpha=.5, cmap='jet_r')\nC = plt.contour(X, Y, Z, colors='black',zorder=4)\nplt.plot(sup_vect[:,0],sup_vect[:,1],'o',color='green')\n\n\nplt.axis('equal')\nplt.show()", 287 | "execution_count": 15, 288 | "outputs": [ 289 | { 290 | "output_type": "display_data", 291 | "data": { 292 | "text/plain": "
", 293 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEPCAYAAACqZsSmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VNX9//HXB7KxJYIQNaAmrdoWRdFGbQ1aLWoXcatQ5Kt0cauUqlgrIKUVW60taovfr1Zbbe3PpaCNVVtrW5fS8iUqGldwX4IowS8oyi6CnN8f506cGWYmk+Qmd5b38/HI45G5c+fOmTt3Pp97zzn3HHPOISIixaVX1AUQEZGep+AvIlKEFPxFRIqQgr+ISBFS8BcRKUIK/iIiRUjBX0SkCCn4i4gUIQV/EZEiVBLVGw8ePNjV1tZG9fY9prX1w6iLIFLwamrKoi5Cj3jiiSfecc4NCWNbkQX/2tpampubo3r7HjNr1rKoiyBS8GbN2i3qIvQIM3sjrG2p2kdEpAgp+IuIFCEFfxGRIqTgLyJShBT8RUSKkIK/iEgRUvAXESlCCv4iIkVIwV9EpAgp+IuIFCEFfxGRIqTgLyJShBT8RUSKkIJ/N3LOcdnsw3nyyXlRF0VEJIGCfzf6zW9+w9ZNLfz1r9O45JLdeeihK6IukuSApqYBtLSUJyxraSmnqWlARCWSYqTg343Gjx+PxT1euPAaJQChpuZDGhsHtyWAlpZyGhsHU1OjiX+k54Qa/M2st5k9ZWb3hrndfDVw4EBevOgipn/xi22z5ixceA3z518VabkkWnV1mxk79h0aGwczf34VjY2DGTv2HerqNkddNCkiYZ/5nwe8EPI281rfsjImH3ooL/7wh5QGlwELFvw38+f/MtqCSaTq6jZTX7+eBQuqqK9fr8AvPS604G9mw4BjgBvD2mYhKS8p4YUZP+Tbe+8MwIIFV6sKqIi1tJTT3Nyfww5bQ3Nz/+3aAES6W5hn/nOAqcC2ELdZUMpLSvjJ2O9w9O47AGoDKFaxOv6xY9/hiCPWtFUBKQFITwol+JvZGGClc+6JdtY7y8yazax51apVYbx1Xvr1qZPbqoCUAIpPa2tZQh1/rA2gtbUs4pJJMQnrzL8BOM7MlgLzgC+a2a3JKznnfuucq3fO1Q8ZMiSkt84/sSogJYCOKZQukg0N67ar46+r20xDw7qISiTFKJTg75y7yDk3zDlXC5wM/Ms5d2oY2y5UqRKAGoEzK4YukoWS4CT3qZ9/hGIJYFB5b8A3AisBpFcMXSSLIcFJbgg9+Dvn/u2cGxP2dgtVeUkJz0yfyRn7qBdQNgq9i2QxJDjJDTrzzxEXn/QddijzVwBqA0ivGLpIFnqCk9yg4J9Dmi+crkbgDIqli2QxJDiJnoJ/DknVCHzJJXX85S/Toi1YjiiGLpLFkuAkegr+OSY5AcA2nnpqHnff/YMoi5UTiqGLZDEkOMkNJe2vIj0tlgCuWbCAspISZs+fzzPP/ImPPvqQk07676iLJ90oVSKrq9usen8JnYJ/jiovKeGCL34RgN8vWsQ7GzeyZMk9AEoAItJlqvbJA0/84AcMKPd1vkuW3MOdd54bcYlEJN8p+OeBXmYsmTZNCUBEQqPgnydSJQA1AotIZyn455FYAjhn1CgAnnnmT7oCEJFOUfDPM73MmDp6NFMOOwxQFZCIdI6Cf546//DD1QYgIp2m4J+n1AgsIl2h4J/HirERWOPdi4RDwT/PxRLAjhUVgG8ELuQEoPHuRcKh4F8Aepnx9LRpzBg9GijsXkAa714kHAr+BWTSqFGM2NlPCrNkyT1cdtmnuPvuCyIuVfjCHu9eVUlSjBT8C8y9Z53V1gawdesHPPNMI7///diISxWusMe7V1WSFCMF/wITawP47LBhbV/um28+zi23nBJpucLSHePdqypJipGCfwHqZcbdp5/O0h//mOE77QTA668vLIgE0F3j3WvqRCk2Cv4FzMz459lnc+1JJwE+AeR7FVB3TeiiqROl2Cj4F4Hj9tmHkUOHAr4KKN8TQNg0daIUIwX/IvGX009n16oqQAkgmaZOlGKk4F8kzIym885jWFwCuOmm4koA6bp0AgU/N7BIMgX/ImJmPHzeeW1XAMuWPc7NN+d3I3BH+ui316VT/f2lmCj4F5nYFcBFwfzALS0LuemmcRGXqvPiA3pD0/U88cr13LLjQTzYsCNzOIQVK2fT0HQ90H6XTvX3l2Ki4F+EzIzvHnoo//O1rwGwbNljedsGEB/Qp5dVcW/tHFzVMjDHGlvOLYOupXHPtQnrp+vSmS45tLaW6YpACo6CfxE7YcSIhEbgfGoDiK+iiQX0J/e6Dso2Jay3qRRur7677XF7XTpTJQddEUghUvAvck1JbQD5kgDiA3JLSzmLFg2AqmUp111DK5Bdl85UyUF3AEshUvAvcrE2gPgEMHv2ftxzz4U45yIuXXqxgHz77YOZO3cI4Oi3dWjKdauoAdrv0pkpOegOYCk0Cv6yXQLYtOl9nn76Dv7nfw7P+QRQU/MhW7b04uCD13PK6uPpsyVxnT5bYPzKE4D27w7OlBx0B7AUGgV/AYJuoFOm8MDZZ/Ot4X48oPfeW5rTCaClpZz/+7+ytoC8x6KRTFw9mSo3FJxR5YYycfVkxr5SmdX20iWHWBWT7gCWQmJR/bDr6+tdc3NzJO/dk96aNSvqInTKcTfcwFOtvq584MDdOeec/2BmEZfKN/TGGlpjARlgyZK+vPhi326pi4+9Z/x2W1rKaW0t041gOWLWrN2iLkKPMLMnnHP1YWxLZ/6S0l/OPJP9a3xd+XvvvZEzVwCxs/AlS/q2Bf7GxsHss8/GbhuSobsGkxOJkoK/pJWYAJZy3XVfijwBxOrhX3yxL0uXViT0vFFAFsleKMHfzHY1s/lm9ryZPWdm54WxXYneX848kwOHDQNg1aqXciYBqOeNSNeEdea/FbjAOTcc+Bww2cyGh7RtidifTz+dxm9+E/AJIOoqoO7seaPxfaRYhBL8nXMrnHNPBv+vA14AUne6lrx0cG0tZxx8MLB9L6CmpgE8tPJe5nAIl1DLHA7hoZX3dkvA7O6x93U3rxSL0Ov8zawW2B9YlOK5s8ys2cyaV61aFfZbSze7+MtfTmgDiCWAD/acy8JBF7LGlreNqbNw0IV8sOfc0MvQ3WPv625eKRahBn8z6w/cCUxxzq1Nft4591vnXL1zrn7IkCFhvrX0kL+ceSYHJCSAL/DskMugdGPiiqUbWVz9s9Dfvyd63qhNQYpBaMHfzErxgf8259yfw9qu5J57zjyTzwbTQr733husva4VUjQBxMbUyTd//etAFi1KbFNQvb8UmrB6+xjwO+AF59wvw9im5La7zziDC7/wBf9gFXAt2yWA2Jg6+aSlpZznnusLGLW1H7SNH3T77UNU7y8FJawz/wZgIvBFM3s6+PtqSNuWHHXu4Ye39QLiHeC/+TgBbOnLiJUzIipZ57W2ljF+/DuMH7+KxsbBLF1aARh7771B1T9SUErC2IhzbiEQ/b3/0uMOrq2lpnJvWtc+B+8BV0L5oZXU115GxasToDq/brqKbzuI1fsfdtgajjhiTYSlEgmf7vCVLlt0/ti2RmA2wOZ/rOXJ//cjDjkkfwOmRvGUQqfgL6G4J+5OYPDDQv/ylwfj3LYIS9U53X0vgUguUPCX0Pz59NN58+KLmXvKKQCsX7+SK6+sDzUB9MQNZd19L4FILlDwl9CN2mMP5k+eDMDGje9yxRX7h5YAeuKGMo3iKcVAwV+6xR6DB3PMcD+806ZN74eWABZX/6zHbigTKWQK/tJtrh83jjEhJ4B0N47l6w1lIlFR8JdudV1SArjqqoO6lADS3TgW1g1lGtVTioWCv3S7+ASwYcOqLvUCGrFyBmzpm7gwxBvKNKqnFAsFf+kR140bx7+DRuD161d2ugqo4pUJjFp9RcIk7aNWX0HFKxM6Xbb4HkQ31+2CnV/LbYvv5+abh6Qc1VNXB1IINIF7N8vXCdy7qn9TE/P2/IBLqxfTyhpqqGLmyhEMXriGcYsXA9Cnzw5ceOFTmEV7DvLQyntZOOjCxIbkrWWweQD0XU0VNYxYOYOKVybQ0LAu4T6AurrN2z2WnqcJ3DtOZ/7SLebt+QEXDFrIcluDM1hua7hg0ELeGVUVeiNwV6XsQVTyIfR7N2V30mzG/NfVgeQ6BX/pFpdWL2ZjaeKyjaV++XXjxnVLN9DOyqqnUOlGHttxZlvwbm/Mf7UdSK5T8Jes9G9q4t6VD/E55rAbl/A55nDvyofo39SUcv1WUo/rE1ue3A00yqEgsu0p9GHv99rO/tsb+0czgkmuU/CXrKSrxpm35wcp16+hqt3l140bx+RRowDfCHzVVQfx7rstPT45fMoeRKkYNFWfy52v/IO5c4e0O/aPZgSTXKbgL1nJVI2TysyVI+i7JXFZ3y1+ebzpo0e39QLasGEV11xzOJde+hm2bu256pHkHkS49D8LZx+xpPZ8PjXuxnbH/tHIoJLLFPwLWP+mJspbWhKWlbe0pK2qyaS9apxkJ79SwVWrR1H5QTU46O2MjSVwWVUzv1i5guuaGtrW/eTgwYzbd9+2x9u2beJnl+/D1q09c6bc0LCO0dVjmMLDXMxSUs5JGa9sI2/u+ZOERclj/2hkUMl1Cv4FbEtNDYMaG9sSQHlLC4MaG9lS0/G7YbOpxom3vqGBMdWj+caaE2BrHz4yBwZvVXzANYNuYe2ejQnr//LEE3l0yhT+a//9AXDbNvOzy0d0SwJob2TQbNoA2msk1sigkusU/AvY5ro6Vo8dy6DGRirnz2dQYyOrx45lc11dh7eVbTVOsruqb4fSTYkLSzf55UmGVlXxi+OOY9IhhwA+Afz8F/uHXgXU3sig41eeQOmHpRm30V6C0MigkusU/Avc5ro6NtTXU7lgARvq6zsV+OHjapyhrgpzMNRVcdXqUZz8SkXG13W0ughgxlFHcdP48dC7Fx9t3cDlPx8ZagJob2TQsa9U8u33zgraAPDtAPHydH5ikXgK/gWuvKWFfs3NrD3sMPo1N2/XBpCtWDXOo0xhGRfzKFMYUz2a9Q0NGdsWOlpdFHPkpz/NU1POBzO2fbQhqzaAhqbrWbFydkJ1zoqVs2louj5hvfZGBm1qOJtdqqcyhYc5smk1o1ZeE+pwEiK5QMG/gMXq+FePHcvaI45oqwLqbAJIJ1Pbwokrx8OWPkkv6OOXt2Nw//5MjqsCaq8NoHHPtdwy6NqE6pxbBl1L455rE9bryMigyY3BU3iY0dVjVH0jeU/Bv4CVtrYm1PHH2gBKW8Md+z5T20LlK2P53uqJCdVF31s9kcpXxma17elHHpnQBuATQOoqoNur72ZTUlX9plK/PF53jwwqkg9Koi6AdJ/1DQ3bLdtcV9fpev9M4tsW1h52WNt7TGpoAnZhGlM+XrkaqM6+u+mMo44C4LqHH8Zt28zll4/goosWU1KS2HMm24leKl6ZwKg9fd3/Glr9wG2r/cBtVOuMXoqDgr+EIrltYXNtbahJZsZRR4FzXPfII2zb9gE//8VIpk97ipKSj/vNV1HDGpZv99rk6hxfZTOG0Yz5eGE1CvxSVFTtI11W3tLCjnPnsm7UqIS2hf6PPJJwQ1lHxwdKNuPoo3ny+99v6wWU3AYwfuUJ9Enqjtpni18uIokU/KXLSltbWXvEEQxYuJDylhY219WxbtQoKufPT7ihrKPjA6UyZMAAms85F/BtAJddNoL7W+9iDodwQ/W19O7Vn7KtA9t65kxcPZmxr1SG/plF8p2qfaTLYm0LW3bemUGNjWyor6dfczPvTpiQUPVzafViNiZ1mY+NDzSG0Vm/305VVUz6/Oe57pFHgM088vspcCFQDut7r4ctfRm18hpGV4+BamiqDuFDihQYnflLaNq7oawzN3ylM+Poo5n0+c/7Bx8BPwcuAzaRcMOWiKSm4C+hae+Gss7e8JXOjKOPhkPiFmwFrgA2ZTlBi0gRU/CXUGRzQ1lnxwfKZOhRVXAWMB5fibkNmA39N6muRyQTBX8JRTY3lHV2fKBMTlw5Hob0gc8A5wcLHayfvZJNmzpenSRSLNTgK6HI5oay9Q0NjIHExt1qWN+Fk/TKV8byvT396KGtfdcw4Jgy1v7tQ8Axe/ZIpk59mj59OletJFLIdOYveW1SQxPTqndpG3DuufqLuPyYY4JntzF79kg2bVqbcRsixUjBXwrOqfX1XP7VrwaPtjH7igOUAESShBb8zezLZvaSmb1qZtPD2q50vzCne8wVpx54INeedBIl5b3BbVECEEkSSvA3s97AtcBXgOHABDMbHsa2pfuFNd1jriWR4/bZhyXfn0rvsl4+AczeTwlAJBDWmf9BwKvOudedcx8C84DjQ9q2hCRdcI711OnqdI9hzhkcln5lZVzxlWODR9uCBKBeQCJhBf+hwJtxj98KlkkOyRScw5juMda9c8e5cxl4110JSSTKK4BxI0fys/g2ADUCi/Rsg6+ZnWVmzWbWvGrVqp58ayHzpCthTfe4ua6OTZ/5DP2efZaPKv2AavFJJqokMPHAAxMbgVUFJEUurOC/HNg17vGwYFkC59xvnXP1zrn6IUOGhPTW0hGpzvDDnO6xvKWFildfZcO++1L69tsMvu02drz9dlaP9TN3RVkNdOqBByZ2A73iAFUBSdEKK/g/DuxpZnVmVgacDPwlpG1LiFKd4Yc13WN8EnnvxBPZuO++2EcfYVu3Ur50aafbElLp7NwAp9bX8/y0aZRUBL2AZu+vKwApSqEEf+fcVuB7wD+BF4A7nHPPhbFtCU+6M/xYnX+8zXV1Ke/azSQ+icRfAeBcl9oSUunK3AADKipYcv7U4NFHzJ49gp9e+mnWrNFgcFI8Qqvzd87d55zbyzn3SefcZWFtV8LT3RO6r29o2K4aaePIkWwrLWVbaSn9Fi3qdFtCskurF7MxabL22NwA2ehXVhbXCAzbPtrEnDmfVwKQoqE7fItILDjH68wZfntiSQZ8Hf/q8eN5d8IENu29d6fbEpKFMTfAxAMP5JZTTmHEzju3LZsz5xAlACkKCv4SuliSib/S2FxXx/vHHhvalUZYcwMcvsce3Ped73DzhAnBEsecOQ1KAFLwFPyl23TnlUbYcwMcsddetMycSVVVL2CbqoCk4Cn4S17KNDdAZ3sClfTuzdPnzGh7rAQghUzj+UteyjQ3wL0rH+KCQQvbJotfju8JxJ6jGNPOdkt69+bmCRP4xty5gE8AU6Y8QlVVdENUiHQHnflLwelqT6Aj9torrg1AVwBSmBT8peCE0RNo+wSgRmApLAr+UnDC6gkUSwBl/UrxjcBKAFI4FPwLSK6Np5+N7ihzmD2BjthrL146fxqVleoFJIVFwb+A5OJ4+u3paJmzSRaZegJ1Rknv3vxx3LfbHisBSCEw51wkb1xfX++am5sjee+e9NasWT36frHguaG+nn7NzaENpNadOlLm+KEjkoeS6O7POf/ll9t6AQHqBZRDZs3aLeoi9Agze8I5Vx/GtnTmX2DCmJSlp3WkzJnmJOhuqXoBzZ69H62tS7r9vUXCpuBfYMKalKUndbTMUSa45ASwadP73HDDMbS2ZteNVCRXKPgXkDAnZekpnSlz1AnuiL324sXp07n3jDMYPNjfJ3nDDWOUACSvKPgXkO4esrk7dLTMuZLg+pWXs9/Qocwbe0bbMp8AVAUk+UHBv4D01JDNYepomXMtwX1qp53451lntT32VUBKAJL7FPwlr+Righu+yy4pEoCqgCS3KfiLhGD7BKA2AMltCv4iIRm+yy688aMfMXRoGaAEILlNwV8kRL169eKR06fTu7d/rEZgyVUK/iIhMzPuO11tAJLbFPxFuoHaACTXKfhLWmGOuJmPI452VeoEoCogyQ0K/pJWmKOE5uOIo2GIJYB+/fxPTfcBSK5Q8Je02htErSNn81EOyBa14bvswgsXzGRYWy8gtQFI9BT8BUgfyEtbW9MOotbRs/l8HHE0LGbGQxO/3/b4hhvGcMMNx7JsWeEPay65ScFfgPSBnF690g6i1tGz+agHZIta3/LyhDaA1tZnuemmk3jppYciLJUUKwX/HBJlo2iqQL5u1CgGLFyYcRC1bM/mc2VAtqjF2gDK4n558+adxssv/yu6QklRUvDPIVE3iiYHcrZta3cQtWzP5nNtQLYoDd9lF1770cUsOv989qj2d4PNnfttXnrpwYhLJsVEwT+HRN0omhzIt9TUZBxErSNn87k4IFvUaiormT9pJvV1pQDMm3e6EoD0GAX/HBNVo2hnqmV0Nh+OP3z944ZgJQDpKQr+OSaqRtHOBHKdzYejqqKCReef3/bYJwA1Akv3UvDPIVE2inZnIC/Gu3s7qqayMikBnMbLL+sKQLqPgn8OKdRqlKgbsvNFcgKYO/d09QKSbqPgn0MKtRol6obsfFJTWcnrM2dy4Cd8I7B6AUl36XLwN7MrzOxFM3vWzO4ysx3CKJgUlmK+u7ejSnv35s8TZzCkvwFqBJbuEcaZ/wPAPs65fYGXgYtC2KYUmGK/u7cz7j1zStv/SgASti4Hf+fc/c65rcHDR4FhXd2mFBbd3ds52zcCKwFIeMKu8z8N+Hu6J83sLDNrNrPmVatWhfzWkqsKtSG7J6RKAPPmncGyZY9HWCopBFkFfzN70MyWpPg7Pm6dHwJbgdvSbcc591vnXL1zrn7IkCFdL73khUJtyO4psQTQv9w/fumlB7jpprE89tgt0RZM8lpJNis5547M9LyZfQsYA4x2zrkQyiUicWoqK3lh+sX8ctFV3Py/G3h3g+Pvf5+JGRx44MSoiyd5KIzePl8GpgLHOec2dr1IIpLO9w++gAWTL2z74d5330wef1xXANJxYdT5XwMMAB4ws6fN7PoQtikiaVT26cOSadOUAKRLwujts4dzblfn3Mjg7+wwCiYi6Q2oqFACkC7RHb4ieSqWAD5RVQH4BKBGYMmWgr9IHhtQUcF/pkzjuE/sCMDf/64rAMmOgr9IAbh24vfoHfyvKiDJhoK/SIFYnNQGoCogyUTBX6RADKioYPHUqW0/alUBSSYK/iIFpLJPn4QEcN99M2luVgKQ7Sn4ixSYyj59eG76dMbUDQLgb3/TFYBsT8FfpAD1Ly/num+cw0E79wfUCCzbU/AXKWA3feO7CVVADz74C95/f3mkZZLcoOAvUsCS2wCamn7N1VcfwoMP/iLSckn0FPxFClxsLKD4IXybmn6tBFDkFPxFisCAigpe+/GPaTr3XM7etwbwCeCBBy6PuGQSFQV/kSLRy4zdBg7khyeeyaE1lQA8/PD13H//zyIumURBwV+kCP3h2+dQbv7/Rx75jRJAEVLwFylCZSUlPD/jh0oARUzBX6RIpUoAagQuHgr+IkUslgB2G1AGqBdQMVHwFylyZSUlNH3/Iibt93EvIFUBFT4FfxEBYMYJZzKg1NcBqQ2g8Cn4i0ibp6fOUCNwkVDwF5E26gVUPBT8RSRBqgTwz3/+hA8/3BBtwSRUCv4isp1YAjj109UAPPro77j88uHcccekiEsmYVHwF5GUykpKuHz8JM4dOaxt2Qsv3KcEUCAU/EUkowuPP52/nnEGA8rLASWAQqHgLyLtGjl0KEumTWNgRQXgE8Cf/jQ54lJJVyj4i0hWepnx9NSpDOzTB4Dnn7+XO+88N+JSSWcp+ItI1nqZ8fSFFzLjyCMBWLLkHlUB5SkFfxHpkF5mTGpoYHJDA6A2gHyl4C8inTJ19OiENgAlgPyi4C8inZLcBqAEkF8U/EWk02JtAPFXAGoEzg8K/iLSJbErgGGVfl7gJUvuUQLIAwr+ItJlvcx45Pzz+fGXvgSoF1A+CC34m9kFZubMbHBY2xSR/HLm5z7HHjvuCKgNINeFEvzNbFfgaGBZGNsTkfz10OTJagTOA2Gd+f8KmAq4kLYnInmqrRE4LgFcckktt946MdqCSYIuB38zOx5Y7px7JoTyiEgBSE4A4HjttQX84Q9fj7Rc8rGsgr+ZPWhmS1L8HQ/MAH6c5XbOMrNmM2tetWpVV8otIjkulgDOO+wwrj3pJADeeGMRN954QsQlE8gy+DvnjnTO7ZP8B7wO1AHPmNlSYBjwpJntnGY7v3XO1Tvn6ocMGRLWZxCRHNXLjB8ccQTH7bMPI3baCYDly5/ixhuPj7hk0qVqH+fcYudctXOu1jlXC7wFHOCcezuU0olIwfjbd75D7cCBACxf/rSuACKmfv4i0iPMjAXnnEPtDjsAsSsAJYCohBr8gyuAd8LcpogUDjPjf887j7q2K4Cn1AgcEZ35i0iPW3DuuVx57LFArBFYbQA9TcFfRCIx/oAD+HXQC0htAD1PwV9EIjNm773VBhARBX8RiUyqNgAlgJ6h4C8ikfvPOeeoEbiHKfiLSOTMjAXnnssngwTwxhuLlAC6mYK/iOSMf597Lr8ZNw7wCeCSS+r43e9OxDmNGRk2BX8RySlfHT6csfvuGzzaxltvPcnVV49SAgiZgr+I5JxfnXgi40eOpH9ZGQBr1rylBBAyBX8RyUlXHn88L1x0EQfvuivgE8C1145WAgiJgr+I5LTG007j8E98AoB3331NCSAkCv4ikvNumTiRu08/HfAJ4OqrG5QAukjBX0TywmeHDePr++0HwJo1y9UG0EUK/iKSN6464QQOimsDUALoPAV/Eckrd552WkIjsBJA5yj4i0jeaUxKAM3NzRGXKP+URF0AEZHOaDztNP70zDPcOXgp9fX1URcn71hUl0tmtgp4I5I3T28wkC8zkams3SefyptPZYX8Km8ulnV359yQMDYUWfDPRWbW7JzLi1MIlbX75FN586mskF/lzaeydobq/EVEipCCv4hIEVLwT/TbqAvQASpr98mn8uZTWSG/yptPZe0w1fmLiBQhnfmLiBQhBf8kZnaOmb1oZs+Z2eyoy5MNM7vAzJyZDY66LOmY2RXBfn3WzO4ysx2iLlMyM/uymb1kZq+a2fSoy5OJme1qZvPN7PngWD0v6jK1x8x6m9lTZnZv1GVpj5ntYGaNwTH7gpl9PuoyhU3BP46ZHQEcD+znnNsbuDK2j1EoAAAPOklEQVTiIrXLzHYFjgaWRV2WdjwA7OOc2xd4Gbgo4vIkMLPewLXAV4DhwAQzGx5tqTLaClzgnBsOfA6YnOPlBTgPeCHqQmTpauAfzrlPA/uRP+XOmoJ/oknAz51zmwGccysjLk82fgVMBXK68cY5d79zbmvw8FFgWJTlSeEg4FXn3OvOuQ+BefgTgZzknFvhnHsy+H8dPjgNjbZU6ZnZMOAY4Maoy9IeM6sCDgN+B+Cc+9A59360pQqfgn+ivYBDzWyRmf3HzA6MukCZmNnxwHLn3DNRl6WDTgP+HnUhkgwF3ox7/BY5HEzjmVktsD+wKNqSZDQHf5KyLeqCZKEOWAXcFFRT3Whm/aIuVNiKbmwfM3sQ2DnFUz/E749B+MvoA4E7zOwTLsIuUe2Udwa+yicnZCqrc+6eYJ0f4qssbuvJshUqM+sP3AlMcc6tjbo8qZjZGGClc+4JMzs86vJkoQQ4ADjHObfIzK4GpgM/irZY4Sq64O+cOzLdc2Y2CfhzEOwfM7Nt+PE9VvVU+ZKlK6+ZjcCfoTxjZuCrUZ40s4Occ2/3YBHbZNq3AGb2LWAMMDrKhJrGcmDXuMfDgmU5y8xK8YH/Nufcn6MuTwYNwHFm9lWgAqg0s1udc6dGXK503gLecs7FrqQa8cG/oKjaJ9HdwBEAZrYXUEbuDewEgHNusXOu2jlX65yrxR+wB0QV+NtjZl/GX/Yf55zbGHV5Ungc2NPM6sysDDgZ+EvEZUrLfMb/HfCCc+6XUZcnE+fcRc65YcFxejLwrxwO/AS/oTfN7FPBotHA8xEWqVsU3Zl/O34P/N7MlgAfAt/MwTPUfHUNUA48EFypPOqcOzvaIn3MObfVzL4H/BPoDfzeOfdcxMXKpAGYCCw2s6eDZTOcc/dFWKZCcg5wW3Ai8Drw7YjLEzrd4SsiUoRU7SMiUoQU/EVEipCCv4hIEVLwFxEpQgr+IiJFSMFfRKQIKfiLiBShTgd/M9vNzNYHQ+GKhM7MaoN5CrK6GdHMLjWzd8zs7Y4cnz19LJvZUjPLOBRGoQvmIDi8G7b7LTNbGPZ2u0NwbO8R1fu3G/yDA3VT8OOI/dU455Y55/o75z7qiYK2U8bhZtZsZu8Ffw/mwdjmHWJmh5vZWxG87x/M7NKeft+OMrPdgAuA4c65nTtyfHbnsZwv+687pdoHzrm9nXP/jqhIQvZn/scGP47YX2t3FirbM704rcBY/Iicg/FjsswLu1xR6cT+yBk9eGW4G/BunszBUDDy+djsTnlRI+Kcy/gHLAWOTLG8Fj+BSEnwuA5YAKwDHsTPinRr8Nzh+FHyUm4XmIUfOe9WYC1wBj4xTQdeA94F7gAGZVHeEmAysLG9deNe8y38+B3rgBbglLhy3ZrhM/8buBx4LCj3PbEyxq17Fj45rQB+ELetcvwY563B3xygPH5/AdOAt4E/AZvwY6GvD/5qkj7DwcG6veOWnQg8G/yfcX8Co4CHgffx49p/Kyj7Fvw4R+uBvwbrfib47O8Dz+EHa4tt5w/AdcB9wAbgSOCr+IGx1uFHyvxBmu+hN372tHeC72Ny0v6uwg9mtiLYzqXBa45M2j9/SPNd/RRoCspxPzA4w/eact3g+W8AbwT78Uek/42k239LgR8AzwJrgNuBirjXjQGeDvbvw8C+afaX4SfzWYk//hbjZ0sDf3xdiZ/h7f+A64E+ScfXjGBfLyU45oPnjwGeCrb5JjArxW/g9GDbC4Llf8Iff2vwcWDvLPZB/O//DuDmYH8/B9THvecBQXnWBe9zO3Bpht/ywuCzv4f/PX8leG4c8ETS+t8H7ok7dq/Hzzq3DvgPsHvcup8OnlsNvAR8vZ3jvr3tOWCPDuzzbwb7/B38MOmx5zsXK0MM/o8EO7wMH0jW0rHgvwU4IfggffBTvsVmfCoHfgPMbaes7+PHit8GzIxb/l8EQTDFa/oFZf1U8HgXPj5wZ9F+8F8O7BNs5864zxxbd27w3Aj80NCxz/yT4PNVA0PwP/Kfxu2vrcAvgs/eJ9U+TPFZXgOOinv8J2B68H/a/Qnsjj84JwClwI7AyLiD+tK4bZYCr+IDRxnwxeC1n4pbfw1+4LFe+CF8VwCHBs8PxI8+mqr8ZwMv4odWHgTMT9rfdwXl7hfst8eA76Q6xtJ8V6/hJ+zpEzz+eSfWHY4PYqOCz38l/tjd7jeSav/FHfuPATXB53wBODt4bn98MD8Yn9i+GaxfnmLbXwKeAHbAJ4LPALsEz/0KfwU8CBgA/BW4POn4+iX+WPgCPmB9Ku75EcH3ty8+eZyQtK9uDr6HWEI5LXif2EnN01nsg/jf/wf4k4Te+BOqR4PnyvCJ9jz8sfc1fCLJFPy3AGcG25qEP7myoGyrgc/Erf8UcFJcOdfhZ/Eqx0/luDAuTryJH+CtJPie3sFXM8Zem3zcp91eiuCfzT6/AX887gdsjn0OOhErOxL81+MD6/vA3ck/GPwl91agb9zrbqVjwX9B0vMv4Md9jz3eJfhSS9opbz/gu8Ax7X22uPXfB04iOJDjnptF+8H/53HPD8cfmL3j1v103POzgd/FBeqvJv2Ql8btrw9JPBvcbh+m+CyX4kejBP9D3EBwppFpf+Ln070rm+AFHIo/w+sVt2wuwZlKsP7NSdtYBnwHqGyn/P8iCILB46PjjrGd8Ad8n7jnJwDzU+2fNN9V/AnBd/FztHZ03R8T98MC+gbfVUeD/6lJx8X1wf/XEZwExD3/EvCFFNv+In4+5M8lfR8WfPefjFv2eaAlbl9tBfrFPX8H8KM0n2EO8KukffWJDN/jDsE6Ve3sg/jf/4NJv6NNwf+H4U+wLO75hcnbi3vuW/jpOOO/HwfsHLd/Lwv+3xt/dVAeV855ca/tD3yEPxkZD/xv0nv9Brg4w3GfdnvB47bgn+U+Hxb3/GPAye39tjP93rKt8z/BObdD8HdCiudrgNUucZz2N1Osl0ny+rsDd5nZ+2b2Pv4DfoQPAmk55zbgL7VuNrPq9t40WH88/qxzhZn9zcw+3clyv4E/Oxmc4fma4P+a4HGq5wBWOec+6EA5AP4IfM3MyvFnSE8652LvkWl/7opPRtmoAd50zsVPx/cGiVMeJn+XJ+HP6t4Ipsf8fKZtJ203Znf8vl0R9xl+g78CyFb8XAcb8T/Gjq6bUMbgmH+3A2Vob/u7AxfEPmPwOXcl8diIvfe/8ENlXwusNLPfmlkl/kqyL/BE3Db+ESyPeS849mPajj8zO9jM5pvZKjNbg/9txB/TELcPzKy3mf3czF4zs7X4wE6K12SSvD8qgvaEGvxUpS7Ve7e3rbiYFNu//w/4r2A+hInAHS6Yszt528659fgrhRr893Jw0vdyCokz16UqV7rtJchyn2c6ZjocK8Pq578CGGRmfeOWxc+KtAF/MAJtjSHxByJsPwH5m/i6uh3i/iqcc9nMrtQreL+s5mB1zv3TOXcUPmO+iL+82q7cpJ6iMP5z7obPuO9keD7WWN6K/9JSPQfb74/kx9txzj2P/xF/BV/V9ce4pzPtzzeBT6bbbNLjVmBXM4s/dnYjcdarhNc45x53zh2PD9R3488yU1nB9vsrvvyb8XXvsfJXOuf2TrOt7rKCuMnnzawPvposnXa/tyRv4s9M47+nvs65uSk37tx/O+c+iz9b3gu4EH/8bcJXX8a2UeWci092A5PmpY0//v6IrzLa1TlXhT+Zsgyf67/wk90fiW+XqQ2WW4p1O2oFMDQI1jG7plu5Pc65R/FXaofiy31L0ipt2w6myByE3y9vAv9J+l76O+cmxW8+xVum216ybPZ5Op2KlaEE/+DsshmYZWZlwZndsXGrvIzP5McEU8/NxNdNZXI9cJmZ7Q5gZkPMT1i+HTM7ysz2D85AKvF1me/hM2BGZraTmR0f/BA246u4Yme1TwOHBf3Aq/DVI8lODbqa9sXX4ze6xC6DPzKzvma2N76+8PZg+VxgZvC5BuOrE27NUNT/A3YMypHJH/F1gIfh6/xjMu3P24AjzezrZlZiZjua2ci49/1E3HYW4c86pppZadBX+1jS9K4KjodTzKzKObcF376SbhLvO4BzzWyYmQ0kbuo859wKfMPrVWZWaWa9zOyTZvaFdvZH2BqBY83sEPMTfcwi8480ef+15wbg7OBM0MysX/C7GZC8opkdGKxXij9R+QDYFlyV3QD8Knb1a2ZDzexLSZu4JPh+DsU3MseOlwH4K/kPzOwgfJDMZAD+t/Mu/mTpZ0nPd3QfxHsEfxb7veDYPB44qJPbirkZf8W0xTmXfE/AV81sVPDd/hTf9vAmcC+wl5lNDI770mD/f6ad90q3vWQd3efxso6V8cK8w/cUfL3iu/i659vxBwTOuTX4etMb8WeIG/C9DTK5Gp8J7zezdfgGjYPTrLsDPpiuwVdffBL4cqzaJAg+6WZl6oVv8W/FX5J9Ad9IhHPugeBzPItvWLs3xetvwdftvY1v5Dk36fn/4BtIHwKudM7dHyy/FJ8wn8X30ngyWJaSc+7F4DO+HlzebXfpGJgbfIZ/Oefir0DS7k/n3DJ8tcwFwT54Gt+oBL53zfDgPe92zn2ID/ZfwZ9h/hr4RlC+dCYCS4MqgbPxx0oqN+Bn0noGvz+S56X9Br4B8Hl8cm/EX631GOdn9zoHn+xW4E8WVhIc6ykk7L8stt+Mb6y8Bv8ZX8XXY6dSid9n7/Fx76MrguemBa99NNjvDwKfinvt28HrWvHJ/+y47/C7wE+C4+THpL9Si7k5eP/l+O/m0aTnO7QP4gXH29fwvYveB07F/w7T7e9s3ILvpJHqZOuPwMX438Fng/fDObcO3wZ1Mn6fvc3HHTIySbm9FDq6z+N1JFa26baZvMzsduBF59zF3fIGOcDM/o1vEL4xxXO1+G5mpc65rT1bMukpwaX8+8CezrmWqMuTjeBq7Vbn3LD21s1FZrYI30B+Uydf3wefsA9wzr0St/wP+E4DM0MqZ6jbC1toZ/7BJdAng8vxL+PrADuU5UXygZkdG1Tl9cN39VzMx42cEjIz+4KZ7RxU+3wT3xXyH13Y5CTg8fjAX4zCvDtvZ/xl+o74Kp1JzrmnQty+SK44Hl91YPiqu5Ndd11CC/jqqjvw3bJfB8YGbUAdZmZL8d9bql6LRUUTuIuIFCEN6SwiUoQU/EVEipCCv4hIEVLwFxEpQgr+IiJFSMFfRKQI/X/JoNMqPrArEAAAAABJRU5ErkJggg==\n" 294 | }, 295 | "metadata": {} 296 | } 297 | ] 298 | }, 299 | { 300 | "metadata": { 301 | "trusted": true, 302 | "_uuid": "3794d4d5b722d19532d5039af6861a077e3a4552" 303 | }, 304 | "cell_type": "markdown", 305 | "source": "## Final remarks\n\nIn this notebook we implemented the support vector machine for non-linearly separable datasets, using a linear decision boundary. This meant that points from different classes were allowed to cross the (soft) margin, up to a certain distance. In the next notebook we will implement SVM with non linear decision boundaries by making use of the kernel trick." 306 | }, 307 | { 308 | "metadata": { 309 | "trusted": true, 310 | "_uuid": "f2c3dfc6902607d04e62cb0c6a9cb2d0a35ee14d" 311 | }, 312 | "cell_type": "code", 313 | "source": "", 314 | "execution_count": null, 315 | "outputs": [] 316 | } 317 | ], 318 | "metadata": { 319 | "kernelspec": { 320 | "display_name": "Python 3", 321 | "language": "python", 322 | "name": "python3" 323 | }, 324 | "language_info": { 325 | "name": "python", 326 | "version": "3.6.6", 327 | "mimetype": "text/x-python", 328 | "codemirror_mode": { 329 | "name": "ipython", 330 | "version": 3 331 | }, 332 | "pygments_lexer": "ipython3", 333 | "nbconvert_exporter": "python", 334 | "file_extension": ".py" 335 | } 336 | }, 337 | "nbformat": 4, 338 | "nbformat_minor": 1 339 | } -------------------------------------------------------------------------------- /13. SVM part 1.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "metadata": { 5 | "trusted": true, 6 | "_uuid": "ea9ab27b57fbc7617ef9a747c0bec9b083adb44b" 7 | }, 8 | "cell_type": "markdown", 9 | "source": "# Support vector machine, part 1: linearly separable data\n\n## Introduction \n\nIn this notebook we explore the support vector machine (SVM) algorithm. This is a supervised learning classification algorithm. For the sake of simplicity, we focus on the binary classification task; for multiple classes classification, one may perform a one-vs-rest or one-vs-one reduction. We start with the case of linearly separable datasets, in which case the main idea is to find the hyperplane that maximizes the margin (distance) to both classes. For non-linearly sepparable datasets, we allow points to cross the decision boundary up to a certain margin (in this case we say that the classificator has a *soft margin*). Finally, we can also use non-linear decision boundaries by making use of the kernel trick, which relies on the idea of implicit feature augmentation.\n\n## The model: linearly separable case\n\nSuppose we have a dataset $S = \\{(x^1,y^1),\\dots,(x^m,y^m) \\}\\subset \\mathbb{R}^n\\times\\{-1,1\\}$ consisting of pairs (features,class). Denote by $w$ the normal vector to the separating hyperplane $H$ and $a$ a position vector of such hyperplane. Then the points $z$ belonging to $H$ satisfy the equation\n\n$$\nw\\cdot(z-a) = w\\cdot z - w\\cdot a = 0 .\n$$\n\nHere $\\cdot$ denotes the (Euclidian) inner product and we write $b = -w\\cdot a$. Suppose that $w$ is a unit vector. The hyperplane separates $\\mathbb{R}^n$ in three regions: $R_1 = \\{ x : w\\cdot x + b >0 \\}$, $R_2 = \\{ x : w\\cdot x + b < 0 \\}$ and $H$. In order for $H$ to be a separating hyperplane for our dataset, it is necessary that $x^j\\in R_1$ for all $j$ such that $y^j =1$ and $x^k \\in R_2$ for all $k$ such that $y^k = -1$ (or vice versa). This condition can be summarized by\n\n$$\ny^j(w\\cdot x^j + b) > 0\n$$\n\nfor all $j$. Assume now that $H$ is a separating plane for our dataset. Then the distance between a point $x^j$ and the hyperplane $H$ is given by \n\n$$\nd(x^j,H) = |(x^j-a)\\cdot w| = |x^j\\cdot w + b| = y^j(w\\cdot x^j + b).\n$$\n\nIf we want to maximize the size of the margin, then we want to find the parameters $w,b$ for the hyperplane $H$ such that $\\min_j d(x^j,H)= \\min_j y^j(w\\cdot x^j + b)$ is maximal. In other words, we want to solve the optimization problem\n\n$$\n\\begin{align*}\n\\max_{w,b} &\\ M\\\\\n\\text{subject to } &y^j(w\\cdot x^j + b) \\geq M \\text{ for all } j, \\\\\n& \\| w \\| = 1.\n\\end{align*}\n$$\n\nWe can remove the constraint $\\|w\\| = 1$ by normalizing the distance by the norm of $w$. That is, the above problem is equivalent to \n\n$$\n\\begin{align*}\n\\max_{w,b} &\\ M/\\| w\\| \\\\\n\\text{subject to } &y^j(w\\cdot x^j + b) > M \\text{ for all } j. \\\\\n\\end{align*}\n$$\nIf we perform the coordinates transformation $w,b \\mapsto w/M ,b /M$, then we get the equivalent optimization problem:\n\n$$\n\\begin{align*}\n\\max_{w,b}\\ &\\|w\\|^{-1}\\\\\n\\text{subject to } &y^j(w\\cdot x^j + b) > 1 \\text{ for all } j. \\\\\n\\end{align*}\n$$\n\nThis is now equivalent to the convex minimization problem\n\n$$\n\\begin{align*}\n\\min_{w,b}\\ &\\dfrac{1}{2}\\|w\\|^2\\\\\n\\text{subject to } &y^j(w\\cdot x^j + b) > 1 \\text{ for all } j. \\\\\n\\end{align*}\n$$\n\nwith linear constraints." 10 | }, 11 | { 12 | "metadata": { 13 | "trusted": true, 14 | "_uuid": "887f6e532cd105f54410f9eea7cd83ddcdb8c779" 15 | }, 16 | "cell_type": "markdown", 17 | "source": "## Lagrangian form\n\nThe previous minimization problem can be formulated in its dual form by considering the lagrangian\n\n$$\n\\mathcal L (w,b,\\alpha) = -(\\dfrac{1}{2}\\|w\\|^2 - \\sum_{i=1}^m \\alpha_i (y^i(w\\cdot x^i + b) - 1 ))\n$$\n\nfor $\\alpha_i \\geq 0$. Solving the original (primal) problem, is equivalent to the minimization of the Lagrangiagian subject to the [Karush-Kuhn-Tucker conditions](https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions). These conditions can be written as \n\n$$\n\\begin{align*}\n\\nabla_w \\mathcal{L} &= 0 , \\\\\n\\nabla_b \\mathcal{L} &= 0, \\\\\n\\alpha_i & \\geq 0, \\\\\n\\alpha_i (y^i(w\\cdot x^i + b) - 1 ) & = 0.\n\\end{align*}\n$$\n\nfor all $i$. The first two conditions can be written as\n\n$$\n\\begin{align*}\n\\nabla_w \\mathcal{L} = -(w - \\sum_{i=1}^m \\alpha_i y^i x^i &)= 0, \\\\\n\\nabla_b \\mathcal{L} = -\\sum_{i=1}^m \\alpha_i y^i &= 0 .\n\\end{align*}\n$$\n\nWith this conditions, the lagrangian can be reduced to\n\n$$\n\\begin{align*}\n\\mathcal L (w,b,\\alpha) &= -(\\dfrac{1}{2}\\|w\\|^2 - \\sum_{i=1}^m \\alpha_i (y^i(w\\cdot x^i + b) - 1 )) \\\\\n&= -(\\dfrac{1}{2}w\\cdot w - \\sum_{i=1}^m (\\alpha_i y^i w\\cdot x^i + \\alpha_i y^i b - \\alpha_i )) \\\\\n&= -(\\dfrac{1}{2} \\sum_{i,j=1}^m \\alpha_i \\alpha_ j y^i y^j x^i\\cdot x^j - \\sum_{i,j=1}^m \\alpha_i \\alpha_ j y^i y^j x^i\\cdot x^j + \\sum_{i=1}^m \\alpha_i )\\\\\n&=-\\sum_{i=1}^m \\alpha_i + \\dfrac{1}{2} \\sum_{i,j=1}^m \\alpha_i \\alpha_ j y^i y^j x^i\\cdot x^j\n\\end{align*}\n$$\n\nThis is a quadatic problem that can be solved using the usual optimization tools. We proceed to implement the algorithm. Once the optimization is solved, for a new observation $x$, we can predict its class by\n\n$$\n\\widehat y = \\mathrm{sign}(w\\cdot x + b).\n$$\n\nFor this, we need the values of $w$ and $b$ as a function of the training data and the parameters $\\alpha_i$. An expression for $w$ follows directly from the KKT conditions. For $b$, we can get a condition by subtracting the equation $(y^i(w\\cdot x^i + b) - 1 ) = 0$ for two indices $i,j$ such that $\\alpha_s,\\alpha_t >0$ and $y^s = 1$, $y^t=-1$, and then solving for $b$. Thus we obtain\n\n$$\n\\begin{align*}\nw &= \\sum_{i=1}^m \\alpha_i y^i x^i ,\\\\\nb &= -\\dfrac{1}{2}w\\cdot (x^s+x^t).\n\\end{align*}\n$$\n\nThe vectors $x^i$ so that $\\alpha_i > 0$ are called the **support vectors**, and the separating hyperplane is completely defined by them." 18 | }, 19 | { 20 | "metadata": { 21 | "trusted": true, 22 | "_uuid": "bad345e3bbb070073830b35876e66e8606756d9f" 23 | }, 24 | "cell_type": "markdown", 25 | "source": "## Implementation\n\nIn this section we implement the algorithm described above. We start by implementing the usual scientific libraries. This time, we will use `scipy` minimizing tool:" 26 | }, 27 | { 28 | "metadata": { 29 | "trusted": true, 30 | "_uuid": "f8a0d13fd5d6c5ad71612970ce8490fa47f02e30" 31 | }, 32 | "cell_type": "code", 33 | "source": "import numpy as np\nimport matplotlib.pyplot as plt\nfrom scipy.optimize import minimize", 34 | "execution_count": 1, 35 | "outputs": [] 36 | }, 37 | { 38 | "metadata": { 39 | "trusted": true, 40 | "_uuid": "617875a12555609e9edfe89819d68a503fc87247" 41 | }, 42 | "cell_type": "markdown", 43 | "source": "We generate two classes of points which are linearly separable. We fix a seed of random numbers so we always get the same dataset:" 44 | }, 45 | { 46 | "metadata": { 47 | "trusted": true, 48 | "_uuid": "1527939ac76db262eeebe28cf3092d647075ed7e" 49 | }, 50 | "cell_type": "code", 51 | "source": "np.random.seed(6)\nclass_0 = 25 \nclass_1 = 25\n\nx0 = np.random.multivariate_normal([-2, 0], [[1, .15],[.15, 1]], class_0)\nx1 = np.random.multivariate_normal([3, 2], [[1, .25],[.25, 1]], class_1)\n\nplt.plot(x0[:,0],x0[:,1],'x',color='red')\nplt.plot(x1[:,0],x1[:,1],'x',color='blue')\nplt.figtext(0.5, 0.01, 'Figure 1: two Gaussian clouds of points. Red points represent class 1 and blue class 0', \n wrap=True, horizontalalignment='center', fontsize=12)\nplt.axis('equal')\nplt.show()\n\ny0 = -np.ones(len(x0))\ny1 = np.ones(len(x1))\n\nX_train = np.vstack((x0,x1))\ny_train = np.append(y0,y1)", 52 | "execution_count": 2, 53 | "outputs": [ 54 | { 55 | "output_type": "display_data", 56 | "data": { 57 | "text/plain": "
", 58 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhUAAAEPCAYAAAD1QTYeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XucHGWd7/HvjyRclJuS8QIBoqLuxgtymCAedXVQVq6CrCIqcFz1Fc3qEVZwJOrkhKCLiYqXXRdl1cMB1DhHBBVFlJMBdQWciQK74baIQLjJAEYSRSHyO388TzM1PdW3SXU/1d2f9+vVr5nqrq76dV1/9TxP1WPuLgAAgK21TeoAAABAbyCpAAAAhSCpAAAAhSCpAAAAhSCpAAAAhSCpAAAAhSCpAAAAhSCpAAAAhSCpAAAAhZibOgA0Nn/+fF+4cGHqMACga6xbt+4Bdx9IHUe/IanoAgsXLtTExETqMACga5jZHalj6EdUfwAAgEKQVAAAgEKQVAAA2mb1amlsbPp7Y2PhffQekgoAQNssXiwde+xUYjE2FoYXL04bF9qDhpoAgLYZGpJGR0MisXSpdPbZYXhoKHVkaAdKKgAAbTU0FBKKM84If0koehdJBQCgrcbGQgnFyEj4W93GAr2DpCIRM5tjZr8ys0tSxwIA7VJpQzE6Kq1cOVUVQmLRm0gq0jlJ0o2pgwCAdhofn96GotLGYnw8bVxoDxpqJmBmCyQdLunjkj6QOBwAaJvh4ZnvDQ3RrqJXUVKRxmclDUt6vNYIZrbEzCbMbGJycrJzkQEAMEskFR1mZkdIut/d19Ubz93PcfdBdx8cGKBPHABA+ZFUdN7LJb3ezG6XtEbSQWZ2QdqQAADYeiQVHebuy9x9gbsvlHScpLXufnzisAAA2GokFQDQJPqxaD+WcXcjqUjI3a9w9yNSxwGgOfRj0X4s4+7GLaUA0CT6sWg/lnF3o6QCAFpAPxbtxzLuXiQVANAC+rFoP5Zx9yKpAIAm0Y9F+7GMuxtJBQA0iX4s2o9l3N3M3VPHgAYGBwd9YmIidRgA0DXMbJ27D6aOo99QUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEALVq9Whobm/7e2Fh4H+hnJBUdZmbbm9kvzOw6M1tvZqenjglAaxYvlo49diqxGBsLw4sXp40LSG1u6gD60J8lHeTum81snqSfmdml7n516sAANGdoSBodDYnE0qXS2WeH4aGh1JEBaVFS0WEebI6D8+LLE4YEYBaGhkJCccYZ4S8JBUBSkYSZzTGzayXdL+nH7n5NzjhLzGzCzCYmJyc7HySAusbGQgnFyEj4W93Gop1o04GyIqlIwN3/4u4vkbRA0gFm9sKccc5x90F3HxwYGOh8kABqqrShGB2VVq6cqgrpVGJBmw6UFUlFQu6+UdKYpENSxwKgeePj09tQVNpYjI93Zv7ZNh3Ll08lOFTBIDUaanaYmQ1IeszdN5rZDpIOlrQqcVgAWjA8PPO9oaHOntSzbTpGRkgoUA6UVHTeMyWNmdn1ksYV2lRckjgmAF0mZZsOoBZKKjrM3a+XtF/qOAB0r2ybjkoJCVUgKANKKgCgy6Ru0wHUYu48IqHsBgcHfWJiInUYANA1zGyduw+mjqPfUFIBAAAKQVIBAAAKQVIBAAAKQVIBAAAKQVIBACVCvx7oZiQVAFCQIhKCVvv1IAlBmZBUAEBBiujoq9V+PehcDGVCUgH0Ai5XS6Gojr6y/XosXVr/+3QuhjIhqQB6AZerpdFKQlBLq/16NJonOSc6xt15lfy1//77O9DQ2rXu8+e7j4yEv2vXpo6oL23taqh8v/K96uHZzHM20+x2kia8BMfvfnslD4BX4xdJBZo2MhJ265GR1JH0hVWrpp+Y165132UX9yVLpoZbPXlXT7MynVWr8sdvNmHot5yTpCLNi+oPoFfQF3bHVdc6rVkjuUvHHReGZ9PR1/DwzOqLoaHwfp5mOxcroloGaIQOxboAHYqhoeq+sKuH0TaVRb10acjlyrrIuyXOotChWBqUVAC9gL6wk+mGEoBsjrly5dTdIhRmoWiUVHQBSiqA8uqGEoDVq0NVTTausbGQc9aqVul2lFSkQVLRBUgqgHKi1qm8SCrSoPoDAGaJWidgOkoqugAlFQDQGkoq0qCkAgBaxBMqgXwkFQDQIp6KDuSbmzoAAOg22U68ynzXB9BplFQAwCx0w/MpgE4jqQCAWeCp6MBMJBUdZmZ7mtmYmd1gZuvN7KTUMQFoDU+oBPKRVHTeFkmnuPsiSQdKeq+ZLUocE4AWlO35FNyNgrIgqegwd7/X3X8Z/98k6UZJe6SNCkArWu1JtN24GwVlQVKRkJktlLSfpGtyPltiZhNmNjE5Odnp0IAkuOKenezdKMuX86hwpENSkYiZ7SjpQkknu/vD1Z+7+znuPujugwMDA50PEEiAK+7Z424UlAFJRQJmNk8hofiau387dTxAWXDFPXvcjYIyIKnoMDMzSV+RdKO7n5U6HqBsuOJuHXejoCxIKjrv5ZJOkHSQmV0bX4elDgooC664W5e9G6XS/iR7NwrtUtApPKa7w9z9Z5IsdRxAGWWvuIeGwosqkMayd51U2qWMjob3s8sUaDdKKgCURtme/9CNaJeClMzdU8eABgYHB31iYiJ1GAC6yPLloV3KyEhoZ9FvzGyduw+mjqPfUFIBAD2GdilIhaQCAHoId4IgJZIKAOghtEtBSrSp6AK0qQCA1tCmIg1KKgAAQCFIKgCgBXR6BtRGUgEALaDTM6A2nqgJAC3IPlxq6dJwyyYPlwICSioAoEV0egbkI6kAgBbxcCkgH0kFALSAh0sBtZFUAEALeLgUUBsPv+oCPPwKQCOrV4c7ULLtO8bGQrKT7Rq9X/DwqzQoqQCAHsCtrigDbikFgB7Ara4oA0oqAKDL1Hqq5/h4a7e68nRQFI2kAqjGkRYlV6uq46c/lT7/+em3utbbdKkyQdFIKoBqHGlRctmqjuXLw99ly0JS4T71+RveIB19dO1NN286VJlga5BUANU40qILVD/Vc8sW6aKLpIsvnsqJ3aXjjqu/6c726aAU6CEPSQWQh+cwo+Sqn+pZuZ10aEjad9+w6Z50kvSlL02Nn3fCn+3TQSnQQy5351Xy1/777+/osLVr3efPdx8ZCX/Xrk0dEfCEyuZZ2Syzw2vXuu+8s/uTnhT+Vt7L24zrTaeVOMq4m0ia8BIcv/vtRUkFekdR5bE8h7k5lH8nU+upnmvWhE314oulSy6RzKTDDw9tK/Jq8Lb26aAU6GGG1FkNr8YvSiqatLWXXRWrVuVf0q1aVUycvaKo5Y3CVG+6IyPukvt++00fr6jNmZIKXtWv5AH020vSVyXdL+k/m/0OSUULynyU60Us79KqrJoTTnA3c//0p6e/v7Wrquw5JUlFmhfVH513rqRDUgfRsyiP7SyWdylla/DOO0/61KekU0+VTjyxuJuZ6FgNeehQLAEzWyjpEnd/YTPj06FYCypHU55T3Bks71LK61zsxBOl888Pd3msXJkutk6hQ7E0KKkoKTNbYmYTZjYxOTmZOpzWpGrARwPLzkq4vGkjWt/w8MzeSi+9tPXbRoFWkVSUlLuf4+6D7j44MDCQOpzWpLqBnfLYzkq4vHlGQvPItdFJVH8k0BfVHxSLo836YRPLq8aodBw2PNy5aXQjqj/SoKQC7UEDPrRZP2xiRZTIVFeFSGG4lxMKpENS0WFm9g1JV0l6vpndZWbvTB1TW8z22b9Ak/phE6MbGnSbuakD6Dfu/pbUMbRdthK30hkBR0MUqJ82sWyJzMhI7/0+9BZKKlA8Gkym0Ue3RPTTJtYPJTLoHTTU7AJd2VCz7Hqx9Vr15Xv1MLoOq3T2aKiZBiUV6E+9eE8iFfA9p59KZNAbKKnoApRUtEmv3pO4fPlUBXw/PDoRyEFJRRqUVKB/9eI9iVTAA0iIpAL9q9dOwDw6EUBiJBXoT714AqYCHkBitKnoArSpaINevPsDwBNoU5EGSUUXIKkoGRISoPRIKtKg+gP9a7YPi+rF21EBoAAkFehfs00OeB4Eauijh5oCuUgq0L+2JjnoxdtRsdUoxEK/I6lAf5ttctBrt6OiEBRiod+RVKC/zSY56MXbUVEYCrHQz0gq0L9mmxzwPAjUQSEW+hm3lHYBbiltE24NRcHoVbQ8uKU0DZKKLkBSAXQH8tTyIKlIg6SiC5BUAEBrSCrSoE0FAAAoBEkFAAAoBEkFAAAoBEkFAAAoBEkFAAAoBEkFyoUemQCga5FUJGBmh5jZzWZ2q5mdljqeUqFHJgDoWiQVHWZmcyR9QdKhkhZJeouZLUobVYl0ukcmSkYAoDAkFZ13gKRb3f02d39U0hpJRyWOqTOaPYF3skcmSkYAoDAkFZ23h6QNmeG74nu9r9kTeCd7ZKKvagAoDElFSZnZEjObMLOJycnJ1OEUo5kTeIpuxbMlI4ceOjMeqkIAoCkkFZ13t6Q9M8ML4nvTuPs57j7o7oMDAwMdC67tGlVtpOhWvFIycsIJ0gUXSGedNfU+VSEA0LS5qQPoQ+OSnmtmz1JIJo6T9Na0IXVQddXG0ND0xCKvK8fqcYqOJ1ti8pKXSKeeKl17rfTtb4fSErqcBICmUFLRYe6+RdL7JF0m6UZJo+6+Pm1UHZKiaqOR6pKRD3xAOv546fzzpWOOkc48k0acANAkSioScPcfSPpB6jg6rl7VRqqGkdUlDmNj0qWXTpWkLFsWEomlS8MwjTgBoCZz99QxoIHBwUGfmJhIHUbvq64KqQwfemgouRgZCSUsqaxeHUpJqI4BGjKzde4+mDqOfkP1B1CRV5KybJl04YWdub21EZ6pAaDkKKnoApRUJFKr5CJlFUglBqpjgLooqUiDkgqglhS3tzbSyaeNAkCLKKnoApRU4AmUVABNoaQiDUoqgG5RxltyASCDpALoFmWsjgGADKo/ugDVHwDQGqo/0qCkAgAAFIKkAgAAFIKkAuhFq1fPbMBJN+4A2oykAuhFPH0TQAJ0KAb0osqdITzTAkAHUVIB9Cqevgmgw0gqgF41NhZKKMrQGRqAvkBSAfQinr4JIAGSCswOdxeUG0/fBJAASQVmh7sL8pUl2RoentmGYmgovA8AbUJSgdnJ3l2wfPlUUXu/NwYk2QLQx0gqMHvcXTBTkclWWUo9AKBJJBWYPe4uyFdUskWpB4AuQ1KB2eHugtqKSraoYgLQZUgqMDvcXZCv6GSLKiYAXcTcPXUMaGBwcNAnJiZSh4FmrF4dqieyJ/+xsZBszebOi0qSwqO2gZaY2Tp3H0wdR78hqegCJBV9KlvqMTQ0cxhATSQVaVD90UFm9iYzW29mj5sZGzvqo4oJQJehl9LO+k9Jx0j6UupA0AXyqkuGhiilAFBaJBUd5O43SpKZpQ4FAIDCUf0BAAAKQUlFwczscknPyPnoI+7+nRams0TSEknaa6+9CooOAID2IakomLu/tqDpnCPpHCnc/VHENAEAaCeqPwAAQCFIKjrIzN5gZndJepmk75vZZaljAgCgKFR/dJC7XyTpotRxAADQDpRUAACAQpBUAACAQpBUAACAQpBUAACAQpBUAACAQpBUoDutXh26As8aGwvv99I8AaCLkFSgOy1eLB177NRJfmwsDC9e3FvzBIAuwnMq0J2GhqTR0XBSX7pUOvvsMNzObsFTzBMAugglFeheQ0Ph5H7GGeFvvZN7UVUXrcwTAPoMSQW619hYKC0YGQl/q5OGrKKqLlqZJwD0GZIKlE8zpQqVpGB0VFq5cqpaotZJPlt1sXz51HdbKWlodZ4A0GdIKlA+zZQqjI9PTwoqScP4eO3pbm3VxWzmCQB9xNw9dQxoYHBw0CcmJpr/wurV4QScPWmOjYWT3/Bw8QG2QyWRKLJBZDumCaCUzGyduw+mjqPfUFLRi3rh1seiG0RSdQEAbUdS0YuKaD+QWtENIqm6AIC2o/qjC7Rc/VGxfHm40h8ZCVfn3SJbqjA0NHMYABqg+iMNSip6VTff+kipAgB0JUoqukDLJRVc6QPoc5RUpEFJRS/iSh8AkAAlFV1g1m0qAKBPUVKRBiUVAACgECQVQDOK6pAMAHoYSQXQjF54oBgAtNnc1AEAXSH7QDEe8w0AuSipAJpV9KPDAaDHkFR0kJl90sxuMrPrzewiM9s1dUxoQTc/UAwAOoCkorN+LOmF7v5iSbdIWpY4HjSLDskAoCGSig5y9x+5+5Y4eLWkBSnjQQt4oBgANERDzXTeIembqYNAk4aHZ743NES7CgDIIKkomJldLukZOR99xN2/E8f5iKQtkr5WZzpLJC2RpL322qsNkQIAUCySioK5+2vrfW5mb5d0hKTXeJ1npLv7OZLOkcJjuouMEQCAdqDvjw4ys0MknSXpVe4+2cL3JiXd0bbAZpov6YEOzq/sWB7TsTymY3lMV5blsbe7D6QOot+QVHSQmd0qaTtJD8a3rnb39yQMKZeZTdARzxSWx3Qsj+lYHtOxPPob1R8d5O77pI4BAIB24ZZSAABQCJIK5DkndQAlw/KYjuUxHctjOpZHH6NNBQAAKAQlFQAAoBAkFajLzE4xMzez+aljSYnO4MIt0WZ2s5ndamanpY4nJTPb08zGzOwGM1tvZieljqkMzGyOmf3KzC5JHQvSIKlATWa2p6S/lXRn6lhKoK87gzOzOZK+IOlQSYskvcXMFqWNKqktkk5x90WSDpT03j5fHhUnSboxdRBIh6QC9XxG0rCkvm94Q2dwOkDSre5+m7s/KmmNpKMSx5SMu9/r7r+M/29SOJHukTaqtMxsgaTDJX05dSxIh6QCuczsKEl3u/t1qWMpoXdIujR1EB22h6QNmeG71Ocn0QozWyhpP0nXpI0kuc8qXIQ8njoQpMPDr/pYvc7PJH1YoeqjbxTVGRz6h5ntKOlCSSe7+8Op40nFzI6QdL+7rzOzV6eOB+mQVPSxWp2fmdmLJD1L0nVmJoWi/l+a2QHufl8HQ+yoojqD61F3S9ozM7wgvte3zGyeQkLxNXf/dup4Enu5pNeb2WGStpe0s5ld4O7HJ44LHcZzKtCQmd0uadDdy9BJUBKz7QyuV5jZXIUGqq9RSCbGJb3V3dcnDSwRC9n2/5H0kLufnDqeMoklFae6+xGpY0Hn0aYCaM6/SNpJ0o/N7Foz+2LqgDopNlJ9n6TLFBoljvZrQhG9XNIJkg6K28O18Sod6GuUVAAAgEJQUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAAox66TCzPYys82xS2R0KTNb3+ln9ZuZm9k+bZjuCjO7oOBp7mBm3zOz35vZ/y142l80s5Eip5lCG9dnx7dNBO1ap2VjZuea2cdqfPZ2M/tZne9eYWbval90ufMs/XppmFSY2e1m9khMICqv3d39Tnff0d3/0olAG8S4rZl9K8bqrR6IOrVxWPA+M7vezP5oZvfFeR/X7nnX4u4vcPcrUs2/C7xR0tMl7ebubypywu7+Hnc/o5lx6x38WlW1T98Xp71jEdMuUivbZvxNdftu6SdmtjAeC0vfv5OZvdDMLjOzB8yMpzGWiJn9YzxGPGxmXzWz7Rp9p9mSiiNjAlF53bOVsdY1yx3hZ5KOl1TmDq8+L+lkSadI2k2h6+iPSjokZVCoa29Jt8THVPeSI919R0kvUei2e1nieEqh6JNwN5zUS+AxSaOS3pk6EEwxs9dJOk2hv5+9JT1b0ukNv+judV+Sbpf02pz3F0pySXPj8LMk/UTSJkmXS/qCpAviZ6+WdFet6UpaIelbki6Q9LCkdykkPKdJ+rWkBxU2uqc2Ee9dkl7daLzM+B+X9BdJf5K0WaGPh9Ml/XP8fJ6kP0j6ZBzeIY771Dj8eknrJW2UdIWkv64xn+fF+Qw2iOfvFfpW2CTpNknvznz2dkk/qxrfJe0T/z9M0g3xu3crdOojSfMlXRJjfEjSTyVtk7MeDpB0VRzv3rgstq2a13sk/Vcc5wuKj3rP+R1zFLpP/3WMZ52kPXNi3kXSeZImJd2hkGRVYltR2YbqbHNXxun/OMZb2ea2j9vTgzHWcUlPrxHrX8d1tzGuy9fH90+X9KjCQW+zpHfmfHeFwrb7zRjHLyXt22ja8bNzJX0su48oJJz3x+X/9/GzJTGGR2Mc34vvfyiu502SblboPbXlfVrSaknfzwxvJ+lTku6U9FtJX5S0Q+bzD8b47pH0juz6zJnXFZLOlPQLhX37O8rsx6qz/2jmMWI0biub4ncG42fnS3pc0iNx+Qy3uP5vj8vyekl/Vui9eXeFHkgnJf1G0vtbWOetTu8ASRNx+fxW0lmZzw6U9PP4G65T5tgWl9cZkv49xvEjSfPjZ3fG9bI5vl62Ffvo4ZJ+FePbIGlFZho1l7PC8eq2OO3fSHpbg+1yH0nexPb7uRjHwzHmV1atm9ztJH6+X1xfm+L6W6O4D+bM5+1x2f6LpN9LukmZfSwu/3c1eazaRdJXFPabuyV9TNKcrTx2tnW9SPq6pH/KDL9G0n0N10+rB6A6C+0qhQPRtpJeEX9oK0nFY5KOVkgmdpB0kqSrFbpY3k7SlyR9o4l4ZyQVkt4q6fo633li44jDB0n6j/j/f48r95rMZ9fF/5+nkHAcrJB8DEu6VZkTcWaa75F0exPxHy7pOZJM0qsk/VHSf8tsDPWSinsVdzBJT8l870yFE8O8+Hqlpvp9ya6H/RUOYnPj+r1R0slV87pE0q6S9lI4QB5S43d8UNJ/SHp+/C37KlQhVMd8nsKJZqc4z1sUT95qvKNepdBz6HaS/kZhJ6lsc++W9D1JT1LYSfeXtHNOnPPiOvuwwrZ7UJzO8/NiyPn+CoVt941xWqcq7Kjzmpj2uZqeVGyRtDJ+77C47p9SPW4cfr7CgWT3zLJ5TqPtK2edL4jr6XOZzz8j6buSnhrXy/cknRk/O0ThxPdCSU9WOPA0Siruzox/YWYd1d1/NPMY8ae4XOYobNNX1zpONbv+M9+9VqFr9x0UjkHrJC2P6+3ZCgfh1zVa57Oc3lWSToj/7yjpwPj/HgonhcPiNA6OwwOZZfvruBx3iMOfyNtXtnIffbWkF8UYXhzX/9H1lnNc1w9ralt/pqQXNNgum00qjlco6Z2rkITfJ2n7RttJXPZ3SPrHuN7eGNdjvaRiS2b8NyskF5ULyivUfFJxkcI57MmSnqaQZL+7xnxLsV4Uktg3Z4bnx/nvVnf9NHkA2qyQ7WyUdHH1QlM4wWyR9KTM9y5Qa0nFT6o+v1HTs8Jnxg2g5k4Sx2uppKJ644jDldKI3RRKSz4cp7ujwtXr5+N4Iwq9NVa+t43CAXTG/BWuwK/OiXVjnNfeNWK7WNJJmY28XlJxZ9yYdq4aZ6XCiXvGgV81ksb42cmSLqqa1ysyw6OSTqvx3ZslHVXjM1c4gMxRuPpelPns3ZKuaLSjZra5J2c+/3pmm3uHwhXeixus+1cqHJS2ybz3DcWsvzqGnO+v0PST2zaKyV0T0z5X05OKR7Lbt0KJxYHV48bhfeLnr1U8mbWwvd+usE9visvz/0naNX5mCif652TGf5mk38T/v6p44orDz1PjpCI7/qK4zueowf6jmceIy6um80it7bjZ9Z/57jsywy+VdGfVOMsk/e9G63yW0/uJwnFlftU4H5J0ftV7l0n6H5ll+9HMZ/8g6YfV+0qd391wH63x2WclfabeclY4eW2U9HfKlHI1WA9NJRU53/udYklRve1E4cLjHmVKV2Ps9ZKK6vF/oakE8Ao1kVQotMn6s6aX9r1F0liZ14tCwnpIZnhenP/Cet9rtk3F0e6+a3wdnfP57pIecvc/Zt7b0OS0a42/t6SLzGyjmW1USDL+orCC2srdH1EojnyVwoZ4pcIKenl878o46u4KmW/le48r/I49cib7oEJilJ3PAoXsbzuFg7nM7FAzu9rMHoq/+7A4TjP+Lo5/h5ldaWYvi+9/UuEK8EdmdpuZnZb3ZTN7npldUmmYI+mfcuadbbPyR4VEK8+eChtlPfMVNtQ7Mu/dofzlV213Sb9z9z9UfbfifIUD8Bozu8fMVpvZvBrT2RDXXasxVDyx7cbp3BWn2+q0H/TpbTdqLl93v1Uh6Vsh6X4zW2Nmu7cQ89HuvpNCMvNXmlrPAwpXN+sy+94P4/uq/Kaq39NI9fjz4vxa2X+kmdve9nXaLDS7/vNi3FvS7pXfH5fBhzX92FNrnc9meu9USM5uMrNxMzsi8703VX3vFZp+HGl2f8zTzD4qM3upmY2Z2aSZ/V6h1LWyveQu57hfvjmOe6+Zfd/M/qqF2OrFc6qZ3RjvyNqoULWQPU7V2k52l3S3xzNk1Gj7zRu/lf1MCutxnsJyqKzHLymUWOQpy3rZrFC6UVH5f1O9uIp6TsW9kp5qZk/KvLdn5v8/KByoJEkWbkMd0HReNbxB0qGZZGZXd9/e3e8uKOZ685ZC4nCQQh3ceBx+nUL950/iOPcobDCSwt0dCr87L8a1khaY2WCtIGLL2gsVqpGe7u67SvqBYsKhmcvxGdN+hPu4ux+lsLFerFCSIHff5O6nuPuzFeqwP2Bmr8kJ4WyFesPnuvvOCgc+yxmvGRsUqnHqeUCh9GnvzHt7aWr5Tfu9krK/915JTzGzJ1d9V5Lk7o+5++nuvkihCusISSfmxHCPpD3NLLsvZGNoxhPbepzOgjjdIqZdMWMbdfevu/srFJafS1rV8kTdr1QoBflUfOsBhRKTF2T2u108NOqUwnLP7tt7qbHq8R+L82ll/2n4U6YNNL/+876/QaFkJnvs2cndD8v7TVXrvOXpuft/uftbFPbbVZK+FbfrDQolFdnvPdndP9Hq8qihmX1UCiWA31Wo199FoSrVYuw1l7O7X+buByskQTdJ+rcm5lWXmb1SoZrsWIWqwV0VqiSaOU7dK2mPuJ1VNNp+88bPu1Gh3rFqg0JJxfzMetzZ3V9QY56Fk4C0AAAE6ElEQVRlWS/rFapeKvaV9Ft3f7BeUIUkFe5+h8KV/Yp4e+fLJB2ZGeUWhWzx8Hi18FGFq/N6vijp42a2tySZ2YCZHVVrZDPbzsy2j4Pbmtn2VRtDPb9VqOfMulJhJdzg7o8qFnUpHBwm4zijkg43s9fE33WKwsbz8+oZuPvNCtnpGjM72MLzD+YorPCKbRWWy6SkLWZ2qKS/zXx+naQXmNlL4m9dkfn925rZ28xsF3d/TKHe7PH42RFmtk9cHr9XKPHJXj1X7BS/tzlmr0trL7KGvizpDDN7rgUvNrPdsiN4uB15VGE97xTX9QcUqs6kUC/9NxaeibKLMncoZLa50+Nvf4Uy25yZDZnZi+IyfljhRJb3m69RuJoZNrN5Fm5HPlKhAVez9jezY+LV0MkK28DVBU27Yto2ambPN7ODYiL6J4VEIO/3NeOzkg42s33jVfe/SfqMmT0tzmsPCy3BpbC+3m5mi+JFxP9qYvrHZ8ZfKelbmXXf1P7ThOrl0+z6z/MLSZvM7EOV/dTCbY+LM+PUWuctT8/MjjezgbjsN8bvPK6wHxxpZq+L39nezF5tZgua+A2TcRrVx7WshvtotJNCSfSfzOwAhTZqirHnLmcze7qZHRWToz8rXPXmLv847+0Vjn+Kv7PW+WEnhWrPSUlzzWy5pl9N13NV/O774/54jMJFYj1Py4z/JoWG1z/IGa/esepehUa0nzaznc1sGzN7jpm9qsY8S7FeFNq7vTPuu7sqnLfPrTHuE4p8oubbFOpeH1Ro2fpNhaDl7r9XqO/7ssJVyB8Uigvr+ZxCFvYjM9uksMO+tM74NyscWPdQKPZ5RPEqKJ5s1zeY1xvN7Hdm9vn43s8V2lZUSiVuUDh4V4YricLxkv5Z4crrSIVb9R6tMZ/3KtxWepbCXRh3KbTefrNCnesmSe9XONj+TmEj+W5mfrcoHJQvV7gDo/rBLCdIut1C1cV7FNaJJD03fmezwo71r+4+lhPfqXGemxROLN+s8TuacVb8HT9S2Ki/orA8q/1Phe3htvh7vq5Qby93/3GM4XqFhm6XVH33rQrbxEMKJ7fzMp89Q6GF/sMKVWdXKhQJThPX1ZGSDlVYh/8q6UR3v6mF3/odhXX4O4V1cEy8Uihi2hVfkbTIQvHpxQrJ5yfidO9TOPgtk5ra3qeJSfJ5Cg0JpVCXf6ukq+O2dLlCozG5+6UKScjaOM7aJmZxvsLB6D6FVunvj9Nqdf+p50xJH43L51Q1uf7zxITnCIXbbX8TY/uyQjF7Re46n+X0DpG03sw2KxyLjnP3R9x9g6SjFEoMJxWuYD+oJo7bsSr645L+PS6TA3NGa3Yf/QdJK+NxeHn8TkWt5byNwgXCPQr756tU+yJlb4XjdWWbfUTheJ7nMoXquFsUqiL+pCar2uN2dYxCW4mHFNbftxt87RqF4+cDCsvzjXlX6k0cq05USJpuUNhmvqWq6vCMUqwXd/+hwp1hYwrt9e5QExcRlTsACmdm35R0k7s3cyUDdCUzW6HQcOr41LGUkZldodCA7cupYykK6xyorbCSCjNbHIt0tjGzQxQy7IuLmj4AACi3Ip/29gyFoqTdFIr1l7r7rwqcPgAAKLG2VX8AAID+QtfnAACgECQVAACgECQVAACgECQVAACgECQVAACgECQVAACgEP8fmw9apJVZZWUAAAAASUVORK5CYII=\n" 59 | }, 60 | "metadata": {} 61 | } 62 | ] 63 | }, 64 | { 65 | "metadata": { 66 | "trusted": true, 67 | "_uuid": "96404aa419f0afcb1b393dcd175c9ffabdc243c1" 68 | }, 69 | "cell_type": "markdown", 70 | "source": "We compute the matrix containing the inner products of the points of the dataset." 71 | }, 72 | { 73 | "metadata": { 74 | "trusted": true, 75 | "_uuid": "a5165bdaa07a0f94d387c2d3b6fb2f5754ca7510" 76 | }, 77 | "cell_type": "code", 78 | "source": "def inner_prods(X_train):\n return np.dot(X_train,X_train.T)", 79 | "execution_count": 3, 80 | "outputs": [] 81 | }, 82 | { 83 | "metadata": { 84 | "_uuid": "15af46d41d392d59fc5988e6213a6dc93034a8e7" 85 | }, 86 | "cell_type": "markdown", 87 | "source": "With this, we can write the cost function as:" 88 | }, 89 | { 90 | "metadata": { 91 | "trusted": true, 92 | "_uuid": "6a83a7499ebe162cc407b85b26493c0d1ef9abbd" 93 | }, 94 | "cell_type": "code", 95 | "source": "def cost(alpha): \n return -(np.sum(alpha) - (1/2)*np.dot(np.multiply(alpha,y_train),\n np.dot(inner_prods(X_train),np.multiply(alpha,y_train))))", 96 | "execution_count": 4, 97 | "outputs": [] 98 | }, 99 | { 100 | "metadata": { 101 | "_uuid": "f8c74947cc7f78563b2a6cfd2ac24b10cad9b3ac" 102 | }, 103 | "cell_type": "markdown", 104 | "source": "We define the constraint given by the KKT conditions. This is of the form $\\sum_{i=1}^m \\alpha_i y^i = 0$." 105 | }, 106 | { 107 | "metadata": { 108 | "trusted": true, 109 | "_uuid": "565e648ae4ec0623dfa1c79171af121bc753e0aa" 110 | }, 111 | "cell_type": "code", 112 | "source": "def cons1(alpha):\n return np.dot(alpha,y_train)\n\ncons = ({'type': 'eq', 'fun': cons1})", 113 | "execution_count": 5, 114 | "outputs": [] 115 | }, 116 | { 117 | "metadata": { 118 | "_uuid": "c329f3b3b2067cc037b09d059618db2a5831cbed" 119 | }, 120 | "cell_type": "markdown", 121 | "source": "We also set the conditions for the Lagrange multipliers: $\\alpha_i \\geq 0$:" 122 | }, 123 | { 124 | "metadata": { 125 | "trusted": true, 126 | "_uuid": "ec4c8ee6382e84eb3d572a1302eb925d04d23407" 127 | }, 128 | "cell_type": "code", 129 | "source": "bds = [(0,None) for _ in range(len(X_train))]", 130 | "execution_count": 6, 131 | "outputs": [] 132 | }, 133 | { 134 | "metadata": { 135 | "_uuid": "0fee47d9ba66f8bc42dec14848557508a9435ae5" 136 | }, 137 | "cell_type": "markdown", 138 | "source": "Now, we need an initial guess for the value of $\\alpha_i$ for the optimizer:" 139 | }, 140 | { 141 | "metadata": { 142 | "trusted": true, 143 | "_uuid": "23cb29a47e86344313477bf65fd09dcd746791a5" 144 | }, 145 | "cell_type": "code", 146 | "source": "alpha = np.random.random(len(X_train))", 147 | "execution_count": 7, 148 | "outputs": [] 149 | }, 150 | { 151 | "metadata": { 152 | "_uuid": "196bbeb5cd0b41db7f4f6059040056ae9a2ac6a4" 153 | }, 154 | "cell_type": "markdown", 155 | "source": "Finally, we optimize the cost function " 156 | }, 157 | { 158 | "metadata": { 159 | "trusted": true, 160 | "_uuid": "58420d8706d87d2695faac8d82f59da1f6352099" 161 | }, 162 | "cell_type": "code", 163 | "source": "res = minimize(cost, alpha , bounds=bds, constraints=cons)", 164 | "execution_count": 8, 165 | "outputs": [] 166 | }, 167 | { 168 | "metadata": { 169 | "_uuid": "692ac353c202e4de221de9388e6e9261a492ba48" 170 | }, 171 | "cell_type": "markdown", 172 | "source": "We recover the values of $\\alpha_i$ that optimize the Lagrangian:" 173 | }, 174 | { 175 | "metadata": { 176 | "trusted": true, 177 | "_uuid": "5c6343f0598677e5eaccaa893d203cb309117a9a" 178 | }, 179 | "cell_type": "code", 180 | "source": "alpha = res.x", 181 | "execution_count": 9, 182 | "outputs": [] 183 | }, 184 | { 185 | "metadata": { 186 | "_uuid": "25f31cfc13796b8a5006daa8531c1d93868a79e7" 187 | }, 188 | "cell_type": "markdown", 189 | "source": "We will set to zero the values $\\alpha_i$ which are smaller than $10^{-7}$:" 190 | }, 191 | { 192 | "metadata": { 193 | "trusted": true, 194 | "scrolled": true, 195 | "_uuid": "5001faa00b97641f2bd3c05792d5bd06cac7a4b1" 196 | }, 197 | "cell_type": "code", 198 | "source": "alpha = alpha*(1*(alpha > 10**(-7)))", 199 | "execution_count": 10, 200 | "outputs": [] 201 | }, 202 | { 203 | "metadata": { 204 | "_uuid": "073f23d46de099555abd7e1d6a2a652dcfe25e2f" 205 | }, 206 | "cell_type": "markdown", 207 | "source": "We can now construct the parameters $w,b$:" 208 | }, 209 | { 210 | "metadata": { 211 | "trusted": true, 212 | "_uuid": "ed71dd719b6a20e6eb2cae497d2e2d5e17cb8e81" 213 | }, 214 | "cell_type": "code", 215 | "source": "w = 0\nfor i in range(len(X_train)):\n w += y_train[i]*alpha[i]*X_train[i,:]\n\nb = (-1/2)* np.dot(w,X_train[((alpha > 0) & (y_train == 1))][0] + \n X_train[((alpha > 0) & (y_train == -1))][0] )", 216 | "execution_count": 11, 217 | "outputs": [] 218 | }, 219 | { 220 | "metadata": { 221 | "_uuid": "0d1d4676674ebce1c8abd1da0ebc30b5e9dc7f34" 222 | }, 223 | "cell_type": "markdown", 224 | "source": "With this, we can build the prediction function:" 225 | }, 226 | { 227 | "metadata": { 228 | "trusted": true, 229 | "_uuid": "1440c5e3027132f81c60c51c40a92db7a6a5da22" 230 | }, 231 | "cell_type": "code", 232 | "source": "def predict(x,w,b):\n return np.sign(np.dot(x,w)+b)", 233 | "execution_count": 12, 234 | "outputs": [] 235 | }, 236 | { 237 | "metadata": { 238 | "_uuid": "4642ebb91cbb509cc2077fe26ad2c86968e4ef5c" 239 | }, 240 | "cell_type": "markdown", 241 | "source": "Finally, we can plot the decision boundary given by the SVM as well as the regions predicted for each class. We can see how the separating hyperplane maximizes the margin to both classes." 242 | }, 243 | { 244 | "metadata": { 245 | "trusted": true, 246 | "_uuid": "98a44a029aeef73ae3b5434bcde4338df1f66e23" 247 | }, 248 | "cell_type": "code", 249 | "source": "plt.plot(x0[:,0],x0[:,1],'x',color='red')\nplt.plot(x1[:,0],x1[:,1],'x',color='blue')\nplt.figtext(0.5, 0.01, 'Figure 2: SVM decision boundary', \n wrap=True, horizontalalignment='center', fontsize=12)\n\nx = np.linspace(-5, 5, 200)\ny = np.linspace(-5, 5, 200)\nX, Y = np.meshgrid(x, y)\nz = np.zeros(X.shape)\nZ = np.array(z)\nfor i in range(Z.shape[0]):\n for j in range(Z.shape[1]):\n Z[i,j] = predict(np.array([x[j],y[i]]).reshape(1,2),w,b)\nplt.contourf(X, Y, Z, alpha=.5, cmap='jet_r')\nC = plt.contour(X, Y, Z, colors='black',zorder=4)\nplt.axis('equal')\nplt.show()", 250 | "execution_count": 13, 251 | "outputs": [ 252 | { 253 | "output_type": "display_data", 254 | "data": { 255 | "text/plain": "
", 256 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEPCAYAAABWc+9sAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xt8FOW9P/DPN5ALBELAoBBAs4paEaNoVDhgFOVgrVeUVmnVQ61arG3t+dlabc/poa/Tyzmtp1Zfaj3US221CMZCwXoB2wiC4DEiBRHQSLgGSrjmBpvb9/fHTJbNspvdTWb3mZn9vF+vvJzd2cx8dw2fefaZZ54RVQUREflHlukCiIjIWQx2IiKfYbATEfkMg52IyGcY7EREPsNgJyLyGQY7EZHPMNiJiHyGwU5E5DN9Tey0qKhIS0pKTOw6rVpqa02XkLHW794NAMjPH4qCgmGGq6FUKy7OMV1CWnzwwQf7VHVovNcZCfaSkhJUVVWZ2HVa7Zw923QJGWvUj38MAGhqqsPdd7/KcPe52bNPNl1CWojItkRex64Y8qXf3XJLaHnRou8ZrIQo/Rjs5EtXnHlmaDkYbDJYCVH6MdiJiHyGwU6+dWZhHgBg796NUO0wXA1R+jDYybd+P3MWAKClpRlz595puBqi9GGwk28VDxqEwrzOVvtmw9UQpQ+DnXytqaUFAHD06CHDlRClD4OdfO284mIAQDDYiIULv2u4GqL0YLCTr71yxx2hP/KNG183WgtRujDYyddEBAV2PztHxlCmYLCT751WVAQAaG1txjvvPGG4GqLUY7CT7y382tdw8xknAgDefvtRw9UQpR6DnTLCAR0EAFBtN1wJUeox2CkjbNm/33QJRGnDYKeMkNPXmqFatY0XKxm2cuVA1NTkdnmupiYXK1cONFSR/zDYKSM8csMNoeWnnrraYCVUXNyCioqiULjX1OSioqIIxcUthivzD8eCXUT6iMiHIvKqU9skcsrZw4djgn3XLtVWs8VkuEAgiOnT96GiogiVlYNQUVGE6dP3IRAImi7NN5xssd8HYKOD2yNy1A1jx4aWW1qaDVZCgUAQZWWNWL58EMrKGhnqDnMk2EVkJICrATztxPaIUmHG+ecju7/V1/6LX15kuJrMVlOTi6qqASgvP4yqqgHH9blT7zjVYv81gAcA8NI+ci0RwY/KpwAA2tsaDFeTuTr71KdP34fJkw+HumUY7s7pdbCLyDUA9qrqB3Fed7eIVIlIVV1dXW93S9QjNQcPhpZV1WAlmau2NqdLn3pnn3ttbY7hyvzDiRb7RADXichWAC8BuFxEXoh8karOUdUyVS0bOnSoA7slSt7FJx+7m/0jj4w3WEnP+GGo4MSJDcf1qQcCQUycyG9RTul1sKvqQ6o6UlVLANwC4G+qemuvKyNKgS+MGYNx9lS+DQ170Np6xHBFycmEoYJ+OHiZxnHslHFuvfDC0LLXZnzMhKGCmXDwSjVHg11V31bVa5zcJpHTivLzQ8svv3yvwUp6xu9DBTPh4JVqbLFTxrn89NPx8HXXAQCqqyvR0LDXcEXJyYShgn4/eKUag50y0vhTTgktt7Q0GawkOZkyVDATDl6pxGCnjLS/2ZtXnmbCUMFMOXilEoOdMpKELVdXv22qjKRlwlDBTDh4pRqDnTLSOcXFoXB/443ZqK/fY7QeOiYTDl6pxmCnjNQ3Kwt/veee0OPdu9cbrIbIWQx2yljFhYWhVrvXRsYQdYfBThkrPycHpedaY9r/8pcforn5gOGKiJzBYKeMtuj6++0lxZo1LxmthcgpDHbKaFlybHxMezvvrET+wGAnsr297FHPTQpGFA2DnTLe9ydPtha0HW+99XOzxRA5gMFOGe+b5eWh5ebmQwYrIXIGg50IQD8PXdTI+copHgY7EYDzT7GSffv29w1XEh/nK6d4GOxEAB4ovw0AUF9fiyVLfmq4mu5xvnKKh8FOBOD8kSNDy9u2rTZYSWKcnK+cXTv+w2AnsnWOaG9q2m+0jkQ4OV85u3b8h8FOZBuRb/WzHz68C+++O8dwNbE5PV85u3b8h8FOZFv+rw+EllevftpgJd1LxXzlvBWdvzDYiWzZffrglMGDAQCqaria2FIxXzlvRecvDHaiMLOvvBIA0Ni4Fxs3vmG4mvTgrej8h8FOFGbKmWfipALrNOqiRd8zXE168FZ0/sNgJ4rQ1tYPANDe3ma4EudFG9pYXNxyXIjzVnTexmAnipDdpw8AoLW1Gfv3bzVbTAKSGYceb2gjx7T7A4OdfGHAypXIranp8lxuTQ0GrFyZ9LaeuPHG0PKTv7mq17WlWjLj0OMNbeSYdn9gsJMvtBYXY0hFRSjcc2tqMKSiAq3FxUlv66KSEpxeVAQA6Ohw/7C/ZMehdze0Mdq2Pve55uO2wVa8uzHYyReCgQAOTJ+OIRUVKKisxJCKChyYPh3BQKBH2zvrpJMcrtBZkV0mgUAQo0cfSWgceryhjZHBP3ZsM1vxHsNgJ98IBgJoKitDwfLlaCor63GoA8BDU6ZYC9qBjo52hyp0TmSXyapVA7BuXT5KSxu7HYeeyNDGyOAHwCtTPYbBTr6RW1OD/Koq1JeXI7+q6rg+92QMKygAsgSA4tFHJzpXpEPCu0wWLBiCJUsGY+rUg5g27UC349DjDW2MFfwAeGWqhzDYyRc6+9QPTJ+O+smTQ90yPQ33vllZKBtu9c/X1+92slTHdHaZrFs3AKWlTZgwoTH0fKxx6PGuWo0V/B991J9XpnoIg518Ibu2tkufemefe3ZtbY+3OSQ/36nyUiK8y6S6ut9xfe49GYceLfgBYNOm/rwy1UMY7OQLjRMnHtenHgwE0Dix590opSNGhJaffnpaj7fjtJUrB2LVqgFdukwmTTqMuXOHpiRseWWq9zDYiWK4r7wcJ/bvDwDYtWuN4WqOKS5uQWVlISZNOoxAIIiamlysWDEIkycfSknYpmLSMUqtvqYLIHKz+y+/HN9/9VXTZXQRCAQxY0YdKiqKcPRoH1RVDeAoFeqi1y12ERklIpUi8rGIbBCR+5wojMgNvnzBBaHlpUt/ZrCSrjh/OnXHia6YNgD3q+oYAOMB3CsiYxzYLpEr3H6WdbHSu+/+L9rajhquxpKq+dM5V4w/9DrYVXW3qq6xlxsAbAQwovvfIvKOi886dgK2tTV6yzidgZjK+dM5V4w/OHryVERKAIwD8F6UdXeLSJWIVNXV1Tm5WyLj0hmIqRylwvuf+oNjJ09FZACAVwB8R1XrI9er6hwAcwCgrKzMvfcdI+rGP/7xMUpKJhz3fHgglpU1pvSEZrTRKIFA0LF9hfffl5cfZqh7kCMtdhHJhhXqL6rqn5zYJpFbXHr66aHl55+/BS0tR6K+zi8nNGtqcrFq1cDj5p1hX7t3ODEqRgA8A2Cjqv6q9yURucugvDy88JWvhB4fPBh9mgI/3BC6swtp8uRDqK7uh0mTDqOioih0QRT72r3Bia6YiQBuA7BeRNbaz/1AVV9zYNtErnDp6NHIyRa0tCqCwabj1oef0AwEgigpOerJ/unw/vthw1pRUVGE0aOPoLKyEDNm1HnqvWQyJ0bFrFBVUdVSVT3P/mGok++cUGS1g5577ma0tXUNOL9cdh9+lWn4JGMTJkSfQ4bciVMKECXohRvutJfa8emnlV3W+fGyez90LWUqBjtRgoYOPHbisL291WAlqZfKsfKUegx2ogTl9T12Smrx4geh2pH2GtJ1IZRfupYyFYOdKEH9srNx/RhrtoyWlka8//4f0l5Dui6E8mPXUiZhsBMl4fEvfjG03NCwN+3755WhlAgGO1GSzhhmdcls2vSmkf375UIoSh0GO1GS/u3yLwEA9u37FKtXP5P2/ad6tApnePQ+BjtRkiaHTTGwdetx892lVDpGq3CGR+9jsBN5SCpGq0S20AOBYOgeqtH68dmidz8GOzluwMqVyK3pOp9Kbk0NBqxcaaii1Glu3p/W/aVitEpkC33x4sFYtqwQZ53VHOrHBxAKbrbo3Y/BTo5rLS7GkIqKULjn1tRgSEUFWouLDVfmnH59rP/u2FGFTz75q9lieilypM2GDfno6FBs3twf5eWH8d57AzFv3rHgTnRkDlv25jDYyXHBQAAHpk/HkIoKFFRWYkhFBQ5Mn45gIGC6NMe8f/8DoeXKyv8xWIkzwkfanHlmM1QFHR2dt01QAII9e7JDoZzIyBy27M1hsFNCku1eCQYCaCorQ8Hy5WgqK/NUqCfyXgf164evjR0OAOjoaEtrfakQPtJm48b+OO+8RgCC5csH4eKLG3HppYdQWVkYCuVERuZwzL05DHZKSLLdK7k1NcivqkJ9eTnyq6qOC0o3S/S9BgblAQD27t2MPXs+TnudTokcaTNjRh3Wrx8AQJGV1YEVKwZi2bJj0/auWjUAc+cOTWhkDsfcm8Fgp4Qk073SGYQHpk9H/eTJod/zSrgn+l5vv+K20PJLL92V7jIdEznSxqI455xmjB3bjI6OLLTac57V1OSisrIQkycfSmhkDmeINIPB7lOpGJmSaPdKdm1tlyDsDMrs2toe7zvdEnmvIhKaGKyt7WjMbbn9JGLkSJva2hzcfPM+jB3bjOrqfigtbURHh2DJkkJUVBRhxow6TJjQ2GUb0UbmcIZIcxjsPpWKkSmJdq80Tpx4XBAGAwE0TpzY432nW6LvNUsEANDUtB9Hj0Yfcui2k4jxDjSdAd0ZytOmHUBpaRP27MnF6NFHEu5O4QyR5jDYfcrpkSle715JRjLvdfZVV9lLiiefnBJ1e247iZjIgSY8lGtqckMt940b+yfc4uYMkeYw2H3MyZEpfuheSVQy73XGuHEYUVAAADhy5FDMbbrpJGIiB5rOUA7vTpk27QBmzKhjd4oHMNh9zMmRKd11r7jlSlOn6ki2K+nXN94IwLqrUqybb7jtJGKiBxp2p3gTg92n0tl14pYrTU3VUTp8OACBajuefPKfj1vvxpOIiR5o2J3iTQx2n0pn14lbrjQ1VUf/nByUjbAOHvv2bTluvdtavW480JCz+sZ/CXlRtG6DYCCQspAL78+vLy83dqWpqTqCHbHvfxqtdRsIBI31s3d3oOEFRP7AFjs5wi1XmpqvQ+O/xDB2r/gfg516rXDxYgyZN69rf/68eShcvDj0mnScYDU5JPPUIUPsJcWf//xAt68lSjUGOzlC4jxOx4lNk0MyH7vpptBVqGvXzkv5/sK5/cpWSj8GO/XaoWuvxf6bb+5y0nL/zTfj0LXXhl6TjhObJq94zRLBpoceSvl+onHbla1kHoOdHJHIxVBenso3ESLHvqd8+OHLaduv265sJfMY7OSIRE5amj+xmVpZIjixwAr3RYu+m9Z9u+nKVjKPwU69lshJy0yZa2bGeZcY2a/brmwlsxjs1GuJnLTMlLlm+mYd+ye1d+/mtOyTFxxRJAY79VoiJy39MJVvIm4ZNy60/JvfXJmW2+a57cpWMo/BTuSgYQUF+PGVV9qPtNsZH53CC44oEoOdyGF3jB8PyYocyU+UPo4Eu4h8XkQ2i0i1iDzoxDYpPdwy5a5ftbenviuGKFKvg11E+gB4AsBVAMYAmCEiY3q7XUoPp64I5QGiK1VrzphHH5scWiZKFyda7BcBqFbVLaraAuAlANc7sF1yUKzg7Ryt0tsrQt0yJ7tbfKn0XABAR3sz9uz52HA1lGmcCPYRAHaEPd5pP0cu0l3wOnFFaOfwxRPmzsXgBQu6HCAyseV+7yXHxrO3t7carIQyUdpOnorI3SJSJSJVdXV16dot2bqbq8WpK0KDgQCOnHUW8tetw9HRo5FdW4sBq1Z1ablnSsj3z84OLS9Y8B2DlVAmciLYdwEYFfZ4pP1cF6o6R1XLVLVs6NChDuyWkhWtZe7kFaG5NTXIq65GU2kp+q9bh7wtWzBoyRI0TJrUZV+Z0D0zrKAAZ514IgDgwIEa1NT4/2BG7uFEsL8P4HQRCYhIDoBbACxyYLvksGgtc6euCA0/QBycNg3NpaXI27IFwVNPxcAVKxyf0dELJ2vfnDULhYP6AADq6/cYroYySa+DXVXbAHwTwJsANgKYr6obertdclaslnlnH3u4nlwRGn6ACG+55+zYgaOjRzs+o6MXTtaKCDpnGNiwYXH3LyZykCN97Kr6mqqeoaqnqepPndgmOSvVc7V0ThkQ2XKvnzwZ/detQ1NpqaMzOrrlBtrx3Hj6BQCATz+txKefVhquhjIFrzzNEOmaqyWy5T5wxQocnjoVrSee6PiMjl6Y3/2HU6eGlj/77B2DlVAm6Wu6APKX8ANF5LcEAKFvCU6EcOQ5g2BJievCvW+fPqHlhobdBiuhTMJgp5SJ9m0gGAg4Furh3S/BkhLXdsdkZwGtHcDHH7+G7dvfx8knX2i6JPI5dsWQJ3V3zsBtI2be/3/H7qb0zjtPGKmBMguDnTypu3MGbhsxc0J+PiYNLwAAqHYYqYEyC7tiyHfCR8w0lZUhv6rKeBdNXnZnXzsnBKPUY4udfMltI2aaW9sBWCNjGhv3G62F/I/BTr7k1Pw3TvnKhCn2kmLevLuM1kL+x2D3CbedMExEqmp2cv4bp1x3zjmhf2xHj9Ybq4MyA4PdJ9x2wjARydac6IEg1VfZ9lTnP7YDB2p4ZyVKKQa7T3jlEvtwydac6IEgXVfZJuur48cDADo62vDcc9ON1kL+xlExPhJ+wrC+vNzVod4pmZrdONolGT+68kqoKp5+7z0cPLjNdDnkY2yx+4jbThgmItma3TbaJVlnDxsGAGhpaTZcCfkZg90n3HjCMJ6e1OzFg1e48SUlAIC2tqP44x+/arYY8i0Gu0+49YRhd5Kt2YsHr0gjCwuRa08MtmXLCsPVkF+xj90nUjnhVqokW3N3BwI3v89IJ+Tno7aeQx4pdRjs5BlePHh1j9MLUGqwK4YozYbm5wMA2ttbsXr1M4arIT9isBOl2byZM9HPnhNsyZKfmS2GfInBTpRm+Tk5WPWv1hztnMaXUoHBTmTAax9/bC91YMeONUZrIf9hsBMZcMlpp4WWn332JoOVkB8x2CkqJ2de9OLMk6lWMmQIrjv7bPsRu2PIWQx2isrJ2SK9OPNkOuT1PTbauLn5oMFKyG8Y7BRVvJkXk2mFe3HmyXS49cILQ8u/fPhCqHJcOzmDwU4xQzq7tjbmhFvJtsK9PnlXKowbMQJPTben79VWjpAhxzDYKWZIIysr5oRbybbCvT55V6pcHepnB1TbDVZCfsJgdwmTJxijhXTDpEkYuGJFtxNuJdoK98PkXakkWQIAePjhC+O8kigxDHaXMH2CMTKk0dERd+bFRFvhXpx5Mp2uONUa+nj06CE0NtYZrob8gJOAuYTpuwNFhnS0fYdPuBXeCg8GAgiWlMTsjvHf5F3OmnnxxXiruhoA0NHB7hjqPbbYXcTUCcaedJWwFU7kXgx2FzF1grEnIe3WG0Z7UX5ubmj59df/w2Al5BcMdpcweYIx1SHNK0+7d8HIkTihf38AwKZNb2D/fp5Upt5hsLuEn7s2TJ8YdjsRwdrvfQ/98qzRMQ0N/zBcEXkdT566hJ9PMJo+MewFzS0tCLZYV55WV7+NkpLxhisiL+tVi11Efikim0RknYgsEJFCpwojf+GVp93Ly85Gln3h6cqVv0FdXbXZgsjTetsVsxTAWFUtBfAJgId6XxL5Ea887V6WCN7/7ndDj7duXWWwGvK6XgW7qi5R1Tb74WoAI3tfEvkNrzxNzICw0TFNTfsNVkJe5+TJ0zsAvB5rpYjcLSJVIlJVV8er6zKJn08MOymnT5/Q8rJlj+DQoZ0GqyEvixvsIvKWiHwU5ef6sNf8EEAbgBdjbUdV56hqmaqWDR061JnqyRM45j0xWSJ48+67kWX/q3z33TlmCyLPijsqRlWndLdeRGYCuAbAFcoJpYl6Zczw4Th7RF+s39HG2R6px3o13FFEPg/gAQCXqmqzMyURZbadB6xAZzuJeqq3feyPAxgIYKmIrBWRpxyoiSijHWyyAn3Nmvloawsaroa8qLejYkar6ihVPc/+meVUYUSZ6l773INqKxYtesBwNeRFnFKAyGUenHLstFZ9PacXoOQx2IlcbM+eDexrp6Qx2Ilc6NLTrLsqBYP1WLjwfsPVkNcw2Ilc6IVbb8XtF1wAANi58wPD1ZDXMNiJXOqz/da0AkeP1huuhLyGwU7kUnUNDQCA5uYDWLr054arIS9hsBO51NJ774XYy2vWzDVaC3kLg53IpbJEMNCe8ZHTC1AyGOxELlZo3ws1GGzEhx/ON1wNeQWDncjF/nLXXehj98e8/vqPzBZDnsFgJ3Kxwn798MbXrZk62to7DFdDXsFgJ3K597dvt5d4BSolhsFO5HLPrF4NANCOFtTX7zZcDXkBg53I5X529dWh5UcfvcxcIeQZDHYil/unU0/FucOHAwA6Ojg/O8XHYCfygPycHHtJ0dbWYrQWcj8GO5EHzBg3LrT8i4f/yWAl5AUMdiIPuOHcc/H2N74BAGgN1hmuhtyOwU7kEX2y+M+VEsO/FCKPWLVtW2j5ySf/2WAl5HYMdiKPmHH++QgMHgwAqKv7BO3tPIlK0THYiTzkzgkTQsuqnGKAomOwExH5DIOdyEOOjWcHVqx4ymAl5GYMdiIPuW7sWFwxqhAAsGzZIwgGGwxXRG7EYCfykOw+ffDErfdA7H+5R44cMlsQuRKDnchj1u/ejc7zprt2rTNbDLkSg53IY84tLg4tV1R8A83NBw1WQ27EYCfymH7Z2Vh4xx2hx7W1bLVTVwx2Ig8aUVgYWuYJVIrEYCfyoMJ+/ULLFa98G8Fgo8FqyG0Y7EQelNe3L/5y553IyhZA27F27XzTJZGLMNiJPOqc4mKckGstt7W1mi2GXIXBTuRRCqCuUQEAf/3b/6C9neFOFkeCXUTuFxEVkSIntkdE8WWJ4F/KygAA2hFEZeWvDFdEbtHrYBeRUQCmAtje+3KIKBk/ufrq0HJjI++sRBYnWuyPAHgA1jdDIiIyrFfBLiLXA9ilqn93qB4i6qFduz40XQK5RNxgF5G3ROSjKD/XA/gBgB8lsiMRuVtEqkSkqq6OXxmJnHJTaSkAYN++aqxY8aThasgN4ga7qk5R1bGRPwC2AAgA+LuIbAUwEsAaERkWYztzVLVMVcuGDh3q5Hsgymi/njYNl4woAABs3rzUcDXkBn17+ouquh7AiZ2P7XAvU9V9DtRFREl4Z1c9AODo0cOGKyE34Dh2Ih8YnGP9U9637zOsXVthuBoyzbFgV9USttaJzFj93QdDy8uW/dpgJeQGbLET+UD/7GxkZ1n/nDs62gxXQ6Yx2Il8Iq+vdcqsvn43tm9/33A1ZBKDncgnFt91F4YWWeE+f/4sw9WQSQx2Ip84ragIP5k8DQDQ1hY0XA2ZxGAn8pEHFy8GAKhyho9MxmAn8pEsEQBAS0sjWluPGK6GTGGwE/nI/ZdfHlp+7LFyg5WQSQx2Ih+5rawMxQMGAAAam/YbroZMYbAT+Ux+rn2/PO1gX3uGYrAT+cyMCy6wlxRz5lzd7WvJnxjsRD5z14QJeGXmTADAnj0fmy2GjGCwE/nQk+++ay+xKyYTMdiJfOjUIUNCyy+9dJfBSsgEBjuRD/371KkYkJMDANi8eYnhaijdGOxEPiQiuGXcONNlkCEMdiKfKsjLCy2vXv2swUoo3RjsRD41a+JEjB5khXvl6r8YrobSSUxcwCAidQC2pX3H3SsC4JU7QHmpVsBb9XqpVsBb9XqpVsCd9Z6iqkPjvchIsLuRiFSpapnpOhLhpVoBb9XrpVoBb9XrpVoB79Ubjl0xREQ+w2AnIvIZBvsxc0wXkAQv1Qp4q14v1Qp4q14v1Qp4r94Q9rETEfkMW+xERD7DYI8gIt8SkU0iskFEfmG6nnhE5H4RUREpMl1Ld0Tkl/bnuk5EFohIoemaIonI50Vks4hUi8iDpuuJRURGiUiliHxs/53eZ7qmRIhIHxH5UEReNV1Ld0SkUEQq7L/XjSIywXRNyWKwhxGRyQCuB3Cuqp4N4GHDJXVLREYBmApgu+laErAUwFhVLQXwCYCHDNfThYj0AfAEgKsAjAEwQ0TGmK0qpjYA96vqGADjAdzr4lrD3Qdgo+kiEvAogDdU9XMAzoU3au6Cwd7VPQD+S1WDAKCqew3XE88jAB6AB+ZmVdUlqtpmP1wNYKTJeqK4CEC1qm5R1RYAL8E6yLuOqu5W1TX2cgOs4BlhtqruichIAFcDeNp0Ld0RkUEAygE8AwCq2qKqh8xWlTwGe1dnALhERN4TkWUicqHpgmIRkesB7FLVv5uupQfuAPC66SIijACwI+zxTrg8LAFAREoAjAPwntlK4vo1rEZIh+lC4ggAqAPwnN1t9LSI5JsuKll9TReQbiLyFoBhUVb9ENbnMQTW19sLAcwXkVPV0NChOLX+AFY3jGt0V6+q/tl+zQ9hdSW8mM7a/EhEBgB4BcB3VLXedD2xiMg1APaq6gcicpnpeuLoC+B8AN9S1fdE5FEADwL4d7NlJSfjgl1Vp8RaJyL3APiTHeT/JyIdsOaLqEtXfeFi1Soi58BqWfxdRACrW2ONiFykqnvSWGIX3X22ACAiMwFcA+AKUwfLbuwCMCrs8Uj7OVcSkWxYof6iqv7JdD1xTARwnYh8AUAegAIReUFVbzVcVzQ7AexU1c5vQBWwgt1T2BXT1UIAkwFARM4AkAP3TQIEVV2vqieqaomqlsD6YzzfZKjHIyKfh/VV/DpVbTZdTxTvAzhdRAIikgPgFgCLDNcUlVhH82cAbFTVX5muJx5VfUhVR9p/q7cA+JtLQx32v6EdInKm/dQVADx349iMa7HH8SyAZ0XkIwAtAP7FhS1Lr3ocQC6Apfa3jNWqOstsSceoapuIfBPAmwD6AHhWVTcYLiuWiQBuA7BeRNbaz/1AVV8zWJOffAvAi/YBfguArxquJ2m88pSIyGfYFUNE5DMMdiIin2GwExH5DIOdiMhnGOxERD7DYCci8hkGOxGRzzDYXUREThaRRnsKWXKAiPxORH7Sy218RUSWJPC6p0TE8TlFRGS2iLzg9HZ7UEeJPfc/L2x0OQZvM/OWAAAIMUlEQVS7ASKyVUSO2CHe+VOsqttVdYCqtrugxvEislREDohInYi8LCLDk/j9s0Vkif37h0TkAxH5goiMEJE2ETktyu8sEJGH7WUVkb3hISIi2fZzab2qTlVfVNW4E66p6ixV/c901ETUHQa7OdfaId75U5vKnfWglTUY1s18SwCcAqABwHNJ/P5iWDfXGAbgRADfBlCvqrsA/BXWJfHh9Q0B8AUAz4c9fRDWjS86XWU/Rx7DVn56MdhdJPKrrj0h1XIRaRCRt0Tkic6v5CJymYjsjPj9rSIyxV6ebd/e6wURqQcwU0SyRORBEflMRPaLyHw7UI+jqq+r6suqWm9P2vU4rDlKEnkfRbBmn/ytfaOCFlVdqaor7Jc8j4hghzU51Mequj7suT8AuD3s8e0Afh9n3+NEZI39mc2DNZtg+PprRGSt/S3iXREpDVs3SkT+ZH9D2S8ij9vPzxSRFfayiMgj9jeHehFZLyJj7XVdun1E5C6xbrN3QEQWiUhx2DoVkVki8qldyxP25F6x5InIPPt9rRGRc8O2dZaIvG1vZ4OIXBe27m0RuTPscei9xKtDrFvZPSwi+0RkC6wbZYR/ll8V69ZxDSKyRUS+HrbuMhHZKSLfF5E9sOY3/0hErg17Tba97XHdvG/qAQa7u/0RwP8BOAHAbBwfhvFcD2va0UJY859/C8ANAC4FUAyr9ftEgtsqBxCaFEtEviwi62K8dj+AagAviMgNInJSxPoFAIpEZFLYc7eha2sdsGbbLBfrHpSDAVwC4M+xChRr0qaFsA4IQwC8DOCmsPXjYE309nVYn+n/AlgkIrlindd4FcA2WN9SRsC6i1KkqbA+izMADALwJfv9RtZyOYCf2+uH29uN3N41sOb9L7Vfd2Ws9wbr/+XL9vv6I4CFdjBmw/p2tATWN6POCazOjLml48Wq4y573TgAZQCmR/zeXnt9AayJsh4RkfPD1g+z6z0FwN2wDsrhszp+AcBuVf0wiVopEarKnzT/ANgKoBHAIftnof18Cazb3PUFcDKsG1L0D/u9FwC8YC9fBmve6MjtTrGXZwNYHrF+I6y50DsfDwfQCqBvnHpLARwAcEkS73EkrFb+Z7DumrMcwOlh658GMMdePh3WbJonhq1XAKPt130dwCwAv7Wf0xj7LAdQC3tyO/u5dwH8xF7+DYD/jPidzbAOdBNgzbt/3GcBYCaAFfby5bDu2ToeQFbE634Xtq9nAPwibN0A+7MuCXt/k8LWzwfwYIz3NRvWbJidj7MA7IZ1oLsEwJ7wWgDMBTDbXn4bwJ3R3ku8OgD8DcCssHVT7ddH/XuBdVC9L+zvswVAXtj6YlhdegX24woAD5j+9+jHH7bYzblBVQvtnxuirC8GcEC7zl2+I8rruhP5+lMALLC/ch+CFfTtACJb1CEiMhrWbezuU9V3Et2xqu5U1W+q6mn2fpvQtRvleQBfFJE8WK31NzX6PWZ/D6sLJm43DKzPbJfaqWHbFrZ8CoD7O9+//RmMsn9vFIBteuy+rLHe199gHbCeALBXROaISEGMWraF/V4jrJZ9+O32wufPb4YV/rGE/l+qagesOfiL7Z8d9nOdtiG52/rFqqMYXf+Gwj9LiMhVIrLa7mo6BKsFXhT2kjpVPRpWdy2AlQBuEpFCWOdMeCetFGCwu9duAENEpH/Yc+F3+GkCEFpndyUMjdhG5OiRHQCuCjugFKpqnlonNI8jIqcAeAtWK/cPPX0jqroDVhCODXt6BaxvAdfD+noe2Q3T6R1Y3yxOsn+nO7sBjIjoqz45bHkHgJ9GvP/+qjrXXneyJHCST1UfU9ULAIyB1SXzvSgvq4V1IAEAiHXfzBPQ87syhf7fi0gWrG9EtfbPKPu5TieH7afL3wmi37owlt3o+jcX+ixFJBfWHZweBnCSqhYCeA1A+GcfbfTS87D+f38RwKpYf3vUOwx2l1LVbQCqAMwWkRwRmQDg2rCXfALrhNrVdj/rv8G6kUV3ngLwUzuwISJDxbop9nFEZASsr+KPq+pTydQuIoNF5MciMlqsE7ZFsG5gvTrs/SmsFvh/wzoHsDjatuzXXQvrzkvxhjmugtV99W27//lGABeFrf8tgFkicrF9EjTf/vwGwjqXsRvAf9nP54nIcSeLReRC+/ezYYXmUUS/QfNcAF8VkfPsEPwZgPdUdWuc9xDLBSJyo33g+Q6AIKzP8z1YrewH7Pd8GazPq7M/fy2AG0Wkv/3t62tJ7HM+rM9ypH2OI/wWcTmw/t7qALSJyFVI7B68C2HdU/Q+xP8GRj3EYHe3r8Dq+90P4CcA5sH6Bw1VPQzgG7D6oHfBCpmd0TcT8iis270tEZEGWMFwcYzX3gngVFgHltB4+86VYl20E+sOQy2wzhe8BaAewEd23TMjXvd7WK3AeaoajFW0qm7QBO5mpKotAG6093MAwM0A/hS2vgrWCcHHYZ04ru6sSa1rB66F1Ye/HdZneXOU3RTAOkAchNU1sR/AL6PU8hasGyC/AuuAcRqskT899We7noOwuq5uVNVW+z1fC6tbYx+AJwHcrqqb7N97BNb/j3/Aai0n0/XxW1h3lPo7gDXo+lk2wBrCOt+u6ctI4FaCqnoE1mcSCN8eOYt3UPIQsYbvbVLV/zBdC1FPiciPAJyhLr3vqR+wxe5i9tf+0+zujM/D6o9eaLouop4S67qJr8G6+I1ShMHubsNgDVdrBPAYgHuUY37Jo0TkLlgnqV9X1eWm6/EzdsUQEfkMW+xERD7DYCci8hkGOxGRzzDYiYh8hsFOROQzDHYiIp/5/w+zZkLAXuH2AAAAAElFTkSuQmCC\n" 257 | }, 258 | "metadata": {} 259 | } 260 | ] 261 | }, 262 | { 263 | "metadata": { 264 | "_uuid": "835d5be8da0bf460e03d0b95a95c6b06fa6727d0" 265 | }, 266 | "cell_type": "markdown", 267 | "source": "We can check the support vectors and plot them together with the separating hyperplane" 268 | }, 269 | { 270 | "metadata": { 271 | "trusted": true, 272 | "_uuid": "282b83ac5feca00c626be63acdb19d2735fbf967" 273 | }, 274 | "cell_type": "code", 275 | "source": "sup_vect = X_train[alpha > 0]\nsup_vect", 276 | "execution_count": 14, 277 | "outputs": [ 278 | { 279 | "output_type": "execute_result", 280 | "execution_count": 14, 281 | "data": { 282 | "text/plain": "array([[0.15695433, 1.0690201 ],\n [0.76728419, 1.90999537],\n [1.06908872, 0.91924923]])" 283 | }, 284 | "metadata": {} 285 | } 286 | ] 287 | }, 288 | { 289 | "metadata": { 290 | "trusted": true, 291 | "_uuid": "dc5a969d0d5e50c1b52abd8c6acbf06eb9e83bbd" 292 | }, 293 | "cell_type": "code", 294 | "source": "plt.plot(x0[:,0],x0[:,1],'x',color='red')\nplt.plot(x1[:,0],x1[:,1],'x',color='blue')\nplt.figtext(0.5, 0.01, 'Figure 3: support vectors defining the separating hyperplane', \n wrap=True, horizontalalignment='center', fontsize=12)\n\nx = np.linspace(-5, 5, 200)\ny = np.linspace(-5, 5, 200)\nX, Y = np.meshgrid(x, y)\nplt.contourf(X, Y, Z, alpha=.5, cmap='jet_r')\nC = plt.contour(X, Y, Z, colors='black',zorder=4)\nplt.plot(sup_vect[:,0],sup_vect[:,1],'o',color='green')\n\n\nplt.axis('equal')\nplt.show()", 295 | "execution_count": 15, 296 | "outputs": [ 297 | { 298 | "output_type": "display_data", 299 | "data": { 300 | "text/plain": "
", 301 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEPCAYAAACqZsSmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XmYFNW5+PHvOysMAwMIBAeEaYUYF4iaiUpADHGJJrhFvErMokmuV5Oo+V2zuSQxud7c3JjF3CdeE41JjAvRTFxuNIkaRc0gLuMSUFBRBoMMyLANDDDDLO/vj1Pd1PR093QP1VO9vJ/n4aG6q6f67erqt06dc+ocUVWMMcYUl5KwAzDGGDP0LPkbY0wRsuRvjDFFyJK/McYUIUv+xhhThCz5G2NMEbLkb4wxRciSvzHGFCFL/sYYU4TKwnrjcePGaV1dXVhvP2T2tLSEHUJR2rprF++0tQEwceLhiEjIEZlsqq2tCDuEIfHCCy9sUtXxQWwrtORfV1dHU1NTWG8/ZN659tqwQyhKi1et4jN33QVAV9cuvvzlxSFHZLLp2munhB3CkBCRt4PallX7mII0b/p0Dh7vCkibN69m585NIUdkTG6x5G8K1ndPPTW23N3dGWIkxuQeS/7GGFOELPmbgjV9/Hjw2nmfe+72cIMxJsdY8jcFa0J1NZfMqAXg6advYtu2d0KOyJjcYcnfFLR/O/mTseUtW9aEF4gxOcaSvylond3dseWNG98IMRJjcoslf1PQJowcGVt++OHvsn37+hCjMSZ3WPI3Ba2spIQnvvSl2OMVK/4SYjTG5A5L/qbgTRkzJrbc3d0RYiTG5A5L/qbglZaUUO4d6Y89dj27d28NNyBjcoAlf1PwSkR47cqrKassBXp54okbwg7JmNBZ8jdFoaKsDO3qAaCry6p+jLHkb4pGT2/YERiTOyz5m6KzYsWDqGrYYRS1JUtG0txc2ee55uZKliwZmeQvTNAs+ZuiccrBBwPQ2dnOQw9dE3I0xa22dg8NDeNiJ4Dm5koaGsZRW7sn5MiKR6DJX0RKReQlEXkwyO0aE4Rbzjsvttzaanf7hikS6WTBgk00NIxj8eIaGhrGsWDBJiIRG3p7qARd8r8cWBnwNo0JTG1VOQDbtq0NORITiXRSX9/OU0/VUF/fbol/iAWW/EVkMvBx4FdBbdOYoP383E8BsH37eh599L9Cjqa4NTdX0tRUzdy5bTQ1VfdrAzDZFWTJ/wbg64D1qTA564NTpjB2+HAAVqx4KORoile0jn/Bgk3Mm9cWqwKyE8DQCST5i8h8YKOqvjDA6y4SkSYRaWptbQ3irY3J2PZOV71gUzuGp6Wlok8df7QNoKWlIuTIikdQJf/ZwOkisgb4PfAREbkj/kWqerOq1qtq/Xhvcm1jhtq4qioA2ts38txzt4UcTWYKpYvk7Nk7+tXxRyKdzJ69I6SIik8gyV9Vr1TVyapaB5wHPK6qnwpi28YE7cnLLostL1784xAjyVyxdJEslJNcLrN+/qboVJWXUzNsGACqPSFHk5li6SJZLCe5MAWe/FX1CVWdH/R2jQlStNG3s7Odt99+LuRoMlMMXSSL5SQXJiv5m6L05KWXEqlzpf/f3X5BqLFkqli6SBbDSS5MlvxNURIR3l89DQDtzZ+qhGLqIlksJ7mwWPI3RWvZunXeUv4M8lYsXSSL6SQXlrKwAzAmLCICgGo3XV27KS8fHnJEA0vUFTIS6Sy4KpFUJ7lC+6xhsZK/KVqXH398bPmH1x8bYiQmnt0HkH2W/E3ROmvmTA7xbjbs7moLORpjhpYlf1PUzpgxw1tSm+DFFBVL/qaoXTx7Nnh1/zfddFLI0RgzdCz5m6JWWlLCN7y6/9bWVSFHY8zQseRvit7GXbvCDsGYIWfJ3xS9KaNHx5bvuutzIUZizNCx5G+K3uePPTY20NuqVY/ZOP+mKFjyN0VPRLj6pL2Nvb29+TXSpzGDYcnfGGDauHFhh5A2G+veBMGSvzG4uX29Hp+8/vqj4QYzABvr3gTBkr8xnhMOdcn03nsvY8+e3O0BZGPdmyBY8jfG868fODe2vHPnphAjGVjQY91bVVLxseRvjKeybO8gt21tLSFGMrCgx7q3qqTiY8nfGM90b5A3gNtuOy9nq36yMda9VSUVH0v+xnhGDRvGLeec4z1SmpufDjWeZLI1oYtNm1hcbDIXY3w+dOCBsWXV7hAjSS5bE7rEVyXV1XXYCaCAWcnfGJ9Rw4YxqbYcgD80XF40d/vatInFx5K/MXH+cv5XAOjt6eCll+4OOZqhUSxzA5u9LPkbE2dMVVVsuaOjsKYNTNalE7BpE4uMJX9jUnjmmVvzYoavdPvpD9Sl0/r7Fw9L/qZgVC9ZQmVzc5/nKpubqV6yJONtzZo6FYBduzbz97/fGEh82ZRuP/2BunRaf//iYcnfFIyu2lrGNjTETgCVzc2MbWigq7Y2423dc8EFseVNm3J/hq9M+umn6tKZbDstLRV2RVBgLPmbgtEZibBlwQLGNjQwavFixjY0sGXBAjojkUFtrzzHfx3xVTSRSCfTpu0esJ/+QHcHJzo52BVB4cnxw9uYzHRGIuysr2fUU0+xs75+0Ikf4KiI6/K5ZcvbQYUXqPiEvHRpNcuWjWDmzPakQz6k06Uz0cnB7gAuPJb8TUGpbG5mRFMT2+fOZURTU782gEx8ZmodAOvWvcTLLzcEFGFw/An5vvvG8sgjYzj55K2cddaWpP30B+rSmerkYHcAFxZL/qZgROv4tyxYwPZ582JVQIM9AZw2Z2Fsuanp9qDCDFQ0IS9bVs3MmTuZNas99nyifvqzZ+9I2aUz1ckh6MHkTLhseAdTMMpbWvrU8UfbAMpbWgZV/SMiCKCQs3f6JquigcEN+ZBs6AigT1VPXV2HVf3kOUv+pmC0z57d77nOSGSf6v2Hlwq7epR3313JW28t4aCD+r9HGJYsGUlJidLYWBNLwMOG9bBo0XgWLmwNPCGnuiKw5J+frNrHmBSeuOwrseUHH/xGiJH0VVu7h8WLRzNnThuRSCfNzZU0NtYwb962rAzJMFB1kck/VvI3JoX9R43iPdXVvNveTk9PV9jhxEQinSxc2EpDwzg6Okppaqq2KhiTkUBK/iJygIgsFpEVIvKqiFwexHaNyQVfnTcPgB07NuTUDF/W+8bsi6CqfbqBK1T1UOBY4EsicmhA2zYmVOcccQTDhrnlW245PdxgfLLV+8bG9ykOgSR/VV2vqi96yzuAlcCkILZtTNhKS0qokuEA7NmzM+nrliwZyWMbH+QGPsR3qeMGPsRjGx/MStLM5vj7djdvcQi8zl9E6oAjgWcTrLsIuAhgypQpQb+1MVkjabymY/oiGsd+DcTN/dvGOhrHfo050wHmBxpPNnvf+G8eq69vt/aEAhVobx8RqQb+CHxFVbfHr1fVm1W1XlXrx/smyzYmX3R3d6Dam3Dd8gnfh/K4Sd/Ld7nnA5bt3jfWnlD4Akv+IlKOS/x3quq9QW3XmFyw8KijAFDt5cYbT0r4mjYSNwYnez6XNTdXsnTpyH7jBFndf+EIqrePALcCK1X1J0Fs05hc8o0TT2T6uHEAbN6SeLiIGhIPHZ3s+VwVreOfN28bb745nDlz2mhoGMfSpdVW919Agir5zwY+DXxERF72/n0soG0bkxNuO//8lOtnbLwKuqr6PtlV5Z7PI9H2hFmz2lmwYBONjTVMm7abxYtHW91/AQmqt0+jqoqqzlTVI7x/fw5i28bkiv2ic/tqD7ff/ul+64etWsicLddTo5NAhRqdxJwt1zNs1cJ+r81l/vYE/8Bxs2b1b2cw+cvu8DUmTVUVFRw4diyrt2xh9eqn+q13ja3zOcHfs2cCMCF/h0CIv5egrq7DTgAFwsb2MSYDB+63X9ghDJls3ktgwmfJ35gMTBi5t6fLY49dH0oMQ3UH7kATv5j8ZsnfmAx879RTY3WljY0/DyWGoboD10byLGyW/I3JQGVZGVeefHKoMdh8uiYIlvyNydBFs2bFlt9884lQYrA7cM2+suRvzCAcc1A5AHfe+dlQxvnP9ny6NrJn4bPkb8wgXHjUmbHlrq5dKV4ZvKHohWMjexY+S/7GDEJXT09o752NXjjxJf1IpJM5c9pYtGh8wnYFuzLIf5b8TVZUL1lCZXPfMXAqm5upXrIkpIiyp7Mz+Rj/2ZCNXjjxJf0//WkMTz45mkMO2RVrVwBiyd2uDPKfJX+TFV21tYxtaIidACqbmxnb0EBXbX4NcpbMEZP2zlX0s58dl3SY53wR34Po1VdH0NurvP56FXPntvHssyO5++69yT2dHkd2dZDbLPmbrOiMRNiyYAFjGxoYtXgxYxsa2LJgAZ2RSNihBaJu7Fi++uEPA6DazaZNb4UbUAD8PYgOPngXqkJvr3prFRA2bCiPJe+BehzZ1UFus+Rv0jKYapzOSISd9fWMeuopdtbX503ir16yhAc3Psax3MAUvsux3MCDGx/r91kvP/54SitKAejtDa8NICj+HkQrV1ZxxBHtgPDUUzUcc0w7xx+/jcWLR8eS90A9jux+hNxmyd+kZTDVOJXNzYxoamL73LmMaGrqd/LIVb+f3sEVYxtZJ22owDpp44qxjfx+eke/11ZVupLxbbedh6r2W58v4nsQLVzYyvLl1YBSUtJLY+NInnxyNAsXthKJdLJ0aTWLFo0fsMeR3Y+Quyz5m7RkWo0TPTlsWbCA7fPmxf42H04A101Yzq7yvs/tKnfPx/vvk1yXz927t9LcnL+N2fE9iBxlxoxdHH74Lnp7S+jybmdobq5k8eLRzJu3bcAeR9m+H8EMng3pXMCqlyyhq7a2T4KubG6mvKWF9tmzM96evxpn+9y5Katxylta+pwcoieP8paWnK/+aaEt7efnHHRQbLmjI/HfLVkyko7pi1g+4fu00UINtczYeBXDVi3MmXFy4uNoaang3HM3AdDQMI6ZM9tZtmwEjzwymu3by2JXAH6RSGef5/xXE5FIJ3V1HVb1k0Os5F/Agu5xk0k1Tvvs2f2SfGckMqiTzlCrpSbt50tL9v6EGhtvSvh3HdMX0Tj2a7TJOhClTdbROPZrdExfFEzAgzBQT5zoySCarM86awszZ+5kw4ZKpk3bnVbytlFBc5sl/wIWZI+bfK7GydQ1G2dQFTdiQ1WXez7eqGHDmFzjTgrr1y9nxYq/9HvN8gnfh/K4u4DLd7nnQ5JOTxx/8m5uruTNN4czc2Y7K1dWpVV9Y6OC5jZL/gUuqB43qapxCs15q4bx4y1zmKQ1iMIkreHHW+Zw3qphCV//9OWXx5Y3bFjRb30bifdRsueHQjo9caLJ2199c9ZZW1i4sNUmdSkAVudf4OKrajrr6gZ1AkhUXdMZidAZiQTetjBYQcXRPns283ETMsZMgPYJiV8vIhw4vpTVrT1s2PBqv/U11NLGuoTPh8nfE2fu3LakVTmpqm+s7j5/Wcm/gA1VVU2u3M0bZhyfmXYAAKtWPcYbbzzeZ92MjVdBV1VcsFXu+RCl2xPHqm8KkyX/AjZUVTW5cjdvmHF8/uTPxpZffvnuPuuGrVrInC3XU6OTQIUancScLdczbNXCrMeVjM3Pa6zap4ClqqoJWibdQLMpV+LwcyXk+ZzA/L1PTgAmhFdytqocYyV/E4hcuZs3F+Lo7s79sWusKsdY8jf7rLK5mf0WLWLHnDl92haqly7tMx5Otod5Drs7qnj/r1r1OJs2FV4XWFNYLPmbfVbe0sL2efMY2dhIZXMznZEIO+bMYdTixX0aW7PdIBt2d9T7Pv/52PIDD1wxJO8JNnSyGRyr8zf7LNq20DVxImMbGthZX8+IpiY2L1zYp87d3yAbfU2QDbJD2caRyAcmT+bCwybym1c30NXVfxC4bInesOW/ISv62JhkLPmbwKTT2JqLDbJBGlXhflLvvruSPXt2UVFRNcBf7Dv/DVv19e00NVXb+DlmQFbtYwKTTmNrLjTIZtMlH/2Ut9TLr3999pC9rw2dbDJlyd8EIp3G1rAbZIfCiMpKKkrdBC/t7a1D9r42dLLJlCV/E4h0GlvDbpAdKiKu38/u3VuHZIIXu2HLDIYlfxOIdIZwzudhnjPxiRlu9M/e3m5uvfUTWX8/GzrZDIYlf2MC9sPTT2dEuZsK7N13+4/yGTS7YcsMhiV/Y7Lg1wvduD0i9hMzuSmwI1NEThGR10XkTRH5ZlDbNdmX7Ttvi5lqT9ghGJNQIMlfREqBG4FTgUOBhSJyaBDbNtkX1J23dhLZa7hX7dPd3clf//q9kKMxpr+gSv5HA2+q6mpV3QP8HjgjoG2bgCRLztFeOPs6FHKujOufC2bsvz+lXq+fZ5/7bbjBGJNAUMl/ErDW9/gd7zmTQ1Il5yCme4x23dxv0SLG3Hdfn5NIsV0BlJWWcsjEie7BEHT3NCZTQ9oaJSIXiUiTiDS1tg7dDTDGSTXZSVB33nZGIuw+5BBGLFtGz6hRQN+TTDGdBGoqo/3se1m+/E+hxmJMvKCS/zrgAN/jyd5zfajqzapar6r148ePD+itTSYSlfCDvPO2srmZYW++yc6ZMynfsIFxd97JfnffzZYFCwCKqhro52fvHd7h3nsvDTESY/oLKvk/D0wXkYiIVADnAf8X0LZNgBKV8IO689Z/Etl61lnsmjkT6elBurupXLMm0GkV86FxeVx1Na9feWXYYRiTUCDJX1W7gS8DDwMrgXtU9dUgtm2Ck6yEH63z9xvMnbf+k4j/CgDVfWpLSCRfGpcryqID5yqbNq0ONRZj/AKr81fVP6vqe1X1IFX9z6C2a4KT7bF1osM3+E8yu444gt7ycnrLyxnx7LOBDeKWK5PGD6SspIQKb5SFG288IdxgjPGx2w+LyFCNrRM9yYCr499y7rlsXriQ3YcdFugonkH0UBoK5874gLfUG2ocxvjZZC4mcNGTSfWSJf2uNHYffjjlLS2BJOr49ovOurqcPAGMq66OLXd17aa8fHiI0RjjWMnfZE02rzTyaW6Aj0yfHlv+wQ/ePyTDPBszEEv+Ji+lar/ItZ5AR0yaxLlHHAFAb28nu3dvDSUOY/ws+Zu8lOqqIhd7An1//nyiA3xayd/kAqvzNwXH3xNoZ309I5qaQu8J5KZ2FMASv8kNVvI3BSnXegKpKtrrEv/tt58faizGgCV/U6CCGqsoKCJCZMwYAN59dyVbtrwdajzGWPIvILnW0JmObMScqz2BfvPJT8aWd+9uCzESYyz5F5RcbOgcSKYxp3OyyPadzINVWrL35/bss78JMRJjLPkXlHwZ8sAv05jTOVkM1Z3MmZo6ZgxVpW6Cl+XL7+Xdd1eGGo8pbpb8C0yuNXSmI5OY8/EEFyUiLP/mVQwf7k4AGzZY8jfhseRfYHKtoTMdmcacjye4qIqyMior3c9u48bXQ47GFDNL/gUkVxs6UxlMzPl4gvM7srYUgKef/gUbN74RcjSmWFnyLyC52tCZSqYx5+MJLt71H/1SbPnll+8JMRJTzOwO3wKSqEGzMxLJ6WqRTGNOdbLI5c/p9x5vbmNjwmTJ3+SVfDzBpdLT0x12CKZIWbWPMSF67rnb6OiwG77M0LPkb0wIfnT66d5SLw899O1QYzHFyZK/MSE498gjOWGKG+vHSv4mDJb8jQnJq607AFiz5mlUbX5fM7Qs+RsTkg+/9zAAurs7ueeeS0KOxhQbS/4mqSBH3MzHEUez7fozz4z9ADdvXh1qLKb4WPI3SQU5Smg+jjg6lHbs2BB2CKbIWPI3SQ00iFompfl8HpAtmz54wAEAdHRs5/77rwg5GlNMLPkbIHkiL29pSTqIWqal+XwekC1b/nDhhRwzZQoAq1YtDjkaU0ws+RsgeSKnpCTpIGqZlubzfUC2bBAR/uXIIwHo7e0KORpTTCz555AwG0UTJfIdc+YwsrEx5SBq6ZbmC2FAtmyZPm4c4Kp+nnzyf0KOxhQLS/45JOxG0fhETm/vgCNupluaz8cRR4fKYRMnIt5yY+ONocZiiocN7JZD/KXvnfX1jGhqGtJG0fhEnui9/YOo+UvznZEInXV1Sat+Cm1AtiBVlJUxceRI1u/YYTd7mSFjJf8cE1aj6GCqZaw0b0z+suSfY8JqFB1MIs/VidLzUVV5OQA9PXtoaXkl5GhMMbDkn0PCbBTNZiK3u3sHdtM558SWb7317BAjMcXCkn8OKdRqlLAbsvPBIRMn8uIV7iavXrUunyb7rME3hxRqo2jYDdn54uHXXnML2sPu3dsZPtymezTZs88lfxG5XkReE5FlInKfiIwOIjBTWOzu3oFNGr33p/OjHx8dYiSmGARR7fMocLiqzgTeAK4MYJumwNjdvQObN20ac6ZOBaC3Z3fI0ZhCt8/JX1UfUdXoLNTPAJP3dZumsNjdvembWFMTW7Y+/yabgm7w/Rzwl2QrReQiEWkSkabW1taA39rkqkJtyM6G46dNiy3/5CdW9WOyJ60GXxH5GzAxwaqrVfUB7zVXA93Ancm2o6o3AzcD1NfXa8bRmrxUqA3Z2XDmjBksbW7mrpdeor29FdVeRKxTngleWslfVU9MtV5ELgDmAyeoqiV1Y/bBf59+One99FLYYZgCt89dPUXkFODrwPGqumvfQzKmuL2+cWPYIZgiEMT15M+BkcCjIvKyiPwigG0aU7SW+dpC7r33KyFGYgpZEL19pqnqAap6hPfv4iACM6ZYnf3+91NdUQHAK688QEfH9pAjMoXIWpKMyTElItx49t7xfbq6rM+/CZ4lf2NyUGlpaWz51VcfCjESU6gs+RuTgz5UV0c0/T/88Hdpb98Uajym8FjyNyYHlZeW8tY111BR4SZ43Lx5dcgRmUJjyd+YHLWnt5fubnfbzLZt74QcjSk0lvyNyVElIvR6w/vcf///Y9eureEGZAqKJX9jclRlWRn3Xnhh7PHKlUmHzTImY5b8jclhh++/f2y5u3tPiJGYQmPJ35gcVu7r8vnXv36PPXtsBBUTDEv+xuSwspISvnvKKd6jHv7+95+HGo8pHJb8jclxnzvmGA6f7MZg7OjYEXI0plBY8jcmD2zc3hN2CKbAWPI3Jg9s3O76+1uPHxMUS/7G5IH6yW5q7J07W3nssR+GHI0pBJb8jckD933+87Hldetsli+z7yz5G5Nntm2zie/NvrPkb0ye2H/kSAC2bl3DkiW/DDkak+8s+RuTJ5Z+5SucMnUMAC++uCjkaEy+s+RvTJ4oLSmhrNQN8dzb2xVyNCbfWfI3Jo80rtsJuCGeV6ywGb7M4FnyNyaPPHHppbHlhx66JsRITL6z5G9MHtlvxAhGVlYC0Ntrd/2awbPkb4wxRciSvzF5pqzE/Ww7Otpob28NORqTryz5G5NnfnHOOVSMKAfgpps+GnI0Jl9Z8jcmz3woEuHHJ88HoLPThng2g2PJ35g8dP3ixQD09Oyhp8f6/JvMWfI3Jg9dePTRseUf/ui4ECMx+cqSvzF56AuzZjF19GgA9nRsCDkak48s+RuTp/avqfGWNNQ4TH6y5G9Mnjp6ypTY8m9+syDESEw+suRvTJ762kc+wvdOOQWAf/7z+ZCjMfnGkr8xeeyMww8POwSTpyz5G5PHfvv83hL/gw9eHWIkJt8ElvxF5AoRUREZF9Q2jTGpXXbccVR6v+IXXriD3t7ucAMyeSOQ5C8iBwAnA/8MYnvGmPSUlZZy9cmnxB7bSJ8mXUGV/H8KfB3rc2bMkBORsEMweWifk7+InAGsU9V/BBCPMSZDJb7k/847L4UYicknaSV/EfmbiLyS4N8ZwFXAt9PczkUi0iQiTa2tNhStMUE47bDDeM973Cift912Ht3dHSFHZPJBWslfVU9U1cPj/wGrgQjwDxFZA0wGXhSRiUm2c7Oq1qtq/fjx44P6DMYUtTFVVSz9wteREgClvX1T2CGZPFC2L3+sqsuBCdHH3gmgXlXt6DNmCD2/di3a65Z3725j9OjJ4QZkcp718zemALx3QqwMxs03z6enZ0+I0Zh8EGjyV9U6K/UbM/TGjRjBdR/7mPeol7ffbgo1HpP7rORvTIE4+eCDY8u9vVbyN6lZ8jemQIyoqIgt//GPl9kNXyYlS/7GFIhRw4bx7ZNOAqCjo43ly+8POSKTyyz5G1NAvjBrFvuPdj/rXbu2hhyNyWWW/I0pID2qrN/WG3YYJg9Y8jemQL3yyv+FHYLJYZb8jSkgZSUlvKe6GoCWln9Yvb9JypK/MQXm+X//99jy6tVLQozE5DJL/sYUGP8Qz21tLSFGYnKZJX9jCtBR46sAaG5u5I03Hg85GpOLLPkbU4Ae+OLXOGw/dwJobPzfkKMxuciSvzEF6tXNuwBsXl+TkCV/YwpUuVf1v27dS7S0vBJuMCbnWPI3pkD9+eJLYsv33ntpiJGYXGTJ35gC9b4JEygvcT/x7m4b5dP0ZcnfmAJW4iX/trZ17N69LeRoTC6x5G9MAbv+tNMorSgBlHXrloUdjskhoqrhvLFIK/B2KG+e3DggX2Yis1izJ5/izadYIb/izcVYp6rq+CA2FFryz0Ui0qSq9WHHkQ6LNXvyKd58ihXyK958inUwrNrHGGOKkCV/Y4wpQpb8+7o57AAyYLFmTz7Fm0+xQn7Fm0+xZszq/I0xpghZyd8YY4qQJf84InKpiLwmIq+KyA/DjicdInKFiKiIjAs7lmRE5Hpvvy4TkftEZHTYMcUTkVNE5HUReVNEvhl2PKmIyAEislhEVnjH6uVhxzQQESkVkZdE5MGwYxmIiIwWkQbvmF0pIrPCjilolvx9RGQecAbwflU9DPhRyCENSEQOAE4G/hl2LAN4FDhcVWcCbwBXhhxPHyJSCtwInAocCiwUkUPDjSqlbuAKVT0UOBb4Uo7HC3A5sDLsINL0M+Cvqvo+4P3kT9xps+Tf1yXAD1S1E0BVN4YcTzp+CnwdyOnGG1V9RFWjYws/A0wOM54EjgbeVNXVqroH+D2uIJCTVHW9qr7oLe/AJadJ4UaVnIhMBj4O/CrsWAYiIjXAXOBWAFXdo6oFNzaGJf++3gscJyLPisiTIvLBsANKRUTOANap6j/CjiVDnwP+EnYQcSYBa32P3yGHk6mfiNT2sk8XAAAOK0lEQVQBRwLPhhtJSjfgCim9YQeShgjQCvzGq6b6lYiMCDuooJWFHcBQE5G/ARMTrLoatz/G4i6jPwjcIyIHaohdogaI9ypclU9OSBWrqj7gveZqXJXFnUMZW6ESkWrgj8BXVHV72PEkIiLzgY2q+oKIfDjseNJQBhwFXKqqz4rIz4BvAt8KN6xgFV3yV9UTk60TkUuAe71k/5yI9OLG92gdqvjiJYtXRGbgSij/8Cbsngy8KCJHq+qGIQwxJtW+BRCRC4D5wAlhnlCTWAcc4Hs82XsuZ4lIOS7x36mq94YdTwqzgdNF5GPAMGCUiNyhqp8KOa5k3gHeUdXolVQDLvkXFKv26et+YB6AiLwXqCD3BnYCQFWXq+oEVa1T1TrcAXtUWIl/ICJyCu6y/3RV3RV2PAk8D0wXkYiIVADnAf8XckxJiTvj3wqsVNWfhB1PKqp6papO9o7T84DHczjx4/2G1orIwd5TJwArQgwpK4qu5D+AXwO/FpFXgD3AZ3OwhJqvfg5UAo96VyrPqOrF4Ya0l6p2i8iXgYeBUuDXqvpqyGGlMhv4NLBcRF72nrtKVf8cYkyF5FLgTq8gsBq4MOR4Amd3+BpjTBGyah9jjClClvyNMaYIWfI3xpgiZMnfGGOKkCV/Y4wpQpb8jTGmCFnyN8aYIjTo5C8iU0Sk3RsK15jAiUidN09BWjcjish1IrJJRDZkcnwO9bEsImtEJOVQGIXOm4Pgw1nY7gUi0hj0drPBO7anhfX+AyZ/70Dd7f04ov9qVfWfqlqtqj1DEegAMR4qIk0istX797c8GNs8IyLyYRF5J4T3/a2IXDfU75spEZkCXAEcqqoTMzk+s3ks58v+y6ZE+0BVD1PVJ0IKyZB+yf8078cR/deSzaDSLen5tAALcCNyjsONyfL7oOMKyyD2R84YwivDKcDmPJmDoWDk87GZTXlRI6KqKf8Ba4ATEzxfh5tApMx7HAGeAnYAf8PNinSHt+7DuFHyEm4XuBY3ct4dwHbgC7gT0zeBt4DNwD3A2DTiLQO+BOwa6LW+v7kAN37HDqAZON8X1x0pPvMTwH8Bz3lxPxCN0ffai3Anp/XAV33bqsSNcd7i/bsBqPTvL+AbwAbgD8Bu3Fjo7d6/2rjPcIz32lLfc2cBy7zllPsTmAM8DWzDjWt/gRd7F26co3bgT95rD/E++zbgVdxgbdHt/Ba4CfgzsBM4EfgYbmCsHbiRMr+a5Hsoxc2etsn7Pr4Ut79rcIOZrfe2c533NyfG7Z/fJvmu/gNY4sXxCDAuxfea8LXe+s8Ab3v78Vsk/40k239rgK8Cy4A24G5gmO/v5gMve/v3aWBmkv0luMl8NuKOv+W42dLAHV8/ws3w9i7wC2B43PF1lbev1+Ad8976jwMvedtcC1yb4DfweW/bT3nP/wF3/LXh8sBhaewD/+//HuB33v5+Faj3vedRXjw7vPe5G7guxW+50fvsW3G/51O9decAL8S9/t+BB3zH7i9ws87tAJ4Epvpe+z5v3RbgdeBfBjjuB9qeAtMy2Oef9fb5Jtww6dH1g8uVASb/pd4Or8Alku1klvy7gDO9DzIcN+VbdManSuCXwKIBYt2GGyu+F7jG9/wn8ZJggr8Z4cV6sPd4f/YeuNcycPJfBxzubeePvs8cfe0ib90M3NDQ0c/8Pe/zTQDG437k/+HbX93Af3uffXiifZjgs7wFnOR7/Afgm95y0v0JTMUdnAuBcmA/4AjfQX2db5vlwJu4xFEBfMT724N9r2/DDTxWghvCdz1wnLd+DG700UTxXwy8hhtaeSywOG5/3+fFPcLbb88B/5boGEvyXb2Fm7BnuPf4B4N47aG4JDbH+/w/wh27/X4jifaf79h/Dqj1PudK4GJv3ZG4ZH4M7sT2We/1lQm2/VHgBWA07kRwCLC/t+6nuCvgscBI4E/Af8UdXz/BHQvH4xLWwb71M7zvbybu5HFm3L76nfc9RE8on/PeJ1qoeTmNfeD//XfgCgmluALVM966CtyJ9nLcsfcJ3IkkVfLvAv7V29YluMKVeLFtAQ7xvf4l4GxfnDtws3hV4qZybPTlibW4Ad7KvO9pE66aMfq38cd90u0lSP7p7PNbcMfj+4HO6OdgELkyk+Tfjkus24D7438wuEvubqDK93d3kFnyfypu/UrcuO/Rx/t7X2rZAPGOAL4IfHygz+Z7/TbgbLwD2bfuWgZO/j/wrT8Ud2CW+l77Pt/6HwK3+hL1x+J+yGt8+2sPfUuD/fZhgs9yHW40SnA/xJ14JY1U+xM3n+596SQv4DhcCa/E99wivJKK9/rfxW3jn8C/AaMGiP9xvCToPT7Zd4y9B3fAD/etXwgsTrR/knxX/gLBF3FztGb62m/j+2EBVd53lWny/1TccfELb/kmvEKAb/3rwPEJtv0R3HzIx8Z9H+J99wf5npsFNPv2VTcwwrf+HuBbST7DDcBP4/bVgSm+x9Hea2oG2Af+3//f4n5Hu73lubgClvjWN8Zvz7fuAtx0nP7vR4GJvv37n97yYbirg0pfnL/3/W010IMrjJwL/D3uvX4JfCfFcZ90e97jWPJPc59P9q1/DjhvoN92qt9bunX+Z6rqaO/fmQnW1wJbtO847WsTvC6V+NdPBe4TkW0isg33AXtwSSApVd2Ju9T6nYhMGOhNvdefiyt1rheRh0TkfYOM+21c6WRcivW13nKt9zjROoBWVe3IIA6Au4BPiEglroT0oqpG3yPV/jwAdzJKRy2wVlX90/G9Td8pD+O/y7Nxpbq3vekxZ6Xadtx2o6bi9u1632f4Je4KIF3+uQ524X6Mmb62T4zeMb85gxgG2v5U4IroZ/Q+5wH0PTai7/04bqjsG4GNInKziIzCXUlWAS/4tvFX7/mord6xHxU7/kTkGBFZLCKtItKG+234j2nw7QMRKRWRH4jIWyKyHZfYSfA3qcTvj2Fee0ItbqpSTfTeA23Ll5Oi+/c24JPefAifBu5Rb87u+G2rajvuSqEW970cE/e9nE/fmesSxZVse32kuc9THTMZ58qg+vmvB8aKSJXvOf+sSDtxByMQawzxH4jQfwLytbi6utG+f8NUNZ3ZlUq890trDlZVfVhVT8KdMV/DXV71i5vEUxT6P+cU3Bl3U4r10cbyFtyXlmgd9N8f8Y/7UdUVuB/xqbiqrrt8q1Ptz7XAQck2G/e4BThARPzHzhT6znrV529U9XlVPQOXqO/HlTITWU///eWPvxNX9x6Nf5SqHpZkW9myHt/k8yIyHFdNlsyA31uctbiSqf97qlLVRQk3rvo/qvoBXGn5vcDXcMffblz1ZXQbNarqP9mNiZuX1n/83YWrMjpAVWtwhSlJ8bk+iZvs/kRcu0yd97wkeG2m1gOTvGQddUCyFw9EVZ/BXakdh4v79riXxLbtTZE5Frdf1gJPxn0v1ap6iX/zCd4y2fbipbPPkxlUrgwk+XulyybgWhGp8Ep2p/le8gbuTP5xb+q5a3B1U6n8AvhPEZkKICLjxU1Y3o+InCQiR3olkFG4usytuDNgSiLyHhE5w/shdOKquKKl2peBuV4/8Bpc9Ui8T3ldTatw9fgN2rfL4LdEpEpEDsPVF97tPb8IuMb7XONw1Ql3pAj1XWA/L45U7sLVAc7F1flHpdqfdwInisi/iEiZiOwnIkf43vdA33aexZU6vi4i5V5f7dNI0rvKOx7OF5EaVe3Cta8km8T7HuAyEZksImPwTZ2nqutxDa8/FpFRIlIiIgeJyPED7I+gNQCniciHxE30cS2pf6Tx+28gtwAXeyVBEZER3u9mZPwLReSD3uvKcQWVDqDXuyq7Bfhp9OpXRCaJyEfjNvFd7/s5DtfIHD1eRuKu5DtE5GhckkxlJO63sxlXWPp+3PpM94HfUlwp9svesXkGcPQgtxX1O9wVU5eqxt8T8DERmeN9t/+Ba3tYCzwIvFdEPu0d9+Xe/j9kgPdKtr14me5zv7RzpV+Qd/iej6tX3Iyre74bd0Cgqm24etNf4UqIO3G9DVL5Ge5M+IiI7MA1aByT5LWjccm0DVd9cRBwSrTaxEs+yWZlKsG1+LfgLsmOxzUSoaqPep9jGa5h7cEEf387rm5vA66R57K49U/iGkgfA36kqo94z1+HO2Euw/XSeNF7LiFVfc37jKu9y7t+l46eRd5neFxV/VcgSfenqv4TVy1zhbcPXsY1KoHrXXOo9573q+oeXLI/FVfC/F/gM158yXwaWONVCVyMO1YSuQU3k9Y/cPsjfl7az+AaAFfgTu4NuKu1IaNudq9LcSe79bjCwka8Yz2BPvsvje034Rorf477jG/i6rETGYXbZ1vZ2/voem/dN7y/fcbb738DDvb97Qbv71pwJ/+Lfd/hF4HvecfJt0l+pRb1O+/91+G+m2fi1me0D/y84+0TuN5F24BP4X6HyfZ3Om7HddJIVNi6C/gO7nfwAe/9UNUduDao83D7bAN7O2SkknB7CWS6z/0yyZUxWZvJS0TuBl5T1e9k5Q1ygIg8gWsQ/lWCdXW4bmblqto9tJGZoeJdym8Dpqtqc9jxpMO7WrtDVScP9NpcJCLP4hrIfzPIvx+OO2EfpaqrfM//Ftdp4JqA4gx0e0ELrOTvXQId5F2On4KrA8zoLG9MPhCR07yqvBG4rp7L2dvIaQImIseLyESv2uezuK6Qf92HTV4CPO9P/MUoyLvzJuIu0/fDVelcoqovBbh9Y3LFGbiqA8FV3Z2n2bqENuCqq+7BdcteDSzw2oAyJiJrcN9bol6LRcUmcDfGmCJkQzobY0wRsuRvjDFFyJK/McYUIUv+xhhThCz5G2NMEbLkb4wxRej/AzQOGmuWa9vcAAAAAElFTkSuQmCC\n" 302 | }, 303 | "metadata": {} 304 | } 305 | ] 306 | }, 307 | { 308 | "metadata": { 309 | "trusted": true, 310 | "_uuid": "3794d4d5b722d19532d5039af6861a077e3a4552" 311 | }, 312 | "cell_type": "markdown", 313 | "source": "## Final remarks\n\nIn this notebook we explored the most basic setup for a support vector machine. In the next notebooks, we will explore the non-linearly separable case as well as the usa of kernels for introducing non-linear decision boundaries." 314 | } 315 | ], 316 | "metadata": { 317 | "kernelspec": { 318 | "display_name": "Python 3", 319 | "language": "python", 320 | "name": "python3" 321 | }, 322 | "language_info": { 323 | "name": "python", 324 | "version": "3.6.6", 325 | "mimetype": "text/x-python", 326 | "codemirror_mode": { 327 | "name": "ipython", 328 | "version": 3 329 | }, 330 | "pygments_lexer": "ipython3", 331 | "nbconvert_exporter": "python", 332 | "file_extension": ".py" 333 | } 334 | }, 335 | "nbformat": 4, 336 | "nbformat_minor": 1 337 | } 338 | --------------------------------------------------------------------------------