├── adadelta.py
├── README.md
├── nesterov.py
└── GD_vs_SGD.ipynb
/adadelta.py:
--------------------------------------------------------------------------------
1 | import sys
2 | import numpy
3 |
4 | from matplotlib import pyplot
5 |
6 | n_steps = 100000
7 | gradients = [1e-4, 1e-3, 1e-2, 1e-1, 1e2]
8 | decay_rate = float(sys.argv[1])
9 | epsilon = float(sys.argv[2])
10 |
11 | for gradient in gradients:
12 | mean_gradient2 = 0
13 | mean_step2 = 0
14 |
15 | steps = numpy.zeros(n_steps)
16 | for i in range(n_steps):
17 | mean_gradient2 = decay_rate * mean_gradient2 + (1 - decay_rate) * gradient ** 2
18 | steps[i] = ((mean_step2 + epsilon) / (mean_gradient2 + epsilon)) ** 0.5 * gradient
19 | mean_step2 = decay_rate * mean_step2 + (1 - decay_rate) * steps[i] ** 2
20 |
21 | pyplot.plot(steps)
22 | pyplot.ylabel('Absolute step')
23 | pyplot.xlabel('Iteration number')
24 | pyplot.legend(gradients, loc='best')
25 | pyplot.savefig('plot.pdf')
26 | pyplot.show()
27 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # The_evolution_of_gradient_descent
2 |
3 | This is the code for "The Evolution of Gradient Descent" by Siraj Raval on Youtube
4 |
5 | ## Coding Challenge - Due Date, Thursday June 8th at 12 PM PST
6 |
7 | This weeks coding challenge is to write out the [Adam](https://stats.stackexchange.com/questions/220494/how-does-the-adam-method-of-stochastic-gradient-descent-work) optimization strategy from scratch. In the process you'll learn about all the other gradient descent variants and why Adam works so well. Bonus points if you add a visual element to it by plotting it in a Jupyter notebook. Good luck!
8 |
9 | ## Overview
10 |
11 | This is the code for [this](https://youtu.be/nhqo0u1a6fw) videon on Youtube by Siraj Raval. In the video, we go over the different optimizer options that Tensorflow gives us. Under the hood, they are all variants of gradient descent.
12 |
13 | ## Dependencies
14 |
15 | * matplotlib
16 | * pyplot
17 | * numpy
18 |
19 | install missing dependencies with [pip](https://pip.pypa.io/en/stable/)
20 |
21 | ## Usage
22 |
23 | Run `jupyter notebook` to see the code that compares gradient descent to stochastic gradient descent run in the browser. I've also got 2 seperate python files, one for adadelta and one for the nesterov method. Run those straight from terminal with the python command.
24 |
25 | ## Credits
26 |
27 | The credits for this code go to [GRYE](https://github.com/GRYE) and [dtnewman](https://github.com/dtnewman/gradient_descent). I've merely created a wrapper to get people started.
28 |
29 |
--------------------------------------------------------------------------------
/nesterov.py:
--------------------------------------------------------------------------------
1 | import numpy as np
2 | import math
3 |
4 | def calc_numerical_gradient(func, x, delta_x):
5 | """Function for computing gradient numerically."""
6 | val_at_x = func(x)
7 | val_at_next = func(x + delta_x)
8 | return (val_at_next - val_at_x) / delta_x
9 |
10 |
11 | def nesterov_descent(func, L, dimension, init_x=None, numerical_gradient=True, delta_x=0.0005, gradient_func=None,epsilon=None):
12 |
13 | assert delta_x > 0, "Step must be positive."
14 |
15 | if (init_x is None):
16 | x = np.zeros(dimension) # todo проверка что функция определена в 0.
17 | else:
18 | x = init_x
19 |
20 | if (epsilon is None):
21 | epsilon = 0.05
22 |
23 | lambda_prev = 0
24 | lambda_curr = 1
25 | gamma = 1
26 | y_prev = x
27 | alpha = 0.05 / (2 * L)
28 |
29 | if numerical_gradient:
30 | gradient = calc_numerical_gradient(func, x, delta_x)
31 | else:
32 | gradient = gradient_func(x)
33 |
34 | while np.linalg.norm(gradient) >= epsilon:
35 | y_curr = x - alpha * gradient
36 | x = (1 - gamma) * y_curr + gamma * y_prev
37 | y_prev = y_curr
38 |
39 | lambda_tmp = lambda_curr
40 | lambda_curr = (1 + math.sqrt(1 + 4 * lambda_prev * lambda_prev)) / 2
41 | lambda_prev = lambda_tmp
42 |
43 | gamma = (1 - lambda_prev) / lambda_curr
44 |
45 | if numerical_gradient:
46 | gradient = calc_numerical_gradient(func, x, delta_x)
47 | else:
48 | gradient = gradient_func(x)
49 |
50 | return x
51 |
--------------------------------------------------------------------------------
/GD_vs_SGD.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": 1,
6 | "metadata": {
7 | "collapsed": false
8 | },
9 | "outputs": [],
10 | "source": [
11 | "%matplotlib inline\n",
12 | "import numpy as np\n",
13 | "import scipy as sp\n",
14 | "import matplotlib.pyplot as plt\n",
15 | "import random\n",
16 | "from scipy import stats\n",
17 | "from scipy.optimize import fmin"
18 | ]
19 | },
20 | {
21 | "cell_type": "markdown",
22 | "metadata": {},
23 | "source": [
24 | "### Gradient Descent"
25 | ]
26 | },
27 | {
28 | "cell_type": "markdown",
29 | "metadata": {},
30 | "source": [
31 | "Gradient descent, also known as steepest descent, is an optimization algorithm for finding the local minimum of a function. To find a local minimum, the function \"steps\" in the direction of the negative of the gradient. Gradient ascent is the same as gradient descent, except that it steps in the direction of the positive of the gradient and therefore finds local maximums instead of minimums. The algorithm of gradient descent can be outlined as follows:\n",
32 | "\n",
33 | " 1: Choose initial guess $x_0$
\n",
34 | " 2: for k = 0, 1, 2, ... do
\n",
35 | " 3: $s_k$ = -$\\nabla f(x_k)$
\n",
36 | " 4: choose $\\alpha_k$ to minimize $f(x_k+\\alpha_k s_k)$
\n",
37 | " 5: $x_{k+1} = x_k + \\alpha_k s_k$
\n",
38 | " 6: end for"
39 | ]
40 | },
41 | {
42 | "cell_type": "markdown",
43 | "metadata": {},
44 | "source": [
45 | "As a simple example, let's find a local minimum for the function $f(x) = x^3-2x^2+2$"
46 | ]
47 | },
48 | {
49 | "cell_type": "code",
50 | "execution_count": 2,
51 | "metadata": {
52 | "collapsed": false
53 | },
54 | "outputs": [],
55 | "source": [
56 | "f = lambda x: x**3-2*x**2+2"
57 | ]
58 | },
59 | {
60 | "cell_type": "code",
61 | "execution_count": 3,
62 | "metadata": {
63 | "collapsed": false
64 | },
65 | "outputs": [
66 | {
67 | "data": {
68 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEACAYAAACnJV25AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHBRJREFUeJzt3Xu4XFWd5vFvhRMCJBEIJIGQQLiI4AUC2BAgNAckzUXk\nIgo4clEcO4/Qoo6jiPpMh54eR+xpfRpoUdthgKEFW+1BkEBzkSASQRxI5A4RkAQkXA3Q9rQI7/yx\nVjyVQ1WdqlO79lq76v08z36yzzn77P2mcvI7u3577bXBzMzMzMzMzMzMzMzMzMzMzMzMzKzSNgLu\nAJYD9wKLG2wzCfgu8AhwO7BdWeHMzGx8Nol/DhEK9z6jvn468PW4fgJwRUm5zMysS5sA/xf4k1Gf\nv46RYj8EPFtmKDMzW9+ENrdZDqwBrgfuHPX1bYBVcf0PwFpgWlEBzcysM+0U9teBecBswpn520Z9\nvdbge9RlLjMzG6ehDrZdCywFDgPuq/v8amBb4Km4v02BFxt8/0pgx3GlNDMbXL8Cdipyh1sCm8X1\njYGfAEeM2uZ04MK4fiLNL55W/Sx+ceoAXVicOkCXFqcO0KXFqQN0aXHqAF1aXMxudBHo9GL21dmB\nO/2GsVoxWwM/BlYAPyf02JcA5wDvidv8T2ALwnDHTwKf6zSEmVkF7A/cljpEO8ZqxdwD7Nng839Z\nt/7vwPGFJTIzy46mA1sR7ufJXjsXTy1YmjpAF5amDtClpakDdGlp6gBdWpo6QJeWFrCP/YDbofZa\nAfvqK1XvsZvZwNK5oP+S6uCdfoPP2M3MxlaZ/nrZfMZuZhWkSaB/BU1NFaDTb/AZu5lZa3sBD0Ht\n5dRB2uXCbmbWWuXaMC7sZmat7UfFCnuZ3GM3s4pRDfQMaE7KEAmPPaasw5mZvZHeDHoidYhOv8Gt\nGDOz5vYHlqUO0SkXdjOz5ip34bRsbsWYWcXoflCj+bJKDZH4+C1lHc7MbH2aAfotaIPUQTr9Brdi\nzMwaOwC4rYoTf7mwm5k1diDh4ULWglsxZlYhWg6anzoFmdfOrMOZmY3Q5qCXQBNTJ8E9djOzQhxA\neLDGq6mDjIcLu5nZG/0p7q+3xa0YM6sI3Qk6IHWKKOvamXU4M7NAbwK9Eh6wkQX32M3MurQf8Auo\n/XvqIOPlwm5mtr4DgVtSh+iGC7uZ2foqX9jL5B67mWVOm8T++iapk9Rxj93MrAv7Aiug9rvUQbrh\nwm5mNsJtmA65FWNmmdNtoIWpU4xSeO2cA9wM3A/cC5zZYJthYC1wd1y+WFY4M7PiaGqG/XXoQe3c\nCpgX16cADwG7jtpmGLiqjX25sJtZxnQE6ObUKRoo/OLp08DyuP4K8AAwq8F2tU4PbGaWmXcBN6UO\nUba5wK8JZ+71DgSeI/wCWAK8tcn3+4zdzDKm5aB9U6dooGe1cwrwC+CYBl+bCqzrSR0OPNxkHy7s\nZpYpTY/PNx1KnaSBjmtnO3+JicAPgMuAKxt8/eW69WuBrwPTgBcabLu4bn1pXMzMUjsIuBVqf0gd\nhHDdcriXB6gBlwJfa7HNTEZ67HsDjzfZzmfsZpYpfRP0qdQpmii8di4AXif0z9cNZzwcWBQXgDMI\nQyGXA8uAZs8IdGE3s0xpJWi31CmayLp2Zh3OzAaVtgM9A8r1TnzPFWNm1qGDgR9D7fXUQYriwm5m\ng24gx68Xxa0YM8uMaqDfgHZInaSFrGtn1uHMbBBp93DhNGvusZuZdeBQ4F9ShyiaC7uZDbJDgetS\nh6gyt2LMLCOaDHoZNHr+q9y4FWNm1qZh4BdQeyV1kKK5sJvZoOrL/nrZ3Ioxs4zoYdAeqVO0Ieva\nmXU4Mxsk2h60JuNpBOq5x25m1oZDgev7aRqBei7sZjaI3F8viFsxZpYBTYxPS5qROkmb3IoxMxvD\nvsCjUHsmdZBecWE3s0HzbuBHqUP0C7dizCwDug+0d+oUHci6dmYdzswGgXao0DDHddxjNzNr4d3A\nNf06zHEdF3YzGyRH4v56odyKMbOENDXO5jg1dZIOuRVjZtbEIcDPoPZy6iC95sJuZoPCbZgecCvG\nzBLRhPjQ6h1TJxmHrGtn1uHMrJ/pnaD7U6cYJ/fYzcwaeA9wTeoQ/chn7GaWiFaAFqROMU5Z186s\nw5lZv9KOoKdBG6ROMk5uxZiZjXIs8EOovZY6SC7mADcD9wP3Amc22e484BFgBdDsGYI+YzezBHQb\n6LDUKbpQeO3cCpgX16cADwG7jtrmCGBJXN8HuL2scGZmrWlr0IugDVMn6ULhrZingeVx/RXgAWDW\nqG2OAi6J63cAmwEzOw1iZtYDRwPXQu33qYOUaaiDbecS2ix3jPr8NsCquo9XA7OBNV0ls3HSTMK7\nqpnADGAq4Tf+68BLhH+X3wAPQe2FVCnNSnIs8A+pQ5St3cI+Bfg+8AnCmXu9WlzqNZsSc3Hd+tK4\n2LipBrwNWBiXdxL+Te8jFO9nCcW8Rnh3tgOh4M8CdgkTIrEcuIVwLeUuX2Cy/qHNCY/BOy51kg4N\nx6WnJhKe5v3JJl//BnBi3ccP0rgV4x57YbQt6Augh0CPgb4Jeh9om1js29lHDbQd6DjQ+aB74wMI\nvgFaCOrk3ZxZhnQS6KrUKQpQeO2sAZcCX2uxTf3F0/n44mkPaXfQFaDnQReC5rdfyNva/w6gz4B+\nDnoSdA5odnH7NyuTfgD6cOoUBSi8di4gtFWWA3fH5XBgUVzWuQBYSRjuuGdZ4QaH3gy6Mhbb/1zO\nfNJ6B+gC0Aug74J26/0xzYqiyaC1oC1TJylA1rUz63B50lTQuaDnQJ8FbZQgw5T4y+Q38ZfLXuVn\nMOuUTgRdlzpFQbKunVmHy48WgB4FXRrG4qamjUEfj+8aLgt9frNc6f/0SRsGMq+dWYfLhyaCvhzP\nkI9KneaNNCX23p8H/bfwltcsJ9oU9FIcFdMPsq6dWYfLg6aDloKWgGakTtOaZoP+MY7KOTR1GrMR\nOqVPRsOsk3XtzDpcetoD9Hg8C67QLHQ6NBb3y8IvJrPUdA3og6lTFCjr2pl1uLR0COgZ0PGpk4yP\nJoP+B2Fq1KNTp7FBpmlxNEwJI8dKk3XtzDpcOjouFvU/TZ2ke9o/XvD9lnvvloY+Esav95Wsa2fW\n4dLQKaCnQhumX+hNoEtAD3topJVPN4DenzpFwbKunVmHK5+OjyNfRk+D3Cd0AuhZ0J9T6N2xZs1o\nBui3oE1SJylY1rUz63Dl0pGEeVl2T52kt/QWwhw0F/fhfzbLjv4C9J3UKXog69qZdbjyaEHsqe+d\nOkk5NDkOi1wB2il1Gutn+nmfDr3NunZmHa4cf3yobj/+8LWgGuiM+HcfTp3G+pF2jder+nFW0qxr\nZ9bhek+bgx4EfSx1knR0cGxB/cfUSazf6Eugv0mdokeyrp1Zh+stDYFuBH01dZL0tHMcMfPVat2I\nZfnSBNAq0DtSJ+mRrGtn1uF6S1+Khd2FDIg3kdwE+hFoSuo0VnU6GHR36hQ9lHXtzDpc7+hI0BO+\n3X40TQRdFC94+bWxLuhi0KdSp+ihrGtn1uF6Q9vHnvK+qZPkSTXQX8fWzPap01gVaXIcu75V6iQ9\nlHXtzDpc8TQEWgb6T6mT5E9nEOZ5n5c6iVWNTgJdkzpFj2VdO7MOVzx9IfbVJ6ROUg16Xxzff3Dq\nJFYluinc5dzXsq6dWYcrlt4Zi5QfBN0RHRhfN88QaW3QTvHnZVLqJD2Wde3MOlxxtHEcr35i6iTV\npL3ijUwfSJ3Ecqcvg/42dYoSZF07sw5XHP130D+lTlFtenvsuX8kdRLLlSbGE4BdUicpQda1M+tw\nxdBu8a1hP1+hL4neDPo16MzUSSxHei/oJ6lTlCTr2pl1uO5pA9Advl2+SNoOtBJ0duoklhtdCzo5\ndYqSZF07sw7XPX0cdItHwRRNs0D3g/4rntfdgPgL/7lwPWsgZF07sw7XHc0mPFTiLamT9CdNB/3S\nxd0CnQM6L3WKEmVdO7MO1x1dFu6gtN7RdNA9Lu6DThsSpud9W+okJcq6dmYdbvy0L2g1nsyqBJoR\ni/tfubgPKn0g3JQ0ULKunVmHGx9NIExiNSgXcTKgGYTH7bm4DyQtAx2TOkXJelI7LwLWAPc0+fow\nsBa4Oy5fbLJdPxb2U+JIGF8wLdUfi/s5qZNYmbQX6HEGb/rrntTOA4A9aF3Yr2pjP31W2DUl3kQz\nP3WSwaQZoPtAi1MnsbLoYtBnU6dIoGe1cy6tC/vVbeyj3wr7F+nPJ6JXiGa6uA8KTQe9CNoidZIE\nkhT2A4HngOXAEuCtTbbro8KuLeI42p1SJzHNBD3gm5j6nT4P+nbqFIl0XDuLeKL3XcC2wO+Aw4Er\ngZ2bbLu4bn1pXKroLOD7UFuZOojV1oAOAW4B/T+ofS11IiuaJgF/ARyWOklJhuPSc3NpfsY+2mPA\ntAaf75Mzds0CPQ/aJnUSq6dtQY+BTk+dxIqmj4QpBAZWklbMTGDdsLO9gcebbNcvhf1C0N+kTmGN\naAfC82VPS53EiqIJhGmwD0qdJKGe1M7LgaeA3wOrgNOARXEBOAO4l9BjXwY0GyXSB4VdO8Te+iBe\nwKkI7RxHK30wdRIrgo4G3Tng9yxkXTuzDtcefSvc0m5509tAvwG9P3US65Zu879j3rUz63Bj0+zY\nW/fZeiVod9Aa0FGpk9h4aX/QrwbwhqTRsq6dWYcbm/7OvfWq0Z8QHnwyKKMp+oyu8sVwIPPamXW4\n1jQT9AJ+MlIFaT/ClMoHp05indCe8VrJRqmTZCDr2pl1uNZ0Luj81ClsvHRgLO4LUiexdumHoI+n\nTpGJrGtn1uGa07TYW982dRLrhhbGtsw+qZPYWHy2PkrWtTPrcM3p86D/lTqFFUFHxguqe6ZOYq3o\nh/gh5vWyrp1Zh2tMkwhPa3lH6iRWFL0X9LT/TXOlPeLZ+qA8z7QdWdfOrMM1plNB/5I6hRVNJ8Zf\n2LukTmKj6SrQJ1KnyEzWtTPrcG+kGmgF6NDUSawXdCrhkYaeoTMbOoDwIA331teXde3MOtwb6RDC\nU3oG+VbmPqePgn4N2jF1ElMN9DPQSamTZCjr2pl1uDfSEjyZ1ADQoljcd0idZLDpONDd+DGTjWRd\nO7MOtz7tGi+w+S3hQNDHYgtg+9RJBpMmgh4G/VnqJJnKunZmHW59Oh/0V6lTWJl0eizuc1MnGTw6\nHXR96hQZy7p2Zh1uhKbE6QNmp05iZdMZhId1zE2dZHBoWry3YF7qJBnLunZmHW6EFoH+OXUKS0Uf\nBz0K2i51ksGgC0BfT50ic1nXzqzDBX8c4rgwdRJLSWfG4u5pJHpK8+LZeqNHadqIrGtn1uEC7R8v\n4vjK/MDTJwlzgbu494RqoJ+C/jx1kgrIunZmHS7QZaBPpU5hudAnY8/d49wLp1NAv8AP0WhH1rUz\n63CgGaAXQZunTmI50SLQqjAE1oqhGbEFs1fqJBWRde3MOhzo06CLU6ewHOlkwjNUPXKjELoCdG7q\nFBWSde3MOJxqoPvCXBVmjei4eJbp+dy7oqNAj+DZGzuRce3MOpzmxx82zwtjLegIwsM6DkydpJq0\nWWxrDadOUjEZ186sw+lboLNTp7Aq0EGxuB+eOkm1qBZbMH7EZOcyrp3ZhtPkeKfprNRJrCo0P84l\ndGrqJNWhk2O70y2YzmVaO4NMw+kU0I9Sp7Cq0S5xbpnPuYU3Fm1PeJj47qmTVFSmtTPINJyWgt6b\nOoVVkWaBfgk6z+Oxm9GGhHnWP506SYVlWjuDDMNpp9gv3TB1EqsqbQa6GfQ9PM1zA7qA8HBq3809\nfhnWzhEZhtNfg76aOoVVnSaBvhtvkZ+eOk0+dHKcomPT1Ekqrie18yJgDXBPi23OAx4BVgB7NNkm\ns8KuCbFH6ptOrACaEE8UHgO9PXWa9DQv9tXfkTpJH+hJ7TyAUKybFfYjgCVxfR/g9ibb5VbYF8Sr\n9L7wZQXSB2N778jUSdLR7Dhe/fjUSfpEz2rnXJoX9m8AJ9R9/CAws8F2uRX2C0GfT53C+pH2AT0J\n+szgnTjoTYSpr89KnaSPJCnsVwP71X18I9Bocp+MCrs2BD2Hn5RjPaM5oLtAl4Ompk5TDk0EXQf6\n5uD9QuupjmvnUAEHrcWl3utNtl1ct740LikcCjwItccTHd/6Xm0VaH/gfOBO0HFQuy91qt7REPAd\n4FXgDKhldCJXOcNx6bm5tG7FnFj3cQVaMboC9LHUKWxQ6EPxQuJJqZP0hjYAfSeerXvIZ/GStGLq\nL57OJ/uLp5oKWgvaMnUSGyTaDfQQ6OL+Gv6nIdCloBs9XUDP9KR2Xg48BfweWAWcBiyKyzoXACsJ\nwx33LDNc53QK6OrUKWwQaQroG3GYbR/MEKlNwv8lXQuanDpNH8ukdjaWSThdBzpx7O3MekXvBj0F\n+kp1WxfaArQMdEm4aGo9lEntbCyDcNoS9NtwpmGWkqbHaQgeBh2SOk1nNI/woO8ve/RLKTKonc1l\nEE4fDbd+m+VCR8bWzD+CGg06yIxOiheCP5A6yQDJoHY2l0E4XQ96X+oUZuvTZNC58d6Kz+V5EVKb\nx4ukD3uagNJlUDubSxxOW8TRML7IY5nSW0A/iLfjf5hspgLWkaDVoPP9/ycJF/YWh/9I6Gma5U7z\nQbfEPvbH0p3Ba1fQEsLzgA9Kk8FIXjtbS13YrwWdMPZ2ZrnQ/qCrCI/h+0vQtiUdd9c43v5Z0Kfw\n8wpSc2FvcuhpsQ0zJV0Gs/HS20F/D3o+Dtf9D6DNCj7GRNAxcVz6GtAXij+GjZMLe5NDfxj0z+mO\nb1YEbUyYFvhHoJcId3t+OrZuJo1jf1uDjo9j0Z8F3UqY/sB99Lx0XDvLHIOqko9Xf+hrgMugdnma\n45sVTZOBhcCfAfsCOxMedvMw4S7w54AXgd8BE4ENgS2BOcB2hDvENwbuAK4JiyfFy1THtXMACrs2\nA54AtoHay+Uf36wMmgLsArwZ2AmYBmwObEKYDuRV4HlgNeH/w3LgMc/CWAkJT4rHlugHSCeDfpjm\n2GZmXeu4dg7Ck8OPBq5MHcLMrB8lOGPXRnE0jJ8cb2ZV5TP2UQ4G7oHas6mDmJmVpd8L+1GA++tm\nZj1ScitGEwhzXu9c7nHNzAqV9cilsgv73qAHyj2mmVnh3GOvczRuw5iZ9VTZZ+z3gvYt95hmZoVz\nKyYeasc4kVEm81mbmY2bWzHR0cDVUHstdRAzs35W5hn7j0FHlXc8M7OecSsG9CbQy5561Mz6hFsx\nwCHAMqj9a+ogZmYp9GNhPwK4NnUIM7NBUEIrRjXQk77b1Mz6yKD32LU7aGUo8GZmfWHge+xHAEv8\nVBgzs9YOAx4kPE/xrAZf/xDwLHB3XE5rsp8yzth/Ajqs98cxMytN4bVzA8KDcecSHoi7HNh11Dan\nAue1sa8eF3ZtHp/cvnFvj2NmVqrCWzF7Ewr744SH4V5BuKuzXo08HrS6ELgVav+WOoiZWUpjFfZt\ngFV1H6+On6sn4DhgBfA9YHZh6ToT++tmZoNtPBdPR78tuBrYDtgduBG4pNtQndMEwrUAj183s4E3\nNMbXVwNz6j6eAzw5apsX6ta/DZzbYn+L69aXxqUIuwEvQe3RgvZnZpbKcFx6Zgj4FeHi6YY0vni6\nVd36scCyJvvq4cVTfQb0973bv5lZMj2pnYcDDxEuop4dP3cO8J64/iXgXkLRvwlodtdnLwv79aBj\nerd/M7Nksr4vp0fhtFGczXHT3uzfzCypgbzzdAFwD9TWpg5iZpaDfijsC4EbUocwMxtEvWrF3AVa\n0Jt9m5klN2g9dk0HrQVNLH7fZmZZGLge+7uAW6D2auogZma5qHphd3/dzCyhglsxqoGeAI2+YcrM\nrJ8MUo9dbwGt9tOSzKzPDVSPPbZh/LQkM7N6VS7sh+D+uplZUgWeWWsD0AugrYvbp5lZlrLuShRZ\n2PcAPVjc/szMsjUwPfaDgJtThzAzy1FVC/swxT2kw8zMxqmgVow2AL0ImlnM/szMsjYIPXbtBbq/\nmH2ZmWVvIHrsw7gNY2bWVBULuy+cmpllooBWjIZAvwXN6H5fZmaV0PetmD2A1VB7JnUQM7NcVa2w\nD+P+uplZS1Us7O6vm5llosseu4biY/CmFxPHzKwS+rrHvifwBNSeTR3EzCxnVSrsB+L+upnZmKpU\n2BcAt6YOYWZmI7rosWsC6HnQNsXFMTOrhH6dK0ZvBT1aXBQzs8roycXTw4AHgUeAsxp8fRLw3fj1\n24HtOg3RBrdhzMwKsgGwEpgLTASWA7uO2uZ04Otx/QTgiib76uaM/X+DPjr+7y/EcOLjd2M4dYAu\nDacO0KXh1AG6NJw6QJeGUwfoUuFn7HsTCvvjwKuEon30qG2OAi6J6z8A3tVpiDYsAH7ag/12Yjjx\n8bsxnDpAl4ZTB+jScOoAXRpOHaBLw6kDlG2swr4NsKru49Xxc822+QOwFphWSDoANBuYSmgHmZnZ\nGMYz3HH024JaG9t0Y3/gp1DL+sqwmVlVzAeuq/v4bN54AfW6uB3AENDsztCVhILvxYsXL17aX1ZS\nsCHgV4SLpxvS/OLphXH9RJpfPDUzs0wcDjxE+K1xdvzcOcB74vok4J8YGe44t+R8ZmZmZmbWqfcD\n9wGvEWZlbGasm59SmQbcADwMXA9s1mS714C743JlOdFayuFmsm6Mlf9DhGs4617z00pLNraLgDXA\nPS22OY/wd1tBeBpYTsbKP0wY8bbutf9iObHaMofwnIb7gXuBM5tsl+vr307+YTJ4/XcBdiaEbVbY\n27n5KZWvAJ+N62cBX26y3cvlxGlLkTeTpdBO/lMJ/zlzdAChWDQrjEcAS+L6PoRfrDkZK/8wcFVp\naTqzFTAvrk8htI5H/+zk/Pq3k3+YDl7/Xs3u+CDhbLeVdm5+SqX+pqtLgGMSZmlXLjeTjVc7+Ws0\nHl6bg1uBF1t8vf61v4PwLnBmr0N1YKz8kO9r/zThRADgFeABYNaobXJ+/dvJDx28/imn7W3n5qdU\nZhLelkJ40Wc02W4j4E7gZ6T/pZTBzWRdaSe/gOMIb6W/B8wuJ1ohGv39qpRfwL6EArQEeGvaOE3N\nJbzzuGPU56vy+s+lcf6OXv+hLgLcQHgLMdrngavHuU+NP07HmuX/Qgf7mEMo/NsDPya8jc1pFsrR\nr2evbyYr2uhsVwPfIZzRLyKcgeX0rqOVRu82Xk8RZJzuArYFfkcYKXclod2akynA94FPEM5861Xh\n9W+Vv6PXv5vCvrCL74XwG3NO3cdzgCe73GcnWuVfQyj6TwNbA8802e7p+OdjhKc77UG6wt7O67ma\n8MPxFOHfflPGfvtdlnbyv1C3/m3g3F6HKtDov99swr9DVdRfT7qWcK1mGuv/m6Q0kdBevIzGAxly\nf/3Hyp/V638zsFeTr7Vz81MqX2FkVMbnaHzxdDPCKBOALQnXFHbpfbSmqn4zWTv5699hHQssKyVZ\n++bS3sXT+eR18W6duTTPP5ORM969CddCclEDLgW+1mKbnF//dvJn8fofS+hn/RvhrPba+PlZwDV1\n2zW6+SkH04AbCcX6BkaGO+4F/ENc3w/4JaEA/RL4cMkZG6n6zWRj5f8SYTjYcuAm8moFXE44A/w9\n4Wf/NEK7aFHdNhcQ/m4raD0MOIWx8p/ByGu/jJFpRHKwgNBWWc7IcMDDqc7r307+nF9/MzMzMzMz\nMzMzMzMzMzMzMzMzMzMzMzMzM+sn/x/uyn9gmMfBYQAAAABJRU5ErkJggg==\n",
69 | "text/plain": [
70 | ""
71 | ]
72 | },
73 | "metadata": {},
74 | "output_type": "display_data"
75 | }
76 | ],
77 | "source": [
78 | "x = np.linspace(-1,2.5,1000)\n",
79 | "plt.plot(x,f(x))\n",
80 | "plt.xlim([-1,2.5])\n",
81 | "plt.ylim([0,3])\n",
82 | "plt.show()"
83 | ]
84 | },
85 | {
86 | "cell_type": "markdown",
87 | "metadata": {},
88 | "source": [
89 | "We can see from plot above that our local minimum is gonna be near around 1.4 or 1.5 (on the x-axis), but let's pretend that we don't know that, so we set our starting point (arbitrarily, in this case) at $x_0 = 2$"
90 | ]
91 | },
92 | {
93 | "cell_type": "code",
94 | "execution_count": 4,
95 | "metadata": {
96 | "collapsed": false
97 | },
98 | "outputs": [
99 | {
100 | "name": "stdout",
101 | "output_type": "stream",
102 | "text": [
103 | "Local minimum occurs at: 1.33342535085\n",
104 | "Number of steps: 17\n"
105 | ]
106 | }
107 | ],
108 | "source": [
109 | "x_old = 0\n",
110 | "x_new = 2 # The algorithm starts at x=2\n",
111 | "n_k = 0.1 # step size\n",
112 | "precision = 0.0001\n",
113 | "\n",
114 | "x_list, y_list = [x_new], [f(x_new)]\n",
115 | "\n",
116 | "# returns the value of the derivative of our function\n",
117 | "def f_prime(x):\n",
118 | " return 3*x**2-4*x\n",
119 | " \n",
120 | "while abs(x_new - x_old) > precision:\n",
121 | " x_old = x_new\n",
122 | " s_k = -f_prime(x_old)\n",
123 | " x_new = x_old + n_k * s_k\n",
124 | " x_list.append(x_new)\n",
125 | " y_list.append(f(x_new))\n",
126 | "print \"Local minimum occurs at:\", x_new\n",
127 | "print \"Number of steps:\", len(x_list)"
128 | ]
129 | },
130 | {
131 | "cell_type": "markdown",
132 | "metadata": {},
133 | "source": [
134 | "The figures below show the route that was taken to find the local minimum."
135 | ]
136 | },
137 | {
138 | "cell_type": "code",
139 | "execution_count": 5,
140 | "metadata": {
141 | "collapsed": false
142 | },
143 | "outputs": [
144 | {
145 | "data": {
146 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAADSCAYAAAC8Yk/kAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcHEX9//HX7G6yuS8SAiSQBSISIJwCQQIsh9ygiApy\nC8olKILIqQb9eqAo/BQVETEoigooh4T7voIgSYBwBgJJCAm5CSTk2s/vj6plJ8vMzuxsd1dPz/v5\neMxj5+jp/lTv1merq6urQUREREREREREREREREREREREREREREREREREREREREREZC1vAnv55xcC\nfwgURzMwM9C2RaS0N1GuiMoNwGdDB1GhZjre/0uBpjLW0wi8BAzpekgibY4EngLeB+YCE4HTIt7G\ndNqSYVSagBagrhPfaSa9yXA88MPQQYh0QLkiHcZTOldsDUyNP5TYNBPd/j8XuCyidSWmM3+skqxz\ngCuAS4Gh/nEqsCvQvch30vb7zIUOQKQGKFdUl1OA60MHkRI3AMcD3UIHItWvP+7I8bASy40HfgdM\n8MvvBRwETAKWADOA77f7zrHAW8B8XHd7/lHkOOAvecuOAZ4AFgGTgT3yPnsI+AHwGPAecDewjv9s\nBu4ocql/7Fwg9p4+/oW4o69zWfsoZgPgZuBd4A3gzLzPdgKe8WWcA/wi77OxeTHPwFVKcN3Cl/my\nz8Httx7+s2ZgFnA27mh9NnCC/+xkYCWwwpfl1gJlEQlFuaL6csXrwKfzXk/JK/9S3P7Y3X92qC/z\nIuBBYPO8743C7dtFwAvAIXmfjQd+i/t9L8Xt+6G4BvZC3CmybfOW72gfltr/7bUAm+TF8RvgP7jf\n/cS8z1q9mldekYrtD6yi9FHheGAxsIt/3YhLWFv616NxFb/1XPoWuEo0Fnck+gu/ndZk+H3akuEw\nXMLc37/ex79uTXgPAa8BI3FJ5UHgJ/6zEZTufv8p8DAwABiOq/gz/Gd1wP+Ai4EGYGNcstnXf/4k\ncLR/3ou2ZLsRrnIeAdQDg4Bt/GdXALf47fUBbgN+7D9r9vthnP/eAcAHuH9KAH/CJX6RtFGuqK5c\n0duXd50in58MvOi3uxmukbu339a5uP3YgOuhmQac71/v6cuzmV/PeGAesB3ud30/rkF0DK5374fA\nA37ZUvuwo/1fSPuG0wLgU74M1+N6mfLdytoNNZGKHAO80+691iOjZbhkBu6PcnyJdV0B/NI//x7w\nt7zPeuGOjgodRZ4H/Lnduu4CjvPPH8QdhbY6DbjTP2+idDLMr5gAX6PtKGZn3NFevguAa/3zh32s\ngwssc3OBbeVwCSj/SGcXXCIBlwyXtYt3Lu5oFVwy1BgnSSPliurKFcNw5S10CnWsX9dI//q7wN/b\nxTYL1+DdjY//3v9GW6/heOD3eZ+dwdrjqkbj/kag9D7saP8Xkt9w+hNwdd5nB+B6u/Jdjytr1WgI\nHYAUtABX0etwf4TQ1rU7k7ZKa7iKlG9n3BHClrjK2Qj803+2Qbvll/ltFTIC+CJrd/820HaUAu4I\ntdVy3FFSuTZg7cqXfwQzwn++KO+9euAR//wk3FHdS7jTB5cAd+COht7g44bgEv//8t7LsXbyW0Db\nvga3bzpTHpEQlCuqK1cs9j/7svb+3BD4B66xOc2/tz5rl9Vw+2EYsJqPN17ewu2L1mXfzfvsw3av\n838HpfZhR/u/HHOLbLdV33bbTr20DRAU50nc0d3nKvju33DdzMNxXatX0TbwcjaugrbqRfEu4xm4\nI8qBeY++wM/KiMHKWOYdXHd5q/znM3FJLn/b/YCD/efTgKNwSe5S4CZcWWYCmxbY1nxchd0ib30D\n/DrLUU55REJQrqiuXPEBrgfnk3nv9cT9Hi7Hjf9qNRvXqGmVw/1OZtH2+8kfVD8CeLvMOPOV2ocd\n7f8ojMKN86oaajil02LckdFvgcNxLfQ63GC+3nnLFboSpQ+u9b4S1318VN5nN+MqQ+vVNj+g+N/A\n9bgjyH1xRx89cN3Uw0psH9y59RYKJ6ZW/8R1B7eeN88/x/1f3Pn67+CSSj2wFe48ObjTE61zfyzB\nJavVuH8E++COfhtwiX4bH8sfcKciWr83jLW7nzsyl48PaBRJA+WK6ssVE1h78Py1uB6x9pfl/xM3\ngH8v3Jimc3A9R0/gyv0BrtzdcPv7YNpO7XXmKsVS+7Cj/V9KqTiG4caXTezEOoNTwym9fo67cuM7\nuMo4B3dE+B3cUSa4JND+COd0XJJ7D3fe+B95n00Fvo5LGrNxV0nkd8Hmr28WbqDohbgu3hm4iptr\nt3yh7y4DfgQ8jkvMO/Fxl+C6lqfjxkP8Oe/7a3CJeFtcd/o83Hny1qO+/XADFJfijtKOxCX/GcCB\nPs4FuCuGtvbfOQ939DkRl0DvpW0gZfuytPdH3BHoIuBfHSwnEoJyRXXliqtpG7AOboD651j7yrpd\ncVebHQP82pfrIF/W1b4Mh+LGDM0DrsRdBflqXozF9nn7cpTahx3t/0LK3S64xvp43ID7zOiBm1Rt\nMu6Pb1yBZRpxFe413B/aiALLiIiEoBwmafRXqnfm8Ki0zhzefuB+JvTyPxtwSaX9PBun47qJwbWc\n/46ISHooh4lIEK1XGuzY7v27aEtEDbhuPhGRtFEOE5EuK2eMUx2um3sucA/wdLvPh9F27ns17pzw\noKgCFBHpIuUwEYlMOfM4teAGjfUH/o2b8yN/Iq1Co+YLDRybRsdXTohItrxO22R+IUWRw5S/RGrP\nFNa+NU1Fvoe7CiHfXbj7FEHH3dxpmwtnXOgA2hkXOoB2xoUOoIBxoQNoZ1zoANoZV9nX7DGwvSON\nxK84hnV2VaU5LI1lidK40AEkYFzoAGI2LnQACRiX8PYK1vtSp+oG4+ZuADe/wz58fLr022i7OeIX\ncPfEEZGqYD1wR1RPhY4kJsphIhKpUqfq1geuw02IVYe7ZHcCbl6HZ4DbcfNW/AV3Ke8C3DwZIlId\ndgBegtz7oQOJiXKYiFSttHV1N4cOoJ3m0AG00xw6gAKaQwfQTnPoANpp7vxX7DywyyOPxK88pvWG\nkKWyFNIcOoAENIcOIGbNoQNIQHPC2wte74MHICLt2e1gh8e18pjWG0KWyiIi5Qle74MHICL5rA5s\nIdh6cW0gpvWGkKWyiEh5KhocLiLZNQpYCLk5oQMREakWajiJ1K6xwGOhgxARqSZqOInULjWcRERS\nTGMERFLFpoNtHucGYlx30rJUFhEpT/B6HzwAEWllw8HmgxW63UhkG4lx3UnLUllEpDwaHC4iH9kV\neAxyahCIiHSCGk4itUnjm0REUk5HtiKpYZPBdol7IzGvP0lZKouIlCd4vQ8egIgA2CCw98C6xb2h\nmNefpCyVRUTKozFOIgLA7sCTkFsVOhARkWqjhpNI7WkGHgwdhIiIdExd3SKpYJPBxiSxoQS2kZQs\nlUVEyhO83gcPQERsnYTGN0G26nyWyiIi5dEYJxFhd+BxjW8SEamMGk4itaUZeChwDCIiUgZ1dYsE\nZ1PAdk5qYwltJwlZKouIlCd4vQ8egEhts8FgS8AaktpgQttJQpbKIiLlqWiM04a4y5ZfBF4AvlFg\nmWZgCTDJPy6uOEQRiVPr+KbVoQNJkHKYiCRqPWBb/7wP8Aowqt0yzcBtZaxLR2wiQdmvwL6T5AYT\n3FYxUeWwNJRFRJJVUY/THGCyf/4+8BKwQYHlcpXHJSIJ2ZPaGxiuHCYiwTQBb+GO2vLtAczHJacJ\nwBZFvq8jNpFgbH2whWD1SW40wW2Vo4nKc1jayiIi8StY78sdJNoHuAn4Ju6oLd+zwEbAMuAA4BZg\nsyLrGZf3/CFq7+hXJJR9gAcgtybGbTT7RxpFkcPG5T1/COUvkaxpJqIc1g24GzirzOWnA4MKvK8j\nNpFg7M9gpyS90YS3V0wUOSwtZRGR5FRU73PAn4HLO1hmKG3jA3YC3owyABHpKsuBvQO2SdIbTnh7\nhUSVw9JQFhFJjA2kwno/FmjBnftvvVT3AOAU/wD4Ou4y38nAE0Cxm4cq8YgEYaPBXg+x4QDbbC+q\nHJaGsohIImwDsKmkoN4HD0CkNtnZYFeF2HCAbcYlS2URkaJsE3egaReSgnofPACR2mR3gh0eYsMB\nthmXLJVFRAqyrcBmgZ3e+kbQcNIQgEjtsR5gS/35+sQ3HmCbcclSWUTkY2wM2Fywo/LfDBZOWgIQ\nqT22F9jEUBsPtN04ZKksIrIW2wdsHthB7T8IEk6aAhCpPfYTsB+G2nig7cYhS2URkY/Y58HeBdu9\n0IeJh5O2AERqjz1TJCEksvFA241DlsoiIgDYiX6qlu2LLZBoOGkMQKS22FCwxWDdQwUQaLtxyFJZ\nRGqc5cC+DfYW2Cc7WjCxkNIagEhtsRPAbgoZQMBtRy1LZRGpYVYHdoWbp8k2LLVwoTfLvVediFSf\ng4A7QgchIpIO1gN3J4GhwFjILQocUEk6YhNJjHUDWwS2XsggAm47alkqi0gNsoFgD4P90zegyvpS\nrCFVQwAitcOawZ4OHUTg7UcpS2URqTG2IdgL/hRdXWe+GFtI1RKASO2wn4NdEjqIwNuPUpbKIlJD\nbDTYDLBz3KDwzn05lpCqKQCR2mEvgu0UOojA249SlsoiUiNsTz8b+JcrXUGk4VRjACK1wTb2yaIz\nXdKxBBJ4+1HKUllEaoCd6PPgnl1ZSWThVGsAIrXBvg42PnQUZKvOZ6ksIhlmdWA/BZtWYo6mslYW\nSUjVHIBIbbAJYF8KHQXZqvNZKotIRlkvsJvBHgEbHMUKI1hHdQcgkn3WG+w9sAGhIyFbdT5LZRHJ\nIFvfXUls14E1RrXSiNZTvQGIZJ8dDnZP6Ci8LNX5LJVFJGNsG3/l3EUVXDnX4YojXFd1BiCSfXY9\n2Gmho/CyVOezVBaRDLFDwd6NaXhC8HofPACRbLNuYAvBhoWOxMtSnc9SWUQywOrAvgs2E2znuDZS\nyZc2BB4EXgReAL5RZLlfAa8BU4DtogxARMplnwGbGDqKPGmo81HlsDSURUQAsD7uBub2pBvbFN+G\nKvnSesC2/nkf4BVgVLtlDgQm+Oc7A8UStxKPSDzWA86F+5+CF34ZOpg8aajzUeWwNJRFpBblgCNy\n8APgaHhvU7DnwK6JcBB4MZHU+1uAvdu9dxVwRN7rl3F3Ho4lABFZy7BeMP946lb0Zpb14JPLgN1C\nB+Wlsc5XmsPSWBaRzOsL124O738PrIl9l8PC5bDmjIgHgRfT5XrfBLyFO2rLdzvw6bzX9wE7xBFA\ntlgD2EiwMWD7g+0Htoe/r07f0NFJdegBvzwLVk1kJxvFVPsb2EAIfXPfVmmr801UnsPSVhaRWjCi\nDyxfAvZLzrKhzLY+7Lkc2Dyh7Res9w1lfrkPcBPwTeD9dp/l/CNfS5H1jMt7/pB/1AjrgzvS3RcY\nC2wGzAHmAYv9Qj2AwcAIsPdx/wAnAvcDT0Gu2H6VGtUdBm0KDf/mMA7j32zs3g41h1Ozf6RRFDls\nXN7zh6ip/CWSvF4w4Gh622n8nqlsyUR2YT/eWvkq9I9pk81ElMO6AXcDZxX5/CrgyLzXOlW3FhsD\ndi3YIrB7wb4N9ik3w2nR7+TAhoN9HuwysBfA3gG7CmynhLoopTocvDO5ZSN51W5he9sJPugNPw4d\nlJeWOh9FDktLWUQyz6DO4AvPsvVrGzO1ZQ/+1PIaPe1XsKY3zAV6JxdK5+WAPwOXd7BM/sDKMWhw\nuGe7gt0HNh3sXLBCjcnOrO8TYBeCvQE2Cewr7vJzqXXXsPuUQUxr6U1uQR+4gvJ7kuOWhjofVQ5L\nQ1lEMs0gZ/A5gynXcOK07ny4JMfz5/cj91QvWNIf/gd09f5znQyp88biuqwnA5P84wDgFP9odSUw\nDXcp7/ZRBlB9bD2wG8DeBDsp+saN1YHt63uv3gI7PYErCySlDD55Fr9c1pcll4WOpYA01Pmoclga\nyiKSSb7BdIjBs8tpnLQtz94KLW+AFWtPJBhajQcQPzsWbB7YTzo+FRfZ9sbgbuj6OthhOoVXe9aQ\nu34ACxeBbR06lgKyVOezVBaRVPANpgMNnjaYcif7nQQtT4Dd7q5zCS54vQ8eQHyslx/H9DLYNgG2\n/xk/DuoBsC2S376EYDDqUXZdWMfql1PaaM5Snc9SWUSC8g2mfQ0mGrxg8IVurNjPj+W90J1ZSYXg\n9T54APGwYWCTwf7ir5wLFUcD2Nd9j9dFGv+UfQZ/35t7HwP7fuhYishSnc9SWUSC8A2mvQ0eM3jJ\n4Ij9uLM72KVgs8D2DB1jO8HrffAAomej/Tij89JzxG8bgd3lB5BvW3p5qUYGo1dTNyfHmtlgSc1p\n0llZqvNZKotI4gz2MHjY4FWDow3qwTYGmwh2B9iQ0DEWELzeBw8gWrYT2Fywo0JH8nGWAzvB9z6d\nmZ5GnUTF4KYrOf03YFNCx9KBLNX5LJVFJDEGYw3uN5hmcJx9dNWvfQnsXbBvpfh/VPB6HzyA6NgO\nvtF0cOhIOmYjwf4H9m+wQaGjkWgYbGswuzsf/gHsgtDxdCBDdT5TZRGJncEuBvcYTDc40dx8avgx\nwVeDveb+l6Za8HofPIBo2GjfaPps6EjKY41gV/hTijuGjka6zuCWBQz8NtgCd2o2tTJS54FslUUk\nNgY7GkwweMvgZIPueZ9uAzYV7K9g/cJFWbbg9T54AF1nw8BmgH05dCSdZ5/3p+6ODh2JVM5gB4O3\nB7DwaLD7Q8dTQgbq/EeyVBaRyBlsb3C7wUyD0wzy5he0erDv+P9Bx6X41Fx7wet98AC6xvr6Adfn\nh46kcjYaN/P4z9wfslQbn5jOwM1zclzoeEqo8jq/liyVRSQyBtsY/NvgbYMzzd1zNX+JjcEeAXsY\nbESYKCsWvN4HD6BylgO7BewPVdRSLsLWcT0VNqFKukrFM9jJYOZuPLwR2GKCTn9Rliqu8x+TpbKI\ndJnBVgY3GrxjcJZBz3ZL5HC3BpuHu0drNR6sB6/3wQOonJ3rL5nsXnrZamDdcDcMngS2fuhopDwG\ndxqc6q9CuS50PGWo4jr/MVkqi0jFDEYZ/N1gjsG3reANd20I2L/cVb82OvkoIxO83gcPoDK2O9ic\nlA/CrYDlcBNlTgcbFToa6ZjBpw3edAMtbRLYXqFjKkOV1vmCslQWkU4z2MzgeoN3Dc4zKNLjbYeB\nzcZNalnt91ENXu+DB9B5NgQ3m+n+oSOJjx3vrxIcGzoSKc7gXoOvgm3tL1BIyy0JOlKFdb6oLJVF\npGwGIw2uM5hncJFBkSEeNgTsH2CvZOj/SfB6HzyAzrEc2E1gPw8dSfxsX9xEZFUyxUJtMdjd4A03\nD4r9GuwHoWMqU5XV+Q5lqSwiJRlsbPBHg/kG3zPoX2TJHNgR/szMz8F6Fl6uKgWv98ED6Bz7Mm6+\niR6ll80C28H/4adwJvTaZvCgwVdwE8elfe6mfFVW5zuUpbKIFGUwwuBqgwUGPzAY2MHS6/mxTFPB\ndk4uysQEr/fBAyifbeBPX6V9VtOI2VZgb4N9LXQk4hjs6W9V0OCvUPlP6Jg6oYrqfElZKovIxxgM\nN/itbzD9yGCdDpbOgR3r/0/+KANjmYoJXu+DB1A+uxXsktBRhGEjwd50V25JSObuJP6IwbH+nYlg\nh4SNqlOqqM6XlKWyiHzEYAODX/sG06UGJW62ayPB7gWbXAOdC8HrffAAymOHgr2c4RZ0GWwjsFfB\nvkfVz1tVvQz2MXjZ9zZtAzYTrCF0XJ1QJXW+LFkqiwgG6xlcbrDQ4BcGQ0t8oxHsu2Dzwc6uslxU\nqeD1PngApVkv3OX5e4eOJDxbD+w5sJ+o8ZQ839v0hIG/vY/9Fuz7YaPqtCqo82XLUlmkhhmsa3CZ\nbzBdYVDGXH62B9hLYLdV0RjLKASv98EDKM1+BPa30FGkhw3GTWD2YzWekmWwv8FUg3qwPmALwYaH\njquTqqDOly1LZZEaZDDY4Kf+lNyVBsPK+NZgsGt9b/dhNfh/oOJ6fy0wF3i+yOfNwBJgkn9cHHUA\nybDNcFPDaybttajxlDTf2/Rfgy/5d84AuzlsVBVJQ52vkfwlUpjBID/Ye4HB7ww2LONb9WCn+sHf\nV4D1jT/SVKq43u8GbEfHiee2OANIhv0L7LzQUaTTR42nH6nxFD+Dgw2eN6gDqwN7jeqcUC4Ndb5G\n8pfI2gwGGFzi52G62mBEmd8ci7s7wSNubGVN61K9b6LjxHN73AHEy8aCvUW2Ju6KmBpPSfC9Tc8a\nfN6/czDYM1W6z9NS55vIdP4SaWPQ309YOd/gWoNNyvzmMLDr/Wm5I6s050QttobTHsB8YDIwAdgi\njgDiYzmwJ8GODR1J+tlgP2D8/1Sp4mHwOYNJrrcJwO4HOzpsVBVLS51vIrP5S8Qx6Gtwoblbo1xn\nMLLMbzaCne+vlvuRG1MpXsF6H8XlhM8CGwHLgAOAW4DNiiw7Lu/5Q/4R2uFAD+CvoQNJv9x83M1l\nH3Cv7buug0Si4BtLlwDfzUEL2NbA5sCNYSMrW7N/VJNqz19S48zdbPfrwDnAvcDYHLxSxjdzuJ7t\nS4GXgDGQmxZfpFWhmQhzWBPFj9jamw4MKvB+Cv/BWj3uhoT7ho6kunzU81Qt90yrCgaHGzxj4Hvz\n7E9gF4aNqkvSUuebyGT+klpm0MvgHIM5Bn+34r2lhb69C9jjuEksPxNflFUvtlN1Q/ko0bMT8GYc\nAcTDjgZ7VKedKmFDwJ6n+uYWSiWDOoMXDA7074zA3Zeug/tEpV5a6nwTmcxfUosMehqcZTDb4CaD\n0Z349qZg//TjmI53nQfSgYrr/Q3AbGAlMBM4ETjFP8B1Eb6AGyPwBDAm6gDi8VFvkya7rJiti7u5\n43dDR1LtDI4wmJjX23Ql2E/DRtVlaajzGc1fUmsMehicYfC2wS0G23bi24PAfunHMV0E1iu+SDMl\neL0PHsDa1NsUDRuKm1H2otCRVCuDeoOXDPbz76yHm/CyxC0QUi9ldb5LslQWqSIGjQanGcw0uN2g\nE/eHs7603SbldxnIKUkLXu+DB9BGvU3RsvVx9/c7P3Qk1cjgaIPH83qbLgX7ddioIpGiOt9lWSqL\nVAGDbgZfM3jL4E5zp5LL/XZPsG/jJrC8HqzMK+ykneD1PngAbexIsMfU2xQl28A3Rs8NHUk1MWgw\neNXAN+JtkB/blIX7QaWozndZlsoiKeZzwlcM3jC4x2CXTny7O9jpYG+D3Qy2ZXyR1oTg9T54AI7l\nwP4HdkjoSLLHhuFmuT4ndCTVwuB4g4fzept+CnZ12Kgik5I6H4kslUVSyDeYjjOYZvCAuVnvy/12\nN7ATcTepvwvsU/FFWlOC1/vgATi2px+TUxc6kmyy4WDTwL4VOpK0813xr5ubhBF/ynMB1Xcz32JS\nUucjkaWySIr4MY5HGbxi8Ih1ah4h6wF2GtibYPeBdaKxJWUIXu+DB+DYHWBfDR1FttmGYK+DfSN0\nJGlmcKLB/Xnv/BbssnARRS4ldT4SWSqLpICfguQIgxf9GMe923qeS367F9hZ/pTcHWCdOJ0nnRC8\n3gcPwJ3vtXdcK13iZSN8t/EZoSNJI4PuBtMN/M17bRN/5cvgsJFFKgV1PjJZKosE5BtMh5u7kfdE\ng/060WDqC3Ye2Bzcjem3jzfamhe83gcPAOxadNl8gqzJdyGfFjqStDE42eCevHeuJ3uTiaagzkcm\nS2WRAMzdn+pzBlPM3SHgwE40mIb58Y/zwf4O1olJL6ULgtf7wAHYULBFYOuEjaPW2MZgb4GdUnrZ\n2uDnZZnRdrWMjfFd7n3DRha54EknQlkqiyTIN5gOMXjW3A28D+1Eg2k02HjcvG7/z+VTSVDweh+6\n4XRRhq5WqjK2KdgMsK+FjiQNDE43mOBf1YH9F+zYsFHFInjSiVCWyiIJ8A2mAwyeNnjO4DBzN/Iu\n9c0c2N5gd4LNBrsArND9EyV+wet9wACswf/j7sQU9RItG4m7P9KJoSMJydxtE2YZ7OjfOQHsyYxe\n5Rk86UQoS2WRGPkG074GTxpMNfhimQ2m3mAn426gPtXlSmuMP2LpQPB6H7Lh9FmwJ8JtXxz7BNgs\n11ioTQbfMLjNv+rnjyh3DBtVbIInnQhlqSwSA99g2svgMYOXDY40KOMmujaStvvI/dv3Nmly5nQI\nXu9DNpzuBjsm3PaljX3Sj+c5LnQkSTN3V/PZBtv5d64EuyZsVLEKnnQilKWySMQM9jB4yOA1g2NK\nN5isDuwA3FQC7/qB302JBCudEbzeBwrAPuH/MDUFQWrY5r7xdHLoSJJk8C2Df/lXY/0+GBg2qlgF\nTzoRylJZJCIGuxrc7yeyPd6gocQ3huNuuvsm2DP+VH3PJGKVigSv96EaTr9wrXlJFxsJ9gbYeaEj\nSYJBb4M5Blu7Rry9DHZ46LhiFjzpRChLZZEuMhhjcLefi+0kg24dLN0N7DDfu7QA7Ddo/qVqEbze\nBwjAGsHmgW2S/LalNBvmB0H+NOvn9A3ONbjRv/oR2M1hI0pE8KQToSyVRSpksKPBBD+dyMkG3TtY\neguwS3GTLj8MdixYr+SilQgEr/chGk5fBLu/9HISjq2Duxz/92BlDKSsPgZ9DeYabOVP0c0BWz90\nXAkInnQilKWySCcZbGdwm78i9nSDIle72fpgZ4M96y+E+Zkb1ylVKni9D9FwulODwquB9QV7ADcj\nbuYuvzW4wOAGsAF+bMMhoWNKSPCkE6EslUXKZLC1wb/8RR1nGhQYK2t9wY4Huwc3UeW1YHtl9UCw\nxgSv9wkHYMP8H7G6RquC9XCnr+xhMjTZm0E/g3kf0n1z3zC8MnRMCQqedCKUpbJICQZbGdxo8I6/\nqKPdAG7r5cct3QC2GOxWf4ZDA72zJXi9T7rhdIE7/SPVw+rALvMDpzcNHU0UDL5r8BewM8Cm1Fhi\nDZ50IpSlskgRBqMMbvCn1r9t0Dvv075gR4LdCLYE7D6wU8nWjbllbRXX+2uBucDzHSzzK+A1YAof\nzVETXQCdZzmw18DGJLdNiY6d5gdU7hI6kq4wGGAw/xBuPdqPa6q1ixTS0NiowvwlSTPYzOB6g3cN\nzjfo4z/yC0zJAAATwUlEQVQZAHac71FaAjYB7CQ1lmpGxfV+N1wyKZZ4DuSj+26xMzAx6gA6z8aC\nvZT1K7WyzQ7EXRF5VOhIKmUwbjJb3+gbgfuEjieANDQ2qjB/SVIMNjUYbzDP4KIFDOwHNgrsHD/u\n8j2wW3BXxA0IHa8krkv1voniiecq4Ii81y8DQ6MOoHPsGrDvJLc9iYdtA/Y62OVgHcyTkio9gcNG\nw1ffYcii7nz4OtiZoYMKJC2NjSaqKn9JxEYAxwAH4+dbMtjY4I8G8xfR/4cb8/rnwX6Nm1tuJthV\n7iIO693hmiXrYms43Q58Ou/1fcAOUQdQPmv0g8KHJ7M9iZcN9N3jD4OtFzqaEvr1gZd3hPfG02fF\nJjzVAtOuDh1UQGlpbDRRNflLIja2J7x/CCzdCpZuB/9bQe4Pk9l68df4/V2NLL/L9yo9CnY+2NY6\nUyF5Ctb7EtPDlyXnH/laiiw7Lu/5Q/4RtQOA5yA3K4Z1S+Jyi8AOBr4PPO26zHMPBQ6qoG7wrYOg\n6Rp6NX6OfzOc51nA17ZYEjqw5DT7RzVJW/6SCPWD8eOh97Y0cR97czd7b78O+2+1jH6zW6ifAdwP\nHAW5hYFDlXRoJsIc1kTHXd1H5r0O3NVt/wA7JZltSbJsf9z93S4F62DG3jD6wp+upq/txsN2Atfa\n09TZAHgzdFwBpaWXpomqyV/SFQaNK+i23b/43HkX8n/3HMQ1LRvzug3lHTuK6+0wTmyBXS4PHadU\njdhO1eUPrhxD0MGV1tdf+bBO/NuSMGyIv8LlWTeIMx0Mcpfxib9uz1N2Er+x5eTsi7C8H4wPHVtA\naWlsNFEV+Us6w2Cwwd4LGHjezRx258X84J2DuW3NQBasHsy772/PM5O25/wZJ7HlqlW4mWeHwgdA\nLV6oIZWpuN7fAMwGVgIzgROBU/yj1ZXANNzlvMVuXphEw+kYsNvj346EZTmwk8Hmg10CVmA23wSj\ngYH3sde96/P2yuFc+2wduVUNsLof3M1HlzXXpDQ0Nqoof0khBnV+uoAvLqfxx4+y6yNXcfLCU/nt\nitFMea+R5SvXZc6bmzDthgEsPLrd+NbB/eCpBljdACsb4exgBZFqFLzeJ9FwmlDNl69LZ9lw3Gzj\nr4a63H8lDbv9llPn9+G9Zf1Z1Pq310DBWzPUnOBJJ0JZKktqGfQ2GGNw6lJ6//5xdplyDScu/ypX\nL9mCFxZ058OVfVny9gAW3pJjzdlge4CVc3DSC9AtUKSzgtf7mAOwIbip72v5CL9G2cH+MuI7wLaN\nays5+NIg+E8/+Ou6sNWrjPzZIdy6vD+LpoONjmu7VSx40olQlsoSp1w3OH0Q3NkHrgGGFVrIIGcw\nzODAZfS4+El2nnAdx759MT9YtR93LtqAWYvrWbWyN0vf7MGyG/28SnuA9Uu4PFLbCtb7JC+7tHi3\nZ6cCe0Duy/FtQ9LLGoGTgYuAB4GfQm5KF1daD/QHFnWDU4bAL34CvRbTrWU6pzGe76+pp+WaBQw+\nG3IfdnFbWRRznU9UlsrSkTpgALCIChqLveHnI+C0i6D3FFh9JSzaEkY/SK+hExnTPJ2Nd53HkK3n\nMnTE62xaP5UtV81kw8ZeLFvQwOqp79HvqdV0ew6YCrwKuRURl0+kMwrW+yw1nO4FroLczfFtQ9LP\n+gBnAGcCrwD/D5gAuVUdfOlK3JiXHPByA8xrgJVrYPd6qKuHhTmwO2hc/1WO4eecS09mMJXLf7+a\nO0+NvUjVK0uNjbSV5VfASf7548CjwHJgHvA0sCcu3luAWcB6jXBOIwx5D/4N3Fpgnfv0gJuBHnWw\ndBkcBDxVKhCD3PNsNeRu9hl9I8/c9VU2bFjIhsxgI15iI+ayib3BJvRk+YoBLJ7TjVWvLqXv/+Yy\n9NkW6l8DpkHug67uEJEYZLnhZIOA6cAGqoDiWHfgC8DpwCeBm3D/LB5t9zcypSdsvS/wBm7ugPeB\nkcB84BZyPM+OTOBInuVItmMS5/Jz7uFhuwy7dDVckGSpqkzaGhtdYWBzgXc7eCwAFuc9lrkzUpF7\nsQ+M2heYhPs7XQY0AlvChy9A416wYiBwM6xcDvv2gtu+AoM2h4Yfw7L58J1V8Ju8dQ7pCdMnQO9m\nXEU5me6Lz+awwxpZb9gS+o98nz4jPqD3sPfot+779FnnPfr1W8yAXvMY0m0uQxnEwtWDeaNhFDNp\nYgYbMYOHeHvlA/T85WL++iPIvR/DvhCJU6YbTicAh0Lu8/GsX6qbNeHm6jkQd9XUZOA54MWeHPLr\ni1nIQbzPchr5GT2ZzYb0ZxNW8CmeY1eG8i4D+Kct4R8rLuOlHm8DZ8EHy9zl6y8EK1b6Za3htD6w\nbpHHUGAQ7jRX66M7sIS1G1OLcW3zZR08luOuAlzd7rEGVq9u4FNP/401fJIcH1LHWdQzjzqGUE8j\ndexIPYuo48vUcx+9Wp5kwMLh1A8cS4/65fRkLj15mj6r+zHqudU09FpOz17v073fMroNWMUAFvvH\nKrrRjyUtffhgRX+WfNCbD5b0ZPn8Hnw4txurZuWw6StofO1Nml56mVHTIbeqL/xhSzjqEug1BVq+\nD0uXwyjgndh/QyLRy3TD6TbgH5D7azzrl+yw3rgGz5bAlnX85+TtGMRK+tCdldSxnO7MpA9vMoJJ\nLORxfsg7bAMrDM7rB0e3wJJFcDFlnMaocRlrOHW2LNYdN0YuvzE1EOiNu8qr9dGz3eteuHuqNXz8\nsao7TN12SxrIYdSzhtW0AGvI0UIda+hJC6tZQ19aqGc5y1mxpj/L6oaxLNeLZdSxnCmsWD2Eze+s\no2VpDz5cuIq317zG3addzsLuI1jMUhazLctXrMCG4XrSytXQCy7uCQevgXcWwznAq53bbyKpETyH\nxXRVivX19xrSnaul0/q7qwpsNdh0N0GeAdYLbF1346rlfeGDbnB86FirUJauREtNWfr6qfNbwCa5\nQX1WB7YZ2L5gO4GdBzYLbDv4oBF+0Qvevx7sKbBd4YO+bsb0tfSGH64DHxwE7/WHD3rCN0OUTyRF\ngtf7uBpOR7j5m0Qq8uV+YPVgDa6LwBqgpTfMqIef4E7xpWaG8ioTPOlEKE1lObNv3t9rnf/ZG1rq\nYFVvmNTdNZiW9YYrcFfKfXog/HcgTOsDP8f1aBXyKeDLwNZJFUYkxYLX+7gaTv8A+2o865YashOw\nGe6fjEQjeNKJUBrLsi1unqQeuLHh9WiSR5EoBa/3MQRgPf2kl0OiX7eIdFHwpBOhLJVFRMpTsN5X\n+9H1Z4BJkJsXOhARERHJvmpvOB2Km+BNREREJFMi7uq2OrB3wDaNdr0iEpEsnd7KUllEpDzB633U\nDadPgb0Y7TpFJELBk06EslQWESlP5sY4HQz8J3QQIiIiInGIusfpGbDdo12niEQoS700WSqLiJQn\neL2PMADbAGyBmwNORFIqeNKJUJbKIiLlydSpugOBuyG3OnQgIiIiUjuqteGk8U0iIiKSSvsDLwOv\nAecV+PwEYB4wyT9OLLKeiLq6rQfYErBB0axPRGKSltNbUeSwtJRFRJJTUb2vB6YBTbibQk7m4zc8\nPR74VVwBFFjN/mCPRrMuEYlRGhobUeWwNJRFRJJV0RinnXBJ501gFfB34LPtlsn5R1IOAu5IcHsi\nUr3SmMNEpIqVajgNA2bmvZ7l38tnwOHAFOBGYHhk0RW2L3B3zNsQkWxIYw4TkSpWyeX87buubgf+\nhjuaOwW4Dti7yHfH5T1/yD86s+kmYAAuwYlIujT7R9pVmsPG5T1/iE7nLxFJuWYiyGFjgLvyXl9A\n4cGVreqBxUU+i2CMgJ0Mdn3X1yMiCUjDuKCoclgayiIiyaqo3jcAr+MGVnan8MDK9fKeHwY8EWUA\n7VZxE9hxXV+PiCQgDY2NqHJYGsoiIsmquN4fALyCG2B5gX/vEuAQ//zHwAu4hHQ/sFnUAfivN4At\nBFu/a+sRkYSkpbERRQ5LS1lEJDnB631XG067gGlsk0j1CJ50IpSlsohIear+liv7AveEDkJEREQk\nCV3tcXoc7DPRhCIiCchSL02WyiIi5Qle77sQgA0AW+putyIiVSJ40olQlsoiIuWp6lN1ewGPQ+7D\n0IGIiIhI7aqWhtNngHtDByEiIiKSlK6cqnsFbNvoQhGRBGTp9FaWyiIi5Qle7ysMwDYAWwBWLb1j\nIuIETzoRylJZRKQ8VTvGaU/gYci1hA5EREREals1NJz2Ah4IHYSIiIhIkio9VfcG2JbRhiIiCcjS\n6a0slUVEyhO83lcQgI0AmwuWiz4cEYlZ8KQToSyVRUTKU5VjnPYEHoSckpaIiIgEl/aG017Ag6GD\nEBEREUlaJ3uNLAc2E2yzeMIRkZhlqac4S2URkfIEr/edbTiNBJul8U0iVSt40olQlsoiIuWpujFO\nGt8kIiIiqZLmhlMz8FDgGERERESC6Oypuhlgn4gnFBFJQJZ6i7NUFhEpT/B634kAbATYHI1vEqlq\nwZNOhLJUFhEpT8VjnPYHXgZeA84r8Hkj8A//+URgRIUB5hsLPBbz+KbmGNddiebQAbTTHDqAAppD\nB9BOc+gA2mkOHUBKhchh1aY5dAAJaA4dQMyaQweQgObQAUDphlM9cCUu8WwBfBkY1W6Zk4AFwCeA\ny4FLI4hrN+DRCNbTkeaY199ZzaEDaKc5dAAFNIcOoJ3m0AG00xw6gBQKlcOqTXPoABLQHDqAmDWH\nDiABzaEDgNINp52AacCbwCrg78Bn2y1zKHCdf34zsHcEce0GPBbBekSktoXKYSKSUaUaTsOAmXmv\nZ/n3ii2zGlgCDKo8JFsH2BCYUvk6RESAIDlMRLKsoYLvtB93VGgAd6GxSa8Xeb/Y6lZ1JqgKfT+B\nbXSG4iktbTEpnuJeDx1AEZXksDLzV1VL099OXLJexqyXD5ItY8EOnFINp1m43p9WGwJvF1hmI2C2\nX19/YFGBdY0sK0wRkehElcOUv0SkLA24I60moDswmY8PrDwd+J1/fiRuDIGISBooh4lI4g4AXsEN\nsLzAv3cJcIh/3gj8k7ZLeZsSjk9EpCPKYSIiIiIiIlnxRWAqsAbYvoPlSk1MF6VBwL3Aq8A9wIAi\ny60BJvnHLTHEkbbJ+ErFcwIwj7Z9cmLM8VwLzAWe72CZX+HinQJsFzieZtxVWK375+KY49kQeBB4\nEXgB+EaR5ZLaR+XE00yy+6hSpX7XR+P25xTgcWDrhOKKSjl1C2BHXB78fOwRRa+cMjbj/g5foPru\nh1qqfP2B23GnpF/A5e9qkrb8lqjNgc1wO6BYw6ke13XeBHSj8NiDKP0M+I5/fh7w0yLLLY0xhnLK\nfDrwW//8COIdb1FOPMfj/kiTshuuIhRLDAcCE/zznXGNy5DxNAO3xRxDvvWAbf3zPrhTUO1/Z0nu\no3LiaSbZfVSpUr/rXXD/mMAdcMT9txe1UuUDlxMeAP4DHJ5EUBErVcYBuIP64f714CSCilCp8l0I\n/MQ/H4yb2LWSq+dDSVt+K6icW65U4mVcz05HypmYLkr5k9xdB3wuxm0Vk7bJ+MqJJ0fhy7Xj8iiF\nr8pslb9/nsIlwqEB44Fk988cXAMX4H3gJWCDdsskuY/KiQeS3UeVKvW7fhLXcwZuvw7vYNk0Kudv\n+UzgJlwvczUqVcajcHl1ln89P/aIolWqfC1AP/+8H67htDruoCKUtvxWUFwNp3KUMzFdlIbiujjB\n/XLWLbJcD+BpXJKMuiGXtsn4yonHcEeeU4AbCf/PolDMIWMyXE/EZNxR0BYJbrsJd/T5VLv3Q+2j\nYvGE3EdxOYm2o96sGIbLeVf511mct+oTuHz6IPAMcGzYcCJ3Ja5+zcbl7G+GDadLmkhXfvtIV7rw\n7sV1q7V3Ie4cayW6WlGLxXRRJ9axIa5htTGuy/p54I0uxtWRSicUjUv7bd0O/A3XI3UKrqUf8pYU\nhXrAWkIE4j2LmwNoGe7qrVtwp6nj1gfXM/BN3JFZvhD7qKN4Qu2juOyJG+u3a+hAInYFcD7ubyXp\nnuakdMMNH9kb6IU7QJ6IGy+TBfvj6tuewKa4/4nbEO8QlDikLb+tpSsNp890cdvlTEzXWR3FNBfX\nqJoDrA+8W2S5Of7ndNzAwe2IruEU5YSiScWzMO/5NYS/AWr7mIfj9lUo+QnpTtz4tEGsvd+i1g13\nuuF6Cl/AkPQ+KhVPiH0Ul62BP+D+QcVVL0PZgbYxlYNxjdxVVMf4tHLNxJ2GXO4fj+AaFllpOJ1A\n2xin13H/xz6J612rFmnLbx+TxKm6Ykctz+C6TZtwE9MdQbwV9DbcQGf8z0K/kAG4q9rAJY5dcQMJ\no1JOmfPj/AJwf4TbrySe/B68Q3FXO4R0G3Ccfz4GWEzbKdgQhtL2N76Tfx5ngyAH/BH3e7iiyDJJ\n7qNy4kl6H8VlI+BfwDG4sYFZswmup31j3NH+aWSr0QRwK26AdT2ux2lnwue0KM0A9vHPh+IaTXGe\nMYla2vJbog7DteyX43pw7vTvbwDckbdcoYnp4jIIuA83aP1e2qYj2AF3BAnwaeA53FiM54CvxBBH\n2ibjKxXPj3GXhU7GNeLiPsVyA+7oYSXub+hE3CnCU/KWudLHO4WOp7tIIp6v07Z/nsBV5DiNxXVL\nT6bt8v4DCLePyokn6X1UqVK/62twg21by/nfADF2RTl1q9WfqM7pCMop47dxB8TPU/xy97QqVb71\ngbtx/7+exw2GryZpy28iIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiISof8Pgh4O\nvVOCOC8AAAAASUVORK5CYII=\n",
147 | "text/plain": [
148 | ""
149 | ]
150 | },
151 | "metadata": {},
152 | "output_type": "display_data"
153 | }
154 | ],
155 | "source": [
156 | "plt.figure(figsize=[10,3])\n",
157 | "plt.subplot(1,2,1)\n",
158 | "plt.scatter(x_list,y_list,c=\"r\")\n",
159 | "plt.plot(x_list,y_list,c=\"r\")\n",
160 | "plt.plot(x,f(x), c=\"b\")\n",
161 | "plt.xlim([-1,2.5])\n",
162 | "plt.ylim([0,3])\n",
163 | "plt.title(\"Gradient descent\")\n",
164 | "plt.subplot(1,2,2)\n",
165 | "plt.scatter(x_list,y_list,c=\"r\")\n",
166 | "plt.plot(x_list,y_list,c=\"r\")\n",
167 | "plt.plot(x,f(x), c=\"b\")\n",
168 | "plt.xlim([1.2,2.1])\n",
169 | "plt.ylim([0,3])\n",
170 | "plt.title(\"Gradient descent (zoomed in)\")\n",
171 | "plt.show()"
172 | ]
173 | },
174 | {
175 | "cell_type": "markdown",
176 | "metadata": {},
177 | "source": [
178 | "You'll notice that the step size (also called learning rate) in the implementation above is constant, unlike the algorithm in the pseudocode. Doing this makes it easier to implement the algorithm. However, it also presents some issues: If the step size is too small, then convergence will be very slow, but if we make it too large, then the method may fail to converge at all. \n",
179 | "\n",
180 | "A solution to this is to use adaptive step sizes as the algorithm below does (using scipy's fmin function to find optimal step sizes):"
181 | ]
182 | },
183 | {
184 | "cell_type": "code",
185 | "execution_count": 6,
186 | "metadata": {
187 | "collapsed": false
188 | },
189 | "outputs": [
190 | {
191 | "name": "stdout",
192 | "output_type": "stream",
193 | "text": [
194 | "Local minimum occurs at 1.33333332845\n",
195 | "Number of steps: 4\n"
196 | ]
197 | }
198 | ],
199 | "source": [
200 | "# we setup this function to pass into the fmin algorithm\n",
201 | "def f2(n,x,s):\n",
202 | " x = x + n*s\n",
203 | " return f(x)\n",
204 | "\n",
205 | "x_old = 0\n",
206 | "x_new = 2 # The algorithm starts at x=2\n",
207 | "precision = 0.0001\n",
208 | "\n",
209 | "x_list, y_list = [x_new], [f(x_new)]\n",
210 | "\n",
211 | "# returns the value of the derivative of our function\n",
212 | "def f_prime(x):\n",
213 | " return 3*x**2-4*x\n",
214 | "\n",
215 | "while abs(x_new - x_old) > precision:\n",
216 | " x_old = x_new\n",
217 | " s_k = -f_prime(x_old)\n",
218 | " \n",
219 | " # use scipy fmin function to find ideal step size.\n",
220 | " n_k = fmin(f2,0.1,(x_old,s_k), full_output = False, disp = False)\n",
221 | "\n",
222 | " x_new = x_old + n_k * s_k\n",
223 | " x_list.append(x_new)\n",
224 | " y_list.append(f(x_new))\n",
225 | " \n",
226 | "print \"Local minimum occurs at \", float(x_new)\n",
227 | "print \"Number of steps:\", len(x_list)"
228 | ]
229 | },
230 | {
231 | "cell_type": "markdown",
232 | "metadata": {},
233 | "source": [
234 | "With adaptive step sizes, the algorithm converges in just 4 iterations rather than 17. Of course, it takes time to compute the appropriate step size at each iteration. Here are some plots of the path taken below. You can see that it converges very quickly to a point near the local minimum, so it's hard to even discern the dots after the first two steps until we zoom in very close in the third frame below:"
235 | ]
236 | },
237 | {
238 | "cell_type": "code",
239 | "execution_count": 7,
240 | "metadata": {
241 | "collapsed": false
242 | },
243 | "outputs": [
244 | {
245 | "data": {
246 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3cAAADfCAYAAABVlCoIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XecJFW5//FPT9jM7gILS97hEiRHyQgNChKUKIIIF0QB\nQQSVe8nKiD8FAxdERLwoIEhQoqAgyV0ECYKw5LSkJe3CAosIeIHd5/fHOUPX9HbP9MxU1ak6832/\nXv2a7q7qqufpmXOmTtWpc0BERERERERERERERERERERERERERERERERERERERERERERERERERERE\nRERERERERERkwJ4DtvbPjwPOCRRHFXgh0L5FRAZqGvDlJsuOJVxdKiLlNA3VKSLR2wu4G/gXMBu4\nCzgk5X08S61xl5YuYD7QNoDPVClu4+584HuhgxCRQpkKHBA6CBGJhuoUydRADsolG0cCpwM/BCb7\nx1eBzYARTT5TtN9bJXQAIiIiIpKJoh13ihTWBNzVul37We984BfAdX79rYEdgfuBt4CZwIl1n9kX\neB6Yg+uGmbxy1w1cmFh3Y+AO4E1gOrBlYtk04CTgduCfwA3Aon7ZTNyVu7f9Y6MGsY/28b8BPAL8\nN72v3C0FXAG8CjwDfD2xbEPgXp/jLODUxLLNEzHPBPbz748EfuJzn4X73kb5ZVXgReBbuCukLwP7\n+2UHAe8D/+dz+UODXERkaPakVl+8jStvU/2yCcAFuLrgOeB4aieOKsAJ/v3ZwG+A8X5ZF64e2h9X\nF7wBHAxsADyIqyN+VhfHAcCjft0/A8sllm0DPA7M9Z+bRvMuVN3U6tKeOP4TV/+8hqt7RSQ7qlNq\nzgfOwh0rvo07bpuMu4DwBvAYsE5i/VV9LG8CDwOfrdtW/XFnX8drIuJtB3xA/2dEzsdVCpv41yNx\nDbDV/es1cQ2Znf3r1XAFe3Pc1b9T/X56GncnUqs8lsY1ALfzrz/lX/c04KYBTwEr4hpJU4GT/bIp\n9N8t8xTgVmAisAyuApnpl7UB/8BVsB3A8sDTwLZ++Z3AF/3zMdQaj8vhGpp7Au3AIsDaftnpwNV+\nf+OAa4Af+GVV/z10+89tD7yD+wcAcB6uISsi2VsIdzB0oH99AXAVMBZXtzxBrevSAbh6qMsvv8Kv\nD7UDoLNw9d02wL+BK4FJuAOS2cAWfv1d/LY+hquDjgf+5pdNwp1M2g1XR3wDV2c060KVrEt74vgl\nro5ey8exSitfhogM2XCvU87HNQDX9evfgmuE7YNrzH4P+ItftxOYARyDO/7aCndctXJiW8njztH0\nfbwmIt4+wCt17/VcjXoX1zgDV8jO72dbpwP/459/B7g4sWwM7mxWoyt3R1Or0Hr8GXemCFxjLnmm\n6BDgev+8i/4bd/WF/0BqV+42wp2NSjoWONc/v9XHOqnBOlc02FcFd4bpPxLvbYKr3MA17t6ti3c2\n7gohuMad7rkTyV4b8Efg5/51OwsetBxE7Qz8Lbju6j1Wxl1pb6NWDy2ZWD4H2CPx+nLgcP/8enof\nWLXhTvIsh6v37qiL9QWaH4h1s+CB2FKJ5XfjTkKJSLZUp7hjmF8mXh+G6zHVY03c8SXAJ1jw+PNi\nar3Azqf3cWd/x2tSIOpDG9bruIZL8vewKbCwX9bzvrHgICQb4SqpV3FnVw6mdrVtKVz3wx7v+u01\nMgVXYb2ZeGwGLJFYZ1bi+Xu4K2KtWqou9pmJ51P88uS+jwUW98u/jKtwHwP+juuKCu4K4DMsaDFc\nQ/Yfie1dT+/G4eu4yrLHuwPMR0SG7vu4s+U9B0eTcGfIkwcPM3E9C8AdZNUv68B1OeoxO/H8vQav\ne8r5FOCn1OqInrpxab+fZN0JAx8AKllfvovLU0SypTrFeTXx/N91r5Mx1x+bgfs+ehqSRu+4+zte\nkwLpCB3AMHcn7oraLrjL/QNxMXAG8Gnc2abTqDXuXsb1pe4xJrGs3kzcWaKDBrh/cIW/P6/gzl49\n5l8n+6G/gLsXcOX6D3kzgL39891xZ8oW9Z/bsMH6c3CV12oseEaqFa3kIyJDsxfuzPMGwDz/3hxc\nV6UuetcVPQcXL/tlJJZ9iDvYStYprZiJu0J/SYNlKwHLJl5X6l7XU50hEp7qlIF72cdRSexzCu7e\nwEaxzKTv4zUpEF25C2su8F1cv+7dcWdU2nA3vCbPzDQajXIc7szJ+7iGzt6JZVcAn6E24uZJNP9d\n/xZ3E+22uG4Mo3DdF5dOrNNsNMzXcFfBVmiyHOD3uLM7PffcJW/A/Tuuj/dRuP7c7cAawMf98n1w\nV+PA9Vk3XOV7Me7ewD1wJygWxd1zNx83P8zpic8tTet9wmfTu0uniKRrXdyAArvSuzfBPFxd8X1c\n3TYF+CaufgJ30PRN3MHYONx9tJfS+yp8f3rqsbNxXc1X868nUOtudR3uXuZdcXXL4fTuxdBsm63s\nV0TSpzplYJ/tcTeu6+hRuPvvqrjjxkubbKu/4zUpEDXuwvsxbvTGo3CNi1m4iuIo3JU9cI2a+rM5\nh+Iabf8Evg38LrHsEeBruEbQy7hRkpKX35PbexE3EMtxuMv3M3HTM1Tq1m/02XdxFeffcA3NRlfT\nvou71P8s7l6+CxKfn4drWK6D62b5GvC/1Eas+jRuAJa3cVcm98I1ZmcCO/g4X8eNGrqW/8zRuCt+\nd+EahDfR+0xTX2fFfo2rnN9k4FdSRaR/O+FO9NxObXS7P/llX8cdbDwD3AZchLuHBNx9HRcCf/XL\n36X3iaJWznb3rHM1buqZS3F1xEO4ugZq99Wc4p+v6GPta5v19WOz/YpI+lSn9P3Z+nV7Xr+P++62\nxx17nYkbZf3JJp+dT9/HaxKZibjuco/hRinaOGw4IhK5UbizjtNxjf/uBuuMxJ3weArX0J+SV3Ai\nMmypbhKRKPyG2qg/HdSGlRcRycoY/7MDd4BUP8fiobjuzuDuxbgUEZHsqW4SkVIbT+NRC0VE8tAz\nOuoGde//mdpBVQeuC4mISF5UN4lIEEO95+4/cBXTecB9uMEsxvT5CRGRoWvDdX2aDdwI3FO3fGlq\n95l+iLsPYpHcohOR4Up1k4gENdSpEDqA9XATJd6DG6XwGNwk2j1m0PdoiiJSPk/jbgwPZT7uxu4J\nwFW40ciSk7U2GjWs/sZy1U0i8XkAVzeEkkbdBKqfRGIU+tipJUvgRkHssTnwx7p1ijRSWHfoABK6\nQweQ0B06gDrdoQNI6A4dQEJ36AASilSuv4MbOTXpz9QGd2rW9alIOaStO3QAGeoOHUBGukMHkKHu\nHPdVpHI92LoJipVHXrpDBxBId+gAAugOHUAguZTroXbLnIXrXtAz1Pyn6H2GSkQkbZNwo/SCm2/n\nU9Qmqe1xDbCff/454JZ8QhORYUx1k4gEN9RumeDmBbkIN1n208CXUtimiEgzS+JG6W3HnaD6HW6i\n2O8C9wLX4uYsvBA33PjruDkSRUSypLpJRIaFInUtqIYOIKEaOoCEaugA6lRDB5BQDR1AQjV0AAlF\nKteDFUMOzVRDB5ChaugAMlINHUCGqjnuK5ZyHUseA1ENHUAg1dABBFANHUAg0ZTraBIRkY/EUK5j\nyEFEeoulXMeSh4jUlOKeOxERERERESkANe5EREREREQioMadiIiIiIhIBNS4ExERERERiYAadyIi\nIiIiIhFQ405ERERERCQCatyJiIiIiIhEQI07ERERERGRCKhxJyIiIiIiEgE17kRERERERCKgxp2I\niIiIiEgEOlLYxnPAP4F5wAfAhilsU0RERERERHL2LLBIH8str0BEJA+2D3GU6xhyEJHeYinXseQh\nIjW5lOu0umVWUtqOiBTfoaEDEBEREZEFpdG4M+BG4F7gwBS2JyKFZWsAU0JHISIiIiILSuOeu02B\nWcBiwE3A48Btdet0J55P8w8RKY+qe+y7Hbw/A1gqbDgiIiIikrUTgSPr3lO/cZEo2GiwOWBdxFGu\nY8hBRHqLpVzHkoeI1JTinrsxwEL++VhgW+ChIW5TRIppd+BeqDwXOhARERERSd/ywHT/eBg4tsE6\nOvskEgW7FWz3nhcBA1kWmAo8iqt3Dm+wThV4C7jfP05osI7qJpH4xFA3geonkRhFU66jSURk+LJV\nwGaBdfa8ETCYJYB1/PNxwBPAqnXrVIFr+tmO6iaR+MRQN4HqJ5EYlaJbpogMD18BzofKB6EDwQ3g\nNN0//xfwGI0HeNEULSKSJ9VNIjIs6OyTSKnZKLBXwVZMvhksnN66gOdxZ8mTtgTm4A60rgNWa/DZ\nouQgIukpSrnuYvB1ExQnDxFJTy7lOo2pEEQkbnsC/4DKjNCB1BkHXA4cgTtLnnQfsBzwLrA9cDWw\ncoNtdCeeT0PTtIiUTdU/iiSNuglUP4mUXZXi1U+p0NknkdKyCti9YDvWLwgSTk0ncAPwjRbXfxZY\npO690DmISPpCl+s06iYIn4eIpC+ach1NIiLDj20M9jRY/f25Ict1BbgAOK2PdSZTu69lQ+C5Buuo\nbhKJTwx1E6h+EomRumWKSHCHAT+HyvzQgSRsBuwDPIgbShzgOFxXJ4BfAp8DDgE+xHV/2ivnGEUk\nd7Z+4LFKVDeJyLCgs08ipWSTwd4EW7jRwtzDSV8MOYgIALYJ2GziKdex5CEiNdGU62gSERle7ASw\n/222MNdQshFDDiKCbeVH9P008ZTrWPIQkZpoynU0iYgMH9YJ9iLYWs1WyDWcbMSQg8gwZ9v5hl21\n542Q0aQoljxEpCaach1NIiLDh30e7Na+VsgtlOzEkIPIMGa7uq6YtnHyzWDhpCuWPESkJppyHU0i\nIsODVcDuBtulr5VyCyc7MeQgMkzZ3mCvgK1XvyBIOOmLJQ8RqYmmXEeTiMjwYJuDPQnW3tdKuYWT\nnRhyEBmG7Mu+2/jqjRbmHk42YslDRGpKVa7bccP+XttgWakSERG7GuyQ/lbKJZRsxZCDyDBjh4M9\nD7ZSsxVyDSc7seQhIjWlKtffAi4CrmmwrFSJiAxvtrIfnGBMfyvmEk62YshBZJiwCtj3wJ4Am9LX\nirmFlK1Y8hCRmtKU62WAm4Gt0JU7kZKzX4Cd1MqKmYeSvRhyEBkGrB3sbLB7wRbrb+VcQspeLHmI\nSE1pyvVlwLrAlqhxJ1JitpiftHxyKytnHk72YshBJHI2EuwysFvAFmrlA5mHlI9Y8hCRmlzKdccQ\nP/8Z4FXc/XbVPtbrTjyf5h8iUiyHAJdDZXaDZVX6LuMiIimzhYCrgLnADlD5v8ABiYhE7wfAC8Cz\nwCvAO8AFdevo7JNI4dlYP1/Uqq1+INNw8hFDDiKRssV8N8yz+xm5d4EPZhZSvmLJQ0RqSleu1S1T\npLTsm67rU+sfyCyU/MSQg0iErMsPnPI9N5DKwD6cRUQBxJKHiNSUrlxviUbLFCkhGwX2Etg6A/lQ\nZuHkJ4YcRCJj6/v66OuD3UCq4YQTSx4iUhNNuY4mEZE42SFgja669/mhTELJVww5iETEdgR7DWyX\noWwktXDCiiUPEamJplxHk4hIfKwT7DmwjQf6wSyiyVkMOYhEwr4K9sog6qIFNpRKOOHFkoeI1ERT\nrqNJRCQ+9iWwmwbzwdRDyV8MOYiUnLWBnQL2JNgKaWwwhW0UQSx5iEhNNOU6mkRE4mLtYE+BVQfz\n4bSjCSCGHERKzEaCXQL2N7BJaW00pe2EFkseIlITTbmOJhGRuNg+YLcPYjQ6iKNcx5CDSEnZomC3\ngl0ONjrNDae4rZBiyUNEaqIp19EkIhIP6wSbMcirdhBHuY4hB5ESstXAngb7keuWme7GU95eKLHk\nISI10ZTraBIRiYcdNMh77T7aQGqhDNyywFTgUeBh4PAm650BPAU8AKzbYLnqJpHc2XZgr4Ltn9UO\nMtpuK9Kqm0D1k0iMoinX0SQiEgcbBfYC2IZD2Uhq4QzcEkDPnHzjgCeAVevW2QG4zj/fCLirwXZU\nN4lkazHgv9rgROhcB+xwPyLm5hnuM4a6CVQ/icQomnIdTSIicbBvgP1hqBtJJZR0XA18su69s4E9\nE68fBybXrVOkHERiM3kMzNoH/v0tOj/s4JwP4K1nwLoy3m+RyvVg6yYoVh4iko5oynU0iYiUn40D\nmwW21lA3lEo4Q9cFPI87S550LbBp4vXNwPp16xQlB5HodMD3DoT357CIbcUttj7X2ngmPpbDrotS\nrrsYfN0ExclDRNKTS7nuyGMnIlIYhwNTofJg6EBSMA64HDgC+Ffdsop/JM1vsI3uxPNp/iEiQ2Aw\n/hTYZB3W7NyIK9mFq9mbo9ma+fUNnTRU/aNI0qibQPWTSNlVKV79lAqdfRIpBFscbA7YSmlsLIVt\nDEUncAPwjSbLzwb2SrxWt0yRDBl0GuxocKnBW6dw0BMjedW6+YI97W6ye2chOD2fUIJKo26C8HmI\nSPqiKdfRJCJSbnYWWFoHVyHLdQW4ADitj3WSgxZsjAZUEUmdQcXg4wY/NZhtcOc7jD5sPHPPBHum\nwgEnjIeXxsIbC8FZuIZPDmEFk1bdBKqfRGJUinI9CrgbmI4b9re7wTqlSEQkbraaH358kbQ2mNJ2\nBmNzXDem6cD9/rE9cLB/9DgTmIEbbny9BttR3SQyCAZTDI4zeMzgaYNugxXBJoHdAnZDinXNIMIL\nJq26CVQ/icSoNOV6jP/ZgTsDtVHd8tIkIhIv+xPYN9PcYIrbCiWGHERyYTDR4CsGtxrMMfiFwab2\n0f1jth7Yc2Ang7WHDTUKseQhIjWlK9djgH8AG9S9X7pEROJi24DNABuR5kZT3FYoMeQgkhmDEQaf\nNfi9wVsGVxrsajCybs0v+Z4Bu4eJtJdYynUseYhITWnKdRuuC8LbwMkNlpcmEZH4WDvYA2C7pb3h\nlLcXQgw5iKTK30e3kcGZBq8Z3G5wsEGDbpY2Fux8sEdc1+9CiKVcx5KHiNSUZiqE+cA6wATgKmB1\n4JG6dboTz6eh4XxF8nIwMBdXNoeiSqTD+YoIGCwP7OMfbcCFwEYVeKbJJ1bFDfd/L7AhVN7JJ1IR\nEcnTd4Aj697T2SeRIGxx31VqjSw2nsE28xZDDiKDZrCwwUEGt/mrdGcabGwLzsNW/8l9wF4DOwCs\nn3VzF0u5jiUPEakpRbmeBEz0z0cDf8UN85tUikRE4mPngZ2a1cYz2m6eYshBZED8fXS7GFzh76O7\nzGAngxbuybXRYOeAPQG2VvbRDkos5TqWPESkphTlek3gPtxwvg8BJzRYpxSJiMTFNgN7EWyhrHaQ\n0XbzFEMOIv3y99FtYnCWH+nyrwYHWu3kbCtbWdXfv3tJhvVKGmIp17HkISI10ZTraBIRKQfr8Adh\ne2a5kwy3nZcYchBpymAFgxMNnjJ43OB4g64BbqUCdjDYHLCDCtgNs14s5TqWPESkJppyHU0iIuVg\nR4LdnPFBWAzlOoYcRHoxWNTgEIM7DF41OMNgg/7vo2u4tUlgV4PdB7ZK+tFmIpZyHUseIlITTbmO\nJhGR4rMV/Rn2FbLeUcbbz0MMOYhgMNJgN4Or/H10vzP4jEHnELb6Sd+1+ydgI/tfvzBiKdex5CEi\nNdGU62gSESk2awObBvatPHaWwz6yFkMOMkz5++g2N/ilwesGUw0OMDct0VC2PALsR75ht0060eYq\nlnIdSx4iUhNNuY4mEZFis6+C3eUmLs9+ZznsI2sx5CDDjMFKBicZPGPwqMGxBsultPU1fBfMa8AW\nS2ebuYulXMeSh4jURFOuo0lEpLhsWT/v1Op57TCn/WQphhxkGDCYZHCYwV0GswxOM1hvcPfRNdxD\nO9hRvg75SgkGTelLLOU6ljxEpCaach1NIiLFZG1gN4I1mooks53muK+sxJCDRMpglMEeBtf4++gu\nNtjeoCPlPa0EdgfYVLCudLcdRCzlOpY8RKQmmnIdTSIixWTfALvTTYGQ305z3FdWYshBImLQZrCF\nwTkGbxjcbLC/wfgM9tYGdpi/Wne4ex2FWMp1LHmISE005TqaRESKx9b0B2dZj465wI5z3l8WYshB\nImCwisH3DZ4zeMjgKINlMtzjFLBb/EmhlbPbTxCxlOtY8hCRmmjKdTSJiBSLjQJ7EOxLIXYeYJ9p\niyEHKSmDxQ0ON7jH4BWDUw3WSe8+uoZ7bfdX6eaAHZ3T4Et5i6Vcx5KHiNREU66jSUSkWOw0sMsD\nDX4QQ7mOIQcpEYPRBnsZ/NFgrsGFBtsa5NDIsjX8aLq3Rni1LimWch1LHiJSE025jiYRkeKw3cCe\nA1s0VACB9pumGHKQgvP30W1lcK7BmwY3GuxrMC6nCEaCneS7bx8U0b11zcRSrmPJQ0RqoinX0SQi\nUgy2kj9Q2yBkEAH3nZYYcpCCMljN4GSDmQbTDf7LYKmco9gM7DGwq8By3ncwsZTrWPIQkZpSlOtl\nganAo8DDwOEN1ilFIiLlYGPAHgA7NHQgAfd9LjAbeKjJ8irwFnC/fzSbIkJ1k6TKYAmDbxr8w+Al\ngx8ZrBkgkkXB/hfsJbDdSz5v3UCFLteqn0SkmVKU6yWAdfzzccATwKp165QiEZEC22IhOG80lbNg\n1pVgFxXgYC1kuf4EsC59Hzxd08J2VDfJkBmMMdjb4Hp/H91vDD6Vz310C0TTBnYg2GywM8Am5h9D\ncKHLteonEWmmlOX6auCTde+VMhGRgthxPLzzP2Db8o35FR6aB59YL3RQhC/XXfR98HRtC9sInYOU\nlEG7wScNzvf30V1v8EWDsQGjWs8PmHIn2Lrh4giuCOW6C9VPIrKg0pXrLuB5FrxJvHSJpMfawJYE\nWx9sa7BdwfZzQ9fb/v75F8F29PdGrAY2qQBXZaQgFoF/XAn2J7a3JXnJjmDKvLHwi9BxEb5cd9H8\n4GlLYA4wHbgOWK3JeqFzkJIxWNN3tXzR4D7fBXOJwFFNBPuZv1r35WEwYEp/ilCuu1D9JCILyqVc\nd6S0nXHA5cARwL8aLO9OPJ/mHxGxCrAisB6uO8ba/vWyuL71LwNz/fO3gXm4uYwqQCcwAZgILAxM\nBkaCPY9rLD+Fu5/xIfez8nZuaUlwG8DCi7M6u3I+f2Bn7uH5tvYwVweq/lEG9wHLAe8C2+N6FDQb\n+r078Xwa0dVNMlQGSwJ7A/sCiwIXAZ+uwCNBA3Nz1B0AnITr5rcaVF4PG1MQVcpTN4HqJ5HhpEq5\n6qePdAI3AN9osjzSs082GewAsN+CvQz2AtiVYN8G2wlsFTf4xaC2Pd7PSbQj2LfAzgO7B+wdsBlg\nF4IdArY2cU5CO+wZfMzg90+yxFsTeWb+CextfwCbAO8AW4eOj/DluovmZ8brPQss0uD90DlIQRmM\n89MV3OC7XZ7rpzMoyFUx29oPrPRX1zNEEopQrrtQ/SQiCypFua4AFwCn9bFOKRJpjS0G9jWwqWBz\nwX7vb15fIZ+ulNYOtjrYV8DOBXsc7C2wa8AOA1tZXTrLzWAZg3MMXnuWKd1tfPhgG1ddvzA8vjA8\nAOwcOkYvdLnuovnB02Rc3QSwIfBck/VC5yAF4u+j29ZPLD7X4E9+wvFBnqTLgq0EdjXYs2CfU33f\nUBHKdReqn0RkQaUo15sD83F9x3uG9d2ubp1SJNKcVcCqYJeAvemv1O0ENip0ZI4tBraXb+y9hJvY\n+pf+qt/I0NFJawwmGfzE4HWDk3/L3kv6s/JnFPQALmS5vgTX1fl94AVc17SD/QPga7iuzNOBO4CN\nm2yn5HWTpMFgbV/2Xja4x+AIcwfgBWITwX4CNgfsmOL8/ymk0OVa9ZOINBNNuS5pItbuG00Pgj0C\n9nUKP6y0VfygLEeC3eavLl4KtifYQqGjkwUZLGTwHYM5Bj939/dYp78ae1GBB0coabnuJYYcZBAM\nljY4yuBBg+cNvm8LTuNTADbKd81/FezXYIEHbymFWMp1LHmISE005bpkiVgb2H+CPQn2N7DtC3rl\npAU2GewgsOvB/gn2B7DPg40OHdlwZzDSXyGYZfBbgxX8kk7f3fdP7nlhlaxcNxRDDtIifyJlP4Ob\nDd4w+JXBlsW5jy7J2nEjKj/v6+01QkdUIrGU61jyEJGaaMp1iRKxKth9vlFXLW+jrhGb4A8WbvLd\nS88D+xQakCVX/r6e/Q2eM7jWYK3E0hG4QXn+WIJuVyUq103FkIP0waDDYHuDi/19dNcY7GFQ0BNc\nVvHd/h/2vS82Cx1RCcVSrmPJQ0RqoinXJUjElvUH1c/6K1sRNeoasSXBvgl2L26kz1N1ZjhbBhWD\nXQ0eNfirQd1Bm430XTGvLsm9kiUo1/2KIQep48vaegan+SvjdxscZrBY6Nj6ZlWw28EeAvtM/P+H\nMhNLuY4lDxGpiaZcFzgRawP7KthrYN8pwdWSDNgqYN/HDcZyt+/GOT50VDEx2NofYE73VxHqDtps\nPNiNYFe4q3elUOBy3bIYchDPYFmDYwweMXjG4CSDj4WOq3+2JW4E5hn+lgD1phiaWMp1LHmISE00\n5bqgidgU/w/1LrDVQ0cTnnWA7eAbGHPBzgfbQmePB89gA4ObDGYYfKHxvT22FNj9uBFOO/KPctAK\nWq4HJIYchjWD8QYHGEz1I83+0mDzBU+gFFGvRt1+JSv/RRZLuY4lDxGpiaZcFzAR2wVsNthROkva\niC2OG3HzUdzAMse4rpzSCoNVDC43eMngYIMmA6PYqripK44vYSO6gOV6wGLIYdgx6DTY0eBSg7cM\nrjbY3aAkPS9sC7C/qFGXmVjKdSx5iEhNNOW6QInYSNy8Yc+CbRQ6muKzCtjGYOfgBmG5Bmxnij2K\nYzAGyxn82uA1c8Os9zH5sX0WN7z5fvlFmKoCletBiyGHYcHfR7eBwRkGsw3uMDjEYNHQsbXG2nyZ\nv9036vZXoy4zsZTrWPIQkZpoynVBErHJYHeAXUXh56srIhsH9iXcSKKvgP0QbOXQURWBwWJ+8IbX\nzc2X1cffl7WDfQ/sBddwLq2ClOshiSGHqBlMMTje4DHfvbnbYMXQcbXOOv19dA/jRmL+vHqLZC6W\nch1LHiJSE025LkAitrbv/tZNcSeFLhFbFezHvmvrX33XorGho8qbv9+n2zfqzjToZ4Jhmwz2Z7Bp\n7nmpFaCd7Zy9AAAb40lEQVRcD1kMOUTHYKLBVwxuNZhj8AuDTctxH10PGwt2OG6eur+AbVvCrtdl\nFUu5jiUPEamJplwHTsQ+gxsNc6+wccTIRoDthptw+w2ws8E2iP0gxmCUwTd997ALDf6jhU/t7K94\n/iCSbq0xVFAx5BAFgxEGOxlc5u+ju8JgF4MyTAuSYMuCneL/51wBtmHoiIahWMp1LHmISE005Tpg\nIrYv2CzdX5cHWwY3MMgzYA+CHQFWkvthWmNuQuQDDGaamwx5zRY+NRHsV2BPE9eExDFUUDHkUFr+\nPrqN/FXv1wxu9wMQLRI6toGxCtimYL/zJ7lOBytR19HoxFKuY8lDRGqiKdeBErGvg810XQglP9YG\ntjXYRbgpFS4F26bM3WH9Qeju/r6fWw02beFTFbB9cJPEnw22UPaR5iqGCiqGHErHYHmDbxs8YfCk\nf97C1e+isRFgXwT7O26QlCPQHKFFEEu5jiUPEakpTbk+F5gNPNRkeYBE7ASwp8C68t+31NjCYIfh\n5nF7DuxksLXK1G3T4FMG9xjcZ7Bda/f92Nr+vrr7Ir5qXJoKqg8x5FAKBgv7q3K3+6t0Z/qrdqWp\nC2rsP3zXy1lgt4DthAZJKZJYynUseYhITWnK9SeAdSlM486OA3sMrJ/BLSRftg5uhM3ncfPnfbvI\nXZcMNjS4xV9Z2NMaTkC+wKdWBrvEH/R9LfIDvtJUUH2IIYfC8vfR7eLvn3vL30+3k8GI0LENnHX6\n+4tv8PfTnQr2sdBRSUOxlOtY8hCRmlKV6y4K0bizb/krdkvlt08ZGGtz957ZmbjRNu8BO9p1nw1/\nRc9gNYMrDV40ONCaTkDe61Nrg/3GH/QdBzYu+0iDK1UF1UQMORSK78K8qR/hco7vxvwV63N6kCKz\nFXBTl7wMdpvval2SydKHrVjKdSx5iEhNqcp1F8Ebd3YobjCPZfPZnwyddfj78X6Om/ftKX9GfAty\nnuDX3Hxa5xm8avBfBqP7+USn7451C9hLYMe6bqjDRqkqqCZiyKEQDFY0Ny3IDIPHzc1N1xU6rsGx\nCWBf8Y25V8F+CrZ66KikZbGU61jyEJGaUpXrLvpu3HUnHtX0d297+sbB8ulvW/JhFbB1wU4E+wdu\n1LmrcAPjrJ7VVT2DxQ1+am6uuv/X9xUGq+CmevipP+j7mz+TX8JuZgNWpXc5LlUF1UQMOQRjsKjB\nIQZ3+JMiZxhsUNL76DrAtvPdqufipjHYaZiU7djEUq5jyUNEakpVrrsIduXOtvAH2mtlux/Jly0J\n9gWwc3DTCMzCDTV+pP+dD6jrYxvctDDMXxjmd8DfDCYYnOQbdWcYNJlU3BYF28XH8TLYE2DdFPh+\nwZyUqoJqIoYccmUw0o8ce7W/j+5Sgx1b675cNB+drPmJL9t3+x4gUU3hMgzFUq5jyUNEakpVrrsI\n0rizVf19W9tktw8pBusC+0+wn4HdBfYO2MO+wfddsL3B1veNsV5XDipwxWK4kRD+4ld+A943+I2B\nv9pr7e7Kr20H9g2w831D7p9gN/r3Vso/78IqVQXVRAw5ZM7fR7e5wS/9yZCp5uZ7nBA6toGzCtjH\nwX4E9qwv4/8PTZkTk1jKdSx5iEhNLuU6je4zlwBbAosCrwLfAc5LLLeU9lPHJgN3ASdC5YL0ty/F\nZiOA1YHVgFWAj/mfywJjgFnAK8DcTq7e5uO807Yq/2Ye7bzFCG5jpL3O7jcCi+Ou2k3CTenxBPA4\n8DBwJ/AIVOblnFwZZFSuW3IusCOuvmk2kfwZwPbAu8D+wP0N1gmZQ+EZrAzs4x/vARcCF1dgZtDA\nBswqwHrA54E9gHnA74DLgAdd21UiErpcq34SkWaiKdcZ/OO0Ebib3U9Kf9tSfjbaX4XbFGy7Mez2\nf/uyt53Dl+089rMD+YKNZY8PwHbwZ/GXBRsZOuqSCXlA3N/0KzsA1/nnG+FOAjWig/o6BpMMDjO4\n22CWwWkG65XvPjobCbatv9L/HG6S8R/gpmQpWS4yQKHLteonEWkmmnKdRePu52DXgLUw95gI3x7j\n+16dAjbW/U2eGjqokgtdQXXR/ODpbGDPxOvHaXxPZegcCsFglMEeBtf4++guNtjeINcRa4fOJvmu\n25fjBkW5AzeK7Rpq0A0rRSjXXah+EpEFRVOuU07EDgB7HKyE93tIQP89Bt4YC2/iRnuUoQldQXXR\n/ODpWmDTxOubgfUbrBc6h2AM2gy2NPiVwRsGNxvsZ7BQ6NhaZx1gG4N9G+x2sLfArgTbH2zx0NFJ\nMEUo112ofhKRBeVSrst2ZnZD4BRgC6i8FToaKZUfvws/Dh2E5KLCgt0I5zdZtzvxfJp/RMvcfan7\nAl8E3sbdR7dWBV4MGlhLrAKsCGzjH1Xc/X83A98DboXKv4OFJ6FUyWSKpcyofhIZPqqUq35qWUqt\nVJvoRzfbLZ3ticgQhD6r3EXf3Z72Srwe1t2e/FyOhxvcY/CywU8M1i7+fXRWAVsBbD+wX/t7517y\nI9l+EWyJ0BFKIRWhXHeh+klEFhRNuU4hEauAXQZ25tC3JSIpCF1BddHagAUbMwwHLDAYbbCXwZ8M\n5hpcaLCtQXvo2Jqzdj/gydfBfo+be+4lsEvBDnPTFejeOelXEcp1F6qfRGRB0ZTrNBp3XwWbDjZq\n6NsSkRSErKAuAV4G3gdeAA4ADvaPHmcCM4AHcMPgNxJNJQsf3Ue3lcG5Bm8a3Giwr8G40LEtyCpg\ny4Dt6kexvNEPgvIY2Dl+YJTl1ZiTQQhdrlU/iUgzpZnnrj9DnNPB1gSmAptB5YmUYhKRoYlhrpYY\ncsDcfI/74O6jewN3H90lFXeAWQBWAZYE1gY28I+P464i3gPc63/eDZXXQkUp0YiiXBNPHiJSk0u5\nLnjjzkbi/vH/D1TO629tEclNDAcepc3BYAngC7jBUSYDFwEXVpp3BcuJLQSsgZu8OfmYDzxIrSF3\nDzBTE4hLBkpbruvEkoeI1KhxB3YysCqwqw4CRAolhgOPUuVgMBbYGdeg2xj4A/BbYGoF5uUYSTsw\nBVg58VgJ+BiwGPAYrpGZfMxWHS45KVW57kMseYhIzXBv3NnGwNXA2lCZnXJMIjI0MRx4FCmH3YDD\ngNeAA4F/AvgBULbCNeh2Bu7Edbv8QwXeySYUa8N1o5wCLOd/9jxfAVgemA086R9PJX4+A5UcG5oi\nCyhSuR6KIuexFW7Ez1eA7wMfhA1HcrYIsC3ub/TPgKYma91wbtzZGOB+4HioXJ5BTCIyNEU+8GiV\ngb2Ja6jMBl5t8HwOMBd40/98L4MrUKeOhm9tCzwNPA/zr4Etqq4xt7ePpec+ulmD24W14SYoXxhY\nHNetcwlcl87k86WApXH37s104fB84vmzwAyovDe4OEQyF0PdBMXN45gxcPKncXM4vAhz33b1x/8F\njkvyMWUM3LMZjP4A7B54+x1Yn0H/bxp2hnXj7jRgMlT2ziIgERmyoh54DISBLYZr1Cxe97PnsSgw\nEdcomgi04Rp5yQbfXOBfwLt1j3cSz/+NO7s9D/gw+RjLun//AR+yFfAh7TxHO6No/2AuK1xyHTvc\n+Fv2nen32wGM9o8xTX72NOB6Yu55Pt7H8SausTjLP2Ynnvc8XtRk4FJiMdRN4OqnNUIHUW80azz4\nU6xtU1xl9t/Aayxy3dbs/MvAoUkObuPyoz/Fvzba23XP59dUPryTCVM/wc4/Dx1bGfyYo6+mJI27\n7YDTcd2HfgX8sG75ACta2wC4BlgDKq+nEJ+IpC+GA6jBnHgaRe/GXs/zMf4xNvE8+d5oXB3ZkXi0\nAx0V7v/4ynTSidHGfN5kHi/Q9j6sMx13/DTf/5wHvIdrpDX7+Q6uAdfz6GmEvgWVDweWq0gpxVA3\ngWvcPRo6iAU9stoquLNN4M5QzaNtfoUOdc0cFj4Y0YFVen7/84EPqcyHTv3+WzCDlUdSgvqpHTdX\nSxfQCUzHDYCSNIAuTNYBdj/YF1OKT0SyEcPgGIXIYQK8fyjYh2DPuEuJBpwTOi6RkipEuU5BIfOY\nAP88Emwe2JNgi7g4Dwodl+RjLPygCu+87Scm3QDeGeku4EprClmu622Cu5myxzH+kTSQxt2RYDdp\n4lqRwitFBdWPouSwwXj4sN2f3RoZfDoDkVIrSrkeqqLmsdZ4eLcDrNM9LgwdkOSqczxc0u7+Z304\nzp2IbA8dVIkUtVz38jl6n2HeB/hZ3TotJmJTwOaArZhOaCKSoVJUUP0oWg5L4npAiMjgFa1cD1bR\n81gYHdQPZyPQ/6vByKVcd2SwzUaBdyeeT/OP5EcqwM9xk5XPyCAmERmaqn9Idl4JHYCISIveDB2A\nBPV+6AAkOxvTu1vmscDRdeu00Eq1XcEeARuRXmgikqGin1VuRQw5iEhvsZTrWPIQkZpSlOsO3NRM\nXbhLtIMYUMVG+XEEPplBfCKSjVJUUP2IIQcR6S2Wch1LHiJSU5pyvT3wBG7UzGMbLO+vcXcM2FXp\nhyUiGSpNBdWHGHIQkd5iKdex5CEiNdGU6z4SsaX8ICor5BeOiKQghgoqhhxEpLdYynUseYhITTTl\nuq/G3flgJ+cWiYikJYYKKoYcRKS3WMp1LHmISE005bpJIrYh2EtgC+UbjoikIIYKKoYcRKS3WMp1\nLHmISE005bpBIlYBux1s/9yjEZE0xFBBxZCDiPQWS7mOJQ8RqYmmXDdq3O0E9iCYJsAUKacYKqgY\nchCR3mIp17HkISI10ZTrukSsHexhsB3DhCMiKYihgoohBxHpLZZyHUseIlITTbmub9ztD/ZX1zVT\nREoqdAW1HfA48BRwdIPl+wOvAff7xwEN1gmdg4ikL3S5TqNugvB5iEj6oinXiURsFNjzYJuEC0dE\nUhCygmrHzavZBXQC04FV69bZDzijn+1EU8mKyEdiqJtA9ZNIjHIp12157CThUOB+qNyZ835FJB4b\n4g6gngM+AC4Fdq5bp+IfIiJ5Ud0kIsHl2Liz8bguCsfnt08RidDSwAuJ1y/695IM2B14ALgMWCaf\n0ERkGFPdJCLBdeS4r8OBG6DySI77FJHhob6rw7XAxbiz5wcDvwE+2eBz3Ynn0/xDRMqj6h9FNdi6\nCVQ/iZRdlWLXT4Nm7qqdvQa2cuhgRCQVIe8H2Rj4c+L1sTQeuKBHOzC3wfu6p0UkPjHUTaD6SSRG\n0ZRrAzse7MLQgYhIakJWUB3A07hBC0bQeNCCJRLPdwXuaLCdaCpZEflIDHUTqH4SiVHhy/UewCPA\nPGC9PtYzf9XuY/mEJSI5CF1BbQ88gRu84Fj/3neBz/rnPwAexh1c3QI06jUQOgcRSV/ocp1G3QTh\n8xCR9BW+XK+Cq5Sm0n/j7qJ8QhKRnBS+gmpBDDmISG+xlOtY8hCRmtKU61Yad6vkFYyI5KI0FVQf\nYshBRHqLpVzHkoeI1MQ0z13l8Xz2IyIiIiIiMjz1NxXCTfS++bfHcbjhfFvVnXg+DQ3nK1I2VSId\nzldEREREalrolikikYmhXMeQg4j0Fku5jiUPEakpVbfMSkrbERERERERkZztCrwAvAfMAq5vsp7O\nPonEJ4ZyHUMOItJbLOU6ljxEpCaach1NIiLykRjKdQw5iEhvsZTrWPIQkZpSdcsUERERERGRgNS4\nExERERERiYAadyIiIiIiIhFQ405ERERERCQCatyJiIiIiIhEQI07ERERERGRCKhxJyIiIiIiEgE1\n7kRERERERCKgxp2IiIiIiEgE1LgTERERERGJgBp3IiIiIiIiEVDjTkREREREJAJDadz9GHgMeAC4\nEpiQSkTZqoYOIKEaOoCEaugA6lRDB5BQDR1AQjV0AAWyHfA48BRwdIPlI4Hf+eV3AVPyC60QqqED\nyFA1dAAZqYYOIEPV0AHkSHXT4FVDBxBINXQAAVRDBxCzoTTubgRWB9YGngSOTSWibFVDB5BQDR1A\nQjV0AHWqoQNIqIYOIKEaOoCCaAfOxB1ErQZ8AVi1bp0vA68DKwGnAT/MM8ACqIYOIEPV0AFkpBo6\ngAxVQweQE9VNQ1MNHUAg1dABBFANHUDMhtK4uwmY75/fDSwz9HBERPq1ITADeA74ALgU2LlunZ2A\n3/jnVwCfzCs4ERm2VDeJSHBp3XN3AHBdStsSEenL0sALidcv+vearfMh8BawSPahicgwprpJRILr\n6Gf5TcASDd4/DrjWPz8eeB+4uMk2ngZsUNFl48TQASQoluaKFI9iWdDToQOoU1/HVFpYp2h1U9qK\n8reShVhzizUvyC+3B3LaT6sGUzdB/PVTMzGXgb4Mx7yHY865HDv117jbpp/l+wE70He3ghUHFJGI\nSN9eBJZNvF4WeKnBOssBL+PquQnAm3XrqG4SkTSlVTeB6icRCWA74BFgUuhARGRY6cCd/eoCRgDT\nWXDQgkOBX/jne+HufRERyZLqJhEptaeA54H7/eOssOGIyDCyPfAEbvCCnpF6vwt81j8fCfye2nDj\nXTnHJyLDk+omEREREREREREpnj1w3TXnAev1sV5/E32mYRHcoDBP4ublm9hkvXnUrkBenXIMRZrQ\ntL9Y9gdeo/ZdHJBhLOcCs4GH+ljnDFysDwDrBoylihvRrOd7OSHDWJYFpgKPAg8DhzdZL4/vppVY\nquT33QxGf7/bL+K+wweAvwFr5RRXGlopQwAb4Oq43TKPKD2t5FbF/c09DEzLPqRU9JfXBNxgZdNx\nee2fT1ipKFLdFcpQ/t8f699/HNi2hW0ehrs6OJ8FR9vM+zsuQt5V8v1flGfOF/n3HwJ+Te/xMmL+\nXTfLu0q8v+tf4+r/B4DLgLEt7CMXqwAr4yr5Zo27dlzh7AI6adwvPQ0/Ao7yz48GTmmy3tsZ7Bta\ny/NQal1a9yS7/vetxLIfrqLIwydwFVGzg5wdqE2vsRHujzlULFXgmgz3n7QEsI5/Pg7Xvaf+95TX\nd9NKLFXy+24Go7/f7Sa4A2pwlW2Wf2dp6y83cOX+L8Afgd3zCCol/eU2EXcSsWd+1bLc+91fXscB\nJ/vnk3CTXfc38FlRFKnuCmEo/+9X8+t3+s/PwI2q2dc218Ed4D1L70ZO3t9xUfKukt//orxz3j6x\n3YuBr/rnsf+um+VdJd7f9UKJ7Z5KreE3oLZCWvPcJT2Ou1LWl1Ym+kxDcrLQ3wC7ZLCPvhRpQtNW\nYqnQeJjmLNxG4xHCeiS/l7txB3OTA8UC+X0vs3AFHeBfwGPAUnXr5PXdtBIL5PfdDEZ/v9s7cWcA\nwX2Xy/SxbtG08nf7deBy3BX5Mukvt71x9eWL/vWczCNKR395zQfG++fjcY27D7MOKiVFqrtCGMr/\n+52BS/znnvPb2aifbU7HjXtQL+/vuCh5Q37/i/LO+frEdu+hNnfizsT9u67PO/n/Odbfdc/Fpgow\nBvc/oa99NJRF464VrUz0mYbJuC4w4P7xLN5kvVG4P5w7SbeRWaQJTVuJxXBn93suB4c80G0Ub6h4\nDHeFZzruLNlqOe23C3eW/+6690N8N81iCfXdZOHL1M6CxmBpXH12tn8d05xZK+HqyanAvcC+YcNJ\nzZm4MvQyrh4+Imw4g9ZFcequvAz2//2iuEbwi4n1ej67VAvbbCWOLL/jouSd5/+iUDl3AvsAf/av\nG30mxt91T97Jxl7Mv+vzgFdwvSB/1sc+mrYVBtvdo5XJzQdqsAcezWI5fgDbWBbX+Fse14XpIeCZ\nQcbTn8FOaJqF+v1ci7v0/QFwMO4sQVZXEvvT6Cri/EYr5uA+3LxE7+K6CVyNK3RZGoe74nIE7ix4\nUt7fTV+xhPhusrAV7h7TzUIHkqLTgWNwfxt5XpXPQyeu2/8ncWc378R1SXoqZFAp2A5XprYCVsD9\nf1ub7G4dyEKR6q7QWv1/3+z9Rifg+zs+KMJ3HCLv0P+L8sj5LOBW3P3hPfsYDr/r+rz/Qdy/6y/5\ndX6Gmy7l/D621dBgr9xtA6zZ4NFqw66ViT6HGss1uKt2PQ2/JYFXm2xjlv/5LO7G/LRuSh3IhKbQ\n94SmecTyBq5hB/ArYP0M4mhVfbzL4M5mh/A2rhIBd+aok2yurvboxF12/y2NB/jJ87vpL5a8v5ss\nrAWcg+v2kEXZC2V9XHePZ3FX5M/C5RiDF4AbgPdwXRf/imsEld3+wJX++dO4393HgkUzcEWqu/I2\n2P/3bzT47DL+s4M5Vsr7Oy5K3nn+LwqR84m4q0Hf6iOOGH/XjfKO/XcNrpH+e2r3yufVVujXVJo3\nDlqZ6DMNP6J2M+IxNB5QZSJuFBpwN7A/iRsUJg1FmtC0lViSV0B3Be7IKJYeXbQ2oMrGZH+jcF+x\nTKZ21mRDXD/prFSAC4DT+lgnr++mlVjy/G4Gq4vmv9vlcH3fN84tmnR10f9omeC6eZRptEzoO7dV\ngJtxN6aP8euVpUtwF83zOgt3MAOubL1IeU6WFKnuCmEo/+97Bl4YgetB9DTu+2xlm8/iDn575P0d\nFyXvPP8X5Z3zV3BXrUbV7SP233WzvGP+Xa/of1aAn+DaMX3tIze74s6qvoe7ItbTR3Yp4E+J9RpN\n9Jm2RXAHAE/iurf0TIWwPu5MPcCmwIO4L/dB3OXQNBVpQtP+YvkBbgjr6cAtZHuZ+xLcGab3cX8v\nB+C6gh6cWOdMH+sD9D2tRtaxfI3a93IH2TYENsedsZlObZjf7Qnz3bQSS57fzWD097v9Fe7KT09+\nfw8Q42C1UoZ6lK1x10pu/4UbMfMhmg+7XzT95bUk7orkg7i89g4Q42AVqe4KZSj/74/zn3sc+HQ/\n2wT3N/8C7m/pJeB/E8vy/o6LkPdh5Pu/KM+cP/DbaTT0f8y/62Z5x/q7bgNup1b/X4jr5t7fPkRE\nRERERERERERERERERERERERERERERERERERERERERERERERERESkbPbAja48j+ajio4C7saNwvkw\n0J1Y9mv//gPAZcBY//5XcaNb3g/cRm1Kg22Ae/2ye4Gt0klDRERERERk+Kjipv9JWgU3vddU+p4y\nYoz/2YGbcmAj/3qhxDqnUptDO/n+Z6lN/7YOtbmjV8fNWZqbjjx3JiIiIiIikhFr8N7jLX72Xf9z\nBNCJm7sT4G3/s4JrANa/D25Oup73pyfefxR3VbATN3df5try2ImIiIiIiEjGKkP4bBuuYTYbuBG4\nJ7HsPOAV3BXAnyXePxQ3GfkPgcMbbHN34D5yatiJiIiIiIiU3V24e9+eAl73z+8Htk2s01+3zB4T\ngL/gulQmtQE/B/Zv8JkvAOfXvbc6ruG3fAv7FBERERERkYQtWfCeux6tNu4AvgMc2WT71zZ4vw2Y\nm3i9DPAEsEmL+0uNumWKiIiIiEgM+uuW2Wz5JGCifz4a+BTwmH+9YuKzn028v1Li8zsCT/rnE4E/\nAccAd7YUtYiIiIiIiPSyJXBu3Xu7Ai8A7wGzqI1quRSuEQawFu7euAeAh4AT/PttwO24aQ0eAi7E\nDZ4CcDpu2oT7gVuoTYVwAvAval1D78c1HkVERERERERERERERERERERERERERERERERERERERERE\nRERERERERERERERERERERERERERERERy9/8BUheOD5BIie8AAAAASUVORK5CYII=\n",
247 | "text/plain": [
248 | ""
249 | ]
250 | },
251 | "metadata": {},
252 | "output_type": "display_data"
253 | }
254 | ],
255 | "source": [
256 | "plt.figure(figsize=[15,3])\n",
257 | "plt.subplot(1,3,1)\n",
258 | "plt.scatter(x_list,y_list,c=\"r\")\n",
259 | "plt.plot(x_list,y_list,c=\"r\")\n",
260 | "plt.plot(x,f(x), c=\"b\")\n",
261 | "plt.xlim([-1,2.5])\n",
262 | "plt.title(\"Gradient descent\")\n",
263 | "plt.subplot(1,3,2)\n",
264 | "plt.scatter(x_list,y_list,c=\"r\")\n",
265 | "plt.plot(x_list,y_list,c=\"r\")\n",
266 | "plt.plot(x,f(x), c=\"b\")\n",
267 | "plt.xlim([1.2,2.1])\n",
268 | "plt.ylim([0,3])\n",
269 | "plt.title(\"zoomed in\")\n",
270 | "plt.subplot(1,3,3)\n",
271 | "plt.scatter(x_list,y_list,c=\"r\")\n",
272 | "plt.plot(x_list,y_list,c=\"r\")\n",
273 | "plt.plot(x,f(x), c=\"b\")\n",
274 | "plt.xlim([1.3333,1.3335])\n",
275 | "plt.ylim([0,3])\n",
276 | "plt.title(\"zoomed in more\")\n",
277 | "plt.show()"
278 | ]
279 | },
280 | {
281 | "cell_type": "markdown",
282 | "metadata": {},
283 | "source": [
284 | "Another approach to update the step size is choosing a decrease constant $d$ that shrinks the step size over time:\n",
285 | "$\\eta(t+1) = \\eta(t) / (1+t \\times d)$."
286 | ]
287 | },
288 | {
289 | "cell_type": "code",
290 | "execution_count": 8,
291 | "metadata": {
292 | "collapsed": false
293 | },
294 | "outputs": [
295 | {
296 | "name": "stdout",
297 | "output_type": "stream",
298 | "text": [
299 | "Local minimum occurs at: 1.33085067409\n",
300 | "Number of steps: 6\n"
301 | ]
302 | }
303 | ],
304 | "source": [
305 | "x_old = 0\n",
306 | "x_new = 2 # The algorithm starts at x=2\n",
307 | "n_k = 0.17 # step size\n",
308 | "precision = 0.0001\n",
309 | "t, d = 0, 1\n",
310 | "\n",
311 | "x_list, y_list = [x_new], [f(x_new)]\n",
312 | "\n",
313 | "# returns the value of the derivative of our function\n",
314 | "def f_prime(x):\n",
315 | " return 3*x**2-4*x\n",
316 | " \n",
317 | "while abs(x_new - x_old) > precision:\n",
318 | " x_old = x_new\n",
319 | " s_k = -f_prime(x_old)\n",
320 | " x_new = x_old + n_k * s_k\n",
321 | " x_list.append(x_new)\n",
322 | " y_list.append(f(x_new))\n",
323 | " n_k = n_k / (1 + t * d)\n",
324 | " t += 1\n",
325 | "\n",
326 | "print \"Local minimum occurs at:\", x_new\n",
327 | "print \"Number of steps:\", len(x_list)"
328 | ]
329 | },
330 | {
331 | "cell_type": "markdown",
332 | "metadata": {},
333 | "source": [
334 | "Let's now consider an example which is a little bit more complicated. Consider a simple linear regression where we want to see how the temperature affects the noises made by crickets. We have a data set of cricket chirp rates at various temperatures. First we'll load that data set in and plot it:"
335 | ]
336 | },
337 | {
338 | "cell_type": "code",
339 | "execution_count": 9,
340 | "metadata": {
341 | "collapsed": false
342 | },
343 | "outputs": [
344 | {
345 | "data": {
346 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEZCAYAAABxbJkKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYHFW9//H3TPaZyUwWEgghMCSBIIiCLIJsoxciqIBg\n2PRGQpRVIKI/QBR/DOsFQVBRFqMg8LBj2BQUyGUA2QnZMYQ1AUI2lmRmEiZb3z++p+3qTlVP1aSr\nq6f783qefqaquqvOt7unz6k659Q5ICIiIiIiIiIiIiIiIiIiIiIiIiIiImXme8A/Q7yuGbgt3lA2\nMgF4Js/zjwDjixOKSHmqTjoASdztwNdDvC7VxeM3E1/h8Y0Yj11ojcAGyuc31wL8IOkgZNOVyz+k\ndE2PCK+tii2KeER5b8XUHT7HniFe09WTgrRS/X5EysIIYAqwFFgOXOu2TwCeBa4GPgIuZuMql52A\nx93zi4Hz3PZmMmfdvYA7gfvc8pbAX116bwNnuNcdDHQAa4BWYHoX4n0GuBL42B37YM9+LWTORv3e\n2/Fu2++AT4F/A1/z7D8BeAtY6Y79XZ/YtgRWAQM923YFlmEZ2WjgKXf8ZcBdAe9xIXYl0OoeX3bb\nJwKvuff3D2Brzz4bgFOB+S7Gi4BRwPPACuBu7PMHaALex76vZcA7Oe+nD3AVsAD7Xq8H+ubsew7w\nIXALMAD4G/adfAw8DAx3r78UWAesdu/ld/hf6bQQ/P1cBPTOE5OIdFEPYCbwa6Af9uP/intuArAW\n+BH2Y+1LdiHQH8sEzsJ+oHXAnu65ZqwQ6Av8HbgJO6utBqYB52NnkNtiGetYt98FwK2bEO8aLCOp\nAk4BPvDs+ySWieZ7b2uBSS6do7HMegBQi2Wk27n9Nwd2DIhxKvBDz/qVwHVu+U4yBWVvT+y5tmHj\nTPLbwBvAGLf9F1hGmbYBeAD7HnbECtQnsAy3HpgLfN+9tsm916uwgmF/oA3Y3j3/G3esAe54DwGX\n5ez7P27fvsAg4Ai3XAfcA9zvic372YN/IdDZ95MvJhHpor2xsze/qr4J2FlX7rZ0IXAclqH7uQB4\nEDvr/Y1n+5d9jnkeVkhA520CncX7hme9Bstohrr13EzG7719kLPtReC/3bE+AY7ECp98foAVBGCF\n0UJgX7d+C3AjmbPkII1snEk+SnZGWg20Y1dGuNfv7Xn+FeBsz/pVwDVuuQnLZL3v5W6scK7CCoSR\nnuf2xq5+0vt2YIVYkF2wK4K0J8luE2ik80LA+/10FpMUidoEys8I7Me2IeD59zrZN+hHWAXsBXwe\nuMKzfRusyuQTz+M8Mhn1psa72LO8yv2tC3it33vLLQQWAMPcsY7Bri4WYVUfYwKO+1csgxqGnWGn\ngH+5587BPpuXgDnACQHH8LMN8Fsyn9tHbru3QFniWV6ds/4Z2Z/FJ+41aen3uhlW6E3zpPWo2562\nDLvqSqvBCrd3sSump4AGsts0orYLeL+fISFikiJQIVB+3sPqlYMa3vL9cBdidc5B+z0GXI6dFQ/1\n7PMOVmeeftQD33LPB2XuYeONwu+95Z6hb4Nl+mDvZyywBTAPmBxw3E/da4/G6tnv8Dy3BDjJpXMy\nVk00MvcAAbEtdPt6P7ta4IWAODo75kAsY01Lv9flWOGwoyedAdj3FHSsn2JVSXtimf8BWAFQFfD6\ndvfXm/4WeeINE5MUgQqB8vMiVq9/OfaD7EtwPXWuv2M/3ElY3Xx/Mm0C6R//lVgmOBUYDLyMNVqe\ng1VF9MCuFnZ3r1+CVRUE9YrZlHjDGAqcidV1H4Wd7T/ith+GZbprsUxsXZ7j3IE1NB9JdiFwFLCV\nW/4Uy+jW++y/DCsQvYXsDcDPybRFNLjj5VMVsJx2IfZe9wO+CdzrYpqMVeMNca8bTqbdxk8dlkmv\nwNoHLsh5fgnZ72UZdtU1HvsfmEjwCQXYZxE1JomBCoHyswE4FOu1shA70z7aPZdi4zM477ZW4CC3\n/4dYr5Qmn9ddgjXoPYEVFIdidcZvY5nBH8mc0d3r/n6E1WkXIl4/fq8FK2S2c3FdDIzDqh6qgZ9g\nGddHWKZ5WsCxwRotR2PVU7M923fHztxbsTaTM9m4bQKs+ulSrOH3E6xwfQCrWrsLy2xnk33Pht/7\nSeUse9cXu2MvwtphTsa+Q4BzgTddrCuwHmDbe/bNTes3WKG+HHgOq6rxvua32Gf5MZk2ohOxNovl\nWMHmbeT2+346i0nKwCTsH3uOWwZrKHwf6y44newufyKFNIH8dxyXkybyt/eI+ApzU0hXfR7rVrcH\ndrn9D6y6IYX1Fb46xrRFRCSEOAuBHbDLvM/c+lNYv2PoHndNSvcXVEVUrirpvUo3sAPwOtaoVIPV\nK/4Oa2B6B7tB6M9YjwAREUlA3GfkE7E7BNuwW+NXY3clLnfPX4z1Y9ZAVCIiCShmtcxlWO+PGzzb\nGrExSXb2ef2b5O9iJiIi2d7CerGVjPQNRVtjg3c1YGf+aWeR3efaqzvUbzYnHUBIzUkHEFJz0gGE\n1Jx0ACE1Jx1ASM1JBxBSc9IBhBA534yzYRhslMnBWO+g07C+wNdifcpTWNvAyTHHICIiAeIuBPb3\n2fZ9n20iIpIA3TG8aVqSDiCklqQDCKkl6QBCakk6gJBakg4gpJakAwipJekAKk13aBMQESklkfNN\nXQmIiFQwFQIiIhVMhYCISAWLu3eQiEgpawCOxeaVeBS7n0lKhBqGRSROA6F2AXyzHU7ugJp2MvNn\ndFdllW+W1ZsRkVJT/XP4bgekUvaYkoIBc5OOahOpd5CISDi9hsLOvTPrOwAbBiUWjmxEVwIiEqex\nMKQdZqRgaQq+sQrq/5x0UJuorPLNsnozIlKKep4ENZ9A79VQfxc2r3J3Vlb5Zlm9GRGRIlCbgIiI\nhKdCQESkgqkQEBGpYCoEREQqmAoBEZEKpkJARKSCqRAQEalgKgRERCqYCgERkQoWphAYGXKbiIiU\noek+26YVIV0NGyEiEk3kfDPfzGKfA3bEZt45EqhyCdQDfbsSnYiIlJZ8hcAY4FCsEDjUs70VODHO\noEREpHTsnVC6qg4SEYkmcr5Zlee5c4ErgGsDEjozamIRpcgfn4iIZIucb+arDnrN/fU2AqcT0Fm6\niEiFqevCPpOA2cActwwwCHgcmA88BgwI2FcFjYhINLHkm1/Brgrec+u7ANeF2O/zWAHQF+iBZfyj\ngV8B57jXnAtcHrC/CgERkWhiyTdfArYm+36BuSH2GwdM9qyfj2X+84DN3bYt3LofFQIiItHENr3k\nwpz1dSH2mQPsj1X/1ADfALbCCoAl7jWLgaEhYxARkQLL1zCcthDYxy33Ac4A/h1iv3lY76LHgTZg\nJuEKDxERKZIwhcCpwG+B4cD7WGPuj0Ie/yb3ALjU7b8EqwZaDAwDlubZv9mz3OIeIiJimtyjZKWr\nerbGrh4GYA3D57rtP0MNwyIihVLQm8XShmLDRDSSuXJIARND7Ps0MBhYC5wFPIm1EdyDFQwLgKOA\nT3321c1iIiLRRM43w7z4eSwznwZs8CT010ihRadCQEQkmljyzRmFPmBIqg4SEYkmli6ifwO+GT0W\nEREpdfkuG9rIlCq1wBqsbh8y8wrESdVBIiLRxNImkBQVAiKStCpsUq2dsPHO7qa0q6pjyzeHY2MI\n7e95xK2UP2gRqQj9r4fRbXDeetixFfrfTmmfnMaSb14BvAs8AjzsecRNhYCIJGkE1K6GT1OQSkFb\nCga2Y1PvlqpY8s352HARxaZCQESStDNstdIKgPRjh09JbrbFMGLJNx8F+sdx4E6oEBCRJPWB2kXw\n6/WwOAXXb4CaZXRtbpViieWO4SnAF4GpQIcnIU0vKSLlbjQ03AUdO0Cft2DFcWRmXSxFsfQOmuA5\neHqfFHBLlIS6QIWAiEg0seWbNcAOcRw4D1UHiYhEE0u+eRjwOtZDCGBX4KE4EsqhQkBEJJpY8s1X\nsSGgvdNLzo4joRwqBEREooll7KB1bDzUszJoEZEyEKYQmAN8D5tLYDvgWuC5OIMSEZHSUQNcBrzi\nHpcCfYuQrq42RESiKXi+2RObDSwJKgRERKIpeJvAOmw2sQFdCkdEREpaz85fQjvWG+hxtwzFuWNY\nRERiFqYQmOIeXqqqERGRWKmgERGJJnK+GeZKYF/gAqDR8/oUMDJqYiIi0v28DhwCbA5s5nnETVcC\nIiLRxJJvvhjHQUNQISAiEk1B5xPYzf09CuiBNQ53eJ5/NWpiEWkoaRGRaAo6n0AL+UuVr0ZJqAtU\nCIiIRFNW+aaqg0REoomldxDAt4AdyR4z6KKoiYmISGkJM4rojcDR2B3CVW55mziDEhGR0pGeQGaW\n+1sH/CvkvmdhQ1HPBu4A+gB/Ad7GJqmZDnwhYF9VB4mIRBNLvvmS+/sCMByrEnozxH7Dscy+j1u/\nGzgeuBk4MsT+KgRERKKJpU3gYWAgcCUwzW2bHOH4NcB693eR2142rdciIuXqdM/y593fvkQbVnoS\n0AosBW5z224G5gEzgauB3gH76kpARCSagt4sNh3Y1Wc5rIHAfVhD8grgXrf+BLAEy/z/CLwFXOyz\nfwq40LPe4h4iImKa3CPtAgpY0zI9YDmso4A/edbHA3/Iec0BWHWTH10JiIhEU9A2gQasAbcqZzmd\nUO4cA7kWAHsB/YDPgAOxRuYtgMXuWEeQ6X0kIqWhCjgc2AXr3HE71q4nZSjfZcNfyJQqVWxcwpwQ\n4vjNwDHYNJWvAicCjwJD3DGnA6cAq3z2Lavbn0W6j7prYMiJ8N0aeGQVvDUVVn4bXZ13B2WVb+of\nTqT4NoO+HfBRClIp+CwFW7QBuycdmIRS8InmRaSy1EPtWuvXAXabz5brsCphkaLSlYBI8fWEurfh\nwnWwKAV/3gA1H5MpFaS0lVW+WVZvRqQb2RoanoN+rdAwm+ChXaT0xJJvHg3Uu+VfAvcDX4ojoRwq\nBKTU7AQ9L4OelwLbJR2MiI9Y8s10F859sZu1vkVxppxUISBe6a7KSfV82B1q2uBnG+Ds9XaW/J87\n6UVKRSz55gz393Lge265KzePRaVCQNL2g34fQ+810O8jYJ/ihzDwn3C96zGTSsEVG6Dh7uLHIZJX\nLL2DPsCGdzga+Ds2fpB6FUmxNEC/v8OUgdDRC+4bBP0eIVNFWSTV9TYwbtpWVdBDPWakItRidwun\n60CHAWOLkK6uBARgT9ju08wZeCoFo1YAuxU3jF6nwMh2eDkFz6dgeDv0+G5xYxDpVGz55n5k7hAe\nAoyMKyEPFQICMAJqVsOHrgD4IGXrWaflxVAFfc+GhvegYSH0Oq3I6YuEEUu+2YwN8jbfrQ8Hno0j\noRwqBMSpOx8GtcNhK+1v7XlJRyRSomLJN2dibQDexuBZAa8tJBUC4rUb1jGhyNVAkewCA16B+kWu\n0VhtBlJssU4vmS4EalEhIJJrGPRbAX/aAHNTMP4zqG9JOiipOLHkm2cDNwLvACdhcw2fGUdCOVQI\nSHdyHByyMtN4vSYFPddi06qKFEsscwxfifUGagW2x+4afjxqQiJlbpVNk5Eeyfej9Pa1iUUkUkCN\nwEFuuQboX4Q0dSUgpa4vMA44HhgN/WfBYavhyhSMbIOaSxKOTypPLPnmScDL2FzAYFcDU+NIKIcK\nASlltdB/DuzRCke0uWEk9oeqs6Hf77HpVctmcg/pNmLrHdSH7N5BxZgSUoWAlLCqn8Khq2GDawO4\nPQUNxegwIZJPLMNGdLhHWs+uJCRSXnpvBV/pmznZ3x3YsHmSEYl0RZhC4CngF1hbwEHAvdjNYyIV\nrONJ+MMqWAisAS7sgOqnko5KJA7VWLvAfe5xIsWp69TVhpS4fudBzw7osR4anqDog9qJbKTg+WYP\nYF6hDxqSCgHpDqqB3kkHIeIUvE1gPfA6sE2XwhEpfxuw+iCRbinMzWKDgLnY8BHtblsKOCyuoERE\npDjCFAK/jD0KERGRHGoTEBGJJpb7BFp9Hu8D91OcyWWk7PX5CfRthV4dUH8nNhxDN9DrNOi70sU9\nBQ0WJ2XqEuBkrPtbPdZd9ArgWKAlxnR1JVAZvm1TNc5LwUcpGLsK+l+XdFAhfB2Gttuw0Z+kbMyg\n+luSDkoqXiz5pt+t8DPc35lxJOioEKgItZPhGs/8wdNTMGBB0lF1rs81cJkn7nkp6L846aik4sVS\nHbQKOMa9tho4GvgsZIJnAXOwsYbuwMYg2hZ4EZuu8i6gV9SgpZx0fAgzPV0s5wIsTyqa8NYshhme\n4VTmAtUfJxaOSIxGAX/DfpjL3fJooB+wb579hgNvYxk/wN3YkLv3YAUJwPXAKQH760qgMgyG2vfg\nkHY4YTX0awP2SjqoEBqg7m04qA0mroaadmD/pIOSildS+eZwbGCVgVhX1IexyWmWkbkC2Qv4R8D+\nJfVmJFYNwA+BM7CTju6iDpiIzbQ3JuFYRCCmfHMMNn/AXLf+BeD8kPtOwnoTLQVuAwYDb3ieH0Hw\nsNQqBEREoollesnJ2DzDN7j12cCdWK+hfAZidxU3Aiuw0Ue/4fO6fEE3e5ZbiLc3UjnqD5wIvTaH\ntU+gaUFFyk2Te8TqFffXO6nMDL8X5jgK+JNnfTzWBuCtDtobVQfFpRbqXofDV8OFG2BIO/Q6Nemg\nRCRWseSbj2INwelCYJzb1pk9sZ5B/bChp28BTscaho9xr7kBNQzH5XhoasvMfDU3ZTdkiUgZiyXf\nHIW1CawGFgHPYlU8YTQD/8aqkG7BuoOmu4i+gfUYCuoiqkJg05wOE1Zl+rGvTEGPtWjeW5FyFmu+\nWYvVMReLCoFNM8a6LT6QgjdTMG41NDyUdFAiEqvI+Wa+s8KfdnLgq6MmFlEKnbVuqiZouBHWD4Ie\nT8CKE4G2pIMSkdhEzjfz9Q7q7w44BtgDeMgd/FvY3AJS+lpghfqvi8gmeYbsaqD+blvcVB0kIhJN\nLGMHDQXWetbXum0iItLNhblZ7Fas+meKWz8C6+kjIiLdXNgGhN2A/bBLjafJvnEsLmoYFhGJpqzy\nTbUJiIhEE0ubgIiIlCkVAiIiFUyFgIhIBQtTCHwHG+dnJTY3QKtbFhGRCvAW8LkE0lXDsIhINLHk\nm8/GcdAQVAiIiERT0AHk0n4LbAE8AKzxJDQlcI/CKKv+riIiRVDQAeTSGrC5BMbmbI+7EBARkQqm\n6iARkWgKOtH8ucAVwLUBCZ0ZNTERESkt+QqB19zfaWSXLlXoLF1ERGKmgkZEJBqNHdQNqQeUiCRG\nhUBiep0GfVuh5xqofxCoTToiEZFSUs7VQV+Hoe3wWgpWpOCI1VB/W9JBiUi3F0u+OQaYCsx1618A\nzo8joRxlXAj0/jVcmoKUe8xPQf+lSUclIt1eLG0Ck4Gfk7lbeDZwXNSExGvtYpjekVmfDfRYnlg4\nIiJ5vOL+eqeUnFGEdMv4SoB6qHsTvtYGJ6yGmnZg/6SDEpFur6A3i6UtA0Z71scBH0ZNSLKshLYv\nwv+OA/oDjwHzE45JRMTXKKxNYBWwCBtVtLEI6ZbzlYCISBwKnm/2AK5yy3VAfaETyEOFgIhINLHk\nmy/Q9RuaxmBtCenHCmAS0Ay879l+sM++KgRK0z4w8AUYOA9qLsJOFESkNMSSb94APASMx6aa/A5w\nZBeOU421JYwALgB+0snrVQiUnp2sEfvWFDyTgt3bofbXSQclIv8RS8NwX+Bj4Gs526POJ3Ag8Cbw\nHnZloeESup2qI+GUPnY+AHBHDXxpAvDTBIMSkU0QphCYUKC0jgXudMsp4HTg+1gX1J8CnxYoHYlN\nqgM+Wc9/qoBWAFVrk4xIRDZNmLPxm3PW05cbEyOk0xv4ANgR63I61P0FuBgYBvzAJ50LPest7iHJ\nGQY1c+DUBhjVAy5aBct/AutuTDowkQrV5B5pFxCxliXMi8eRyfj7AUdgXUXPiJDO4cCp+DcANwIP\nAzvnbNccw6VpBNScDb02gxX3YHNPi0hpKEq+WQ08H3Gfu4DjPevDPMtnAXf47KOGYRGRaIqSb+6A\nNfCGVQMsx+6MTbsVmAXMxM4kN/fZT4WAiEg0seSbbUCre6wE3sC6icZNhYCISDRllW+W1ZsRESmC\nWIaSnhpym4iIdDP57hPoh9XnDwEGebbXA1vGGZSIiCTvx8A7QIf7m37Mwm70ipuqg0REookl3zwz\njoOGoEJARCSayPlm2JsKPo/d7dvXs+3WqIlFpJvFRESiiSXfbAaeBJZiQ0gsBu4rdCI+dCUgIhJN\nLPnmHGzAsJlufXPgiTgSyqFCQEQkmli6iK4G1gPrgAbsimBE1IRERKT0hBlK+mVgIDAZG/a5HXgu\nzqBERKQ0VAFbe9a3Bb5YpLRVHSQiEk3B880qrE0gCSoERESiKXibQAqYBuzZpXBERKSkhelP+jow\nGliAtQeAFQ5fiCsoTxq6T0BEJLzI+WaYFzcGbH83SkJdoEJARCSayPlmmC6i72JdQr/qltujJiIi\nIt1XMzYH8Hy3Phx4tgjpqmFYRCSaWPLNmdgVw3TPtllxJJRDhUBlqcLuTBeRrovljuEOYINnvTZq\nIiL59fsl9PoMqjug/iFsHgsRKRFnAzdicwmcBLxAcYaX1pVAZTgKtmmHhSloT8Hhq6H+z0kHJdJN\nxZZvjgWuco+D4kokhwqBilA3GX6TgpR7zEhBw4KkoxLppiLnm2HGDgKYjU03mXLLIgXy2fvwUgek\n+lizwLQUVC1JOioRyfghsBC4xT0WAD8oQrq6EqgMDVD3FuzXCuPaoF8r8KWkgxLppmLJN+cDgz3r\ng8l0F42TCoHKUQscB0xEw5SLbIpY8s3ngD6e9T4UZyhpFQIiItHEMsfwbdgcww+69cOx+wRmuQSv\njppoSBo2QkQkmsj5ZpiG4bfcI13CPOiW6yKFJiIiEoGqg0REoonljuE9gPuxYSNmu0eYYSPGuH3S\njxXYTWaDgMexxuXHgAFRgxYRkeKZDxwGjMSGlU4/oqgGPsR6fvwKOMdtPxe4PGAfXQmIiEQTS775\nrwIcYyzwjFueB2zulrdw635UCIiIRBNL76ADgWOBqcAaT0JTIqRzE/AKcB3wCTDQ89zHWBVRLvUO\nEhGJJpbeQROAHYBeZI8mGrYQ6A0cilX9SNfVQf0fIXUgVH8EK04Gnk46KBHp3sIUAntghUBXq2cO\nwSarX+bWl2DVQIuBYcDSPPs2e5Zb3KNC1d8JYw+Ey/rC7CEw/hFYtSvwRtKRiUhimtwjVjcDO23C\n/ncBx3vWf0XmquBnqGE4jCrosRZaPaNtfn8VcFoR0t4LGm6HhjuBfYqQnoh0XSz55jxgLdZLKEoX\nUbDJQZYD/T3bBgFPuOM9TnAXURUCWfq0wTxXAGxIwQFtwPiYE90HatvhmhT8LgV17RThrENEuiyW\nfLMx4BE3FQJZep8OQ9vhkhQcsRrq5hP7DFwDH4TrPVcfN6Vg4OPxpikimyCW+QTeBfYDRmNVQ0PQ\nkBEJWPN7WPoGXHQgrFkM/BFYFW+aVX2yy5laoLpXvGmKSKlpBh4mM3z0cODZIqSrK4HkHQqD2+H+\nFDyUgiHtUD0u6aBEJFAs+eZM7I7f6Z5tYdsENoUKgZJQ/R0Y9DIMmgZVxyYdjYjkFUu++ZL7my4E\nalEhICJSimIZQO5e4EasF89J2J3Df4qakIiIlJ6wtxePdQ+Af2JdO+OmYSNERKKJJd+8IuS2QlN1\nkIhINLHkm9N9ts2OI6EcKgRERKIp6H0Cp2LDEowiO9PvT3G6iIqISIIasDuD7wK2IXOn8OAipa8r\nARGRaMoq3yyrNyMiUgSxdBEVEZEypUJARKSCqRAQEalgKgRERCqYCgERkQqmQkBEpIKpEBARqWAq\nBEREKpgKARGRCqZCQESkgqkQEBGpYCoEREQqmAoBEZEKpkJARKSCqRAQEalgKgRERCqYCgERkQoW\ndyEwALgP+DfwGrAX0Ay8j01gPx04OOYYREQkIbcAE91yT2ze4guAn4TYtztML9mUdAAhNSUdQEhN\nSQcQUlPSAYTUlHQAITUlHUBITUkHEEJJTS9ZD+wH3OTW1wEr3HJVjOkWU1PSAYTUlHQAITUlHUBI\nTUkHEFJT0gGE1JR0ACE1JR1AHOIsBEYCy4CbgVeByUCNe+50YCbwZ6zKSEREEhBnIdAT+BJwnfvb\nDvzMrY8EdgE+BH4dYwwiIpJHnNUyWwDPA9u69X2xQuBbntc0Ag8DO/vs/yYwKsb4RETKzVvA6Cg7\n9IwpEIDFwHvA9sB84EBgLlY4LHavOQKYHbB/pDciIiKl54vAy1j9/xSs/v9WYJbb9gCweWLRiYiI\niIhIcm4ClpBdLXQxdqUwHfgnMCyBuHL5xZn2/4ANwKCiRuTPL85mSu8GvaDP8wxgHjAHuKLYQfnw\ni/MuMp/lO+5vkvxi3AV4AYvtZWCPBOLK5RfnF7G2w1nAQ0D/BOLKNQJ4ErvBdQ5wpts+CHgcq95+\njOR7NgbFeRRW9b4e65RT8vYDdiX7H8P7j3AGcH1RI/LnFyfYF/EPLDMohULAL86wN+gVk1+cX8V+\nZL3c+pBiB+Uj6HtPuwo4v3jh+PKL8THg6275ECyzSJpfnC+77QAnABcVOygfW2CFKEAd8DrwOeBX\nwDlu+7nA5cUPLUtQnDtgbbFPEqIQKIWxg54BPsnZ1upZrsPOspPmFyfA1WT+MUpBUJyldoOeX5yn\nAv8DrHXry4oakb+gzxPsMz0auLN44fjyi3EDdoc+2BnrB0WNyJ9fnNu77QBPAN8pakT+FgMz3HIb\nNuzNcOAwbBQE3N9vFz+0LH5xboldSc8Pe5BSKASCXAosBL4L/P+EYwlyGFbNMivpQELoDjfobQfs\nj1VjtAC7JxpN5/bDqjfeSjoQHz8GrsR+Q1cC5yUbTqA52O8IrBpjRIKx+GnErl5exDqxLHHbFwND\nE4rJTyOZOCMp5ULgF8DWwO1YlVCpqcFivMCzrdTOttOup3vcoNcTK6D2As4G7kk2nE4dB9yRdBAB\nTsMKgq2Bs8gM31JqJgI/Al7BrvrXJBtOljrgr8AksmsnSk0dNlDnJOyKoFtqJLjOdZs8zxVbI5lY\ndsbOCt5950Y+AAAHkklEQVRxj7XAu5TG2UEjwZ9ZvueKrZHsWB7FrgTS3gQGFzOgAI1s/Jn1xM4G\ntyx6NP4ayY7xU89yFZlxu5LWSPD/3/Z04Uw2Jr2wTik/9mybh9XDg3VWmVfsoHz4xZnWbdoE/Gzn\nWT4Mq+sqNbOxy8Nt3eN97ANfmmRQAby9q/LdoJe0B4D/csvbA72Bj5ILJ68Dsf/LRUkHEmARcIBb\n/hoR6oiLLN34X401sJdCJ5AqrNr0NeA3nu0PAce75eOx/9ckBcWZ+5qSdyf2D7sGu8N4InZpMxur\nw36Q0ugimo6zA4vzhJzn36Y0egf5fZ6leIOe3+fZC7gN++6nURqjNgZ97zcDJyUVVI7c7/wEYB+s\nimUG1gVz18Siy/D73zwT69XyOnBZcqFl2RdrWJ9BdrfqQVjj9XysF1vSbWt+cR6CNVi/B6zGrlYf\nTSpAERERERERERERERERERERERERERERKai/4D8Y15bAvQVMZ2/gjwU8nted2L0Nkwp4zG2w4R6C\nFPLzmQBcW6BjFdq7dP2+lt2A33bymijDFxyA/R9JAuKcXlKSlQrYvggbqCtXT2BdF9I5mHhuRtkC\nG0Buu85e6NEDG0M9n22xQQn9Rv7sSfDnU2zpOz2DvsdN1dXj9sRu5JtWwON/FRub5/kuxiRS8b6P\nnTHPIDPU7c3Y2dqz2CiX6auCRjLDRkzAboWfio3aeQDwNPA3bFyU67HMqAd2ZTEbu/PYO07JM9j8\nDzth475Md7GMcs//t2f7DWSGKjkYy0hmYHdh5poFrHL77UtmkhTvVKW4uK/BxqU/K+cYB5C5k3Ia\nNtDWC9i4OtPd+zje8xk8iV0pzPF8Pg+67a+TPZpt0Ps6wb32RewKye9KYAh2x+kcYDKZs/JGt+8t\n7rmtsRFA05/70W7/JuBhz/F+T2Y4g3exSYSmuX3GuO2DsTkGctPM5f1eHnfbmrE7uf+FDZh3gCf9\nOux/LX1H+hFue3rAtc2A57A7WYdgowG85B5fwT7vD8lMerQvVgjPdjE85ROjiHjshGXY6R90OnP8\nC3C3W/4c8IZbbiS7EHjPs08Tdqt5I5apPYYVHl9yy2npceo3A/7XLV+LnWGDnS32dek+hBUiANcB\n47HMYCGWAXhj9sodOHAWmclHLsQyfrAM+vc+++PSTlcz1Lg4vBkYbPwZNJL9+SwCBrr3MxurCgl6\nX8OABViG2wvLNH/nE9fvsUlJwCZ+Sc9K14hdyezpnvsO9rlXYQMTLsCukJpy3sO12IkA2GCGP3LL\np2IZPi6O9OQ338B/Jryg76UZK2T7uHVv+ldgc2qQs0+ri/kFMuNB3YENZwFWwL3mlnMnPZpFZqiY\neiRWqg7q/r6G1WF/7NbTo0emyAxw9W+Cxwt6jOwRJ1/CzhLBqkz2xc6SR2IZyd/JFAhjsREMwc72\nfgFshZ2pv4n9+HfDxrABy0gXA1/GrjgW5MTs5R34qsE90pOP3EJ2vf3d+HsWKyxudzF9wMYDaqXY\n+DPweozMRChTsM9jPf7va0/syiQ96N3d2EB4ufYhMyHJP8meaGUB9h2kX3eHi3Epdla8B7AyINa0\nKe7vq8CRbnk/Mmfpj+A/Uc5eLo3c7yWFFXodPvv8F3CMZz29T2/s/+Y0Mt/bgVgBmtYfu5KA7O/l\nWew7vsfzXiQmpTqKqBSGd2z2oNEEV+Wse+tyq9z6p9hcsC3AKcCf3PMHY1NrghUYh2JXEo9g9bxg\nP+Zd3eNz2PSB6eN2Ve57aQ943RXAD4B+WMYyJuB1uZ9BvnTTcfu9L9j48ws6TtBz7Z28LoW13Xh/\nu/1yXpPOrNdTuBO9oM8o6H2sxQrJg3NeuxeZz20E/g3Ip2JXLSOwqqlSGJixbKkQ6P6mYnXF6R/K\nwE083p5kqoOOxs7iBmNVH1OAX5IZkfILWD0wWIPrO1jVxIPYfAtTgXFkhgsehFUDPI9VyzR6tuez\nAjtz3detj8cKpM6Mwibc/hVWnTEGO4v2zmHd2VC7B2GfaT/gcKyKJ+h9vYhVlQzCqoOCGpj/RaZ+\nfyzB39nT2Fl2tUtrf+wqYSGwI3a23YBdDXbmaeB7bvmQgDRfINr3AnaldLpnPV0dlMJGCd2BTNXX\nY2RPEJWeH7eV7O9kFPY+L8CmGN0qRBzSRaoO6v5ew6bifAo783sV+/FB9lmp33LKZ/vLWJ31aKy+\n/wEss7+JzEnDz7CeO9M9+x6DNZauxRr6LsWuIM7HfvzV7rnTsB/4SVihUo1NzvN1NuaN7XisAbYG\na+jOHcrbzyTsimQ9Vhg8SuZMegbWbvIJG1+VeNdfwmaX2gprHH3VbQ96X81YIZdufPa74rkQu3Ia\n7167GMsI63Nefz/WpjHTbT+bzHwV92CNvO94Ysrl/X7TaR6HVd0t8Hn9MoK/l9z/k/T6JcAfsPaS\n9e79P+B5zbFY+8EKbNjoP7j30xP7nz3NPX8fNnfImVgD/3ZYAf0E3WP6VpGy0ER2g2M+vyBzNluu\nJhBPP//eZBqV9yY4ExeJna4ExCv3yiCfS+MMpERE+Tyi2Bo7k6/G2m1OjCENERERERERERERERER\nERERERERERERqUz/BzAcR0lG1IibAAAAAElFTkSuQmCC\n",
347 | "text/plain": [
348 | ""
349 | ]
350 | },
351 | "metadata": {},
352 | "output_type": "display_data"
353 | }
354 | ],
355 | "source": [
356 | "#Load the dataset\n",
357 | "data = np.loadtxt('SGD_data.txt', delimiter=',')\n",
358 | " \n",
359 | "#Plot the data\n",
360 | "plt.scatter(data[:, 0], data[:, 1], marker='o', c='b')\n",
361 | "plt.title('cricket chirps vs temperature')\n",
362 | "plt.xlabel('chirps/sec for striped ground crickets')\n",
363 | "plt.ylabel('temperature in degrees Fahrenheit')\n",
364 | "plt.xlim([13,21])\n",
365 | "plt.ylim([65,95])\n",
366 | "plt.show()"
367 | ]
368 | },
369 | {
370 | "cell_type": "markdown",
371 | "metadata": {},
372 | "source": [
373 | "Our goal is to find the equation of the straight line $h_\\theta(x) = \\theta_0 + \\theta_1 x$ that best fits our data points. The function that we are trying to minimize in this case is:\n",
374 | "\n",
375 | "$J(\\theta_0,\\theta_1) = {1 \\over 2m} \\sum\\limits_{i=1}^m (h_\\theta(x_i)-y_i)^2$\n",
376 | "\n",
377 | "In this case, our gradient will be defined in two dimensions:\n",
378 | "\n",
379 | "$\\frac{\\partial}{\\partial \\theta_0} J(\\theta_0,\\theta_1) = \\frac{1}{m} \\sum\\limits_{i=1}^m (h_\\theta(x_i)-y_i)$\n",
380 | "\n",
381 | "$\\frac{\\partial}{\\partial \\theta_1} J(\\theta_0,\\theta_1) = \\frac{1}{m} \\sum\\limits_{i=1}^m ((h_\\theta(x_i)-y_i) \\cdot x_i)$\n",
382 | "\n",
383 | "Below, we set up our function for h, J and the gradient:"
384 | ]
385 | },
386 | {
387 | "cell_type": "code",
388 | "execution_count": 10,
389 | "metadata": {
390 | "collapsed": false
391 | },
392 | "outputs": [],
393 | "source": [
394 | "h = lambda theta_0,theta_1,x: theta_0 + theta_1*x\n",
395 | "\n",
396 | "def J(x,y,m,theta_0,theta_1):\n",
397 | " returnValue = 0\n",
398 | " for i in range(m):\n",
399 | " returnValue += (h(theta_0,theta_1,x[i])-y[i])**2\n",
400 | " returnValue = returnValue/(2*m)\n",
401 | " return returnValue\n",
402 | "\n",
403 | "def grad_J(x,y,m,theta_0,theta_1):\n",
404 | " returnValue = np.array([0.,0.])\n",
405 | " for i in range(m):\n",
406 | " returnValue[0] += (h(theta_0,theta_1,x[i])-y[i])\n",
407 | " returnValue[1] += (h(theta_0,theta_1,x[i])-y[i])*x[i]\n",
408 | " returnValue = returnValue/(m)\n",
409 | " return returnValue"
410 | ]
411 | },
412 | {
413 | "cell_type": "markdown",
414 | "metadata": {},
415 | "source": [
416 | "Now, we'll load our data into the x and y variables;"
417 | ]
418 | },
419 | {
420 | "cell_type": "code",
421 | "execution_count": 11,
422 | "metadata": {
423 | "collapsed": false
424 | },
425 | "outputs": [],
426 | "source": [
427 | "x = data[:, 0]\n",
428 | "y = data[:, 1]\n",
429 | "m = len(x)"
430 | ]
431 | },
432 | {
433 | "cell_type": "markdown",
434 | "metadata": {},
435 | "source": [
436 | "And we run our gradient descent algorithm (without adaptive step sizes in this example):"
437 | ]
438 | },
439 | {
440 | "cell_type": "code",
441 | "execution_count": 12,
442 | "metadata": {
443 | "collapsed": false
444 | },
445 | "outputs": [
446 | {
447 | "name": "stdout",
448 | "output_type": "stream",
449 | "text": [
450 | "Local minimum occurs where:\n",
451 | "theta_0 = 25.1285525586\n",
452 | "theta_1 = 3.29726475625\n",
453 | "This took 565859 steps to converge\n"
454 | ]
455 | }
456 | ],
457 | "source": [
458 | "theta_old = np.array([0.,0.])\n",
459 | "theta_new = np.array([1.,1.]) # The algorithm starts at [1,1]\n",
460 | "n_k = 0.001 # step size\n",
461 | "precision = 0.001\n",
462 | "num_steps = 0\n",
463 | "s_k = float(\"inf\")\n",
464 | "\n",
465 | "while np.linalg.norm(s_k) > precision:\n",
466 | " num_steps += 1\n",
467 | " theta_old = theta_new\n",
468 | " s_k = -grad_J(x,y,m,theta_old[0],theta_old[1])\n",
469 | " theta_new = theta_old + n_k * s_k\n",
470 | "\n",
471 | "print \"Local minimum occurs where:\"\n",
472 | "print \"theta_0 =\", theta_new[0] \n",
473 | "print \"theta_1 =\", theta_new[1]\n",
474 | "print \"This took\",num_steps,\"steps to converge\""
475 | ]
476 | },
477 | {
478 | "cell_type": "markdown",
479 | "metadata": {},
480 | "source": [
481 | "For comparison, let's get the actual values for $\\theta_0$ and $\\theta_1$:"
482 | ]
483 | },
484 | {
485 | "cell_type": "code",
486 | "execution_count": null,
487 | "metadata": {
488 | "collapsed": false
489 | },
490 | "outputs": [
491 | {
492 | "name": "stdout",
493 | "output_type": "stream",
494 | "text": [
495 | "Actual values for theta are:\n",
496 | "theta_0 ="
497 | ]
498 | },
499 | {
500 | "ename": "AttributeError",
501 | "evalue": "'tuple' object has no attribute 'intercept'",
502 | "output_type": "error",
503 | "traceback": [
504 | "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
505 | "\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)",
506 | "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0mactualvalues\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstats\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinregress\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;32mprint\u001b[0m \u001b[0;34m\"Actual values for theta are:\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0;32mprint\u001b[0m \u001b[0;34m\"theta_0 =\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mactualvalues\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mintercept\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 4\u001b[0m \u001b[0;32mprint\u001b[0m \u001b[0;34m\"theta_1 =\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mactualvalues\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mslope\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
507 | "\u001b[0;31mAttributeError\u001b[0m: 'tuple' object has no attribute 'intercept'"
508 | ]
509 | }
510 | ],
511 | "source": [
512 | "actualvalues = sp.stats.linregress(x,y)\n",
513 | "print \"Actual values for theta are:\"\n",
514 | "print \"theta_0 =\", actualvalues.intercept\n",
515 | "print \"theta_1 =\", actualvalues.slope"
516 | ]
517 | },
518 | {
519 | "cell_type": "markdown",
520 | "metadata": {},
521 | "source": [
522 | "So we see that our values are relatively close to the actual values (even though our method was pretty slow). If you look at the source code of [linregress](https://github.com/scipy/scipy/blob/master/scipy/stats/_stats_mstats_common.py), it uses the convariance matrix of x and y to compute fastly. Below, you can see a plot of the line drawn with our theta values against the data:"
523 | ]
524 | },
525 | {
526 | "cell_type": "code",
527 | "execution_count": null,
528 | "metadata": {
529 | "collapsed": false
530 | },
531 | "outputs": [],
532 | "source": [
533 | "xx = np.linspace(0,21,1000)\n",
534 | "plt.scatter(data[:, 0], data[:, 1], marker='o', c='b')\n",
535 | "plt.plot(xx,h(theta_new[0],theta_new[1],xx))\n",
536 | "plt.xlim([13,21])\n",
537 | "plt.ylim([65,95])\n",
538 | "plt.title('cricket chirps vs temperature')\n",
539 | "plt.xlabel('chirps/sec for striped ground crickets')\n",
540 | "plt.ylabel('temperature in degrees Fahrenheit')\n",
541 | "plt.show()"
542 | ]
543 | },
544 | {
545 | "cell_type": "markdown",
546 | "metadata": {},
547 | "source": [
548 | "Notice that in the method above we need to calculate the gradient in every step of our algorithm. In the example with the crickets, this is not a big deal since there are only 15 data points. But imagine that we had 10 million data points. If this were the case, it would certainly make the method above far less efficient.\n",
549 | "\n",
550 | "In machine learning, the algorithm above is often called batch gradient descent to contrast it with mini-batch gradient descent (which we will not go into here) and stochastic gradient descent."
551 | ]
552 | },
553 | {
554 | "cell_type": "markdown",
555 | "metadata": {},
556 | "source": [
557 | "### Stochastic gradient descent"
558 | ]
559 | },
560 | {
561 | "cell_type": "markdown",
562 | "metadata": {},
563 | "source": [
564 | "As we said above, in batch gradient descent, we must look at every example in the entire training set on every step (in cases where a training set is used for gradient descent). This can be quite slow if the training set is sufficiently large. In stochastic gradient descent, we update our values after looking at each item in the training set, so that we can start making progress right away. Recall the linear regression example above. In that example, we calculated the gradient for each of the two theta values as follows:\n",
565 | "\n",
566 | "$\\frac{\\partial}{\\partial \\theta_0} J(\\theta_0,\\theta_1) = \\frac{1}{m} \\sum\\limits_{i=1}^m (h_\\theta(x_i)-y_i)$\n",
567 | "\n",
568 | "$\\frac{\\partial}{\\partial \\theta_1} J(\\theta_0,\\theta_1) = \\frac{1}{m} \\sum\\limits_{i=1}^m ((h_\\theta(x_i)-y_i) \\cdot x_i)$\n",
569 | "\n",
570 | "Where $h_\\theta(x) = \\theta_0 + \\theta_1 x$\n",
571 | "\n",
572 | "Then we followed this algorithm (where $\\alpha$ was a non-adapting stepsize):\n",
573 | "\n",
574 | " 1: Choose initial guess $x_0$
\n",
575 | " 2: for k = 0, 1, 2, ... do
\n",
576 | " 3: $s_k$ = -$\\nabla f(x_k)$
\n",
577 | " 4: $x_{k+1} = x_k + \\alpha s_k$
\n",
578 | " 5: end for\n",
579 | "\n",
580 | "When the sample data had 15 data points as in the example above, calculating the gradient was not very costly. But for very large data sets, this would not be the case. So instead, we consider a stochastic gradient descent algorithm for simple linear regression such as the following, where m is the size of the data set:\n",
581 | "\n",
582 | " 1: Randomly shuffle the data set
\n",
583 | " 2: for k = 0, 1, 2, ... do
\n",
584 | " 3: for i = 1 to m do
\n",
585 | " 4: $\\begin{bmatrix}\n",
586 | " \\theta_{1} \\\\ \n",
587 | " \\theta_2 \\\\ \n",
588 | " \\end{bmatrix}=\\begin{bmatrix}\n",
589 | " \\theta_1 \\\\ \n",
590 | " \\theta_2 \\\\ \n",
591 | " \\end{bmatrix}-\\alpha\\begin{bmatrix}\n",
592 | " 2(h_\\theta(x_i)-y_i) \\\\ \n",
593 | " 2x_i(h_\\theta(x_i)-y_i) \\\\ \n",
594 | " \\end{bmatrix}$
\n",
595 | " 5: end for
\n",
596 | " 6: end for\n",
597 | "\n",
598 | "Typically, with stochastic gradient descent, you will run through the entire data set 1 to 10 times (see value for k in line 2 of the pseudocode above), depending on how fast the data is converging and how large the data set is.\n",
599 | "\n",
600 | "With batch gradient descent, we must go through the entire data set before we make any progress. With this algorithm though, we can make progress right away and continue to make progress as we go through the data set. Therefore, stochastic gradient descent is often preferred when dealing with large data sets.\n",
601 | "\n",
602 | "Unlike gradient descent, stochastic gradient descent will tend to oscillate near a minimum value rather than continuously getting closer. It may never actually converge to the minimum though. One way around this is to slowly decrease the step size $\\alpha$ as the algorithm runs. However, this is less common than using a fixed $\\alpha$.\n",
603 | "\n",
604 | "Let's look at another example where we illustrate the use of stochastic gradient descent for linear regression. In the example below, we'll create a set of 500,000 points around the line $y = 2x+17+\\epsilon$, for values of x between 0 and 100:"
605 | ]
606 | },
607 | {
608 | "cell_type": "code",
609 | "execution_count": null,
610 | "metadata": {
611 | "collapsed": false
612 | },
613 | "outputs": [],
614 | "source": [
615 | "f = lambda x: x*2+17+np.random.randn(len(x))*10\n",
616 | "\n",
617 | "x = np.random.random(500000)*100\n",
618 | "y = f(x) \n",
619 | "m = len(y)"
620 | ]
621 | },
622 | {
623 | "cell_type": "markdown",
624 | "metadata": {},
625 | "source": [
626 | "First, let's randomly shuffle around our dataset. Note that in this example, this step isn't strictly necessary since the data is already in a random order. However, that obviously may not always be the case:"
627 | ]
628 | },
629 | {
630 | "cell_type": "code",
631 | "execution_count": null,
632 | "metadata": {
633 | "collapsed": false
634 | },
635 | "outputs": [],
636 | "source": [
637 | "from random import shuffle\n",
638 | "\n",
639 | "x_shuf = []\n",
640 | "y_shuf = []\n",
641 | "index_shuf = range(len(x))\n",
642 | "shuffle(index_shuf)\n",
643 | "for i in index_shuf:\n",
644 | " x_shuf.append(x[i])\n",
645 | " y_shuf.append(y[i])"
646 | ]
647 | },
648 | {
649 | "cell_type": "markdown",
650 | "metadata": {},
651 | "source": [
652 | "Now we'll setup our h function and our cost function, which we will use to check how the value is improving."
653 | ]
654 | },
655 | {
656 | "cell_type": "code",
657 | "execution_count": null,
658 | "metadata": {
659 | "collapsed": false
660 | },
661 | "outputs": [],
662 | "source": [
663 | "h = lambda theta_0,theta_1,x: theta_0 + theta_1*x\n",
664 | "cost = lambda theta_0,theta_1, x_i, y_i: 0.5*(h(theta_0,theta_1,x_i)-y_i)**2"
665 | ]
666 | },
667 | {
668 | "cell_type": "markdown",
669 | "metadata": {},
670 | "source": [
671 | "Now we'll run our stochastic gradient descent algorithm. To see it's progress, we'll take a cost measurement at every step. Every 10,000 steps, we'll get an average cost from the last 10,000 steps and then append that to our cost_list variable. We will run through the entire list 10 times here:"
672 | ]
673 | },
674 | {
675 | "cell_type": "code",
676 | "execution_count": null,
677 | "metadata": {
678 | "collapsed": false
679 | },
680 | "outputs": [],
681 | "source": [
682 | "theta_old = np.array([0.,0.])\n",
683 | "theta_new = np.array([1.,1.]) # The algorithm starts at [1,1]\n",
684 | "n_k = 0.000005 # step size\n",
685 | "\n",
686 | "iter_num = 0\n",
687 | "s_k = np.array([float(\"inf\"),float(\"inf\")])\n",
688 | "sum_cost = 0\n",
689 | "cost_list = []\n",
690 | "\n",
691 | "for j in range(10):\n",
692 | " for i in range(m):\n",
693 | " iter_num += 1\n",
694 | " theta_old = theta_new\n",
695 | " s_k[0] = (h(theta_old[0],theta_old[1],x[i])-y[i])\n",
696 | " s_k[1] = (h(theta_old[0],theta_old[1],x[i])-y[i])*x[i]\n",
697 | " s_k = (-1)*s_k\n",
698 | " theta_new = theta_old + n_k * s_k\n",
699 | " sum_cost += cost(theta_old[0],theta_old[1],x[i],y[i])\n",
700 | " if (i+1) % 10000 == 0:\n",
701 | " cost_list.append(sum_cost/10000.0)\n",
702 | " sum_cost = 0 \n",
703 | " \n",
704 | "print \"Local minimum occurs where:\"\n",
705 | "print \"theta_0 =\", theta_new[0] \n",
706 | "print \"theta_1 =\", theta_new[1]"
707 | ]
708 | },
709 | {
710 | "cell_type": "markdown",
711 | "metadata": {},
712 | "source": [
713 | "As you can see, our values for $\\theta_0$ and $\\theta_1$ are close to their true values of 17 and 2.\n",
714 | "\n",
715 | "Now, we plot our cost versus the number of iterations. As you can see, the cost goes down quickly at first, but starts to level off as we go through more iterations:"
716 | ]
717 | },
718 | {
719 | "cell_type": "code",
720 | "execution_count": null,
721 | "metadata": {
722 | "collapsed": false
723 | },
724 | "outputs": [],
725 | "source": [
726 | "iterations = np.arange(len(cost_list))*10000\n",
727 | "plt.plot(iterations,cost_list)\n",
728 | "plt.xlabel(\"iterations\")\n",
729 | "plt.ylabel(\"avg cost\")\n",
730 | "plt.show()"
731 | ]
732 | },
733 | {
734 | "cell_type": "markdown",
735 | "metadata": {},
736 | "source": [
737 | " "
738 | ]
739 | }
740 | ],
741 | "metadata": {
742 | "kernelspec": {
743 | "display_name": "Python 2",
744 | "language": "python",
745 | "name": "python2"
746 | },
747 | "language_info": {
748 | "codemirror_mode": {
749 | "name": "ipython",
750 | "version": 2
751 | },
752 | "file_extension": ".py",
753 | "mimetype": "text/x-python",
754 | "name": "python",
755 | "nbconvert_exporter": "python",
756 | "pygments_lexer": "ipython2",
757 | "version": "2.7.10"
758 | }
759 | },
760 | "nbformat": 4,
761 | "nbformat_minor": 0
762 | }
763 |
--------------------------------------------------------------------------------