├── .gitignore ├── 1.jpg ├── Albert-Bayesian_Computation.ipynb ├── Background_Detection_with_Gaussians.ipynb ├── Bayesian_Estimation_and_Simulation.ipynb ├── Bayesian_Linear_Regression.ipynb ├── EM_Algorithm.ipynb ├── Hoeffdings_Inequality.ipynb ├── Metropolis_Hastings.ipynb ├── Mixture_of_Gaussians.ipynb ├── Perceptron Learning Algorithm.ipynb ├── Pillow_examples.ipynb ├── Poisson-Point-Processes.ipynb ├── README.md ├── em.py ├── empty-closeup.png ├── empty-table.jpg ├── license.html ├── table1.jpg └── test.ipynb /.gitignore: -------------------------------------------------------------------------------- 1 | .ipynb_checkpoints 2 | *_2* 3 | -------------------------------------------------------------------------------- /1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rlabbe/tutorials/a338c1319d0c5b6a07dfa9da01bc025c1ffa1446/1.jpg -------------------------------------------------------------------------------- /Albert-Bayesian_Computation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "Examples from Albert's *Bayesian Computation in R*" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 1, 13 | "metadata": { 14 | "collapsed": true 15 | }, 16 | "outputs": [], 17 | "source": [ 18 | "%matplotlib inline\n", 19 | "import math\n", 20 | "import matplotlib.pyplot as plt\n", 21 | "import numpy as np\n", 22 | "import numpy.random as random\n", 23 | "import scipy.stats as stats" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": 2, 29 | "metadata": { 30 | "collapsed": false 31 | }, 32 | "outputs": [], 33 | "source": [ 34 | "def tstat(x, y):\n", 35 | " m, n = len(x), len(y)\n", 36 | " \n", 37 | " # pooled standard deviation\n", 38 | " sp = np.sqrt(((m-1)*np.std(x)**2 + (n-1)*np.std(y)**2)/(m+n-2))\n", 39 | " return (np.mean(x) - np.mean(y)) / (sp * np.sqrt(1./m + 1./n))" 40 | ] 41 | }, 42 | { 43 | "cell_type": "code", 44 | "execution_count": 3, 45 | "metadata": { 46 | "collapsed": false 47 | }, 48 | "outputs": [ 49 | { 50 | "name": "stdout", 51 | "output_type": "stream", 52 | "text": [ 53 | "2.1783893005\n", 54 | "2.17837840853\n", 55 | "2.18054528798\n" 56 | ] 57 | } 58 | ], 59 | "source": [ 60 | "x = stats.norm.rvs(loc=50,scale=10,size=10000)\n", 61 | "y = stats.norm.rvs(loc=50,scale=100,size=10000)\n", 62 | "x = random.randn(100000)*1 + 50\n", 63 | "y = random.randn(100000)*1 + 50\n", 64 | "\n", 65 | "print(tstat(x,y))\n", 66 | "\n", 67 | "# scipy's version, using Welch, which does not assume equal variance\n", 68 | "print(stats.ttest_ind(x, y, equal_var=False)[0])\n", 69 | "print(stats.ttest_rel(x, y)[0])" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 4, 75 | "metadata": { 76 | "collapsed": false 77 | }, 78 | "outputs": [ 79 | { 80 | "data": { 81 | "text/plain": [ 82 | "1.0011533138554058" 83 | ] 84 | }, 85 | "execution_count": 4, 86 | "metadata": {}, 87 | "output_type": "execute_result" 88 | } 89 | ], 90 | "source": [ 91 | "np.std(x)" 92 | ] 93 | }, 94 | { 95 | "cell_type": "code", 96 | "execution_count": 5, 97 | "metadata": { 98 | "collapsed": false 99 | }, 100 | "outputs": [ 101 | { 102 | "data": { 103 | "text/plain": [ 104 | "(2.1783784085286322, 0.029379039784604235)" 105 | ] 106 | }, 107 | "execution_count": 5, 108 | "metadata": {}, 109 | "output_type": "execute_result" 110 | } 111 | ], 112 | "source": [ 113 | "stats.ttest_ind(x, y, equal_var=False)" 114 | ] 115 | } 116 | ], 117 | "metadata": { 118 | "kernelspec": { 119 | "display_name": "Python 3", 120 | "language": "python", 121 | "name": "python3" 122 | }, 123 | "language_info": { 124 | "codemirror_mode": { 125 | "name": "ipython", 126 | "version": 3 127 | }, 128 | "file_extension": ".py", 129 | "mimetype": "text/x-python", 130 | "name": "python", 131 | "nbconvert_exporter": "python", 132 | "pygments_lexer": "ipython3", 133 | "version": "3.4.3" 134 | } 135 | }, 136 | "nbformat": 4, 137 | "nbformat_minor": 0 138 | } 139 | -------------------------------------------------------------------------------- /Bayesian_Estimation_and_Simulation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [ 10 | { 11 | "data": { 12 | "text/html": [ 13 | "\n", 220 | "\n" 239 | ], 240 | "text/plain": [ 241 | "" 242 | ] 243 | }, 244 | "execution_count": 1, 245 | "metadata": {}, 246 | "output_type": "execute_result" 247 | } 248 | ], 249 | "source": [ 250 | "# style the notebook\n", 251 | "from IPython.core.display import HTML\n", 252 | "import urllib.request\n", 253 | "response = urllib.request.urlopen('http://bit.ly/1LC7EI7')\n", 254 | "HTML(response.read().decode(\"utf-8\"))" 255 | ] 256 | }, 257 | { 258 | "cell_type": "markdown", 259 | "metadata": {}, 260 | "source": [ 261 | "# Bayesian Estimation\n", 262 | "\n", 263 | "\n", 264 | "This is from Candy, not me.\n", 265 | "\n", 266 | "\n", 267 | "\n", 268 | "Bayes. Variable or parameter X, and data Y.\n", 269 | "\n", 270 | "$$P(X|Y) = \\frac{P(Y|X)\\times P(X)}{P(Y)} = \\frac{P(Y|X)\\times P(X)}{\\int P(Y|X)\\times P(X) \\, \\mathrm{d}X}$$\n", 271 | "\n", 272 | "$P(Y|X)$ is the *likelihood*.\n", 273 | "\n", 274 | "$P(X)$ is the *prior*.\n", 275 | "\n", 276 | "$P(Y)$ is the *evidence*.\n", 277 | "\n", 278 | "\n", 279 | "# MAP\n", 280 | "\n", 281 | "Estimation requires finding most probable value of P(X|Y). Logical to find the most probably value - its maximum. *Maximu a posteriori* (MAP) estimate is the value of x that maximixes the posterior density:\n", 282 | "\n", 283 | "$$ \\max\\limits_x P(X|Y)$$\n", 284 | "\n", 285 | "Find this by differentiating, set to 0, and solve:\n", 286 | "\n", 287 | "$$\\nabla_x P(X|Y)\\bigr\\rvert_{X=\\hat{X}_{MAP}}=0$$\n", 288 | "\n", 289 | "where $\\rvert_x$ means *evaluated* at x.\n", 290 | "\n", 291 | "The *gradient vector* $\\nabla_x \\in R^{N_X \\times 1}$\n", 292 | "\n", 293 | "$$\\nabla_X := \\begin{bmatrix}\\frac{\\partial}{\\partial X_1}\\cdots\\frac{\\partial}{\\partial X_{N_X}}\\end{bmatrix}^\\mathsf{T}$$\n", 294 | "\n", 295 | "(not sure what $N_X$ is)\n", 296 | "\n", 297 | "\n", 298 | "$\\ln$ is useful since we often work with exponential densities. Logarithm is monotonically increasing so maximums hap[en at the same points. \n", 299 | "\n", 300 | "\n", 301 | "$$\\nabla_x \\ln P(X|Y)\\bigr\\rvert_{X=\\hat{X}_{MAP}}=0$$\n", 302 | "\n", 303 | "Apply Bayes rule:\n", 304 | "$$\\ln P(X|Y) = \\ln P(Y|X) + \\ln P(X) - \\ln P(Y)$$\n", 305 | "\n", 306 | "P(Y) is not a function of X, so map equation is\n", 307 | "\n", 308 | "$$\\nabla_x \\ln P(X|Y)\\bigr\\rvert_{X=\\hat{X}_{MAP}} = \\nabla_x (\\ln P(Y|X) + \\ln P(X))\\bigr\\rvert_{X=\\hat{X}_{MAP}} = 0$$" 309 | ] 310 | }, 311 | { 312 | "cell_type": "code", 313 | "execution_count": null, 314 | "metadata": { 315 | "collapsed": true 316 | }, 317 | "outputs": [], 318 | "source": [] 319 | }, 320 | { 321 | "cell_type": "code", 322 | "execution_count": null, 323 | "metadata": { 324 | "collapsed": true 325 | }, 326 | "outputs": [], 327 | "source": [] 328 | }, 329 | { 330 | "cell_type": "code", 331 | "execution_count": null, 332 | "metadata": { 333 | "collapsed": true 334 | }, 335 | "outputs": [], 336 | "source": [] 337 | }, 338 | { 339 | "cell_type": "code", 340 | "execution_count": null, 341 | "metadata": { 342 | "collapsed": true 343 | }, 344 | "outputs": [], 345 | "source": [] 346 | }, 347 | { 348 | "cell_type": "code", 349 | "execution_count": null, 350 | "metadata": { 351 | "collapsed": true 352 | }, 353 | "outputs": [], 354 | "source": [] 355 | } 356 | ], 357 | "metadata": { 358 | "kernelspec": { 359 | "display_name": "Python 3", 360 | "language": "python", 361 | "name": "python3" 362 | }, 363 | "language_info": { 364 | "codemirror_mode": { 365 | "name": "ipython", 366 | "version": 3 367 | }, 368 | "file_extension": ".py", 369 | "mimetype": "text/x-python", 370 | "name": "python", 371 | "nbconvert_exporter": "python", 372 | "pygments_lexer": "ipython3", 373 | "version": "3.4.3" 374 | } 375 | }, 376 | "nbformat": 4, 377 | "nbformat_minor": 0 378 | } 379 | -------------------------------------------------------------------------------- /Bayesian_Linear_Regression.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Bayesian Linear Regression\n", 8 | "\n", 9 | "Why not MLE? They overfit.\n", 10 | "\n", 11 | "Why not MAP? the prior fits the overfitting, but no representation of uncertainty.\n", 12 | "ex: fit data to something (financial data), then have to predict value for new x. How certain are you? MAP doesn't say\n", 13 | "\n", 14 | "Bayesian approach gets you level of uncertainty. Optimize the appropriate loss function.\n", 15 | "\n", 16 | "gives us $p(y \\mid x, D) $" 17 | ] 18 | }, 19 | { 20 | "cell_type": "markdown", 21 | "metadata": {}, 22 | "source": [ 23 | "## Setup\n", 24 | "\n", 25 | "given data $D = [[x_1, y_1], ... [x_n, y_n]]$\n", 26 | "\n", 27 | "model: $y_1, ... y_n$ independent given $w, Y_i \\sim N(w^Tx_i, a^{-1})$\n", 28 | "\n", 29 | "$a$ is the precision $1/\\sigma^2$\n", 30 | "\n", 31 | "\n", 32 | "$w \\sim N(\\boldsymbol 0, b^{-1}I), b > 0$\n", 33 | "\n", 34 | "w is $w = (w_1, w_2, ... w_d)$, each is independent, i.i.d.\n", 35 | "\n", 36 | "Assume a, b are known. So $\\theta = w$, the only unknown.\n", 37 | "\n" 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "## To model nonlinearies in Xs\n", 45 | "\n", 46 | "\n", 47 | "Can replace $x_i$ by $\\Phi(x_i)$ for some basis function $\\Phi(x_i) = (\\Phi_1(x_1) ... \\Phi_n(x_i))$\n" 48 | ] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "metadata": {}, 53 | "source": [ 54 | "## Compute Posterior Distribution\n", 55 | "\n", 56 | "First need likelihood: $P(D|\\theta) = P(D|w) \\propto exp(-a/2(y-Aw)^T(y-Aw))$\n", 57 | "\n", 58 | "A is the design matrix:\n", 59 | "\n", 60 | "$$A =\\begin{bmatrix} x_1^T \\\\ \\vdots \\\\ x_n^T\\end{bmatrix}$$\n", 61 | "\n", 62 | "$y = (y_1, ... y_n)^T$\n", 63 | "\n", 64 | "\n", 65 | "Posterior is the prob of w given the data $P(w|D) \\propto p(D|w) P(w)\\, \\, \\, (likelihood * prior )$\n", 66 | "\n", 67 | "$\\propto exp(-a/2(y-Aw)^T(y-Aw)-\\frac b 2 w^tw)$\n", 68 | "\n", 69 | "this is quadratic in w, so it is Gaussian:\n", 70 | "\n", 71 | "$$a(y-Aw)^T(y-Aw) = bw^Tw = a(y^Ty - 2w^TA^Ty + w^TA^TAw) + bw^Tw \\\\\n", 72 | "= ay^Ty - 2sw^TA^Ty + w^T(aA^TA + bI)w \n", 73 | "$$\n", 74 | "\n", 75 | "Want to make this look like a gaussian. Gaussian, in general has $(w - \\mu)\\Lambda (w-\\mu) = w^T\\Lambda w - 2w^T\\Lambda \\mu + const$ Lambda is the precision matrix (inverse of covariance).\n", 76 | "\n", 77 | "$\\Lambda = aA^TA + bI$\n", 78 | "\n", 79 | "$$2sw^TA^Ty = w^T\\Lambda\\mu \\\\\n", 80 | "aA^Ty = \\Lambda \\mu \\\\\n", 81 | "\\mu = a\\Lambda^{-1}A^Ty \\\\\n", 82 | "\\Lambda = aA^TA + BI$$\n", 83 | "\n", 84 | "Therefore, posterior of w given data is\n", 85 | "$$p(w|D) = N(w, \\mu, \\Lambda^{-1}) \\\\\n", 86 | "\\mu = a\\Lambda^{-1}A^Ty \\\\\n", 87 | "\\Lambda = aA^TA + bI$$\n", 88 | "\n", 89 | "Aside, is precision matrix invertible? In form $B+cI$. Think of eigenvalues. $Bu =\\lambda u, cIu = cu$\n", 90 | "\n", 91 | "$(B+cI)u = Bu + cIu = \\lambda u + cu = (\\lambda + c)u$. If all eigenvalues are strictly positive, then it is invertible.\n", 92 | "\n", 93 | "\n", 94 | "### MAP estimate of w\n", 95 | "\n", 96 | "Given $p(w|D) = N(w, \\mu, \\Lambda^{-1})$, we can get MAP. MAP is maximum a posteriori, value of w that maximizes the posterior distribution. MAP for a Gaussian is just the mean (top of bell curve!)\n", 97 | "\n", 98 | "so, $w_{MAP} = \\mu = a(aA^TA + bI)^{-1}A^Ty = \\boxed{(A^TA + \\frac b a I)^{-1}A^Ty}$\n", 99 | "\n", 100 | "Compare to MLE:\n", 101 | "\n", 102 | "$w_{MLE} = (A^TA)^{-1}A^Ty = A^+ y$\n", 103 | "\n", 104 | "Difference is the $\\frac b a I$, which is the *regularization parameter*. this may not correspond to a proper prior, you may make it from an improper prior, so more general than Bayesian. \n", 105 | "\n", 106 | "\n", 107 | "If you go back to $\\propto exp(-a/2(y-Aw)^T(y-Aw)-\\frac b 2 w^tw)$, $\\frac b 2 w^tw$ is the regularization term. You are regularizing according to the squared norm of $w$.\n", 108 | "\n", 109 | "test\n", 110 | "\n", 111 | "\n" 112 | ] 113 | }, 114 | { 115 | "cell_type": "markdown", 116 | "metadata": {}, 117 | "source": [ 118 | "## Predictive Distribution\n", 119 | "\n", 120 | "$p(y\\mid x, D)$. This is a *new* y that corresponds to a new $x$. \n", 121 | "\n", 122 | "In practice, use linear Gaussian models. But, to calculate ourselves:\n", 123 | "\n", 124 | "$p(y\\mid x, D) = \\int p(y \\mid x, D, w)p(w|x,D) dw$\n", 125 | "\n", 126 | "x is just notational, the x for the y, so\n", 127 | "\n", 128 | "$p(y \\mid x, D, w) = p(y \\mid x, w)$\n", 129 | "\n", 130 | "$p(w \\mid x, D) = p(w \\mid D)$\n", 131 | "\n", 132 | "Goal is to set $= \\int N(w \\mid ...) g(y) dw = g(y) \\propto N(y \\mid ...)$\n", 133 | "\n", 134 | "$$p(y\\mid \\mathbf x, D) = \\int N(y \\mid w^T\\mathbf x, a^{-1})N(w \\mid \\mu, \\Lambda^{-1}) dw \\\\\n", 135 | "\\propto \\int exp(-\\frac a 2 (y-w^T\\mathbf x)^2) exp(-\\frac 1 2 (w-\\mu)^T\\Lambda(w-\\mu))\\\\\n", 136 | "= \\int exp(-a/2 (y^2 - 2(w^T\\mathbf x)y + (w^T\\mathbf x)^2) - \\frac 1 2 (w^T\\Lambda w - 2w^T\\Lambda \\mu + \\mu^T\\Lambda\\mu))\n", 137 | "$$\n", 138 | "\n", 139 | "pull out 1/2, expression for exp is:\n", 140 | "\n", 141 | "$ay^2 - 2w^T\\mathbf xya + aw^T\\mathbf{xx}^Tw + w^T\\Lambda w - 2w^T\\Lambda\\mu$ (dropped constant $mu^T\\Lambda\\mu$)\n", 142 | "\n", 143 | "$= w^T(a\\mathbf{xx}^T + \\Lambda)w -2w^T(\\mathbf xya - \\Lambda\\mu) + ay^2$\n", 144 | "\n", 145 | "Now want to turn this into form gaussian form $(w-m)^TL(w-m) = w^TLw - 2w^TLm +m^TLm$.\n", 146 | "\n", 147 | "\n", 148 | "let $L = a\\mathbf{xx}^T + \\Lambda$\n", 149 | "\n", 150 | "$Lm = \\mathbf xya + \\Lambda \\mu$\n", 151 | "\n", 152 | "$m = L^{-1}(ay\\mathbf x + \\Lambda \\mu$ (assuming L inverse exists)" 153 | ] 154 | }, 155 | { 156 | "cell_type": "markdown", 157 | "metadata": {}, 158 | "source": [ 159 | "Plug in\n", 160 | "\n", 161 | "$$= w^TLw - 2w^TLm + m^TLm - m^TLm + ay^2 \\\\\n", 162 | "= (w-m)^TL(W-m) - m^TLm + ay^2$$\n", 163 | "\n", 164 | "\n", 165 | "$p(y \\mid x,D) \\propto \\int \\exp(-\\frac 1 2 (w-m)^TL(w-m)) \\times \\exp(frac 1 2 m^TLm - \\frac 1 2 ay^2) dw \\\\\n", 166 | "\\propto \\exp(\\frac 1 2 m^TLm - \\frac 1 2 ay^2)$. \n", 167 | "\n", 168 | "this is g(y). Now rearrange to look like a normal!\n", 169 | "\n", 170 | "$ay^2 - m^TLm$\n", 171 | "\n", 172 | "$$m^TLm = (ay\\mathbf x + \\Lambda\\mu)^TL^{-1}LL^{-1}(ay\\mathbf x + \\Lambda\\mu) \\\\\n", 173 | "= ay\\mathbf x^TL^{-1}(ay\\mathbf x) + 2ay\\mathbf x^TL^{-1}\\Lambda \\mu + \\mu^TL^{-1}\\Lambda \\mu \\\\\n", 174 | "= (a^2\\mathbf x^TL^{-1}x)y^2 + 2(a\\mathbf x^TL^{-1}\\Lambda \\mu) y + const $$\n", 175 | "\n", 176 | "So,\n", 177 | "\n", 178 | "$$ay^2 - m^TLm = (a - a^2\\mathbf x^TL^{-1}x)y^2 - 2(a\\mathbf x^TL^{-1}\\Lambda \\mu) y +c $$\n", 179 | "\n", 180 | "let $\\lambda = a(1-ax^TL^{-1}x)$\n", 181 | "\n", 182 | "$u = \\frac 1 \\lambda ax^TL^{-1}\\Lambda \\mu$\n", 183 | "\n", 184 | "$$ p(y \\mid \\mathbf x, D) \\propto \\exp(- \\frac \\lambda 2(y-u)^2)$$\n", 185 | "\n", 186 | "\n", 187 | "Can clean up:\n", 188 | "\n", 189 | "$\\boxed{u = \\mu^Tx \\\\ \\\\\n", 190 | "\\frac 1 \\lambda = \\frac 1 a + \\mathbf x^T\\Lambda^{-1} \\mathbf x \\\\\n", 191 | "p(y \\mid \\mathbf x,D) = N(y \\mid u, \\frac 1 \\lambda)}$\n", 192 | "\n", 193 | "So, it is a Gaussian!!!!\n", 194 | "\n", 195 | "\n" 196 | ] 197 | }, 198 | { 199 | "cell_type": "markdown", 200 | "metadata": {}, 201 | "source": [ 202 | "Show value for $\\lambda$:\n", 203 | "\n", 204 | "$$L^{-1} = (axx^T + \\Lambda)^{-1}$$\n", 205 | "\n", 206 | "Use Sherman Morrison Theorem.\n", 207 | "\n", 208 | "$$L^{-1} = \\Lambda^{-1} - \\frac{\\Lambda^{-1}axx^T\\Lambda^{-1}}{1+ax^T\\Lambda^{-1}x}$$\n", 209 | "\n", 210 | "$$x^T\\Lambda^{-1}x = \\alpha - \\frac{a\\alpha^2}{1+a\\alpha} = \\frac{\\alpha + a\\alpha^2 - a\\alpha^2}{1+a\\alpha}$$\n", 211 | "\n", 212 | "$$\\lambda = a(1-a\\frac{a\\alpha}{1+a\\alpha}) = \\frac{a}{1+a\\alpha}$$\n", 213 | "\n" 214 | ] 215 | }, 216 | { 217 | "cell_type": "markdown", 218 | "metadata": {}, 219 | "source": [ 220 | "$$\\mu^Tx = \\frac 1 \\lambda ax^TL^{-1}\\Lambda \\mu = \\mu^T \\frac 1 \\lambda ax^TL^{-1}\\Lambda\n", 221 | "$$\n", 222 | "\n", 223 | "$$\\mathbf x = \\frac 1 \\lambda ax^TL^{-1}\\Lambda$$\n", 224 | "\n", 225 | "$$L \\Lambda^{-1}x = \\frac 1 \\lambda a \\mathbf x\\\\\n", 226 | "(axx^T + \\Lambda)\\Lambda^{-1}x = axx^T\\Lambda^{-1}x + x = (a\\alpha + 1)x$$\n", 227 | "\n", 228 | "$$\\frac 1 \\lambda = \\frac{a\\alpha + 1}{\\alpha}$$" 229 | ] 230 | }, 231 | { 232 | "cell_type": "code", 233 | "execution_count": null, 234 | "metadata": { 235 | "collapsed": true 236 | }, 237 | "outputs": [], 238 | "source": [] 239 | }, 240 | { 241 | "cell_type": "code", 242 | "execution_count": null, 243 | "metadata": { 244 | "collapsed": true 245 | }, 246 | "outputs": [], 247 | "source": [] 248 | }, 249 | { 250 | "cell_type": "code", 251 | "execution_count": null, 252 | "metadata": { 253 | "collapsed": true 254 | }, 255 | "outputs": [], 256 | "source": [] 257 | }, 258 | { 259 | "cell_type": "code", 260 | "execution_count": null, 261 | "metadata": { 262 | "collapsed": true 263 | }, 264 | "outputs": [], 265 | "source": [] 266 | }, 267 | { 268 | "cell_type": "code", 269 | "execution_count": null, 270 | "metadata": { 271 | "collapsed": true 272 | }, 273 | "outputs": [], 274 | "source": [] 275 | }, 276 | { 277 | "cell_type": "code", 278 | "execution_count": null, 279 | "metadata": { 280 | "collapsed": true 281 | }, 282 | "outputs": [], 283 | "source": [] 284 | }, 285 | { 286 | "cell_type": "code", 287 | "execution_count": null, 288 | "metadata": { 289 | "collapsed": true 290 | }, 291 | "outputs": [], 292 | "source": [] 293 | } 294 | ], 295 | "metadata": { 296 | "kernelspec": { 297 | "display_name": "Python 3", 298 | "language": "python", 299 | "name": "python3" 300 | }, 301 | "language_info": { 302 | "codemirror_mode": { 303 | "name": "ipython", 304 | "version": 3 305 | }, 306 | "file_extension": ".py", 307 | "mimetype": "text/x-python", 308 | "name": "python", 309 | "nbconvert_exporter": "python", 310 | "pygments_lexer": "ipython3", 311 | "version": "3.5.1" 312 | } 313 | }, 314 | "nbformat": 4, 315 | "nbformat_minor": 0 316 | } 317 | -------------------------------------------------------------------------------- /EM_Algorithm.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# EM Algorithm\n", 8 | "\n", 9 | "## Maximum Likelihood Estimation\n", 10 | "\n", 11 | "Everyone starts with theory, I'm going to start with simple example of flipping coins. It is rare that we actually want to know something about flipping coins, but we *do* care about distributions in disease in populations and many other things which are structured the same way (bad wording).\n", 12 | "\n", 13 | "Start with a cois with unknown bias $\\beta$. Want to estimate bias.\n", 14 | "\n", 15 | "Easy. Flip it multiple times, and compute bias as $\\frac{\\text{number of heads}}{\\text{number of flips}}$. \n", 16 | "\n", 17 | "A coin flip is a discrete **bernoulli distribution**, which takes the value 1 with probability $p$, and 0 with probability $1-p$. For a coin, $p$ would be the bias of the coin, and 1 would indicate heads. " 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 56, 23 | "metadata": { 24 | "collapsed": false 25 | }, 26 | "outputs": [ 27 | { 28 | "name": "stdout", 29 | "output_type": "stream", 30 | "text": [ 31 | "bias: 0.628\n" 32 | ] 33 | } 34 | ], 35 | "source": [ 36 | "from scipy.stats import bernoulli\n", 37 | "\n", 38 | "def compute_coin_bias(bias, trials):\n", 39 | " heads_count = sum(bernoulli(bias).rvs(trials))\n", 40 | " return heads_count / trials\n", 41 | "\n", 42 | "trials = 1000\n", 43 | "bias = 0.612\n", 44 | "\n", 45 | "estimate = compute_coin_bias (bias, trials)\n", 46 | "print('bias: {}'.format(estimate))" 47 | ] 48 | }, 49 | { 50 | "cell_type": "markdown", 51 | "metadata": {}, 52 | "source": [ 53 | "**something about convergence rate**\n", 54 | "\n", 55 | "I won't motivate this yet, but now suppose we have 2 coins, where coin $i$ has some bias $\\beta_i$. We can compute the bias of each coin randomly selecting one, flipping it 10 times, and computing the . I'll make it easier by putting the computation in a function." 56 | ] 57 | }, 58 | { 59 | "cell_type": "code", 60 | "execution_count": 5, 61 | "metadata": { 62 | "collapsed": false 63 | }, 64 | "outputs": [], 65 | "source": [ 66 | "def compute_coin_bias(bias, trials):\n", 67 | " heads_count = sum(bernoulli(bias).rvs(trials))\n", 68 | " return heads_count / trials" 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": 31, 74 | "metadata": { 75 | "collapsed": false 76 | }, 77 | "outputs": [ 78 | { 79 | "data": { 80 | "text/plain": [ 81 | "0.60799999999999998" 82 | ] 83 | }, 84 | "execution_count": 31, 85 | "metadata": {}, 86 | "output_type": "execute_result" 87 | } 88 | ], 89 | "source": [ 90 | "compute_coin_bias(.612, 1000)" 91 | ] 92 | }, 93 | { 94 | "cell_type": "code", 95 | "execution_count": 53, 96 | "metadata": { 97 | "collapsed": false 98 | }, 99 | "outputs": [ 100 | { 101 | "name": "stdout", 102 | "output_type": "stream", 103 | "text": [ 104 | "set 1 has computed bias 0.94 for the actual bias 0.943038\n", 105 | "set 2 has computed bias 0.94 for the actual bias 0.962455\n", 106 | "set 3 has computed bias 0.04 for the actual bias 0.077691\n", 107 | "set 4 has computed bias 0.08 for the actual bias 0.087271\n", 108 | "set 5 has computed bias 0.84 for the actual bias 0.865581\n", 109 | "set 6 has computed bias 0.39 for the actual bias 0.464947\n", 110 | "set 7 has computed bias 0.60 for the actual bias 0.673961\n", 111 | "set 8 has computed bias 0.67 for the actual bias 0.680888\n", 112 | "set 9 has computed bias 0.42 for the actual bias 0.380928\n", 113 | "set 10 has computed bias 0.85 for the actual bias 0.819090\n" 114 | ] 115 | } 116 | ], 117 | "source": [ 118 | "from numpy.random import random\n", 119 | "for i in range(10):\n", 120 | " # create a random bias for this set of coins\n", 121 | " bias = random()\n", 122 | " beta = compute_coin_bias(bias, 100)\n", 123 | " \n", 124 | " print('set {:2d} has computed bias {:.2f} '\n", 125 | " 'for the actual bias {:.6f}'.format(i+1, beta, bias)) " 126 | ] 127 | }, 128 | { 129 | "cell_type": "markdown", 130 | "metadata": {}, 131 | "source": [ 132 | "There is any number of problems for which this applies. We may want to count cancer incidence in different age groups, sexes, or geographic locations. We may be doing quality control on samples drawn from different runs at a factory, such as batches of pharmaceutical drugs. It doesn't matter. \n", 133 | "\n", 134 | "So far this is very straightforward. Let me introduce some terminology and concepts. First, the computation heads / flips is based on an assumption of a **uniform, stationary process** of coin flip results. I.e., we are assuming that the proportion of heads to tails is equal at the start of the trial as at the end, and that the bias of a coin does not vary over time. To put this in the context of a real problem, suppose we are computing cancer incidence for the year 1997. We assume that the cancer incidence rate on on any day in the year is the same.\n", 135 | "\n", 136 | "Next, the bias of the coin in this experiment is called a **parameter**, and our attempt to estimate the bias is known as **parameter estimation**.\n", 137 | "\n", 138 | "Combining the ideas of the last two paragraphs we come up with this. We are trying to estimate the parameters of a statistical model for a process. That is, I make a statistical model of our process, the coin flip. I assume that the flips are uniformly distributed and stationary. I don't know this is true, I am assuming it. Given that assumption, I inspect the data of the coin flips and *estimate* the parameter *bias*. \n", 139 | "\n", 140 | "What we have done is known as **maximum likelihood estimation**, or MLE. MLE estimates the parameters of a statistical model. The literature uses $\\theta$ to represent the set of parameters (there is usually more than one) which we are estimating, and the MLE equation is written as\n", 141 | "\n", 142 | "$$\\begin{aligned}\n", 143 | "\\hat{\\theta} &= \\underset{\\theta}{\\operatorname{argmax}}\\,\\bigg[ P(x_{1..n}|\\theta)\\bigg] \\\\\n", 144 | "&= \\underset{\\theta}{\\operatorname{argmax}}\\,\\bigg[\\prod\\limits_{i=1}^N P(x_i|\\theta)\\bigg]\n", 145 | "\\end{aligned}\n", 146 | "$$" 147 | ] 148 | }, 149 | { 150 | "cell_type": "markdown", 151 | "metadata": {}, 152 | "source": [ 153 | "## Explaining the MLE Equation\n", 154 | "\n", 155 | "From the last section the MLE equation is\n", 156 | "\n", 157 | "$$\\hat{\\theta} = \\underset{\\theta}{\\operatorname{argmax}}\\,\\bigg[\\prod\\limits_{i=1}^N P(x_i|\\theta)\\bigg] $$\n", 158 | "\n", 159 | "This requires some explanation. $x_i$ is the i$^{th}$ data point. For our coin experiment it is one coin flip, so $x_1$ would be the result of the first coin flip. $x_{1..n}$ is the notation for the set of all $N$ flips. Next, $P(x_{1..n}|\\theta)$ is the notation for the probability for the set of coin flips $x_{1..n}$ *given* the parameters with the value $\\theta$. The bar in the equation ($|$) means *given*, and is pronounced as the English word *given*. \n", 160 | "\n", 161 | "Let's be specific to ensure you understand this. For our coin flip experiments $\\theta$ is a single parameter, the bias of the coin, and $x_{1..n}$ is the set of flips in one experiment. Then, $P(x_{1..n}|\\theta=.6)$ would be the probability that the set of coin flips would happen *if* the coin bias was 0.6. We are not asserting that the bias *is* 0.6, we are merely computing the probability of the flips occurring *if and only if* the bias was 0.6.\n", 162 | "\n", 163 | "We are not interested in the probability of the coin flips given some arbitrary bias, we want to *estimate* the bias that gives us the highest probability for the set of coin flips. Naively, you can think of us wanting to compute:\n", 164 | "\n", 165 | "```python\n", 166 | "max_prob = 0\n", 167 | "for bias in np.arange(0, 1, step=1.e-8):\n", 168 | " prob_bias = compute_probability(flips, bias)\n", 169 | " max_prob = max(max_prob, prob_bias)\n", 170 | "```\n", 171 | "This would compute the bias to within 8 digits of precision, but it is a terribly slow and inefficient algorithm compared to our simple computation of the number of heads divided by the number of flips. But most interesting problems do not have such a easy way to compute the results. \n", 172 | "\n", 173 | "The notation $\\underset{\\theta}{\\operatorname{argmax}}\\,\\bigg[ P(x_{1..n}|\\theta)\\bigg]$ expresses the idea" 174 | ] 175 | }, 176 | { 177 | "cell_type": "markdown", 178 | "metadata": {}, 179 | "source": [ 180 | "> compute the value for the parameters ($\\theta$) using some unspecified algorithm that has the highest probability (maximum likelihood) of being true for the data set." 181 | ] 182 | }, 183 | { 184 | "cell_type": "markdown", 185 | "metadata": {}, 186 | "source": [ 187 | "This is a *maximum likelihood estimator* because it is estimating the paramaters $\\theta$ which has the maximum likelihood of being true given the data set and assumed statistical model. \n", 188 | "\n", 189 | "That is pretty easy. What about $\\underset{\\theta}{\\operatorname{argmax}}\\,\\bigg[\\prod\\limits_{i=1}^N P(x_i|\\theta)\\bigg]$? This is based on the way to compute the probability for the set of $N$ independent trials is the product of the probabilities for each trial:\n", 190 | "\n", 191 | "$$\\begin{aligned}\n", 192 | "P(x_{1..n}|\\theta) &= P(x_1|\\theta)\\, \\times\\, P(x_2|\\theta)\\, \\times\\, P(x_2|\\theta)\\, \\times\\, ...\\, \\times\\, P(x_n|\\theta) \\\\\n", 193 | "&= \\prod\\limits_{i=1}^N P(x_i|\\theta)\n", 194 | "\\end{aligned}$$\n", 195 | "\n", 196 | "For our simple coin flipping example we can compute $\\underset{\\theta}{\\operatorname{argmax}}\\,\\bigg[ P(x_{1..n}|\\theta)\\bigg]$ with \n", 197 | "```python\n", 198 | "def compute_coin_bias(bias, TRIALS):\n", 199 | " rands = random(TRIALS)\n", 200 | " return np.sum(rands <= bias) / TRIALS\n", 201 | "```\n", 202 | "\n", 203 | "but most problems are not so easy. There is a lot I could say about solving this problem in general, but the EM Algorithm that we will be learning side-steps this problem nicely. " 204 | ] 205 | }, 206 | { 207 | "cell_type": "markdown", 208 | "metadata": {}, 209 | "source": [ 210 | "## Log-likelihood\n", 211 | "\n", 212 | "I will add one concept to this before I continue. Some of the difficulties in computing the MLE can be reduced by introducing the concept of the **log-likelihood**, which is just the logarithm of the likelihood function. Why would we do this? \n", 213 | "\n", 214 | "To find the maximum value for some function $f(x)$ we take the derivative of it and set it to zero, like this: $f'(x) = 0$. This requires that the function be differentiable, which is often either very difficult or impossible to do. \n", 215 | "\n", 216 | "In our case the function is a probability distribution. The logarithm is a *monotonically increasing* function. In other words, if we take the logarithm of a likelihood function the logarithm will reach its maximum value at the same points that the likelihood function does. This is true both for the local maxima and global maxima. \n", 217 | "\n", 218 | "Let's see this in code and a graph." 219 | ] 220 | }, 221 | { 222 | "cell_type": "code", 223 | "execution_count": 63, 224 | "metadata": { 225 | "collapsed": false 226 | }, 227 | "outputs": [ 228 | { 229 | "data": { 230 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEACAYAAABRQBpkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4k+X6wPHv01KQvSm0tJQNRcpQEVSg4gBRERcyFEFE\njwKCx6OM40BFxYEDByJTPfxERVFAhoIUEJCC7E2ZBUrZoy3Q9fz+eNJaatskbZI3Se/PdeVqknfd\neSHvnfeZSmuNEEKI4i3A6gCEEEJYT5KBEEIISQZCCCEkGQghhECSgRBCCCQZCCGEwIFkoJTqopTa\nqZTao5Qans86423LNymlWtneC1NKLVVKbVNKbVVKPZNj/dFKqcNKqQ22RxfXfSQhhBDOKlHQQqVU\nIPAJcCtwBFirlJqjtd6RY52uQAOtdUOl1PXABKAtkAY8q7XeqJQqB/yllPpVa70T0MD7Wuv33fOx\nhBBCOMPenUEbIE5rfUBrnQbMBO7JtU434EsArfUaoJJSKlhrfUxrvdH2fhKwAwjNsZ1yxQcQQghR\ndPaSQSgQn+P1Ya68oOe3Tu2cKyilIoBWwJocbw+xFStNUUpVciJmIYQQLmYvGTg6VkXuX/nZ29mK\niGYBQ213CGCKkuoCLYEEYJyDxxFCCOEGBdYZYOoJwnK8DsP88i9ondq291BKBQE/AP/TWv+UtYLW\n+njWc6XUZGBuXgdXSsnASUII4SSttdPF8PbuDNYBDZVSEUqpksBDwJxc68wB+gIopdoCZ7XWiUop\nBUwBtmutP8y5gVKqVo6X9wJb8gtAay0PrXnllVcsj8EbHnIe5FzIuSj4UVgF3hlordOVUoOBRUAg\nMEVrvUMp9aRt+USt9XylVFelVByQDPS3bX4j8DCwWSm1wfbeSK31QuBtpVRLTHHSfuDJQn8CIYQQ\nRWavmAit9QJgQa73JuZ6PTiP7f4gnzsPrXVf58IUQgjhTtID2UdER0dbHYJXkPPwNzkXf5NzUXSq\nKGVM7qaU0t4cnxBCeBulFLoQFch2i4mEEL7HtN8Q/s6VP5YlGQjhp+Su2r+5OuFLnYEQQghJBkII\nISQZCCGEQJKBEMKDIiIiWLJkCW+99RYDBw4E4MCBAwQEBJCZmenUvnJv17VrV77++msApk+fTvv2\n7V0bfD4CAgLYt2+fR47lTlKBLITwGKUUSilGjhzp8n3Pnz/f5fssTuTOQAghhCQDIYRnaa0ZPXo0\njzzySJ7Lf/jhB+rWrcv27dvRWjN27FgaNGhAtWrVeOihhzhz5kye20VHRzNlypQr3nv++eepUqUK\n9erVY+HChdnvHz16lG7dulG1alUaNmzI5MmTs5ddvnyZYcOGERoaSmhoKM8++yypqanZy999911C\nQkKoXbs2U6dOLcqp8CqSDIQQHpdXG3mtNdOmTWPEiBEsWbKEyMhIxo8fz5w5c1i+fDkJCQlUrlyZ\nQYMG5bvPnPtds2YNTZo04dSpU7zwwgsMGDAge1nPnj0JDw8nISGBWbNmMWrUKJYuXQrAG2+8QWxs\nLJs2bWLTpk3ExsYyZswYABYuXMi4ceNYvHgxu3fvZvHixa48LdayerhVO0OxaiGE8+x9d8A1D2dF\nREToxYsX69GjR+uHH35Ya631/v37tVJKv/vuuzoyMlIfOXIke/2mTZvqJUuWZL8+evSoDgoK0hkZ\nGdnbZWRkaK21jo6O1lOmTNFaaz1t2jTdoEGD7O2Sk5O1UkonJibqQ4cO6cDAQJ2UlJS9fOTIkbpf\nv35aa63r1aunFyxYkL1s0aJFOiIiQmutdf/+/fXIkSOzl+3evVsrpfTevXudPxlFlN+/se19p6+3\nUoEsRDHkjZ2Tx40bx0svvURISEj2ewcOHODee+8lIODvQowSJUqQmJhod381a9bMfl6mTBkAkpKS\nOHHiBFWqVKFs2bLZy8PDw/nrr78ASEhIoE6dOlcsO3r0aPay66677opl/kKKiYQQXuHXX39lzJgx\n/Pjjj9nvhYeHs3DhQs6cOZP9SElJoVatWgXsqWAhISGcPn2apKSk7PcOHTpEaGho9vIDBw7kuaxW\nrVocOnToimX+QpKBEMLjdB63Js2aNWPhwoUMGjSIuXPNTLj/+te/GDVqVPZF98SJE8yZk3uyReeE\nhYVxww03MHLkSC5fvszmzZuZOnUqDz/8MAC9evVizJgxnDx5kpMnT/Laa69lL+vRowfTp09nx44d\npKSk8OqrrxYpFm8iyUAI4VFZFb05K3uznkdFRTFv3jwGDhzIokWLGDp0KN26deP222+nQoUKtGvX\njtjY2H9sl98xcr+X5ZtvvuHAgQOEhIRw33338dprr9GpUycAXnzxRa699lqioqKIiori2muv5cUX\nXwSgS5cuDBs2jE6dOtGoUSNuueUWvxkhVuYzEMIP2ca0tzoM4Ub5/RsXdj4DuTMQQgghyUAIIYQk\nAyGEEEgyEEIIgSQDIYQQSDIQQgiBJAMhhBBIMhBCCIEkA6925gyMGgWNG8O//gWXL1sdkRBFlzX1\npavlnM9gxowZdO7cOXtZYaemzLndU089lT2UdUxMDGFhYS6I2j53na/cZNRSL6U1DBgAQUEwfTq8\n9x506gRLlsBVV1kdnRCFl9dQEa7eb58+fejTp49L9z9hwgSX7s9R7jpfucmdgZeaORN27YIvv4R2\n7eD776FyZZg40erIhBD+SJKBF0pOhmHDzB1B1l1AQAC88QaMHWuWC+EP7E0x+c4772RPMTl58mSH\ni3umT59O+/bt81z2xx9/EB4ezvLlywGYOnUqkZGRVKlShS5duuQ7LHW/fv146aWXrnjv/fffJzg4\nmJCQEKZPn579/rlz5+jbty81atQgIiKCN954I3scIa01Y8aMISIiguDgYB599FHOnz+fve3XX39N\nnTp1qFatGm+++abdz+oqkgy80Pffw3XXmUdOLVpAx44wfrw1cQnhavammPzggw9YsmQJe/bsISYm\npsjFJQsXLqR37978+OOPdOjQgZ9//pm33nqL2bNnc/LkSdq3b0+vXr3y3DZ3cc2xY8c4f/48R48e\nZcqUKQwaNIhz584BMGTIEC5cuMD+/ftZtmwZX331FdOmTQNg2rRpfPnll8TExLBv3z6SkpIYPHgw\nANu3b+fpp59mxowZHD16lFOnTnH48OEifWaHFWZ6NE89KKbTXt50k9Y//pj3so0btQ4L09o2058Q\nebL33WE0LnkURkRERPZUlvXr1y9wislRo0ZlL4uLiytwisnc017edNNN2cuUUvrNN9/UderU0du2\nbct+v0uXLtnbaK11RkaGLlOmjD506FD2dlnH69evn37xxRe11lovXbpUly5dOnvKTa21rlGjhl6z\nZo1OT0/XJUuW1Dt27MheNnHiRB0dHa211rpTp056woQJ2ct27dqlg4KCdHp6un711Vd1r169spcl\nJyfrkiVLXjH1Z5b8/o2RaS/9w65dsGcP3HVX3sujoqBsWYiNhbZtPRub8B/6Fe8Y3vro0aMFTjHZ\npk2b7GW1a9cu0rHGjx9P3759iYyMzH7v4MGDDB06lOeee+6KdY8cOWK3tVDVqlWvmI6zTJkyJCUl\ncfLkSdLS0v7xuY4cOZL9uXIvS09PJzExkYSEhCs+Z5kyZahatWrhPrCTpJjIy0ybBn37mlZEeVEK\nevSA777zbFxCuIO9KSbj4+Ozl+V8Xhjff/89s2fPZnyOctbw8HC++OKLK6bVTE5Opm0+v7QcKaaq\nVq0aQUFB//hcWRf5vD5ziRIlqFmz5j8+c0pKCqdOnXLykxaOJAMv89NP0LNnwes8+CDMmgWZmZ6J\nSQh3sTfF5LRp09i5cycpKSm8/vrrRTpWSEgIS5Ys4aOPPuLzzz8HzLSab775Jtu3bwdMxe/333+f\n5/b67+LrAgUGBtKjRw/++9//kpSUxMGDB/nggw+umFbzgw8+4MCBAyQlJTFq1Ch69uxJQEAA999/\nP/PmzWPlypWkpqby8ssvk+mhL7okAy9y4IDpaNayZcHrNWv2d1GREL7M3hSTzzzzDDfffDONGjWi\nXbt2AJQqVcrufvObVjMsLIwlS5YwduxYpk6dSvfu3Rk+fDg9e/akYsWKNG/enEWLFv1ju4L2mZeP\nP/6YsmXLUq9ePdq3b0+fPn3o378/AI899hiPPPIIHTp0oF69epQpU4aPP/4YMPNAf/rpp/Tu3ZuQ\nkBCqVKnisc5tdqe9VEp1AT4EAoHJWuu381hnPHAHkAL001pvUEqFAV8BNQANfKG1Hm9bvwrwLVAH\nOAD00FqfzWO/2pFM7C8mToQVK+B//7O/7qhRprmpreGFEFfwx2kvd+zYQfPmzUlNTb2irL648ui0\nl0qpQOAToAsQCfRSSjXNtU5XoIHWuiHwBJDVTS8NeFZr3QxoCwxSSjWxLRsB/Ka1bgQssb0u9hYt\nghw96AvUqRPExLg1HCEsN3v2bC5fvsyZM2cYPnw43bp1k0TgJvbOahsgTmt9QGudBswE7sm1Tjfg\nSwCt9RqgklIqWGt9TGu90fZ+ErADCM29je1v9yJ/Eh+Xlga//w633+7Y+u3awcaN0gFN+LcvvviC\n4OBgGjRoQFBQkGVDQhQH9pqWhgI5q/APA9c7sE5tIDHrDaVUBNAKWGN7K1hrnbU8EQh2Jmh/tGYN\n1K0LwQ6eibJlTd3CqlVw223ujU0IqyxYsMDqEIoNe8nA0ULH3OVT2dsppcoBs4ChtjuEK1e0dezI\nb8ejR4/Ofh4dHU10dLSDIfmWZctM0Y8zbr7ZFBVJMhCi+IqJiSHGBWXG9pLBESBnVXYY5pd/QevU\ntr2HUioI+AH4n9b6pxzrJCqlamqtjymlagHH8wsgZzLwZ3/+CY8+6tw20dGQa6gUIUQxk/tH8quv\nvlqo/dirM1gHNFRKRSilSgIPAXNyrTMH6AuglGoLnNVaJyrT7moKsF1r/WEe22Rd+h4FfqIY09ok\nA1vLOYe1awebN0PSP+63hBDCOQXeGWit05VSg4FFmKalU7TWO5RST9qWT9Raz1dKdVVKxQHJQH/b\n5jcCDwOblVIbbO+N1FovBMYC3ymlBmBrWurqD+ZL4uKgdGkIDbW/bk5lysDVV8P69dChg3tiE77L\nE2PgC/9ht5+BlYpLP4Ovv4a5cws3xMTgwabiOdfQKkKIYsot/QyEZ6xe7XwRUZbrroO1a10bjxCi\n+JFk4AWKmgzWrXNtPEKI4keKiSyWkgLVq8Pp0+DAkCv/kJFhpsM8cACqVHF5eEIIHyPFRD5q82Zo\n0qRwiQAgMBBat5a7AyFE0UgysNjGjfZHKbVH6g2EEEUlycBikgyEEN5AkoHFNm6EVq2Kto/Wrc1+\nhBCisKQC2UIZGVChAiQkmL+FlZlptj9yBCpWdF18QgjfIxXIPmj3bqhVq2iJAMwkN1dfDVu2uCYu\nIUTxI8nAQq6oL8gSFWVaJgkhRGFIMrCQK+oLskgyEEIUhSQDC23cCC1auGZfUVFSTCSEKDxJBhba\nutWU9btC8+YmGWRmumZ/QojiRZKBRc6ehfPnITzcNfurXBkqVYKDB12zPyFE8SLJwCLbtkHTpqYl\nkKs0by71BkKIwpFkYJFt26BZM9fuU5KBEKKwJBlYxB3JoFkz2LHDtfsUQhQPkgws4o5k0LQpbN/u\n2n0KIYoHSQYWcUcyaNLE9GrOyHDtfoUQ/k+SgQVOn4bkZAgLc+1+y5WDGjXMRDdCCOEMSQYW2LYN\nIiNBOT2UlH1SVCSEKAxJBhbYvt31RURZIiOlElkI4TxJBhbYvRsaN3bPviMj5c5ACOE8SQYW2L0b\nGjVyz76bNpU7AyGE8yQZWMATycCP5wQSQriBJAMPS0834wfVr++e/VeubFoVHT7snv0LIfyTJAMP\nO3DAzG5WqpT7jtG4sbn7EEIIR0ky8DB3FhFladRIkoEQwjmSDDzMU8lg1y73HkMI4V8kGXiYJ5KB\nFBMJIZwlycDD9uyBhg3dewwpJhJCOEuSgYd54s6gbl3Tmig11b3HEUL4D0kGHnTxIiQmQp067j1O\nUJCZTnPfPvceRwjhPyQZeFBcHNSrB4GB7j+WVCILIZwhycCDPFFfkEXqDYQQzrCbDJRSXZRSO5VS\ne5RSw/NZZ7xt+SalVKsc709VSiUqpbbkWn+0UuqwUmqD7dGl6B/F+3miviCLtCgSQjijwGSglAoE\nPgG6AJFAL6VU01zrdAUaaK0bAk8AE3IsnmbbNjcNvK+1bmV7LCzCZ/AZnkwGcmcghHCGvTuDNkCc\n1vqA1joNmAnck2udbsCXAFrrNUAlpVRN2+sVwJl89u2GqV28myQDIYS3spcMQoH4HK8P295zdp28\nDLEVK01RSlVyYH2f58k6g5AQuHABzp/3zPGEEL7NXjJwdCDk3L/y7W03AagLtAQSgHEOHsdnnT1r\n5j2uVcszx1PKJB65OxBCOKKEneVHgJzTtodhfvkXtE5t23v50lofz3qulJoMzM1v3dGjR2c/j46O\nJjo62k7I3mnPHlN04455j/OTVVR07bWeO6YQwrNiYmKIiYkp8n6ULmAWFKVUCWAXcAtwFIgFemmt\nd+RYpyswWGvdVSnVFvhQa902x/IIYK7WunmO92pprRNsz58FrtNa987j+Lqg+HzJjBkwdy7MnOm5\nY778MgQEQI58KoTwc0optNZO/+wssJhIa50ODAYWAduBb7XWO5RSTyqlnrStMx/Yp5SKAyYCT+cI\n6htgFdBIKRWvlOpvW/S2UmqzUmoT0BF41tnAfY0n6wuySCWyEMJRBd4ZWM2f7gx694Y77oBHHvHc\nMWNj4emnYd06zx1TCGEtt9wZCNfxZLPSLFkVyH6ST4UQbiTJwAO0NhdlTxcTVa4MV10Fx4559rhC\nCN8jycADjh+HkiWhShXPH1vqDYQQjpBk4AFWFBFlkTGKhBCOkGTgAVYmg4YNZShrIYR9kgw8wMpk\n0KiRadYqhBAFkWTgAVb0McjSsKEkAyGEfZIMPMDKO4P69WH/fsjIsOb4QgjfIMnAzTIyYO9eaNDA\nmuOXKQPVq8OhQ9YcXwjhGyQZuFl8PFSrBmXLWheDFBUJIeyRZOBmVtYXZJFkIISwR5KBm1lZX5BF\nkoEQwh5JBm7mDclAeiELIeyRZOBmWZPaWEnuDIQQ9kgycDMrBqjLrV49U5GdlmZtHEII7yXJwI1S\nU+HwYahb19o4SpaEkBDT30AIIfIiycCN9u2DsDBzMbaaDEshhCiIJAM38ob6gixSbyCEKIgkAzfy\nhvqCLFmzngkhRF4kGbiRNzQrzSLFREKIgkgycCNvSgZSTCSEKIgkAzfypjqDOnXMXMiXLlkdiRDC\nG0kycJOkJDh9GmrXtjoSo0QJkxD27rU6EiGEN5Jk4CZxcWYugQAvOsNSbyCEyI8XXar8izfVF2SR\negMhRH5KWB2Av/Km+oIsDRvC+vVWRyHi4+H4cdMZsXFj7+iUKITcGbiJN/UxyCLFRNY5cQJefhnC\nw+Gaa2DgQOjRA6pWhbvvhj/+sDpCUdxJMnATKSYSAFrDl19CZCQkJsKCBebv+vWwY4e5S7jnHujb\nF+6/3zQ6EMIKSmttdQz5Ukppb46vIFWrmi97jRpWR/K3zEwoV84UUZQrZ3U0/i8tDR5/3Fz4Z8yA\nqKj81718GUaNgu+/hzlzoGVLz8Up/ItSCq21cnY7uTNwg1OnICPDTETvTQICzHDWcXFWR+L/Ll6E\ne+81/xfWrCk4EQCUKgXjxplH587w55+eiVOILJIM3CBr3mPldG52P6k3cL+MDHj4YShTBmbPNn8d\n9eCDMH06dOsGW7a4LUQh/kGSgRt4Y31BFqk3cL/nnzdl/19/DUFBzm9/xx3w4YemYjkhwfXxCZEX\naVrqBt6eDKTlivvMnAlz50JsrCn6KazevU3SfvBBiIkxPciFcCe5M3ADb+xjkEWKidwnLg6eeQa+\n+w4qVy76/l56CUqXhtdeK/q+hLBHkoEbeGMfgyxSTOQeWfUEL70ErVq5Zp8BAaaoafJkuZsT7ifJ\nwMUyM00yaNzY6kjyVrOmaely9qzVkfiXjz4yv+IHDXLtfmvWhE8+MU1UZcRZ4U52k4FSqotSaqdS\nao9Sang+64y3Ld+klGqV4/2pSqlEpdSWXOtXUUr9ppTarZT6VSlVqegfxTvEx0OlSlC+vNWR5E0p\naNBA7g5cae9eePNNmDTJPQMT3ncfNGsGY8a4ft9CZCnwv65SKhD4BOgCRAK9lFJNc63TFWigtW4I\nPAFMyLF4mm3b3EYAv2mtGwFLbK/9wq5d0KSJ1VEUTOoNXGvYMNOCqEED9x3jk09g4kTz/0sId7D3\nO6YNEKe1PqC1TgNmAvfkWqcb8CWA1noNUEkpVdP2egVwJo/9Zm9j+9u9cOF7n507vT8ZyHzIrjN/\nvrlADxvm3uPUqgUjRsCzz7r3OKL4spcMQoH4HK8P295zdp3cgrXWibbniUCwnfV9xs6d3ltfkKVx\nY/mF6Qppaebi/OGHRWtG6qghQ2DfPvjlF/cfSxQ/9pKBowMD5e5r6/CAQrbBh3xzAKI8+EIxUdOm\nJmmJopk61YxC2rWrZ45XsiS89x4MH25aLwnhSva6shwBwnK8DsP88i9ondq29wqSqJSqqbU+ppSq\nBRzPb8XRo0dnP4+OjiY6OtrOrq3lC8VEjRubYqLMTO+aic2XXLpkKnR/+MGzx73zTlNZPXMm9Onj\n2WML7xQTE0NMTEyR91PgqKVKqRLALuAW4CgQC/TSWu/IsU5XYLDWuqtSqi3woda6bY7lEcBcrXXz\nHO+9A5zSWr+tlBoBVNJa/6MS2ddGLT1/3pTtXrjg/RfZsDBYvhzq1rU6Et/0wQewbBn89JPnj/37\n7/Dkk7B9e+GGuxD+zS2jlmqt04HBwCJgO/Ct1nqHUupJpdSTtnXmA/uUUnHARODpHEF9A6wCGiml\n4pVS/W2LxgK3KaV2A51sr31e1jAU3p4IwBQV7dhhfz3xTxcuwNix8Prr1hy/UydTPPXVV9YcX/gn\nmc/Ahf73P1O59803Vkdi39ChUKcO/PvfVkfie8aMMYl0xgzrYli9Gnr2ND9APFF5LXyHzGfgBXyh\nJVGWJk3kzqAwTp82rYdefdXaONq1g+bNTUc3IVxBkoEL+ULlcRYpJiqcjz6C7t3d28HMUa+/biqT\nZZgK4QqSDFzIF5qVZslKBj5UCme5pCT47DN44QWrIzFatYLWrc0cy0IUlSQDF8nIMEMYe+vQ1bnV\nqGESwcmTVkfiOyZPhuho7/o3Hj7c9D2QfgeiqCQZuMiBA+YC68wUh1ZSSoqKnJGaauYnHp7nUI3W\nuekmM9f2jz9aHYnwdZIMXMSXioiySCWy4775xjQOuPZaqyO5klImQb39thT5iaKRZOAivlR5nEXu\nDByTmQnvvON9dwVZ7r4bUlJMZzQhCkuSgYv4UrPSLDJGkWPmzTNt+W+91epI8hYQYIbQfvttqyMR\nvkySgYvInYH/evddc1egnO7G4zl9+pjhKTZssDoS4askGbiIL9YZ1KkDJ06YJpMib3/9BQcPwv33\nWx1JwUqWhGeeMR3ihCgMSQYucPq0mVe4Vi2rI3FOYKCZ6EbmNsjf+PFmXuMS9sb39QKPPw5z5sCx\nY1ZHInyRJAMXyKov8OZihPxIvUH+EhPNxXXgQKsjcUyVKvDQQ/D551ZHInyRJAMX2LYNrr7a6igK\nR+oN8jdxIjz4oLnI+opnnjHJ4PJlqyMRvkaSgQts3eq7yUD6GuQtNRUmTDAXV18SGQlRUfDtt1ZH\nInyNJAMX2LYNmjWzOorCkTuDvH3/vbmw+mKSHzrUDKgnndCEMyQZuIAv3xk0agT795tfwsLQ2lxM\nhw61OpLCueMOMwHPypVWRyJ8iSSDIjp50gwhHBpqdSSFc9VVEBEhlcg5rVlj/l3vvNPqSAonIMAU\nb330kdWRCF8iyaCIsoqIfLElUZaoKNiyxeoovMdHH8GQIabpra969FEzPMXBg1ZHInyFJIMi8uX6\ngizNm0syyHLkCCxcCP3721/Xm5UvbxLCp59aHYnwFZIMisiX6wuySDL424QJZmiHSpWsjqToBg+G\nqVMhOdnqSIQvkGRQRJIM/MelS2ZO4SFDrI7ENerVM/MdfP211ZEIXyDJoAi09o9kEBEBZ86YR3H2\nzTdmGklfG322IEOHmiE1pJmpsEeSQRHEx5uhjWvUsDqSogkIMAlt61arI7GO1uai6WudzOyJjjbj\nKi1ebHUkwttJMiiCTZugRQuro3CN4l5UtGKFmSCmc2erI3EtpaSZqXCMJIMi2LgRWra0OgrXaN4c\nNm+2OgrrfPSRuWgG+OE3ok8fiI2FPXusjkR4Mz/8r+85/nRn0KqVSW7F0cGDEBMDfftaHYl7lC5t\nhrf++GOrIxHeTGkvrllSSmlvjq9BAzPEcWRk4feRlJpE/Ll44s/Hczz5OEmpSSSlJnHh8gWSUpO4\nnHGZQBVIiYAS2Y+rSlxF1TJVqVamGtXLVKdamWoElwumepnqqEL2frtwAWrWhHPnfGPsfld64QVI\nT4f337c6Evc5fNh0Lty/HypWtDoa4U5KKbTWTl8IJBkU0oULEBwM58/bv3hmZGYQdzqOLce3sCVx\nC1tPbGX3qd0cPn+YS+mXCKsQRljFMILLBlO+ZHnKlSxH+VLmb6nAUmToDDIyM0jPTCc9M52L6Rc5\nlXKKEyknOJlykpMpJ0lISiAtI40GVRpkPxpXbcw1IdfQtFpTAgPsd6dt3Bh+/NH3O9E5IznZzPgW\nG2uaYvqznj2hXTvfHXNJOKawyaCY/QZ0nS1bzEUzr0Rw9tJZVh5ayYpDK1h+cDmbEjcRXDaY5sHN\nubr61TwY+SCNqzYmvGI4VUpXKfSv+dzOXDzD3jN72XNqD3Gn41i0dxFvrHiDhKQEWtZsybW1ruX6\n2tdzc8TNBJcL/sf2rVrB+vXFKxl89RXceGPRE8Hl9MtcSL3A+cvnuXDZ/E1OSyY1I5W0jDTSMtOy\n/6ZmmFEBA1UggQHmri/n8zJBZSgbVJZyJctlP8qWNK9LBBT+K/vMM6YobPBg3x5qQ7iH3BkU0mef\nmQvn5MnOb2WXAAAcjUlEQVSQlpHGyviVzN01lyX7l7D3zF7ahLahfXh72oe3p01oG8qXKm9ZrGcv\nnWV9wnrWHlnL6sOrWXZwGeEVw7m17q3cVv822oe3p2zJsrzzDiQkwAcfWBaqR2VmmiG8v/gCOna8\ncpnWmjOXznDw7EEOnTvEsaRjHE8+bh4px7Ofn0g+wbnL59BaU75UeSqUqkCFUhUoX7I8ZUuWpWRg\nSYICgggKDLrir0KZO74cd30Z2vy9mHYxu7gwKTWJ5LRk8zc1mfKlylOtTLUrH6VNMWFYhTBqV6hN\nWMUwapWrRVBgUK7PBG3awMsvw913e/BEC4+SYiIP6/fkOVSjBVyOmMPCuIXUq1yPbo27cXv927mm\n1jX/+CJ6k/TMdNYdXcfifYtZvG8x6xPWEx0RTZPMB1g57W5WLq5sdYge8fPcNEa+u5c3v9jF7lO7\n2HdmH4fOHeLgOZMAAlUgdSrVIbxiOLXK1SK4bDA1yta44lGtTDUqXlWRUoGlXHaHl59MncnZS2ez\niwazHieST3As6RiHLxzm8PnDxJ8z9U/Vy1YnrEIYdSvXpWGVhjSs0pC9axuxdFZDli30oenbhFMk\nGXhAWkYav+79la83f833GxfQNqQ9j17fjbsa3UVI+RCrwyu0s5fOMm/3PP5v4ywW7PydzpE38mDk\nAzzY7EEqlKpgdXhFlpqRyo4TO9iUuIktiVvYeWonu07uYu/JQ1QrWZtr6zamcdXG1K9cnzqV6lCn\nokkAFa/y3ZrW9Mx0Ei4kEH8+nn1n9rH71G72nN7D7pN72HBwDxXKl6BpjUY0q96MqOAomtdoTlRw\nFFXLVLU6dFFEkgzcaEviFqZumMo3W7+hXuV69Irsywt39uD0kSqULm11dK5Vu/4FXvhiPktPzmTp\n/qXc3fhuHmv5GB0jOhKgvL8lcnJqMmuPrmVDwgY2Jm5k07FN7Dq1i7qV6tKyZkua12hOk2pN0Ccb\nM6hXfQ7uK0XJklZH7VmjX9XsPXacJ0bsZuvxrWw5voXNiZvZcnwLZYPKEhUcRYvgFlwXeh1tQtsQ\nViHM7Xc9wnUkGbhYemY6c3bN4ePYj9lzag/9W/anb4u+NKzakNhYePJJ2LDBktDcqnt36NULHnoI\nTiSfYMaWGUzbOI3zl8/Tr0U/+rfqT3jFcKvDzHYi+QQr41ey4uAK/oj/g63HtxIVHMU1ta6hZc2W\ntAhuwdU1rqZ00JVZu39/aNgQRo2yKHALJSaaua/37oUqOUqLtNYcOneILce3sCFhA2uPrmXNkTUo\nFG1C21zxqHSVHwzr6qckGbjI6Yunmbx+Mp+u/ZSwCmEMaTOE+5red0UdwGefmUQwaZJHQ/OIN9+E\nU6dg3Li/39Nas+HYhuy7o+tDr2dg64Hc1eguj9aNaK3Zf3Y/fxz6I/vif/TCUdrVbkf78PbcFH4T\nbULb/OPCn9uxY6biOC4OqhbTUpFHHzX9Y4YPL3g9rTXx5+OJPRJL7JFY1hxZw/qE9TSo0oCOdTrS\noU4HOtTpQLUy1TwTuLBLkkERnUg+wbjV45i0fhJ3NbqLZ9o8wzUh1+S57mOPwfXXm7sDf7NkCbzy\nCvzxR97LL6ZdZNb2WUxaP4k9p/fwaItHebz14zSo0sDlsWRkZrDl+BZz8T+0gj8O/UGmzsxupXVT\n+E1EBUc51Icip1deMb+OP//c5SH7jL/+gnvvhX37nO9kmJqRyrqj61h+cDnLDi5jVfwqaleoTcc6\nHbml7i3cUu8WuXOwkNuSgVKqC/AhEAhM1lq/ncc644E7gBSgn9Z6Q0HbKqVGA48DJ2y7GKm1XpjH\nft2eDI4lHeO9Ve8xdcNUel3di+E3DbdbDBIVBdOmwTV55wqfdv48hISY4ayD7Pzo33lyJ1PWT+Gr\nzV/RrHozHm/9OPc2udfuL/P8XEq/ROyR2OyL/+r41dQsV5Obwm/KvvjXq1yvSOXXFy+aIbuXLTNF\nJcVZhw7w1FOmWLAo0jPT2XhsI8sOLOO3fb+xMn4lUcFRdK7fmc71O3NtyLVOJ2xReG5JBkqpQGAX\ncCtwBFgL9NJa78ixTldgsNa6q1LqeuAjrXXbgrZVSr0CXNBaFzgAgDuTwemLp3lj+RtM2ziNh6Me\nZviNwwmtYH9W+5QUqFbNXCxLlXJLaJZr1sxMiNK6tWPrp2ak8vPOn5m8YTKr4ldxY9iN3NHgDm6p\ndwuNqzbOsygpKTWJA2cPsPHYxuwiiC3Ht9CserPsi/+N4TdSo6xrxwefPBlmz4ZffnHpbn3SvHnw\n0kumv4wr64cvpl1kxaEVLIpbxKK9i0hISuDWerfSrVE3ujbsSuXSxaPpslXclQzaAa9orbvYXo8A\n0FqPzbHO58BSrfW3ttc7gWigbn7b2pJBktY6R8l0nsd3eTK4nH6ZT9d+ylt/vMUDTR/g5Y4vU6t8\nLYe3X7kShg2DtWtdGpZXeewxuO4686vRWecunWPJ/iUs2LOAZQeXEX8+npDyIVQtXZUAFcDF9Isc\nPn+Yi2kXCa8YTouaLbguxLRaaV2rNeVKlnP9B7LR2ozO+uGHcOutbjuMz8jMNHe5H3wAt93mvuMc\nOX+EhXEL+XnXz8QciKFNaBvuaXwP9zS5x6saI/gLdw1HEQrE53h9GLjegXVCgRA72w5RSvUF1gHP\naa3POhG307TWzNo+ixFLRhBZPZJl/ZYRWd35EeZWrTLju/iz66+H1asLlwwqXlWR+5rex31N7wPM\nr8SjF45yMuUkmTqTq0pcRVjFMKqWrurx5ooLF5ohqm+5xaOH9VoBAfD88/DOO+5NBqEVQhnQegAD\nWg8gOTWZ3/b9xk87f+K15a8RViGMe5vcy0NXP0Sjqo3cF4Swy14ycPRnubPf6gnAa7bnrwPjgAF5\nrTh69Ojs59HR0URHRzt5KNNP4KlfniIlLYVJd0+iU91OTu8jy6pVptmlP7v+etdNhlI6qDT1q9Sn\nfpX6rtlhEbz1FowY4doiEV/Xqxf897+mqMjRYsGiKFuyLN2bdKd7k+6kZ6azKn4Vs7bPouP0jtQq\nV4ueV/ekR7MeRFSKcH8wfiImJoaYmJgi78deMVFbYHSOop6RQGbOSmRbMVGM1nqm7fVOoCOmmKjA\nbW3vRwBztdbN8zh+kYqJklOTeW3Za0zbOI0xncbweOvHi9RxSmszzPPatRDux3e36emm/fmBA1e2\nQ/dlK1ZAv36wa1fxG6LbnnHjYN06Mwe0VTIyM1h+cDkzt87kx50/0qBKA3o268lDVz9EzXI1rQvM\nBxW2mAitdb4PzJ3DXiACKAlsBJrmWqcrMN/2vC3wp71tgVo5tn8W+L98jq8La+6uubrOB3V0nx/6\n6GMXjhV6PznFxWldu7ZLduX1brtN659/tjoK17njDq0nTrQ6Cu907pzWVatqvXev1ZEYqempesGe\nBbrv7L660thK+s4Zd+pZ22bpS2mXrA7NJ9iumwVe2/N6ONK09A7+bh46RWv9llLqSduVeqJtnU+A\nLkAy0F9rvT6/bW3vfwW0xBRD7Qee1Fon5nFsbS++3I4nH+fpX55mc+JmPrvzM26t57qawq+/Ni0w\nvv3WZbv0WmPGwNmz8N57VkdSdBs2mFE69+713xZgRTVqFJw+7X19L5JSk/hxx49M2ziNrce30rNZ\nT/q17EfrWq1liIx8SKczYPaO2Tw9/2kebfEoo6NHc1WJq1waz1NPmbbpxWFykOXL4T//MZO++Loe\nPUyl/7PPWh2J9zpxwkxutHGj9xaB7j+zn682fcX0TdMpX7I8A1sP5JEWj0gHt1yKdTI4e+kszyx4\nhtWHV/Nl9y+5IewGt8QTFQVTpphml/7u0iXTn+LYMSjnvtaebrdrF7Rvb3ra+vLn8IQRI0ynw88+\nszqSgmXqTJYdWMbEvyayaO8i7m96P09d+1S+IwYUN4VNBt4/DKUdi/ctJmpCFOVLlmfjkxvdlghO\nnTIVqi1bumX3Xueqq0zrktWrrY6kaN58E4YMkUTgiOeeg5kzIT7e/rpWClAB3Fz3ZmY+MJOdg3ZS\nr3I97v/uftpMasO0DdNISUuxOkSf5LN3BumZ6bz0+0t8vflrpt0zjdvqu7GhNGZu4MmTYf58tx7G\nq7z4ommG+frrVkdSONu3Q3S0GZCugu9Py+ARw4dDUhJ8+qnVkTgnIzODhXELmbBuAn8e/pMBrQYw\n5Poh1K5Q2+rQPK5Y3RnEn4sneno0GxM3suHJDW5PBAC//w433+z2w3iV6GgzcJ2vevllU+8hicBx\n//mPuTs4fNjqSJwTGBDInY3uZF7vecQOjOVyxmWiJkTR+4ferD3ix8MFuJDP3Rn8svsXBswZwLNt\nn+X5G5/32IQrkZGmNZE/Dk6Xn0uXoEYNOHQIKvlYHd369XDXXeauoEwZq6PxLS+8AMnJvnd3kNu5\nS+eYvH4y42PHE1YhjGfbPkv3Jt39ftA8v69ATs9MZ9SSUczcOpNv7v+GG8Nv9FgcCQlm8LYTJyDQ\nv/8f/cMdd8Djj8P991sdiXO6djWPwYOtjsT3nDhh5ntYswbqW99xvMjSM9OZvWM27//5PseTjzP8\nxuE82uJRSpXwz3bGfl1MdCrlFF3+14VNiZvY8OQGjyYCgJgY6Nix+CUCgM6dYdEiq6NwzsqVpr5g\n4ECrI/FN1aubwRj9ZRa4EgEleLDZg6wesJrp90znp50/UW98PcatGkdSapLV4XkNr08GWxK30GZy\nG1rVbMUvvX+xZMLuxYuhU+GHM/Jpt98Ov/5qhuLwBVqbi9grr0gHs6L4979NUl2zxupIXKt9nfbM\n7zOfeb3mEXs0lnof1ePVmFc5ffG01aFZzuuTQaevOvFa9Gu8e/u7lAjw/KAymZmmBdEdd3j80F6h\naVPIyIDdu62OxDGzZ5tmwI88YnUkvq1MGdOK7D//8Z0fAs5oVasV3z7wLX889gfx5+NpML4Bw38b\nzsmUk1aHZhmvTwYL+yykT1Qfy46/YYNpjdLA9bM6+gSlTCKcN8/qSOy7eNG0lf/oIxmMzhX69oVz\n5+Dnn62OxH0aVW3E5G6T2fSvTVxIvUDjTxrz4u8vFss7Ba9PBlb3KvzlF9MqpTi7917zi9vbjRsH\nrVrJfAWuEhgI775rWhelpVkdjXuFVQzjszs/468n/iIxKZFGHzdidMxozl5y6zQrXsXrk4HV5s2D\nO++0Ogpr3XKLqZBNSLA6kvwdPmxm7BpX4Nx5wlmdO0O9eq6b38LbRVSKYFK3Sax5fA0Hzx2k4ccN\nGbN8TLGoaPaZpqVWSEw0g3cdPw4lS1oWhld4+GG48cbCzX7mCb17m4vWmDFWR+J/9u41Ex799RfU\nqWN1NJ61+9RuRseMZumBpbzU4SUGth6Y55ze3sSvm5ZaZc4c05qmuCcCgPvuM0NyeKPly83kNSNH\nWh2Jf6pf34z4OmiQf1YmF6RR1Ub83/3/xy+9f2H2ztlEfhbJ99u+x5t/RBeW3BkU4JZbzBfgvvss\nC8FrJCdDaKhpVVSjhtXR/C0lxYwm+9570L271dH4r9RUM0jj66/7XgdEV/pt728MXzycoMAg3r71\nbaIjoq0O6R/8vgeypx07ZppVHj0KpUtbEoLX6dvXVNB607wAw4aZHrMzZlgdif9bvtwUx23fXrzH\ne8rUmczcOpP//v5fmlVvxnu3v0eTak2sDiubFBO52KxZphWRJIK/9e8P06Z5T1HBihXw3XcwfrzV\nkRQPHTqYCuXhw62OxFoBKoDezXuzc9BObo64mZum3sSwhcM4c/GM1aEViSSDfHz7LTz0kNVReJeO\nHeHCBTMInNVSUuCxx2DCBKjq+U7pxda4cbBwoX/3PXBUqRKleO6G59g+aDsX0y7S5NMmfLb2M9Iz\n060OrVCkmCgPcXFwww2muaJUHl/ptddMKyurR7R85hnT01iKhzxv1SrT92T9elOPJIxNxzYxbNEw\nTqac5IPOH7h0/nVnSJ2BCw0fboahePddjx/a6x05As2bm+aGlStbE8NPP5m6gvXroUoVa2Io7l57\nDZYtg99+gwApX8imtWb2ztn859f/0KJmCz7s/CF1Knm2Pa7UGbhIaipMny4jXuYnNBTuvhsmTrTm\n+Pv3wxNPmGI8SQTWGTXKfFfkB9OVlFLc1/Q+tg/aTuuarWn9RWvG/jGW1IxUq0OzS+4McvnuO/j8\nczOzmcjb5s3QpYu5MHtyZNCkJFN89/jjpphIWOvQIWjTxkz6dJv7Jxv0SfvO7GPIgiHsO7OPT7t+\nSqe67h/+WIqJXEBraN8ehg6FBx/02GF9UufOpr35E0945niZmebfpGJFmDLFDKAnrLd8OTzwgCky\natrU6mi8k9aan3f9zNCFQ7kx7EbG3T6OWuVrue14UkzkAitWmMpR6WRm35gxMHq0+bXublqb0UiP\nHzethyQReI8OHUxR0V13mf4e4p+UUnRv0p3tT2+nTsU6NJ/QnE9iPyFTZ1od2hXkziCHzp2hRw8Y\nMMBjh/RpffqYob1ffdW9x3nnHfjqK5Osraq0FgV78UVYuhSWLIGrrrI6Gu+2/cR2Bs4diNaayd0m\nE1k90qX7l2KiIoqNNbe7cXHSnNRRBw9C69awdq0ZJM4dPvgAPvnEFEPUru2eY4iiy8w0Pw6Sk02H\nTfkOFSxTZ/L5us95JeYVBl83mBE3jXDZnMySDIogq66gXz9TOSkcN26cmetg2TLXzhGtNYwdC5Mn\nm1+c4eGu27dwj7Q0U68TEAAzZ0pCcMTh84d5+pen2XtmL5PunsQNYTcUeZ9SZ1AE331nftH07291\nJL7n2WfNl37sWNftMz3dtBaaOdNUUEoi8A1BQabJb2Ym3HOP6SUuCla7Qm1+7vkzozuO5oHvHmDw\n/MGWzZ1Q7JPBhQtmJqePPnLtL9viIiAAvvzSVOx+913R93f8uKm72bXLJALp4epbSpUyxUQ1asDN\nN5tOiqJgSikebPYg257eRnJaMlETolh2YJnH4yj2yWDYMNNGukMHqyPxXWFhMH8+DB5s/hbWvHmm\nDuL662HBAtOMVPieEiVMx83u3eG660zFv7CvcunKTLtnGuPvGE/vH3szdMFQklOTPReA1tprHyY8\n9/nhB60bNND6wgW3HqbYWLVK65o1tR43TuvMTMe327dP6x49tK5bV+ulS90WnrDAggVa16ih9ccf\nO/d/org7lXJK9/mhj24wvoH+4+AfTm1ru246fb0tthXIW7aYyWvmzjW/RIVrHDxoKhGDgkyT0Btu\nyLtfgNawbp0Z8G7uXHOH9u9/Q9myno9ZuNfevabJdtWqpne/u1qe+aPZO2bz9Pyn6dO8D6/f/Dql\ng+yPqS+tiZxw+LC5SL3zDvTs6fLdF3sZGaZfwBtvmERw223QsKEpTz5/3kyOEhNjXj/+uBkHSsYZ\n8m/p6aaZ8Ntvm4Egn33WFCcJ+04kn2DQ/EFsOb6FGffNoHWt1gWuL8nAQXFxZl7jQYNMr1bhPlm/\n/v/800yXmZFhfvk3bQpt25q/0pu4eNm7F556yoxr9dJLZuY0SQqOmbF5BsMWDeP5G57nuXbPERiQ\nd4sXSQYOWLrUdIwZPdpzY+oIIf4pJgZeecVMKzt8uLlDL1fO6qi838GzB3lk9iMEBgTyVfevCKsY\n9o913NbPQCnVRSm1Uym1RymV54R3SqnxtuWblFKt7G2rlKqilPpNKbVbKfWrUqqSs4E7IzkZRo40\niWD6dEkEQlgtOtp0VJw0ycyaFhZmigtXr/aeaVW9UZ1KdVj66FJur3c713xxDd9u/dZl+y4wGSil\nAoFPgC5AJNBLKdU01zpdgQZa64bAE8AEB7YdAfymtW4ELLG9drmUFNP+vXFjM9zu+vWmiMgXxcTE\nWB2CV5Dz8Dd/OBfR0aYBwbZtUL++6fgZGmrqkn78Ec6edWw//nAuHBUYEMjI9iOZ32c+L8e8TN/Z\nfTl36VyR92vvzqANEKe1PqC1TgNmAvfkWqcb8CWA1noNUEkpVdPOttnb2P52L/Insbl40cy+9K9/\nQZ06sGiR6QQzYwbUrOmqo3hecfrPXhA5D3/zp3MREgIjRsDOneaOoXlzM4FSWJipW+rXz7Q8W7oU\njh37592DP50LR10bci3rn1hPmaAytP6iNbFHYou0P3tVN6FAfI7Xh4HcDTHzWicUCClg22CtdaLt\neSIQ7ETMpKaanqqJiZCQYHqrbtsGW7ealirNm5sOL2vXQkSEM3sWQlitYUMzp8jQoaYV0rZtsGaN\naYzw7bcmYaSmmmFKQkPNY88eqFXLtEqrWBEqVDCPrOdly/rn9JxlS5bl87s+54ftP3DX/93Ff274\nT6H3ZS8ZOFp650hlhcprf1prrZTK9ziRkebX/sWLcOmS+ZuZCdWrm1/6wcHQqJFpnTJgAERFQfny\nDkYthPBqJUpAixbmkbOu79QpiI83w10cPWpaCa5bB2fOmObL586Zv1nPU1JM35dSpcxYWnk9AgNN\n67aAgCv/5vc853tZD0c524rO/vr3c3XJaxmb2Me5HedUUI80oC2wMMfrkcDwXOt8DvTM8Xon5pd+\nvtva1qlpe14L2JnP8bU85CEPecjDuUdheiDbuzNYBzRUSkUAR4GHgF651pkDDAZmKqXaAme11olK\nqVMFbDsHeBR42/b3p7wOXpjmUUIIIZxXYDLQWqcrpQYDi4BAYIrWeodS6knb8ola6/lKqa5KqTgg\nGehf0La2XY8FvlNKDQAOAD3c8NmEEEI4yKs7nQkhhPAMy+vXi9Kpzd/YOxdKqT62c7BZKbVSKRVl\nRZye4Mj/C9t61yml0pVS93kyPk9y8DsSrZTaoJTaqpSK8XCIHuPAd6SaUmqhUmqj7Vz0syBMt1NK\nTVVKJSqlthSwjnPXzcJUNLjqgSk+igMigCBgI9A01zpdgfm259cDf1oZs8Xnoh1Q0fa8S3E+FznW\n+x2YB9xvddwW/r+oBGwDatteV7M6bgvPxWjgrazzAJwCSlgduxvORXugFbAln+VOXzetvjMobKc2\np/ol+Ai750JrvVprndXVcA3gr1PEO/L/AmAIMAs44cngPMyRc9Eb+EFrfRhAa33SwzF6iiPnIgGo\nYHteATiltU73YIweobVeAZwpYBWnr5tWJ4P8OqzZW8cfL4KOnIucBgBFmFfMq9k9F0qpUMyFYILt\nLX+t/HLk/0VDoIpSaqlSap1S6hGPRedZjpyLSUAzpdRRYBMw1EOxeRunr5tWDx7r6Bc4dxNTf/zi\nO/yZlFI3A48BN7ovHEs5ci4+BEZorbVSSuFYx0df5Mi5CAJaA7cAZYDVSqk/tdZ73BqZ5zlyLkYB\nG7XW0Uqp+sBvSqkWWusLbo7NGzl13bQ6GRwBco7BGobJYAWtU9v2nr9x5FxgqzSeBHTRWhd0m+jL\nHDkX12D6toApG75DKZWmtZ7jmRA9xpFzEQ+c1FpfBC4qpZYDLQB/SwaOnIsbgDcAtNZ7lVL7gcaY\nPlPFidPXTauLibI7tSmlSmI6puX+Ms8B+gLk7NTm2TA9wu65UEqFAz8CD2ut4yyI0VPsngutdT2t\ndV2tdV1MvcFTfpgIwLHvyM/ATUqpQKVUGUyF4XYPx+kJjpyLncCtALYy8sbAPo9G6R2cvm5aemeg\ni9Cpzd84ci6Al4HKwATbL+I0rXUbq2J2FwfPRbHg4Hdkp1JqIbAZyAQmaa39Lhk4+P/iTWCaUmoT\n5sfuC1rr05YF7SZKqW+AjkA1pVQ88AqmuLDQ103pdCaEEMLyYiIhhBBeQJKBEEIISQZCCCEkGQgh\nhECSgRBCCCQZCCGEQJKBEEIIJBkIIYQA/h8B5h0FPi98JgAAAABJRU5ErkJggg==\n", 231 | "text/plain": [ 232 | "" 233 | ] 234 | }, 235 | "metadata": {}, 236 | "output_type": "display_data" 237 | } 238 | ], 239 | "source": [ 240 | "%matplotlib inline\n", 241 | "import matplotlib.pyplot as plt\n", 242 | "from numpy.random import randn\n", 243 | "import numpy as np\n", 244 | "\n", 245 | "def norm(x, x0, sigma):\n", 246 | " return np.exp(-0.5 * (x - x0) ** 2 / sigma ** 2)\n", 247 | "\n", 248 | "# compute some arbitrary mixture of gaussians\n", 249 | "x = np.linspace(0, 1, 200)\n", 250 | "likelihood = .65* norm(x, .2, .05) + .35*norm(x, .55, .08)\n", 251 | "likelihood /= np.sum(likelihood) # normalize it\n", 252 | "\n", 253 | "log_likelihood = np.log(likelihood)\n", 254 | "# make it positive and normalized.\n", 255 | "log_likelihood += -np.min(log_likelihood)\n", 256 | "log_likelihood /= np.sum(log_likelihood)\n", 257 | "\n", 258 | "# plot them\n", 259 | "plt.plot(x, likelihood, label='likelihood')\n", 260 | "plt.plot(x, log_likelihood, label='log likelihood')\n", 261 | "plt.legend();" 262 | ] 263 | }, 264 | { 265 | "cell_type": "markdown", 266 | "metadata": {}, 267 | "source": [ 268 | "We can show that the maximum value of each is at the same index with:" 269 | ] 270 | }, 271 | { 272 | "cell_type": "code", 273 | "execution_count": 61, 274 | "metadata": { 275 | "collapsed": false 276 | }, 277 | "outputs": [ 278 | { 279 | "name": "stdout", 280 | "output_type": "stream", 281 | "text": [ 282 | "40\n", 283 | "40\n" 284 | ] 285 | } 286 | ], 287 | "source": [ 288 | "print(np.argmax(likelihood))\n", 289 | "print(np.argmax(log_likelihood))" 290 | ] 291 | }, 292 | { 293 | "cell_type": "markdown", 294 | "metadata": {}, 295 | "source": [ 296 | "Indeed they are the same. I haven't proven that this relationship is valid, but it is. I don't want to get bogged down in a proof. See any statistics or machine learning textbook if you can't work it out yourself.\n", 297 | "\n", 298 | "If you see log-likelihood being used anywhere just think back to this chart. They have almost certainly introduced the logarithm not for any reason related to the problem, but just to make the computations easier." 299 | ] 300 | }, 301 | { 302 | "cell_type": "markdown", 303 | "metadata": {}, 304 | "source": [ 305 | "## Dealing with Incomplete Data\n", 306 | "\n", 307 | "All of the work above is to prepare us for this, the actual problem. It is easy to compute the bias of 2 coins when we know the flips for each coin. What if we lost or don't have that association? In other words, we have a sequence of heads and tails recorded, but do not know which coin produced the flips?\n", 308 | "\n", 309 | "This is not a silly question, this is a very common situation in practice. For example **some example here**\n", 310 | "\n", 311 | "Let's look as a plot:" 312 | ] 313 | }, 314 | { 315 | "cell_type": "code", 316 | "execution_count": 30, 317 | "metadata": { 318 | "collapsed": false 319 | }, 320 | "outputs": [ 321 | { 322 | "name": "stdout", 323 | "output_type": "stream", 324 | "text": [ 325 | "[ 10. 4. 4. 10. 10. 4. 10. 10.]\n" 326 | ] 327 | }, 328 | { 329 | "data": { 330 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEZCAYAAAB7HPUdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGshJREFUeJzt3XmcXHWd7vHPAwFi2DfZBe/IpnJZJwgMpsHIZYBRca6M\nBBzgMozOwOhc9MoyziSAL1lc4DLOEEUIERBHE+ACKsgFGmREwpIAsihyCQRiQiAhIayBfO8f59dS\ntNWdqu46darq97xfr36l9vp20zx16neeOq2IwMzM8rBa1QOYmVn7OPTNzDLi0Dczy4hD38wsIw59\nM7OMOPTNzDLi0LeOJ2kzSXdIWibp63Wuv0zSWW2aZa6kj7TjuVpN0kWSvtLgbfslHV/2TNZ+Y6oe\nwHqXpFnAJGAl8OOI2HOED/W3wHMRsd4Q10f6aod2PldDJB0LHB8R+w93u4j4uyYetuO+T2sNb+lb\nKSStAbwnIn4H7AncN4qH2xZ4tCWDdTlJI9pQk+T/1w1w6Ft5Pgg8kk7vBcwe7saS9pV0j6QXJc2S\ntE+6/DLgr4EvS3pJ0oFDPMRGkm5IS0C/kvRfah57J0k3S3pB0mOSPlVz3aGSZktaKulpSZMHzfUZ\nSU9Jel7S6YOuGy/p3nTfBZK+OcT31ifpGUmnSVok6UlJk2quX0vSN9LzLEjLMGMH3ffLkn4PXDLo\nsXcGLgL2ST+fxQM/t/Q4P5W0HDigdhlM0obp5/WcpMWSrpe01XD/jaxHRIS//NWyL+BYYAnwMvBK\nOr0CWAYsBratc5+N0u2OotgQ+XS67Ybp+mnAmcM852XA8xQvLqsDVwBXpevWBuYBx6TH3g1YBOyc\nrp8AfCCd3gVYAHw8nX8/8BLwZ8CawDfT93Jguv4u4Kh0ehyw9xDz9aX7fQNYA/gwsBzYIV1/PnAt\nsAGwDnAd8LVB9z073Xdsncc/BvhFnZ/Ji8A+6fxatT/H9DM/HBibnvNHwDU1978N+B9V/z75q/Vf\n3tK3loqIyyJiQ4rlnH2AXYFfR8R6EbFRRDxV526HAr+JiCsjYmVE/BB4DPhYzW003NMCV0fEvRHx\nFnAlRbgDHAY8GRHT02PPAa4GPpXmvT0iHk6nHwJ+SPFCAPDfgesj4s6IeAP4Z4r9EwPeALaXtElE\nvBIRd6/ix/PPEbEiIu4AfgIcIUnACcDJEfFiRCynCPhP19xvJTA53fe1Oo9b72cTwLURcVf63l6v\nvW1ELI6IayLitfScX6v5vq2HOfStZSRtlJZnXgT2BfopwntHSUskfWGIu24JPD3osqfS5Y1aWHP6\nVYqtVyj2B+ydnn+JpCUUO5c3SzPvLem2tMzxIvBZYOOauZ4ZeNCIeAV4oeZ5jgd2AB5NS1KHDjPf\nkoh4ddD3twWwCcW7hPtq5vtZunzAovSi06x5Q10haZyk76Q20lLgdmD99CJkPcyhby2Tth43oAjO\ni9MW/43AYRGxYUT87yHu+ixFONfaNl0+Wk8Dt6fnH/haNyJOTNf/gGJpZes0+1Te3nKeD2wz8ECS\nxvH2CwIR8buImBQRmwLnAjMkvWuIOTZM96/9/uZTLEu9Cry/Zr4N4p1NpVW1aJpp2Qzc9osUL1jj\nI2J9iq18Mfw7KusBDn0rw17A/en07qy6ufNTYAdJR0oaI+mvgJ2AG9L1qwqi4a7/SXrsoyWtkb7+\nVNJO6fp1KLbC35A0nuJdwICZwGGS9pO0JnAmNf/PpMfcNJ1dShGotcs/g52Rnn9/iiWtH0dEABcD\nFww8lqStJB20iu+51gJg69SY+sN4dW5XG+rrULzYLJW0ETB5iNtbj3HoWxn2AO6XtDHwZkQsHe7G\nEbGYYu39ixRbvl+ieHeweOAmDL81W+/6SI/9EnAQxRr5s8DvKdbM10y3+3vgTEnLKNbs/6NmroeB\nEyneDcyn2Llcu2Ty34BfS3qJYmfsp2vWzgdbQLGzej5wOfDZiPhtuu4U4HfAr9JSy80UW+Hv+F6G\ncSvwMLBA0nM196n3Mxm47ALgXRQ/719SLCnV/Rlab1GxoVHCA0vbAN8H3k3xy/PdiLgwbVX8B8Xb\n27nAERHxYilDmHUASX3A5RGxzapua1a2Mrf0VwD/MyI+AHwIODF1ik8Fbo6IHYBb0nkzM2uD0kI/\nIhakehypEvYosBVFDW96utl04BNlzWDWQbxUYh2htOWddzyJtB1FJeyDwNOp1UGqhy0eOG9mZuUq\nfUeupHUoWhBfSDvV/iA1F7wFZGbWJqUeZTNVyGZS7MS6Nl28UNLmEbFA0hbAc3Xu5xcCM7MRiIhh\nq7albemnpZtLgEci4oKaq66jOFYI6d9rB98XuvuYQJMnT658Bs9f/Ryev/u+unn2iMa2lcvc0t8P\nOBp4UNLAERZPA84BfqTiDzTMBY4ocQYzM6tRWuhHxJ0M/U5iYlnPa2ZmQ/MnckvQ19dX9Qij4vmr\n5fmr082zN6otlc1mSYpOnMvMrJNJIqrakWtmZp3HoW9mlhGHvplZRhz6ZmYZceibmWXEoW9mlhGH\nvplZRhz6ZmYZceibmWXEoW9mlhGHvplZRhz6ZmYZceibmWXEoW9mlhGHvplZRhz6ZmYZceibmWXE\noW9mlhGHvplZRhz6ZmYZceibmWXEoW9mlhGHvplZRhz6ZmYZceibmWXEoW9mlhGHvplZRhz6ZmYZ\nceibmWXEoW9mlhGHvplZRhz6ZmYZceibmWXEoW9mlhGHvplZRhz6ZmYZceibmWXEoW9mlhGHvplZ\nRhz6ZmYZceibmWXEoW9mlhGHvplZRhz6ZmYZceibmWWk1NCXdKmkhZIeqrlsiqRnJM1OXweXOYOZ\nmb2t7C39acDgUA/gWxGxe/q6seQZzMwsKTX0I+IXwJI6V6nM5zUzs/qqWtP/B0kPSLpE0gYVzWBm\nlp0xFTznRcCZ6fRZwDeB4wffaMqUKX843dfXR19fXxtGMzPrHv39/fT39zd1H0VEOdMMPIG0HXB9\nROzS6HWSouy5zMx6jSQiYtjl87Yv70jaoubs4cBDQ93WzMxaq9TlHUlXAROATSTNAyYDfZJ2o2jx\nPAl8tswZzMzsbaUv74yEl3fMzJrXkcs7ZmZWHYe+mVlGHPpmZhlx6JuZZcShb2aWEYe+mVlGHPpm\nZhlx6JuZZcShb2aWEYe+mVlGHPpmZhlx6JuZZcShb2aWEYe+mVlGHPpmZhlx6JuZZcShb2aWEYe+\nmVlGHPpmZhlx6JuZZcShb2aWEYe+mVlGHPpmZhlx6JuZZcShb2aWEYe+mVlGHPpmZhlx6JuZZcSh\nb2aWEYe+mVlGHPpmZhlpKvQlbSTpv5Y1jJmZlWuVoS/pdknrSdoIuA/4nqTzyx/NzMxarZEt/fUj\nYhnwSeD7ETEemFjuWGZmVoZGQn91SVsARwA/SZdFeSOZmVlZGgn9M4GbgCciYpakPwEeL3csMzMr\ngyI6b6NdUnTiXGZmnUwSEaHhbjNmmDv/a83ZAFRzmoj4/KgnNDOzthpueee+9LUWsAfwW4plnd2B\nNcsfzczMWm2VyzuS7gb+LCJWpPNrAHdGxN6lDeXlHTOzpjWyvNPIjtwNgPVqzq+bLjMzsy4z5Jp+\njXOA+yX1p/MTgCllDWRmZuVpqL2Tevp7U+zEvTsiFpQ6lJd3zMya1sjyTqOhvyGwAzCWt9s7d7Ri\nyCGez6FvZtakUVU2ax7kBODzwNbAHOBDwF3Aga0Y0szM2qeRHblfAMYDT0XEARSVzaWlTmVmZqVo\nJPRfi4hXASSNjYjHgB3LHcvMzMrQSOjPS2v61wI3S7oOmNvIg0u6VNJCSQ/VXLaRpJsl/VbSzyW5\n/mlm1iZNHXtHUh9FZ//GiHijgdvvDyynOCTzLumy84DnI+I8SacAG0bEqYPu5x25ZmZNamV7Z3/g\nfRExTdKmwDoR8WSDQ2wHXF8T+o8BEyJioaTNgf6I2GnQfRz6ZmZNalV7ZwqwJ8U6/jSK4+5cAew3\nwrk2i4iF6fRCYLMRPo7ZH3n2Wbj//qqnGLnVV4ePfATWWqvqSfITAbfdBi+/XPUk5WrkE7mHUzR2\n7gOIiGclrduKJ4+IkFR3k/7oo6fwvvcVp/v6+ujr62vFU1oPW74cDjgAttuue0Pz6adhn31g6tSq\nJ8nP1Klw3nnwwQ9WPUnjnn++nxde6G/qPo0ccG1WRIyXNDsidpe0NnBXRDT0B9KHWN7pi4gF6ZO+\nt9Vb3tlii2DOHHj3u5v6fixjxx8PK1fCtGlVTzJyy5bBbrvBt74Fn/hE1dPk45FHYMIEuPNO2LGL\nu4mtOuDajyV9B9hA0t8CtwDfG8Vc1wHHpNPHULSC/sixx8JxxxVvucxWZcYMuOMOuPDCqicZnfXW\ngyuugM99DubPr3qaPLz+OkyaBF/7WncHfqMa3ZF7EHBQOntTRNzc0INLV1EcoG0TivX7fwH+D/Aj\n4D0U1c8jIuLFQfeLN94I9t0XjjkGTjqpwe/GsjRvHuy1F1x/PYwfX/U0rXHGGcVW5003wWqNbJrZ\niH3xi/DkkzBzJmjYbeTO17L2TrsNtHcefxz23bfYudJN62zWPm+9BRMnwkc/CqefXvU0rfPmm9DX\nB4cfXoSSlePnPy+WBefMgY03rnqa0WvJ8o6kv5T0uKRlkl5KX8taN+bQtt8ezj23eOv12mvteEbr\nNl//erEEeMopVU/SWmPGFMs8554Ls2dXPU1vWrSoWEKePr03Ar9RjezIfQI4LCIebc9I7+zpR8AR\nR8BWW8EFF7RrAusG99wDhx4K990H22xT9TTl+MEP4Kyziu9x3Liqp+kdEfDxj8POOxcvrL2iVTty\nF7Qz8AeT4LvfhauvhhtvrGoK6zTLl8NRR8G//VvvBj4U73L32gtOPrnqSXrL1KnFjvKzzqp6kvYb\ncktf0l+mkx8GNqdo2QwceiEi4urShqrzidzbb4cjj8Q1TgN6o57ZKNc4W6tX6pn1jGpHrqTLSH8w\nBVDNaQAi4rgWzDjUc9c9DMPpp8MDD8ANN3T/XnYbuRkz4LTTik/ertuSjwl2vl/+Ej75yeJ73nLL\nqqfpXq+/DnvvDSeeCCecUPU0rdf17Z3BVqzANc7M9WI9s1GucY5eL9Uz6+m50Adc48xYr9YzG+Ua\n5+j0Wj2znp4MfYBLLy2aPLNmwdixbRzMKnXOOcXO/FtuKQ5MlqO5c4t3ODfdBLvvXvU03WPRomK/\nyOWXw4E9/Ideezb0XePMTw71zEa5xtmcXq1n1tOqD2d9peZ0R2xXu8aZl1zqmY1yjbM5Odcz6xmu\nvXMqcAdwUUTsmi67PyL2KH2oBv+IimucefibvynW83OoZzbKNc7G9HI9s57Rbuk/BnwKeK+kOyVd\nDGwiaadh7tNWEyb4aJy9bubM4sW924+e2Wo+Gueq5Xb0zEYNt6XfB/wKuAv4U2Bn4AbgVmCniNin\ntKGa+HOJrnH2rpzrmY1yjXNovV7PrGe0H846GxhPEfjTgAeBL0XEzq0etM5zN/U3cl3j7D251zMb\n5RpnfTnUM+tpSXtH0gPA8RR/J/erwG+BxRHxF60atM5zNv2H0V3j7C2uZzbONc53yqWeWU+rQv+8\niPhyOj3wJxM3jYhFLZx18HM2HfqucfYO1zOb5xpnIad6Zj0t7+lL2jUiHhj1ZKt+nqZDH2DJEth1\n16LOefDBJQxmpVu+HPbYA7761eJF3Br3mc/A2mvn/UfVL7oILrmkOFbRmmtWPU379eyHs4bjGmd3\ncz1z5HKvceZWz6wny9AHH42zW82cCaeemtfRM1st16Nx9vrRMxuVbei7xtl9XM9snRxrnDnWM+vJ\nNvTBNc5u4npma+VW48y1nllP1qEPrnF2C9czWy+XGmfO9cx6sg991zg7n+uZ5en1Gmfu9cx6sg99\ncI2zk7meWb5ernHmXs+sx6GfuMbZmVzPLF+v1jhdz6zPoV/DNc7O4npm+/RajdP1zKE59Gu4xtk5\nXM9sv16qcbqeOTSH/iCucVbP9cxq9EqN0/XM4Tn063CNs1rnnAM/+xnceqvrme3W7TVO1zNXzaFf\nh2uc1RmoZ957L7znPVVPk6durXG6ntkYh/4QXONsP9czO0c31jhdz2yMQ38YrnG2l+uZnaPbapyu\nZzbOob8KrnG2h+uZneeuu4qdup1e4xyoZ550UrHhYMNz6K+Ca5zlcz2zc3VDjdP1zOY49BvgGmd5\nXM/sbJ1e43Q9s3kO/Qa5xlkO1zM7X6fWOF3PHBmHfoNc42y9e++FQw5xPbMbdFqN0/XMkXPoN8E1\nztZxPbP7dFKN0/XMkXPoN8k1ztZwPbP7dEqN0/XM0XHoj4BrnKPjemb3qrrG6Xrm6Dn0R8A1zpFz\nPbP7VVnjdD1z9Bz6I+QaZ/Ncz+wNVdU4Xc9sDYf+KLjG2RzXM3tHu2ucrme2jkN/FFzjbJzrmb2n\nXTVO1zNby6E/Sq5xrprrmb2rHTVO1zNby6HfAq5xDs/1zN5Vdo3T9czWc+i3iGuc9bme2fvKqnG6\nnlmOjg59SXOBZcBbwIqIGF9zXUeFvmucf8z1zHyUUeN0PbMcnR76TwJ7RsTiOtd1VOiDa5y1XM/M\nS6trnK5nlqeR0K/6KNpd8xq//fZFu2DSJHjttaqnqdY3vgErV8Ipp1Q9ibXDmDFwxRXF7//s2aN7\nrEWL4LjjYPp0B35VqtzS/3/AUorlne9ExMU113Xclj64xgmuZ+ZstDVO1zPL18iW/ph2DVPHfhHx\ne0mbAjdLeiwifjFw5ZQpU/5ww76+Pvr6+to/4SBSUd/cddeiwplbjXP58uKdzre/7cDP0aRJxQfw\nTj55ZDXOqVNh/nyYMaP1s+Wqv7+f/v7+pu7TEe0dSZOB5RHxzXS+I7f0B+Ra43Q900Za43Q9sz06\ndk1f0jhJ66bTawMHAQ9VMctITJgAxx5brE128GtTS82cWbzYXXhh1ZNYldZbD668Ej73uWKrvRGv\nv168Szj7bAd+J6hkS1/Se4Fr0tkxwJURcXbN9R29pQ951Thdz7TBmqlxup7ZPh1d2RxON4Q+5FHj\nHKhnTpwI//RPVU9jnaLRGqfrme3l0G+DXj8a57nnwk9/6qNn2h9b1dE4ffTM9nPot0Ev1zhdz7RV\nGarG6XpmNRz6bdKLR+P00TOtUfWOxumjZ1bDod9GvVbjdD3TGjW4xul6ZnUc+m3WK0fj9NEzrVkD\nR+Mc+NdHz6yGQ7/NBmqcBxwAe+5Z9TQjs2JF0cZwPdOadcYZcP75xU5b1zOr4dCvwBNPwOTJRXh2\nq0MOKT5/YNaMN98sgv8f/9H1zKo49M3MMtKxh2EwM7NqOPTNzDLi0Dczy4hD38wsIw59M7OMOPTN\nzDLi0Dczy4hD38wsIw59M7OMOPTNzDLi0Dczy4hD38wsIw59M7OMOPTNzDLi0Dczy4hD38wsIw59\nM7OMOPTNzDLi0Dczy4hD38wsIw59M7OMOPTNzDLi0Dczy4hD38wsIw59M7OMOPTNzDLi0Dczy4hD\n38wsIw59M7OMOPTNzDLi0Dczy4hD38wsIw59M7OMOPTNzDLi0Dczy4hD38wsIw59M7OMOPTNzDJS\nSehLOljSY5Iel3RKFTOYmeWo7aEvaXXg28DBwPuBIyXt3O45ytTf31/1CKPi+avl+avTzbM3qoot\n/fHA7yJibkSsAH4IfLyCOUrT7b84nr9anr863Tx7o6oI/a2AeTXnn0mXmZlZyaoI/ajgOc3MDFBE\nezNY0oeAKRFxcDp/GrAyIs6tuY1fGMzMRiAiNNz1VYT+GOA3wEeA+cAs4MiIeLStg5iZZWhMu58w\nIt6UdBJwE7A6cIkD38ysPdq+pW9mZtXpuE/kdvMHtyRdKmmhpIeqnmUkJG0j6TZJD0v6taTPVz1T\nMySNlXS3pDmSHpF0dtUzNUvS6pJmS7q+6lmaJWmupAfT/LOqnqdZkjaQNEPSo+n350NVz9QoSTum\nn/vA19Kh/v/tqC399MGt3wATgWeBe+ii9X5J+wPLge9HxC5Vz9MsSZsDm0fEHEnrAPcBn+iWnz+A\npHER8Urad3Qn8KWIuLPquRol6WRgT2DdiPhY1fM0Q9KTwJ4RsbjqWUZC0nTg9oi4NP3+rB0RS6ue\nq1mSVqPIz/ERMW/w9Z22pd/VH9yKiF8AS6qeY6QiYkFEzEmnlwOPAltWO1VzIuKVdHJNin1GXRNA\nkrYGDgG+BwzbwOhgXTm3pPWB/SPiUij2PXZj4CcTgSfqBT50Xuj7g1sdQtJ2wO7A3dVO0hxJq0ma\nAywEbouIR6qeqQnnA/8LWFn1ICMUwP+VdK+kE6oepknvBRZJmibpfkkXSxpX9VAj9GngB0Nd2Wmh\n3zlrTRlLSzszgC+kLf6uERErI2I3YGvgw5L6Kh6pIZIOA56LiNl06dYysF9E7A78OXBiWu7sFmOA\nPYB/j4g9gJeBU6sdqXmS1gT+AvjxULfptNB/Ftim5vw2FFv71iaS1gBmAldExLVVzzNS6a35T4C9\nqp6lQfsCH0vr4lcBB0r6fsUzNSUifp/+XQRcQ7Fc2y2eAZ6JiHvS+RkULwLd5s+B+9J/g7o6LfTv\nBbaXtF16xfor4LqKZ8qGJAGXAI9ExAVVz9MsSZtI2iCdfhfwUWB2tVM1JiJOj4htIuK9FG/Pb42I\nv656rkZJGidp3XR6beAgoGtabBGxAJgnaYd00UTg4QpHGqkjKTYahtT2D2cNp9s/uCXpKmACsLGk\necC/RMS0isdqxn7A0cCDkgbC8rSIuLHCmZqxBTA9tRdWAy6PiFsqnmmkum2pczPgmmK7gTHAlRHx\n82pHato/AFemDc4ngOMqnqcp6cV2IjDs/pSOqmyamVm5Om15x8zMSuTQNzPLiEPfzCwjDn0zs4w4\n9M3MMuLQNzPLiEPfLJG0vqS/G+b6/2zgMbrqsBWWH4e+2ds2BP5+8IXpMLtExH4NPIY/+GIdraM+\nkWtWsXOAP0mfRl4BvE5xaOYdgZ0kLY+IddIB6a6leJFYA/hKRPhwIdYV/Ilcs0TStsANEbGLpAkU\nB2z7QEQ8la5/KSLWTX/sZ1xEvCRpE+CuiNi+9jaVfRNmq+AtfbO3adDpWQOBP8hqwNnp0MErgS0l\nvTsinmvHkGaj4dA3G9rLQ1x+FLAJsEdEvJUOhzy2fWOZjZx35Jq97SWgkaWZ9Sj+4Mlbkg4Ati13\nLLPW8Za+WRIRL0j6T0kPAa8CCwbfJP17JXC9pAcp/gbEo3VuY9aRvCPXzCwjXt4xM8uIQ9/MLCMO\nfTOzjDj0zcwy4tA3M8uIQ9/MLCMOfTOzjDj0zcwy8v8Bx3OqAwcrNtMAAAAASUVORK5CYII=\n", 331 | "text/plain": [ 332 | "" 333 | ] 334 | }, 335 | "metadata": {}, 336 | "output_type": "display_data" 337 | } 338 | ], 339 | "source": [ 340 | "num_trials, num_flips = 8, 20\n", 341 | "data = np.zeros(num_trials)\n", 342 | "bias1, bias2 = .7, .3\n", 343 | "theta = [bias1, bias2]\n", 344 | "\n", 345 | "def run_trials(data, theta):\n", 346 | " num_trials, = data.shape\n", 347 | " flips = random(num_flips)\n", 348 | " for i in range(num_trials):\n", 349 | " if random() < .5:\n", 350 | " data[i] = np.sum(flips < theta[0])\n", 351 | " else:\n", 352 | " data[i] = np.sum(flips < theta[1])\n", 353 | "\n", 354 | "run_trials(data, theta)\n", 355 | " \n", 356 | "\n", 357 | "plt.plot(data)\n", 358 | "plt.title('# of heads per trial')\n", 359 | "plt.xlabel('trial')\n", 360 | "plt.ylabel('# heads')\n", 361 | "plt.ylim(0, 20);\n", 362 | "print(data)" 363 | ] 364 | }, 365 | { 366 | "cell_type": "markdown", 367 | "metadata": {}, 368 | "source": [ 369 | "By inspection of the code we see that one coin had a bias of $\\beta=0.57$ and the second had a bias of $\\beta=0.43$. At first glance this problem seems intractable. For example, probability allows for a coin with bias $\\beta=.99$ to flip tails 10 times in a row, so we cannot assume that any given trial will have a high number of heads reflects a large bias ($\\beta >> 0.5$). If in the code I made the bias for one coin much higher than the other, such as `bias_1 = 0.97`, `bias_2 = 0.007`, we might be able to solve this problem by inspection just by assuming that trials that are all heads are due to coin one and trials with almost all tails are due to coin two. But the differences in data are not generally so stark.\n", 370 | "\n", 371 | "So, in general the problem seems intractable. But, is it? Let's just take a naive approach and see where it gets us.\n", 372 | "\n", 373 | "We need to start somewhere, so lets start by assuming some bias for each coin. We have no reason to assume any value, so I will choose a value at random: 0.75 for the first coin, and 0.3 for the second coin. This is *entirely* arbitrary - I did not inspect the data and somehow infer these numbers. We need to start somewhere.\n", 374 | "\n", 375 | "Let's get this into standard notation. First, we say that\n", 376 | "\n", 377 | "$$\\begin{aligned}\\beta_1 &= 0.6\\\\\\beta_2 &= 0.3\\end{aligned}$$\n", 378 | "\n", 379 | "where $\\beta_i$ is the bias for the i$^{th}$ coin. Now we collect them into $\\theta$ for notational convenience:\n", 380 | "\n", 381 | "$$\\theta = (\\beta_1, \\beta_2) = (0.6, 0.3)$$\n", 382 | "\n", 383 | "In code we might express this as" 384 | ] 385 | }, 386 | { 387 | "cell_type": "code", 388 | "execution_count": 31, 389 | "metadata": { 390 | "collapsed": false 391 | }, 392 | "outputs": [ 393 | { 394 | "data": { 395 | "text/plain": [ 396 | "array([ 0.6, 0.3])" 397 | ] 398 | }, 399 | "execution_count": 31, 400 | "metadata": {}, 401 | "output_type": "execute_result" 402 | } 403 | ], 404 | "source": [ 405 | "theta = np.array((0.6, 0.3))\n", 406 | "theta" 407 | ] 408 | }, 409 | { 410 | "cell_type": "markdown", 411 | "metadata": {}, 412 | "source": [ 413 | "These are very unlikely to be close to accurate, but let's press on. We can now inspect each trial and compute if it is more likely to have been caused by the first coin or second coin based on these biases." 414 | ] 415 | }, 416 | { 417 | "cell_type": "code", 418 | "execution_count": 32, 419 | "metadata": { 420 | "collapsed": false 421 | }, 422 | "outputs": [ 423 | { 424 | "name": "stdout", 425 | "output_type": "stream", 426 | "text": [ 427 | "coin 1 is the closest match for bias 0.5\n", 428 | "coin 2 is the closest match for bias 0.2\n", 429 | "coin 2 is the closest match for bias 0.2\n", 430 | "coin 1 is the closest match for bias 0.5\n", 431 | "coin 1 is the closest match for bias 0.5\n", 432 | "coin 2 is the closest match for bias 0.2\n", 433 | "coin 1 is the closest match for bias 0.5\n", 434 | "coin 1 is the closest match for bias 0.5\n" 435 | ] 436 | } 437 | ], 438 | "source": [ 439 | "def compute_assignment(biases, theta):\n", 440 | " num_trials = len(biases) \n", 441 | " assignment = np.zeros(num_trials, dtype=int)\n", 442 | " for i, bias in enumerate(biases):\n", 443 | " assignment[i] = np.abs(theta - bias).argmin() \n", 444 | " return assignment\n", 445 | "\n", 446 | "bias = data / num_flips\n", 447 | "assignment = compute_assignment(bias, theta)\n", 448 | "\n", 449 | "for i in range(len(data)):\n", 450 | " print('coin {} is the closest match for bias {}'\n", 451 | " .format(assignment[i]+1, bias[i]))" 452 | ] 453 | }, 454 | { 455 | "cell_type": "markdown", 456 | "metadata": {}, 457 | "source": [ 458 | "Now that we have the assignments, we have an outline for an algorithm. We could start with an initial estimate for $\\theta$ and compute which coin is most likely \n", 459 | "\n", 460 | "If we know there is a probability $p$ for an event, and we observe $d$ occurances of that event over $N$ trials, the probability of that occuring is given by\n", 461 | "\n", 462 | "$$P(\\text{data} \\, \\lvert\\, p) = \\begin{pmatrix}N\\\\d\\end{pmatrix} p^d (1-p)^{N-d}$$\n", 463 | "\n", 464 | "We will turn that into code with:" 465 | ] 466 | }, 467 | { 468 | "cell_type": "code", 469 | "execution_count": 22, 470 | "metadata": { 471 | "collapsed": false 472 | }, 473 | "outputs": [], 474 | "source": [ 475 | "from math import log\n", 476 | "from scipy.misc import comb\n", 477 | "\n", 478 | "def coin_likelihood(N, d, bias):\n", 479 | " return comb(N, d, exact=True) * bias**d * (1-bias)**(N-d)\n", 480 | "\n", 481 | "def coin_log_likelihood(N, d, bias):\n", 482 | " return np.log(comb(N, d, exact=True) * bias**d * (1-bias)**(N-d))" 483 | ] 484 | }, 485 | { 486 | "cell_type": "code", 487 | "execution_count": 23, 488 | "metadata": { 489 | "collapsed": false, 490 | "scrolled": true 491 | }, 492 | "outputs": [ 493 | { 494 | "name": "stdout", 495 | "output_type": "stream", 496 | "text": [ 497 | "0.10291934520000003\n", 498 | "0.24609375\n", 499 | "-2.27380965381\n", 500 | "-1.40204271809\n" 501 | ] 502 | } 503 | ], 504 | "source": [ 505 | "print(coin_likelihood(N=10, d=5, bias=.7))\n", 506 | "print(coin_likelihood(N=10, d=5, bias=.5))\n", 507 | "print(coin_log_likelihood(N=10, d=5, bias=.7))\n", 508 | "print(coin_log_likelihood(N=10, d=5, bias=.5))" 509 | ] 510 | }, 511 | { 512 | "cell_type": "code", 513 | "execution_count": 19, 514 | "metadata": { 515 | "collapsed": false 516 | }, 517 | "outputs": [ 518 | { 519 | "data": { 520 | "text/html": [ 521 | "\n", 728 | "\n" 747 | ], 748 | "text/plain": [ 749 | "" 750 | ] 751 | }, 752 | "execution_count": 19, 753 | "metadata": {}, 754 | "output_type": "execute_result" 755 | } 756 | ], 757 | "source": [ 758 | "# style the notebook\n", 759 | "from IPython.core.display import HTML\n", 760 | "import urllib.request\n", 761 | "response = urllib.request.urlopen('http://bit.ly/1LC7EI7')\n", 762 | "HTML(response.read().decode(\"utf-8\"))" 763 | ] 764 | } 765 | ], 766 | "metadata": { 767 | "kernelspec": { 768 | "display_name": "Python 3", 769 | "language": "python", 770 | "name": "python3" 771 | }, 772 | "language_info": { 773 | "codemirror_mode": { 774 | "name": "ipython", 775 | "version": 3 776 | }, 777 | "file_extension": ".py", 778 | "mimetype": "text/x-python", 779 | "name": "python", 780 | "nbconvert_exporter": "python", 781 | "pygments_lexer": "ipython3", 782 | "version": "3.4.3" 783 | } 784 | }, 785 | "nbformat": 4, 786 | "nbformat_minor": 0 787 | } 788 | -------------------------------------------------------------------------------- /Hoeffdings_Inequality.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [ 10 | { 11 | "data": { 12 | "text/html": [ 13 | "\n", 220 | "\n" 239 | ], 240 | "text/plain": [ 241 | "" 242 | ] 243 | }, 244 | "execution_count": 1, 245 | "metadata": {}, 246 | "output_type": "execute_result" 247 | } 248 | ], 249 | "source": [ 250 | "# style the notebook\n", 251 | "from IPython.core.display import HTML\n", 252 | "import urllib.request\n", 253 | "response = urllib.request.urlopen('http://bit.ly/1LC7EI7')\n", 254 | "HTML(response.read().decode(\"utf-8\"))" 255 | ] 256 | }, 257 | { 258 | "cell_type": "markdown", 259 | "metadata": {}, 260 | "source": [ 261 | "# Hoeffding's Inequality\n", 262 | "\n", 263 | "Suppose we have a container of marbles. Some are red, some are green, and we don't know the ratio. We draw $N$ independent samples with replacement from the container. \n", 264 | "\n", 265 | "We define the ratio of $\\nu = \\frac{\\text{# red marbles drawn}}{N}$ \n", 266 | "\n", 267 | "$\\mu$ is the actual ratio of red marbles to all marbles in the container.\n", 268 | "\n", 269 | "We want to know if $\\nu$ is close to $\\mu$. Formally,\n", 270 | "\n", 271 | "$$\\lvert \\nu - \\mu \\lvert < \\epsilon$$.\n", 272 | "\n", 273 | "Hoeffding's Inequality gives us a lower bound:\n", 274 | "\n", 275 | "$$\\Large P( \\lvert \\nu - \\mu \\lvert > \\epsilon ) \\le 2e^{-2N\\epsilon^2}$$\n", 276 | "\n", 277 | "Very powerful as the bound does not depend on $\\mu$!\n", 278 | "\n", 279 | "However, as $\\epsilon$ gets smaller you need exponentially larger $N$ to get the same error bounds because of the $\\epsilon^2$ term." 280 | ] 281 | }, 282 | { 283 | "cell_type": "code", 284 | "execution_count": 2, 285 | "metadata": { 286 | "collapsed": false 287 | }, 288 | "outputs": [], 289 | "source": [ 290 | "import math\n", 291 | "\n", 292 | "def hoeffding(epsilon, N):\n", 293 | " return 2*math.exp(-2 * N * (epsilon**2))" 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": 3, 299 | "metadata": { 300 | "collapsed": false, 301 | "scrolled": true 302 | }, 303 | "outputs": [ 304 | { 305 | "data": { 306 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEACAYAAAC9Gb03AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGblJREFUeJzt3Xt0VOW5x/HvAwl3y00EBeQmICC1SCtBakkFewJSe6qu\npWKr9iimVmyrtUe7VlVa23VOb4pWRYsHULygxS4PtNYip8ZLRWwREeRivFBQK1pBpCgI8p4/3omJ\nMclMZvbMO3vP77PWXslk9sz8Zq/Jkyfv3vvd5pxDRESSq03oACIikl8q9CIiCadCLyKScCr0IiIJ\np0IvIpJwKvQiIgnXYqE3s/5m9oiZPW9m68zs282sd4OZ1ZrZGjMbk5+oIiKSjbI09+8DLnHOPWtm\nXYBVZvawc25D3QpmNhU4wjk31MzGAXOAivxFFhGR1mixo3fOveGcezb1/b+ADcBhjVY7Gbg9tc5K\noJuZ9c5DVhERyULGY/RmNhAYA6xsdFdfYGuD268C/XINJiIi0cio0KeGbRYD30l19p9YpdFtzasg\nIlIk0o3RY2blwP3Anc65B5pY5TWgf4Pb/VI/a/w8Kv4iIllwzjVuplsl3VE3BvwPsN45N7uZ1ZYA\nZ6fWrwDecc5ta2pF59xHy+bNjiuvdPTp47joIseuXe5j9yd5ufrqq4NnKJZF20LbQtui5SUK6YZu\nJgBfA75oZqtTyxQzqzaz6lTxfhB42cxeBG4FvpXJCw8YAD/+MaxfD7t3w+jRsHx5Lm9FRESa0uLQ\njXPuCTIYx3fOzcw2QPfuMH8+PPQQnHcefO1r8JOfgOX0j4qIiNQpmjNjq6pg1Sp4+GGYMQP27w+d\nKH8qKytDRyga2hb1tC3qaVtEy6IaA0r7QmYuk9f617/g1FOhQwdYtAg6dixAOBGRImVmuHzujA2h\nSxdYuhQ6d4Zp0+CDD0InEhGJt6Lr6Ot8+CGcdhocdBDcfrvG7EWkNCWyo6/Tti3cdRds3Ag/+lHo\nNCIi8ZX2hKmQOnXywzjjx8OgQXDOOaETiYjET9EO3TS0YQNMnOiL/rhxEQcTESliiR66aWjECJgz\nB6ZPh3ffDZ1GRCReYtHR16mu9odf3nmnds6KSGkomY6+znXXwerVsHBh6CQiIvERq44e4LnnYNIk\nePJJGDo0gmAiIkUsio4+doUeYPZseOABeOQRDeGISLKV3NBNnYsv9jNeLlgQOomISPGLZUcPfqy+\nqgrWrYNevSJ7WhGRolKyQzd1LrsMtm3TzlkRSa6SL/S7d8OoUTB3Lpx4YqRPLSJSFEp2jL5O585w\n001w0UWa5VJEpDmxLvQAJ50EgwfDLbeETiIiUpxiPXRTZ906OOEE2LTJX5pQRCQpSn6MvqHqan/R\nkl/9Km8vISJScCr0DWzb5nfMrlwJQ4bk7WVERAqq5HfGNtS7N1xyCVxxRegkIiLFJTEdPcD778Ow\nYfDb30JFRV5fSkSkINTRN9KxI/zwh3D11aGTiIgUj0QVeoBvfANeeAGeeCJ0EhGR4pC4Qt+uHVx5\nJVx1VegkIiLFIXGFHuDrX4ctW/w0xiIipS6Rhb683I/TX3UVFGhfs4hI0UpkoQd/IfG33oKHHw6d\nREQkrMQW+rZt/Vj9T38aOomISFiJLfQAp58OW7f668uKiJSqRBf6sjL4/vfhv/4rdBIRkXASdWZs\nU/bsgUGDYNkyGD264C8vIpITnRmbgQ4d/Bw4//3foZOIiISR+I4e4N13/cVJnn7afxURiQt19Bn6\n1Kfgm9+EX/widBIRkcIriY4e/DH1w4f7q1D16hUshohIq6ijb4VeveC002DOnNBJREQKq2Q6eoD1\n6/21ZTdv9jtpRUSKnTr6Vho5Eo45Bu6+O3QSEZHCKalCD3DppXDttZrsTERKR8kV+kmToE0bfwKV\niEgpKLlCb1bf1YuIlIKS2hlbZ+9eGDgQli+HUaNCpxERaZ52xmapfXuoroYbbwydREQk/0qyowf4\nxz/8UTivvALduoVOIyLSNHX0OTj0UJgyBRYsCJ1ERCS/0hZ6M5tnZtvMbG0z91ea2U4zW51afhh9\nzPyYORNuugkOHAidREQkfzLp6OcDVWnWedQ5Nya1/CSCXAUxfryf8Oyhh0InERHJn7SF3jn3OLAj\nzWo5jR+FYgYXX6ydsiKSbFGM0TvgODNbY2YPmtnICJ6zYM44A/72N6itDZ1ERCQ/yiJ4jmeA/s65\n98xsCvAAMKypFWfNmvXR95WVlVRWVkbw8rnp0AHOOw9uvhmuuy50GhEpdTU1NdTU1ET6nBkdXmlm\nA4Glzrm0V101s1eAsc657Y1+XlSHVza0eTN89rOwZQt06hQ6jYhIvaI4vNLMepuZpb4/Fv/HY3ua\nhxWVgQOhogLuvTd0EhGR6GVyeOU9wJPAcDPbamb/YWbVZladWuU0YK2ZPQvMBs7IX9z8ufBCXZRE\nRJKpZM+MbezDD2HIELj/fhg7NnQaERGvKIZukqJtWz//jbp6EUkadfQNvPmmv4D4yy9D9+6h04iI\nqKOP3CGHQFUV3HFH6CQiItFRoW/kwgvhllt0qUERSQ4V+kaOP95PjfD446GTiIhEQ4W+ETO44AL4\nzW9CJxERiYZ2xjZh+3YYPBheegl69gydRkRKmXbG5kmPHvDlL2unrIgkgwp9M+qGb2LyT4iISLNU\n6Jvx+c9rp6yIJIMKfTO0U1ZEkkI7Y1vw9tt+/hvtlBWRULQzNs969oRp02DhwtBJRESyp0Kfxvnn\nw9y52ikrIvGlQp/GxImwbx+sWBE6iYhIdlTo0zCr7+pFROJIO2MzsG2bn77473+Hrl1DpxGRUqKd\nsQXSuzdMngx33x06iYhI66nQZ2jGDA3fiEg8qdBn6MQT/WRnq1aFTiIi0joq9Blq0wbOO09dvYjE\nj3bGtsLWrXD00f5r586h04hIKdDO2ALr3x/Gj4fFi0MnERHJnAp9K82YAbfdFjqFiEjmNHTTSvv2\nweGHwyOPwJFHhk4jIkmnoZsAysvhnHPU1YtIfKijz0JtLUyYAK++Cu3ahU4jIkmmjj6QoUNh5EhY\nsiR0EhGR9FTos6SJzkQkLjR0k6X334d+/eCZZ2DAgNBpRCSpNHQTUMeOMH06zJsXOomISMvU0edg\nzRp/qcHNm6Ft29BpRCSJ1NEHdvTRcOihsGxZ6CQiIs1Toc+RdsqKSLHT0E2Odu3yZ8pu2AB9+oRO\nIyJJo6GbInDQQXDKKXDHHaGTiIg0TR19BJ56Cs4+GzZt8hcTFxGJijr6IjFunJ8K4dFHQycREfkk\nFfoImGn6YhEpXhq6icj27TB4MLz8MvToETqNiCSFhm6KSI8ecNJJsHBh6CQiIh+nQh+hGTP8MfUJ\n/sdFRGJIhT5CEyfCBx/AihWhk4iI1FOhj1DdTlmdKSsixUQ7YyP25pswfLif6Kxr19BpRCTutDO2\nCB1yCEyeDHfdFTqJiIinQp8HF1wAv/mNdsqKSHFIW+jNbJ6ZbTOztS2sc4OZ1ZrZGjMbE23E+Jk0\nyU929vTToZOIiGTW0c8Hqpq708ymAkc454YCFwBzIsoWW23a1Hf1IiKhpS30zrnHgR0trHIycHtq\n3ZVANzPrHU28+Dr3XPjd72DnztBJRKTURTFG3xfY2uD2q0C/CJ431nr3hhNPhDvvDJ1EREpdWUTP\n0/jQnyZ3Q86aNeuj7ysrK6msrIzo5YtTdTVccgl861uavlhEMlNTU0NNTU2kz5nRcfRmNhBY6pwb\n3cR9twA1zrlFqdsbgYnOuW2N1iuJ4+gbOnDAH1O/cCFUVIROIyJxVCzH0S8Bzk4FqgDeaVzkS1Xd\nTtlbbw2dRERKWdqO3szuASYCBwPbgKuBcgDn3K2pdW7EH5mzG/iGc+6ZJp6n5Dp6gLfegmHD/PTF\n3buHTiMicRNFR68pEArgrLPgc5+D7343dBIRiRsV+ph44gk47zzYuFE7ZUWkdYpljF7SmDAB2reH\nP/85dBIRKUUq9AVgBhdeCHNK/pxhEQlBQzcFsmsXDBgAa9dC376h04hIXGjoJkYOOgjOOANuuy10\nEhEpNeroC2jtWqiq8hclKS8PnUZE4kAdfcyMHg2DB8OSJaGTiEgpUaEvsJkz4cYbQ6cQkVKioZsC\n27cPBg6Ehx7yHb6ISEs0dBND5eV+Vkt19SJSKOroA3jjDRgxQvPfiEh66uhjqk8fmDoV5s8PnURE\nSoE6+kCeespPdlZb66czFhFpijr6GBs3zg/b/PGPoZOISNKp0AdiBhdfDDfcEDqJiCSdhm4C2rvX\nz3/zf/8Ho0aFTiMixUhDNzHXvr2f1fL660MnEZEkU0cf2Jtv+guI19bCwQeHTiMixUYdfQIccgh8\n9au6gLiI5I86+iLw3HP1s1q2axc6jYgUE3X0CfHpT/szZe+7L3QSEUkiFfoiccklcN11oH96RCRq\nKvRFYupU2L0bampCJxGRpFGhLxJt2sD3vge/+EXoJCKSNNoZW0T27IFBg2DZMs1VLyKedsYmTIcO\nflqEX/4ydBIRSRJ19EVmxw4YMsQfctmvX+g0IhKaOvoE6t4dzjkHZs8OnUREkkIdfRHasgXGjIGX\nXoJu3UKnEZGQ1NEn1OGH+8Mt58wJnUREkkAdfZF6/nmYNMlfV7ZTp9BpRCQUdfQJNmoUTJgAc+eG\nTiIicaeOvoitWgVf+Yofq2/fPnQaEQlBHX3CjR3rT5y6/fbQSUQkztTRF7m//AXOPhs2bYKystBp\nRKTQ1NGXgAkToH9/WLQodBIRiSt19DGwfDlcdJE/EkddvUhpUUdfIiZNgt694Z57QicRkThSRx8T\nNTVw/vmwcaO6epFSoo6+hFRW+jNmFy4MnURE4kYdfYw8/rif8GzTJigvD51GRApBHX2JOf54OOII\nmD8/dBIRiRN19DGzYgWcfjq88IK/UImIJJs6+hI0fjx85jOa2VJEMqeOPobWrYMTToDaWujaNXQa\nEckndfQl6qijYNo0+PnPQycRkThQRx9TW7f6IZznnoO+fUOnEZF8KUhHb2ZVZrbRzGrN7PIm7q80\ns51mtjq1/DCXQJKZ/v39CVQ/+lHoJCJS7Frs6M2sLbAJmAy8BvwVONM5t6HBOpXApc65k1t8IXX0\nkduxA4YNg8cegxEjQqcRkXwoREd/LPCic26zc24fsAj4SlNZcgkh2eneHa64Ai67LHQSESlm6Qp9\nX2Brg9uvpn7WkAOOM7M1ZvagmY2MMqC07OKL/dE3f/xj6CQiUqzSTY+VyVjLM0B/59x7ZjYFeAAY\n1tSKs2bN+uj7yspKKisrM0spzWrXDq69Fi69FCZP1tQIInFXU1NDTU1NpM+Zboy+ApjlnKtK3f4B\ncMA597MWHvMKMNY5t73RzzVGnyfOwZQpfvnOd0KnEZEoRTFGn67Ql+F3xk4CXgee5pM7Y3sDbzrn\nnJkdC9znnBvYxHOp0OfRhg3whS/4rwcfHDqNiEQl7ztjnXP7gZnAn4D1wL3OuQ1mVm1m1anVTgPW\nmtmzwGzgjFwCSXZGjIDp0+HKK0MnEZFioxOmEmTHDhg5Ev73f+HYY0OnEZEoaAoE+Zju3eGXv4Tq\nati/P3QaESkWKvQJM3069OwJv/516CQiUiw0dJNAL7wAxx0Hq1f7qRJEJL40dCNNGjYMZs7UoZYi\n4qnQJ9QVV8Dzz8P994dOIiKhaegmwZ58Ek491U9l3KtX6DQiko28nzAVJRX6MC67zM9df++9oZOI\nSDY0Ri9pXXMNrFkDixeHTiIioaijLwErVsApp2gIRySONHQjGbv8cti4ER54AExXDxCJDQ3dSMau\nuQZeew3mzAmdREQKTR19Camt9SdS/fnPMHp06DQikgl19NIqQ4f6uXDOPBPeey90GhEpFHX0JcY5\nOOssOOgguPXW0GlEJB119NJqZnDLLVBTAwsWhE4jIoWgjr5ErV8PEyfCQw/B2LGh04hIc9TRS9ZG\njvRH4Jx6Kvzzn6HTiEg+qaMvcZdfDqtW+c6+rCx0GhFpTCdMSc7274eTToIhQ+Cmm3QylUix0dCN\n5KysDO67D554Aq69NnQaEckH/bMudO0Kf/gDjB8PAwbAaaeFTiQiUVKhF8BfcnDpUvjSl+Cww/wZ\ntCKSDBq6kY+MGQMLF8JXv+qvNysiyaBCLx9TVQU33wxTp8KGDaHTiEgUNHQjn3DqqX4unC99CR59\nFAYPDp1IRHKhQi9N+vrXfbE/4QRYvhyOOCJ0IhHJlgq9NKu6Gtq08VMl/OlPcNRRoROJSDZU6KVF\nM2ZAly4webI/KudznwudSERaSztjJa0zz4S5c/0ZtA8/HDqNiLSWCr1k5MtfhsWL/di95rEXiRfN\ndSOtUlsL06b55ec/h7ZtQycSSTZNaiZBbN/uD8Hs2NGfYNWzZ+hEIsmlSc0kiB49YNkyP6f92LGw\ncmXoRCLSEhV6yUp5ub/Q+OzZfvz++uvhwIHQqUSkKRq6kZy9/DJMn+4Pw5w3Dw4/PHQikeTQ0I0U\nhcGD/Xz2kyf7oZx580B/00WKhzp6idRzz8G550K3bvDrX8OoUaETicSbOnopOp/+NDz9NJxyClRW\nwmWXwa5doVOJlDYVeolcWRnMnAnr1sHbb8PQoXDDDbB3b+hkIqVJhV7ypndvmD/fT4i2bBkMHw4L\nFsC+faGTiZQWjdFLwTz+OFx1lT9K59JL4fzzoXPn0KlEipvG6CVWjj8eHnkEfvtbeOwxGDQILr8c\nXnopdDKRZFOhl4I79li4/374y1/gww+hogL+7d/8pGl79oROJ5I8GrqR4Pbs8V3+ggXwzDPw7//u\np0b+4hf9GbgipUyTmknivP46LFrklxdf9BcrP/lkf/3aHj1CpxMpPBV6SbTXX4ff/x6WLPFj+sOG\nwaRJvtOvqPAnZYkkXUEKvZlVAbOBtsBtzrmfNbHODcAU4D3gXOfc6ibWUaGXrH3wgZ8lc/lyePRR\nWLUKBg6E8ePhmGP8Mnq0nzpZJEnyXujNrC2wCZgMvAb8FTjTObehwTpTgZnOualmNg643jlX0cRz\nqdCn1NTUUFlZGTpGUch2W+zb56dbWLHCj+uvXg2bNsGAAX765JEj4cgj4YgjYMgQP2e+5fSrkn/6\nXNTTtqgXRaFPd3HwY4EXnXObUy+4CPgKsKHBOicDtwM451aaWTcz6+2c25ZLsCTTh7hettuivNxP\noDZ2bP3P9u71V8Bav94vS5f6Qzdra/39Awb4mTUPPxwOO8wvhx4KffpAr15+adcumveVDX0u6mlb\nRCtdoe8LbG1w+1VgXAbr9ANU6KWg2reHo47yS0PO+atibdnil7//3Y//19TAP/4Bb7wBb70F//yn\nP4GrRw/o3t0v3bpB167wqU/5pUsXv3Tu7JdOnfzSsSN06OCX9u3rl3bt/B+l8vLi/49Ckitdoc90\nrKXxR1hjNFI0zPzQTc+eMGZM8+s5B++84/8o7Njhv+7c6Zd33/XLtm3+zN5du+D99+G992D3bn+I\naMNl716/X2HvXj/MtH+/v75uXdEvK/Nf27atX3buhLvv9t+3aVP/tW4xa/5rw6XuPTd1u+E2aern\nDe9raju2tI1bq6XHbNrk98OUouHD/UV9opRujL4CmOWcq0rd/gFwoOEOWTO7Bahxzi1K3d4ITGw8\ndGNmKv4iIlnI9xj934ChZjYQeB04HTiz0TpLgJnAotQfhneaGp/PNaiIiGSnxULvnNtvZjOBP+EP\nr/wf59wGM6tO3X+rc+5BM5tqZi8Cu4Fv5D21iIhkrGAnTImISBg5T2pmZlVmttHMas3s8mbWuSF1\n/xozG9Oax8ZJttvCzPqb2SNm9ryZrTOzbxc2efRy+Vyk7mtrZqvNbGlhEudPjr8j3cxssZltMLP1\nqeHR2MpxW/wg9Tuy1szuNrP2hUsevXTbwsyONLMVZrbHzL7Xmsd+gnMu6wU/nPMiMBAoB54FRjRa\nZyrwYOr7ccBTmT42TkuO26IP8JnU913wJ6mV5LZocP+lwF3AktDvJ+S2wJ+j8h+p78uArqHfU4ht\nkXrMy0D71O17gXNCv6c8b4tewGeBnwDfa81jGy+5dvQfnVDlnNsH1J1Q1dDHTqgCuplZnwwfGyfZ\nbovezrk3nHPPpn7+L/wJaYcVLnrkst4WAGbWD/8LfxufPHQ3brLeFmbWFTjeOTcvdd9+59zOAmaP\nWi6fi3eBfUAnMysDOuHP1o+rtNvCOfeWc+5v+Pfdqsc2lmuhb+pkqb4ZrnNYBo+Nk2y3Rb+GK6SO\ncBoDrIw8YeHk8rkAuA74PnAgXwELKJfPxSDgLTObb2bPmNlcM+uU17T5lfXnwjm3HfgVsAV/BOA7\nzrnlecyab5lsi8gem2uhz/aEqiTK+eQyM+sCLAa+k+rs4yrbbWFmNg140/mJ8ZLwucnlc1EGHAPc\n7Jw7Bn9U2xURZiu0rOuFmQ0BvosfrjgM6GJmZ0UXreByOQqm1Y/NtdC/BvRvcLs//q9LS+v0S62T\nyWPjJNtt8RqAmZUD9wN3OuceyGPOQshlWxwHnGxmrwD3ACeY2R15zJpvuWyLV4FXnXN/Tf18Mb7w\nx1Uu2+KzwJPOubedc/uB3+E/K3GVS/1r/WNz3KFQBryE/yvbjvQ7Vyqo37mS9rFxWnLcFgbcAVwX\n+n2E3haN1pkILA39fkJuC+AxYFjq+1nAz0K/pxDbAvgMsA7omPp9uR24KPR7yue2aLDuLD6+M7bV\ntTOKwFPwR4m8CPwg9bNqoLrBOjem7l8DHNPSY+O8ZLstgM/jx6OfBVanlqrQ7yfU56LB/ROJ+VE3\nuW4L4Gj89OBr8F1sbI+6iWBb/CfwPLA2VejLQ7+ffG4L/NF4W4GdwA78/okuzT22pUUnTImIJFzO\nJ0yJiEhxU6EXEUk4FXoRkYRToRcRSTgVehGRhFOhFxFJOBV6EZGEU6EXEUm4/wfBP/gC3tmsPQAA\nAABJRU5ErkJggg==\n", 307 | "text/plain": [ 308 | "" 309 | ] 310 | }, 311 | "metadata": {}, 312 | "output_type": "display_data" 313 | } 314 | ], 315 | "source": [ 316 | "%matplotlib inline\n", 317 | "import matplotlib.pyplot as plt\n", 318 | "import numpy as np\n", 319 | "eps = 0.1\n", 320 | "N = 1000\n", 321 | "\n", 322 | "eps_range = np.arange(1.e-3, .1, .001)\n", 323 | "hs = []\n", 324 | "for eps in eps_range:\n", 325 | " hs.append(hoeffding(eps, N))\n", 326 | "plt.plot(eps_range, hs); " 327 | ] 328 | }, 329 | { 330 | "cell_type": "code", 331 | "execution_count": 4, 332 | "metadata": { 333 | "collapsed": false 334 | }, 335 | "outputs": [ 336 | { 337 | "data": { 338 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAEACAYAAABS29YJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGylJREFUeJzt3XmUVOWd//H3l30VRIUADaICKohsBloi0BpUBAIOwRFN\nRM0MOo5m/EVPIjiekcn5neMyMS5RMxlcwLhgghEhg6AGG3EMi7IjW6uEvfkNCNgqsj2/P55qu2ia\n3qiq51bdz+uce+rWrdt1v/Uc+NZT3/vc55pzDhERyW11QgcgIiLpp2QvIhIDSvYiIjGgZC8iEgNK\n9iIiMaBkLyISA5UmezPrYGbvmtkaM1ttZv9ygv2eMLONZrbCzHqnJ1QREamtelW8fgj4mXNuuZk1\nAz4ys7edc2tLdzCzYUBn51wXM+sP/BbIT1/IIiJSU5X27J1zO51zyxPrJcBaoF253UYCUxP7LAJa\nmlmbNMQqIiK1VO2avZl1AnoDi8q91B7YkvR8K5B3soGJiEjqVCvZJ0o404E7Ez3843Yp91xzMIiI\nREhVNXvMrD7wGvCic25GBbtsAzokPc9LbCv/PvoCEBGpBedc+Q51jVU1GseAZ4GPnXOPnWC3mcC4\nxP75wF7nXHFFOx4+7HBOy/333x88hqgsagu1hdqi8iVVqurZfw/4MbDSzJYltt0LdARwzv3OOTfb\nzIaZWRHwJXDzid7sgw9g4MAURC0iIjVSabJ3zr1PNer6zrk7qnOwN95QshcRCSGjV9DOmAEp/FWS\ntQoKCkKHEBlqizJqizJqi9SzVNaEKj2QmevY0TF7NnTvnpFDiohkPTPDpfsEbapdfbXv3YuISGZl\nNNmPGuXr9iIiklkZLeMcPOj4zndg5Upo3z4jhxURyWpZWcapXx+GDYOZMzN5VBERyfh89qrbi4hk\nXkbLOM45SkqgXTvYsgVatMjIoUVEslZWlnEAmjWDQYPgzTczfWQRkfgKclvCUaNUyhERyaSMl3EA\ndu2Crl1hxw5o3DgjhxcRyUpZW8YBaN0a+vaFuXNDHF1EJH6CJHuAMWPgj38MdXQRkXgJUsYBKC6G\n887zpZxGjTISgohI1snqMg5AmzbQsye89VaoCERE4iNYsgdfypk+PWQEIiLxEKyMA7B9u5/ueOdO\naNgwI2GIiGSVrC/jgL+S9oIL4J13QkYhIpL7giZ7gGuu0agcEZF0C1rGAdi61Z+o3bEDGjTISCgi\nIlkjJ8o4AHl5fgjmvHmhIxERyV3Bkz34UTl/+EPoKEREclfwMg6UlXK2b9eoHBGRZDlTxgFfyunR\nA+bMCR2JiEhuikSyB7juOnj55dBRiIjkpkiUcQB274azz/YlnebNMxKSiEjk5VQZB+C00/wdrN54\nI3QkIiK5JzLJHlTKERFJl8iUcQBKSqB9eygqgjPOyEhYIiKRlnNlHPA3Ix8+XDNhioikWqSSPaiU\nIyKSDpEq4wAcPOhnw1y6FDp2zEBgIiIRlpNlHPCToY0eDdOmhY5ERCR3RC7ZA/zoR/Dii5ChHx0i\nIjkvksl+4EDYvx+WLw8diYhIbohksq9TB8aNgylTQkciIpIbIneCttQnn8DFF/vpE3RTExGJq5w9\nQVvqnHPg3HPhzTdDRyIikv0im+wBbrwRpk4NHYWISPaLbBkHYN8+OPNMX9I57bQ0BSYiEmE5X8YB\naNEChg2DV14JHYmISHaLdLIHlXJERFIh8sl+yBB/b9o1a0JHIiKSvSKf7OvWhRtuUO9eRORkRPoE\nbal16+DSS2HzZqhfP8WBiYhEWCxO0JY67zzo3BlmzQodiYhIdsqKZA9wyy0weXLoKEREslOVyd7M\nnjOzYjNbdYLXC8xsn5ktSyz3pT5MGDMGFi+Gv/0tHe8uIpLbqtOzfx4YWsU+851zvRPL/01BXMdp\n3Biuvx6eey4d7y4iktuqTPbOuQXA51XsdtInD6pj/Hif7I8cycTRRERyRypq9g4YYGYrzGy2mXVL\nwXtW6MIL/S0L58xJ1xFERHJTvRS8x1Kgg3PuKzO7CpgBdK1ox0mTJn27XlBQQEFBQY0PNn68P1E7\nfHitYhURibTCwkIKCwtT/r7VGmdvZp2AWc65HtXY9zOgr3NuT7nttR5nn6ykxN+IfPVq38sXEcll\nkRlnb2ZtzMwS6/3wXyB7qvizWmvWDK65Bp5/Pl1HEBHJPVX27M3sFWAwcDpQDNwP1Adwzv3OzG4H\nbgMOA18BdznnFlbwPinp2QN89BH88Id+6uO6dVPyliIikZSqnn1WTJdQkfx8uPdeGDkyZW8pIhI5\nkSnjhHL77fDUU6GjEBHJDlnbsz9wwN/FasEC6Frh2B8RkewX+559o0bwk5/A00+HjkREJPqytmcP\nfp6cPn381MdNm6b0rUVEIiH2PXvwZZyBA+Gll0JHIiISbVmd7KHsRG2GfqCIiGSlrE/23/++P1n7\n/vuhIxERia6sT/Z16vje/ZNPho5ERCS6svoEban9+6FTJ1i+3M+bIyKSK3SCNskpp8BNN8FvfhM6\nEhGRaMqJnj3Apk3Qt69/bN48bYcREcko9ezL6dTJn6zVbQtFRI6XMz17gIUL4brroKhIs2GKSG5Q\nz74C+fn+hiYzZoSOREQkWnIq2QPcdRf8+tehoxARiZacS/ZXXw07dviSjoiIeDmX7OvWhTvvVO9e\nRCRZTp2gLfXFF3D22fDXv0Lnzhk5pIhIWugEbSWaN4fbboOHHw4diYhINORkzx7gf//X38Fq9Wo/\nQkdEJBupZ1+F00+HceNUuxcRgRzu2QNs2QI9e/qLrFq1yuihRURSQj37aujQwQ/F1PTHIhJ3Od2z\nB1i/3t+68NNPoVmzjB9eROSkqGdfTeeeC4MHw+TJoSMREQkn53v2AEuXwsiRvnbfqFGQEEREakU9\n+xro08cvzzwTOhIRkTBi0bMH+OgjGDVKvXsRyS7q2ddQ375+Ue1eROIoNj178LX7H/zA9+4bNw4a\niohItahnXwt9+sB3v6vevYjET6x69gDLlsGIEerdi0h2UM++lnr3hn794L/+K3QkIiKZE7uePcDy\n5XDVVbBxo66qFZFoU8/+JPTqBQUF8PjjoSMREcmMWPbswdfs8/P93DmnnRY6GhGRiqWqZx/bZA/w\nz/8MTZrAr34VOhIRkYop2afAjh1wwQW+ht+hQ+hoRESOp2SfIv/6r7BzJzz7bOhIRESOp2SfInv3\n+nvVzp8P558fOhoRkWNpNE6KtGwJv/gF3Htv6EhERNIn9j17gK+/9r36F16AQYNCRyMiUkY9+xRq\n3BgefBDuuguOHg0djYhI6inZJ1x7LTRoAL//fehIRERST2WcJAsXwpgx/kKrpk1DRyMiojJOWuTn\n+5r9ww+HjkREJLXUsy9n82Y/M+aKFZCXFzoaEYm7jPXszew5Mys2s1WV7POEmW00sxVm1vtkgwqp\nY0e47TaYMCF0JCIiqVOdMs7zwNATvWhmw4DOzrkuwC3Ab1MUWzATJviLrN5/P3QkIiKpUWWyd84t\nAD6vZJeRwNTEvouAlmbWJjXhhdGsGTzyCNx+Oxw+HDoaEZGTl4oTtO2BLUnPtwJZX+2+5ho44wx4\n6qnQkYiInLx6KXqf8icPKjwTO2nSpG/XCwoKKCgoSNHhU88MnnwSBg6Ev/97aNs2dEQiEgeFhYUU\nFham/H2rNRrHzDoBs5xzPSp47T+BQufctMTzdcBg51xxuf2yYjROeRMmwLZtuthKRMKI0jj7mcA4\nADPLB/aWT/TZ7L77/Mna+fNDRyIiUntVlnHM7BVgMHC6mW0B7gfqAzjnfuecm21mw8ysCPgSuDmd\nAWdas2bw6KP+ZO2yZVC/fuiIRERqThdVVYNzcNVV/iblGn8vIpmkm5dk2KZNcNFF8Ne/QpcuoaMR\nkbiIUs0+Fjp18vX78eM1DbKIZB8l+xr46U/9jU6eeSZ0JCIiNaMyTg2tWgWXXeYnSmvXLnQ0IpLr\nVMYJpEcP+Kd/8qNzcuC7S0RiQsm+Fu67D9atg+nTQ0ciIlI9KuPU0gcfwOjRvpzTJqunfRORKNPQ\nywiYOBHWroXXX/dz6YiIpJpq9hEwaRJ89hlMnRo6EhGRyqlnf5JWroTvfx8+/BDOPDN0NCKSa9Sz\nj4gLL4S774abb9bFViISXUr2KfDzn8M338BvfhM6EhGRiqmMkyJFRXDxxTBvnh+LLyKSCirjREzn\nzvCrX8G118KXX4aORkTkWOrZp5BzMG4cNGoEkyeHjkZEcoF69hFkBk8/DYWF8OqroaMRESmjnn0a\nfPSRv9nJokVw1lmhoxGRbKaefYT17euvrr3uOjh0KHQ0IiLq2aeNc/CDH0DXrvDrX4eORkSylXr2\nEWcGL7wAM2aofi8i4alnn2bLlsEVV/iTtt27h45GRLKNevZZondvP/5+9GjYvz90NCISV+rZZ8ht\nt0FxMbz2mqZDFpHqU88+yzz2GGzfDg89FDoSEYmjeqEDiIuGDf1tDPPzoVs3GDkydEQiEicq42TY\n4sUwfDj85S9+emQRkcqojJOl+vXzUyGPHOlr+CIimaBkH8DYsX7CtNGj4cCB0NGISByojBPI0aN+\nOuTGjf09bDVCR0QqojJOlqtTxyf5devg/vtDRyMiuU6jcQJq0gT+/GcYMADy8uCWW0JHJCK5Ssk+\nsNatYc4cGDQI2rb1k6eJiKSaavYRsWQJDBsGs2b5sfgiIqCafc757ndhyhS4+mpYvz50NCKSa5Ts\nI2T4cHjwQbj8cti0KXQ0IpJLVLOPmJtugpISGDIE3nsP2rULHZGI5AIl+wi6446yhD9/PpxxRuiI\nRCTbKdlH1IQJPuFfeSXMmwctW4aOSESymUbjRJhz8LOf+cnT5syBU04JHZGIZJpG48SAGTz6KPTs\n6Xv4+/aFjkhEspWSfcSZwdNP+6GZQ4bA55+HjkhEspGSfRYwg8cf91fZXnYZ7N4dOiIRyTZK9lnC\nzN+4/Mor4dJLYdeu0BGJSDbRaJwsYgYPPAANGsDgwTB3LnTsGDoqEckGSvZZxgx++Uto0QIuucQn\n/PPPDx2ViESdkn2Wuvtuf7HVpZfCG29A//6hIxKRKFPNPouNGwfPPuunRZ47N3Q0IhJlVSZ7Mxtq\nZuvMbKOZ3VPB6wVmts/MliWW+9ITqlRk+HCYMcMn/hdeCB2NiERVpWUcM6sLPAkMAbYBS8xspnNu\nbbld5zvnRqYpRqnCgAHw7rswYgQUFcG//7vuaSsix6qqZ98PKHLObXLOHQKmAaMq2E+pJbBu3WDh\nQnjnHbj+ejhwIHREIhIlVSX79sCWpOdbE9uSOWCAma0ws9lm1i2VAUr1tW7tJ00Df/GVxuKLSKmq\nRuNUZ+aypUAH59xXZnYVMAPoWtGOkyZN+na9oKCAgoKC6kUp1daoEbz0Ekya5G9v+Prrfm4dEckO\nhYWFFBYWpvx9K5310szygUnOuaGJ5xOBo865hyr5m8+Avs65PeW2a9bLDJs2DX76Uz/VwvXXh45G\nRGojU7Nefgh0MbNOZtYAuBaYWS6QNmb+dKCZ9cN/gew5/q0k08aOhb/8Bf7t3/xUyYcOhY5IREKp\nNNk75w4DdwBzgY+BV51za83sVjO7NbHbGGCVmS0HHgPGpjNgqZkLL4QlS/xNzC+/HIqLQ0ckIiHo\n5iUxceSIH5I5ZQq8/LKfakFEoi9VZRwl+5j57/+Gf/gHX8ufMAHq1g0dkYhURsleam3bNvjRj3yi\nf/FFaNs2dEQiciK6LaHUWvv2/sTtoEHQpw+8+WboiEQk3dSzj7n33oMf/xhGjYIHH4SmTUNHJCLJ\n1LOXlBg0CFasgL17oXdv+OCD0BGJSDqoZy/f+tOf4Pbb4cYb/cidhg1DRyQi6tlLyo0e7Xv569fD\nRRfB4sWhIxKRVFGyl2O0bu17+BMn+jr+nXfCF1+EjkpETpaSvRzHzM+ls3q1T/Tdu8PMmVX/nYhE\nl2r2UqV334Vbb4UePeCJJ/zQTRHJDNXsJWMuvRRWrvQ9/J494YEHdHMUkWyjZC/V0qgR/PKX/qTt\n4sU+8c+YAfqxJpIdVMaRWnn7bX/yNi8PHnvM3xZRRFJPZRwJ6vLL/TDN4cNh8GAYP97PuSMi0aRk\nL7VWv77v3W/YAK1a+bnz77kHPv88dGQiUp6SvZy0U0+Fhx7yPf09e6BrV3j4Yfj669CRiUgpJXtJ\nmbw8mDwZFiyARYvgnHPg0Ufhq69CRyYiSvaScuedB6+9BrNnw/vvw9lnw3/8B5SUhI5MJL6U7CVt\nevXySf/tt+HDD31P/4EHYN++0JGJxI+SvaRdjx7w6qv+StzVq31P/+67YfPm0JGJxIeSvWRMt27w\n0kuwbJl/3quXn4Nn6dKwcYnEgZK9ZFzHjvDII/DZZ/62iKNG+SkZXn8dDh8OHZ1IbtIVtBLcoUPw\nxz/CU0/50s748fCP/wjt2oWOTCQ8XUErOaN+fV/O+Z//gT//GbZv93PvXHMNzJun+XdEUkE9e4mk\n/fvhxRfht7+Fb77xt0q84QZfAhKJk1T17JXsJdKcgyVLYMoUP6KnTx+46Sb4u7+DJk1CRyeSfkr2\nEjsHDvg7Zk2ZAgsX+nvmjh0LBQVQr17o6ETSQ8leYm37dnj5Zd/b37wZfvhDuPZauOQSqFs3dHQi\nqaNkL5LwySfwhz/4xL9rF4wZ40/uDhigxC/ZT8lepALr1/vEP3267/2PGAEjR8IVV0DTpqGjE6k5\nJXuRKmzaBLNm+Tr/okUwaJC/gGvECGjbNnR0ItWjZC9SA3v3wpw5PvHPmQMdOvje/hVXwMCB/h67\nIlGkZC9SS0eO+OGcb70Fc+fCypXwve+VJf/u3cFO+r+WSGoo2YukyN69/krdt97yS0mJL/kMHuyX\nCy6AOrrWXAJRshdJk82bYf78smX3bl/qGTzYfwn07OmneBDJBCV7kQzZvh3ee88n/gUL/Inf3r0h\nPx/69/ePeXmho5RcpWQvEsj+/b7mv3Bh2dKwYVny79vXfxmcemroSCUXKNmLRIRz8OmnfnjnwoX+\nZiwrVsDpp/ukn7y0a6eTv1IzSvYiEXb0KBQV+btyLVvmvwCWLfOJvlcvf9K3e3e/dOsGp5wSOmKJ\nKiV7kSzjHGzb5nv9a9aULWvXQqtWZcm/dDn3XGjZMnTUEpqSvUiOOHrUn/RN/gJYswY2bIDGjaFr\nV+jS5filWbPQkUsmKNmL5DjnYOdO2LjRLxs2lK1/8onv9XfpAmefDZ06wZln+sdOnfzoIE37nBuU\n7EVi7OhRXxLasMH/Kihd/vY3/1hc7Of/KU3+Z57pl7w8aN/eLy1a6GRxNlCyF5ETOngQtm499gtg\n0yb/BVG6HDlSlvjbtStbT97WurUvJUk4SvYiclK++OLY5F9+2bHD3x+gYUOf9Nu08Y/J6+UfW7bU\n1BKppmQvImnnnL+IrLjYJ/5du8rWy28rLvbzCp16qh9d1KoVnHZa5eulj6ecopLSiWQs2ZvZUOAx\noC7wjHPuoQr2eQK4CvgKuMk5t6yCfZTsRXLcoUN+Yrndu2HPnuMfK9q2ezd89RU0b+7PI7Ro4ZN/\n6Xp1njdr5m9O07hx7n1pZCTZm1ldYD0wBNgGLAGuc86tTdpnGHCHc26YmfUHHnfO5VfwXkr2CYWF\nhRQUFIQOIxLUFmXi3BaHD/tfEPv3w759vi3OOaeAffv889LtJ1ovKfHLwYPQpElZ8i99TF4/0WPp\nl0Xjxv7+Bidaz/QkeKlK9lUNzuoHFDnnNiUOOg0YBaxN2mckMBXAObfIzFqaWRvnXPHJBper4vyf\nujy1RZk4t0W9emUlHoDXXy9kxIiCGr/PkSPw5Zd+KSk59rGibcXFfqqL0udff+2XAweOfUxeh6q/\nEErXGzb0S4MGtXtM5UV1VSX79sCWpOdbgf7V2CcPULIXkYyqW9eXeNI5/cThw8d/AZzoC+Kbb/yv\njW++OXa9pOTEryU/du6curirSvbVrbuU/4mheo2I5KR69fz5hebNM3O8qVNT8z5V1ezzgUnOuaGJ\n5xOBo8knac3sP4FC59y0xPN1wODyZRwz0xeAiEgtZKJm/yHQxcw6AduBa4Hryu0zE7gDmJb4cthb\nUb0+FcGKiEjtVJrsnXOHzewOYC5+6OWzzrm1ZnZr4vXfOedmm9kwMysCvgRuTnvUIiJSIxm7qEpE\nRMJJ+4XNZjbUzNaZ2UYzuyfdxwvNzDqY2btmtsbMVpvZvyS2tzKzt81sg5m9ZWYtk/5mYqJ91pnZ\nFeGiTw8zq2tmy8xsVuJ5LNsiMSx5upmtNbOPzax/jNtiYuL/yCoze9nMGsalLczsOTMrNrNVSdtq\n/NnNrG+i/Taa2eNVHtg5l7YFX/opAjoB9YHlwPnpPGboBfgO0Cux3gx/Udr5wMPALxLb7wEeTKx3\nS7RL/UQ7FQF1Qn+OFLfJXcBLwMzE81i2Bf56lJ8k1usBLeLYFonP8ynQMPH8VeDGuLQFMBDoDaxK\n2laTz15akVkM9EuszwaGVnbcdPfsv70oyzl3CCi9KCtnOed2OueWJ9ZL8BegtSfp4rPE49WJ9VHA\nK865Q85fvFaEb7ecYGZ5wDDgGcqG6MauLcysBTDQOfcc+PNhzrl9xLAtgP3AIaCJmdUDmuAHgMSi\nLZxzC4DPy22uyWfvb2ZtgebOucWJ/V5I+psKpTvZV3TBVfs0HzMyEqOYegOLgOSriouBNon1dvh2\nKZVrbfQo8HPgaNK2OLbFWcD/M7PnzWypmU02s6bEsC2cc3uAR4DN+CS/1zn3NjFsiyQ1/ezlt2+j\nijZJd7KP7dlfM2sGvAbc6Zz7Ivk15393VdY2OdFuZjYC2OX8xHgVDr2NS1vgyzZ9gKedc33wI9cm\nJO8Ql7Yws3OA/4MvS7QDmpnZj5P3iUtbVKQan71W0p3stwEdkp534Nhvo5xkZvXxif73zrkZic3F\nZvadxOttgV2J7eXbKC+xLRcMAEaa2WfAK8BlZvZ74tkWW4GtzrkliefT8cl/Zwzb4iLgA+fcbufc\nYeBPwMXEsy1K1eT/xNbE9rxy2yttk3Qn+28vyjKzBviLsmam+ZhBmZkBzwIfO+ceS3ppJv4kFInH\nGUnbx5pZAzM7C+iCP/GS9Zxz9zrnOjjnzgLGAvOcczcQz7bYCWwxs66JTUOANcAsYtYWwDog38wa\nJ/6/DAE+Jp5tUapG/ycS/572J0Z0GXBD0t9ULANnnq/Cj0gpAiaGPhOegc97Cb4+vRxYlliGAq2A\nd4ANwFtAy6S/uTfRPuuAK0N/hjS1y2DKRuPEsi2Anvhpwlfge7MtYtwWv8B/2a3Cn5CsH5e2wP/K\n3Q4cxJ/TvLk2nx3om2i/IuCJqo6ri6pERGJAd4sUEYkBJXsRkRhQshcRiQElexGRGFCyFxGJASV7\nEZEYULIXEYkBJXsRkRj4/x5DQxFUB5EaAAAAAElFTkSuQmCC\n", 339 | "text/plain": [ 340 | "" 341 | ] 342 | }, 343 | "metadata": {}, 344 | "output_type": "display_data" 345 | } 346 | ], 347 | "source": [ 348 | "ns = np.arange(1, 1000)\n", 349 | "hs = []\n", 350 | "for N in ns:\n", 351 | " hs.append(hoeffding(0.05, N))\n", 352 | "plt.plot(ns, hs);" 353 | ] 354 | } 355 | ], 356 | "metadata": { 357 | "kernelspec": { 358 | "display_name": "Python 3", 359 | "language": "python", 360 | "name": "python3" 361 | }, 362 | "language_info": { 363 | "codemirror_mode": { 364 | "name": "ipython", 365 | "version": 3 366 | }, 367 | "file_extension": ".py", 368 | "mimetype": "text/x-python", 369 | "name": "python", 370 | "nbconvert_exporter": "python", 371 | "pygments_lexer": "ipython3", 372 | "version": "3.4.3" 373 | } 374 | }, 375 | "nbformat": 4, 376 | "nbformat_minor": 0 377 | } 378 | -------------------------------------------------------------------------------- /Metropolis_Hastings.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%matplotlib inline\n", 12 | "import math\n", 13 | "import random\n", 14 | "import matplotlib.pyplot as plt\n", 15 | "\n", 16 | "\n", 17 | "def std_normal(x):\n", 18 | " return (1/math.sqrt(math.pi * 2)) * math.exp(-x**2 / 2)\n", 19 | "\n", 20 | "def next_state(x):\n", 21 | " if random.random() <= .5:\n", 22 | " return x + .1\n", 23 | " else:\n", 24 | " return x - .1\n", 25 | "\n", 26 | "def metropolis_hastings(distribution, state, next_state): \n", 27 | " possible_state = next_state(state)\n", 28 | " # Hastings ratio\n", 29 | " transition_odds = distribution(possible_state) / distribution(state)\n", 30 | " \n", 31 | " # Metropolis rejection step\n", 32 | " if random.random() <= transition_odds:\n", 33 | " return possible_state\n", 34 | " else:\n", 35 | " return state\n", 36 | " " 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 2, 42 | "metadata": { 43 | "collapsed": true 44 | }, 45 | "outputs": [], 46 | "source": [ 47 | "state = .1\n", 48 | "xs = []\n", 49 | "for _ in range(1000000):\n", 50 | " state = metropolis_hastings(distribution=std_normal, state=state, next_state=next_state)\n", 51 | " xs.append(state)\n", 52 | " " 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": 3, 58 | "metadata": { 59 | "collapsed": false, 60 | "scrolled": true 61 | }, 62 | "outputs": [ 63 | { 64 | "data": { 65 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEACAYAAAC+gnFaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGj5JREFUeJzt3X+sXGed3/H3J7/dkk1kgYzjmMTS3mhjFjbZbDFaEEyU\nJTJoFTtSSpzVpm7XrRCmgNCK1ga1uYC6JK0gZFvF/2CIky1urE03P5bEiTGZXaTKsVg5weTixpEw\n9TWx0xo2AVFau/n0j3mu78n1+M7cuT9m5p7PSxrdZ75znjvPuXdmvuf5cc7INhERUV/n9bsBERHR\nX0kEERE1l0QQEVFzSQQRETWXRBARUXNJBBERNddVIpB0vqQDkp4o95dK2iPpJUnPSLq8su1WSYcl\nHZJ0cyV+g6SD5bH7KvGLJT1c4vskXTWXOxgREdPrtkfwaWAMmDjpYAuwx/Y1wN5yH0mrgduB1cBa\n4H5JKnW2AZtsjwAjktaW+CbgZInfC9wzu12KiIiZ6JgIJF0JfAT4OjDxoX4LsKOUdwDrS3kdsNP2\nKdtHgJeBNZKWA5fa3l+2e7BSp/q7HgFu6nlvIiJixrrpEdwLfBZ4oxJbZvtEKZ8AlpXyFcB4Zbtx\nYEWb+LESp/w8CmD7NPCapKUz2IeIiJiFaROBpD8EXrV9gMnewJu4dY2KXKciImJIXdDh8d8HbpH0\nEeAS4DckPQSckPR228fLsM+rZftjwMpK/Stp9QSOlfLU+ESddwA/lXQBcJntn01tiKQkm4iIHthu\neyA/Ydoege3P2V5pexWwAfiu7TuBx4GNZbONwKOl/DiwQdJFklYBI8B+28eB1yWtKZPHdwKPVepM\n/K7baE0+n6s9A3W76667+t6GtGlxtSttSpvm+taNTj2Csz6Ly8+7gV2SNgFHgI+WD+oxSbtorTA6\nDWz2ZEs2Aw8AS4Anbe8u8e3AQ5IOAydpJZyIiFggXScC238D/E0p/wz4g3Ns92fAn7WJ/x3wrjbx\n/0NJJBERsfByZvEsNBqNfjfhLGlT9waxXWlTd9KmuaVux5D6TZKHpa0RvZg89/Jsee1HryTh2UwW\nR8RCq67GzsrsWBhJBBERNTfTVUMR0YOpwz4Z6olBkkQQsWAmPvynHa5tq5pIkkRirmVoKGIoZL4g\n5k96BBHzZLpVQBGDJD2CiHmVI/kYfEkEERE1l0QQEVFzSQQRETWXRBARUXNZNRQxh7JSKIZREkHE\nnOv9xLGZyNnKMVeSCCKG2mTSydnH0avMEUQsGjlnIXqTRBARUXNJBBERNTdtIpB0iaTnJD0vaUzS\nl0t8VNK4pAPl9uFKna2SDks6JOnmSvwGSQfLY/dV4hdLerjE90m6aj52NCIi2ps2Edj+NXCj7euA\ndwM3Sno/rYHIr9q+vtyeApC0GrgdWA2sBe7X5AzWNmCT7RFgRNLaEt8EnCzxe4F75nYXIyJiOh2H\nhmz/qhQvAs4Hfl7ut1sbtw7YafuU7SPAy8AaScuBS23vL9s9CKwv5VuAHaX8CHDTTHciIiJ61zER\nSDpP0vPACeBZ2y+Whz4p6QVJ2yVdXmJXAOOV6uPAijbxYyVO+XkUwPZp4DVJS3vdoYiImJluegRv\nlKGhK4EPSGrQGuZZBVwHvAJ8ZT4bGREzI+nMLaKTrk8os/2apG8Dv2e7ORGX9HXgiXL3GLCyUu1K\nWj2BY6U8NT5R5x3ATyVdAFxm+2ft2jA6Onqm3Gg0aDQa3TY/omYW5uzmGDzNZpNmszmjOpruDERJ\nbwVO2/57SUuAp4EvAC/aPl62+Qzwj2z/UZks/hbwHlpDPt8BftO2JT0HfArYD3wb+HPbuyVtBt5l\n++OSNgDrbW9o0xbnbMkYdK0j8OqHcPtyu9fyTOvO5rmiPiRhe9ojgk49guXADknn0RpGesj2XkkP\nSrqO1qvtx8DHAGyPSdoFjAGngc2VT+/NwAPAEuBJ27tLfDvwkKTDwEngrCQQERHzZ9oewSBJjyCG\nQXoEMWi66RHkzOKIiJpLIoiIqLlchjqiS7n+fyxW6RFEzEgu9RyLTxJBRETNZWgoYpYG/ezdfHNZ\ndJJEEDEnejuTd2GSSM4yjullaCii7zLvEP2VRBARUXNJBBERNZdEEBFRc0kEERE1l1VDEdMY9KWh\nEXMhPYKIjrKqJxa3JIKIiJpLIoiIqLkkgoiImksiiIiouSSCiIiamzYRSLpE0nOSnpc0JunLJb5U\n0h5JL0l6RtLllTpbJR2WdEjSzZX4DZIOlsfuq8QvlvRwie+TdNV87GhERLQ3bSKw/WvgRtvXAe8G\nbpT0fmALsMf2NcDech9Jq4HbgdXAWuB+TS7E3gZssj0CjEhaW+KbgJMlfi9wz1zuYERETK/j0JDt\nX5XiRcD5wM+BW4AdJb4DWF/K64Cdtk/ZPgK8DKyRtBy41Pb+st2DlTrV3/UIcFPPexMRETPWMRFI\nOk/S88AJ4FnbLwLLbJ8om5wAlpXyFcB4pfo4sKJN/FiJU34eBbB9GnhN0tLediciImaq4yUmbL8B\nXCfpMuBpSTdOedySFuS0y9HR0TPlRqNBo9FYiKeNiBgazWaTZrM5ozqayVfXSfo3wP8G/jnQsH28\nDPs8a/u3JG0BsH132X43cBfwk7LNtSV+B/AB2x8v24za3ifpAuAV229r89zO1+zFQmtNcVW/4Wuy\nPPF6nG6bQa0b9SEJ29NeNKvTqqG3TqwIkrQE+BBwAHgc2Fg22wg8WsqPAxskXSRpFTAC7Ld9HHhd\n0poyeXwn8FilzsTvuo3W5HNE30g6c4uog05DQ8uBHZLOo5U0HrK9V9IBYJekTcAR4KMAtsck7QLG\ngNPA5sph/GbgAWAJ8KTt3SW+HXhI0mHgJLBhrnYuonf5nt+ojxkNDfVThoZioXQ75NLv4Z0MDUU3\nZj00FBERi1++mCai5qbOhaTXUD/pEUQE+fKdeksiiIiouSSCiIiaSyKIiKi5JIKIiJrLqqGIGspZ\n01GVHkFEbWWlULQkEURE1FwSQUREzSURRETUXCaLI8jkadRbegQRZ2TyNOopiSAiouYyNBQRb1Id\nJsuVSOshPYKImCJDZHWTRBARUXNJBBERNdcxEUhaKelZSS9K+qGkT5X4qKRxSQfK7cOVOlslHZZ0\nSNLNlfgNkg6Wx+6rxC+W9HCJ75N01VzvaEREtNdNj+AU8Bnb7wTeC3xC0rW0BhG/avv6cnsKQNJq\n4HZgNbAWuF+Ts0/bgE22R4ARSWtLfBNwssTvBe6Zo/2LiIgOOiYC28dtP1/KvwR+BKwoD7c7C2cd\nsNP2KdtHgJeBNZKWA5fa3l+2exBYX8q3ADtK+RHgph72JSIiejCjOQJJVwPXA/tK6JOSXpC0XdLl\nJXYFMF6pNk4rcUyNH2MyoawAjgLYPg28JmnpTNoWERG96fo8AklvAf4S+LTtX0raBnyxPPwl4Cu0\nhnjmzejo6Jlyo9Gg0WjM59NFRAydZrNJs9mcUR11c8KIpAuBvwaesv21No9fDTxh+12StgDYvrs8\nthu4C/gJ8Kzta0v8DuADtj9ethm1vU/SBcArtt825Tmck1titqZeU2jiNdWKT7y+uivXoW7ec8NP\nEranvZhWN6uGBGwHxqpJoIz5T7gVOFjKjwMbJF0kaRUwAuy3fRx4XdKa8jvvBB6r1NlYyrcBezvu\nXUTPcsJURFU3Q0PvA/4Y+IGkAyX2OeAOSdfRekf9GPgYgO0xSbuAMeA0sLlyKL8ZeABYAjxpe3eJ\nbwceknQYOAlsmO2ORUREd7oaGhoEGRqKuTB1SGRYhmgyNBS96mZoKBedi4hzygXo6iGXmIiIaWQ+\npQ6SCCIiai6JICKi5pIIIiJqLokgIqLmkggiImouiSAiouaSCCIiai6JICKi5nJmcSx6U684GhFv\nlh5B1ETOkI04lySCiIiaSyKIiKi5JIKIiJpLIoiIqLkkgoiImksiiIiouSSCiIia65gIJK2U9Kyk\nFyX9UNKnSnyppD2SXpL0jKTLK3W2Sjos6ZCkmyvxGyQdLI/dV4lfLOnhEt8n6aq53tGIiGivmx7B\nKeAztt8JvBf4hKRrgS3AHtvXAHvLfSStBm4HVgNrgfs1eWrnNmCT7RFgRNLaEt8EnCzxe4F75mTv\nIiKio46JwPZx28+X8i+BHwErgFuAHWWzHcD6Ul4H7LR9yvYR4GVgjaTlwKW295ftHqzUqf6uR4Cb\nZrNTERHRvRnNEUi6GrgeeA5YZvtEeegEsKyUrwDGK9XGaSWOqfFjJU75eRTA9mngNUlLZ9K2iIjo\nTdcXnZP0FlpH65+2/YvqhbxsW9K8X8hldHT0TLnRaNBoNOb7KWOI5WJzUUfNZpNmszmjOrI7f35L\nuhD4a+Ap218rsUNAw/bxMuzzrO3fkrQFwPbdZbvdwF3AT8o215b4HcAHbH+8bDNqe5+kC4BXbL9t\nShvcTVsjJrQSgYGJn5xVnnhNTW579jap++a6MVwkYXvao6JuVg0J2A6MTSSB4nFgYylvBB6txDdI\nukjSKmAE2G/7OPC6pDXld94JPNbmd91Ga/I5IiIWQMcegaT3A38L/IDJw4OtwH5gF/AO4AjwUdt/\nX+p8DvgT4DStoaSnS/wG4AFgCfCk7YmlqBcDD9GafzgJbCgTzdV2pEcQM5IewfzUjeHSTY+gq6Gh\nQZBEEDOVRDA/daumzsPkPTp45mRoKCJievnSn2GXRBARUXNJBBERNZdEEBFRc0kEERE11/WZxRER\nkDO2F6P0CCKiB1kptJgkEURE1FwSQUREzSURRETUXBJBRETNZdVQLCpZ0dJfU76npI8tiZlIjyAW\noaxo6Z/87YdREkFERM0lEURE1FwSQUREzSURRETUXBJBRETNJRFERNRcx0Qg6RuSTkg6WImNShqX\ndKDcPlx5bKukw5IOSbq5Er9B0sHy2H2V+MWSHi7xfZKumssdjIiI6XXTI/gmsHZKzMBXbV9fbk8B\nSFoN3A6sLnXu1+QZJtuATbZHgBFJE79zE3CyxO8F7pnVHkUtSHrTLSJ61zER2P4e8PM2D7V7960D\ndto+ZfsI8DKwRtJy4FLb+8t2DwLrS/kWYEcpPwLc1H3zo95y8lLEXJjNHMEnJb0gabuky0vsCmC8\nss04sKJN/FiJU34eBbB9GnhN0tJZtCsiImag12sNbQO+WMpfAr5Ca4hnXo2Ojp4pNxoNGo3GfD9l\nRMRQaTabNJvNGdVRNxeGknQ18ITtd033mKQtALbvLo/tBu4CfgI8a/vaEr8D+IDtj5dtRm3vk3QB\n8Irtt7V5HuciVjGhNS8w8XrQmQucTY23yu1i3dSbvpy6nepOynu3fyRhe9qJtJ6GhsqY/4RbgYkV\nRY8DGyRdJGkVMALst30ceF3SmjJ5fCfwWKXOxlK+DdjbS5siYtBkDmdYdBwakrQT+CDwVklHaR3h\nNyRdR+u//GPgYwC2xyTtAsaA08DmymH8ZuABYAnwpO3dJb4deEjSYeAksGGO9i0iIrrQ1dDQIMjQ\nUFRlaGg468bCm7ehoYiIWDySCCIiai6JICKi5pIIIiJqLl9eH0Mj1xSKmB/pEcSQydr0iLmWRBAR\nUXNJBBERNZdEEBFRc0kEERE1l0QQEVFzSQQRETWXRBARUXNJBBERNZczi2Og5WziiPmXHkEMgZxN\nHDGfkggiImouQ0MRsWCmDvXlm8sGQxJBRCywya+wrCaGJIX+6Tg0JOkbkk5IOliJLZW0R9JLkp6R\ndHnlsa2SDks6JOnmSvwGSQfLY/dV4hdLerjE90m6ai53MCIGWeZ/BkE3cwTfBNZOiW0B9ti+Bthb\n7iNpNXA7sLrUuV+TKX8bsMn2CDAiaeJ3bgJOlvi9wD2z2J+IiJihjonA9veAn08J3wLsKOUdwPpS\nXgfstH3K9hHgZWCNpOXApbb3l+0erNSp/q5HgJt62I+IiOhRr6uGltk+UcongGWlfAUwXtluHFjR\nJn6sxCk/jwLYPg28Jmlpj+2KiIgZmvVksW1LWpBBvtHR0TPlRqNBo9FYiKeNiBgazWaTZrM5ozq9\nJoITkt5u+3gZ9nm1xI8BKyvbXUmrJ3CslKfGJ+q8A/ippAuAy2z/rN2TVhNBREScbepB8he+8IWO\ndXodGnoc2FjKG4FHK/ENki6StAoYAfbbPg68LmlNmTy+E3isze+6jdbkc9SYpDO3iJh/HXsEknYC\nHwTeKuko8G+Bu4FdkjYBR4CPAtgek7QLGANOA5s9uTh4M/AAsAR40vbuEt8OPCTpMHAS2DA3uxbD\nbXKteUTMLw3LSRySPCxtjdlp9QSqiaBzeeK10b5ur/Vm85ypO9O6eX/PD0nYnvaIKtcaioiouVxi\nIiIGQi430T/pEUTEgMjlJvoliSAiouaSCCIiai5zBNE3uTZ9xGBIjyD6LOPCEf2WRBARUXNJBBER\nNZdEEBFRc5ksjoiBk5PLFlZ6BBExgLKIYCElEURE1FyGhmJB5TsGIgZPegTRB+n2RwySJIKIiJpL\nIoiIqLnMEUTEUMi1qebPrHoEko5I+oGkA5L2l9hSSXskvSTpGUmXV7bfKumwpEOSbq7Eb5B0sDx2\n32zaFBGLWeaX5sNsh4YMNGxfb/s9JbYF2GP7GmBvuY+k1cDtwGpgLXC/JlP8NmCT7RFgRNLaWbYr\nBoikM7doyd8jBslczBFMfTXfAuwo5R3A+lJeB+y0fcr2EeBlYI2k5cCltveX7R6s1IlFI0dyb5a/\nRwyOuegRfEfS9yX9ixJbZvtEKZ8AlpXyFcB4pe44sKJN/FiJR0TEApjtZPH7bL8i6W3AHkmHqg/a\ntqQc9kREDLBZJQLbr5Sf/1PSXwHvAU5Iervt42XY59Wy+TFgZaX6lbR6AsdKuRo/1u75RkdHz5Qb\njQaNRmM2zY+IWHSazSbNZnNGddTrEixJ/wA43/YvJP1D4BngC8AfACdt3yNpC3C57S1lsvhbtJLF\nCuA7wG+WXsNzwKeA/cC3gT+3vXvK8znLxYZTa1J04n937vLE/7fb7buv24/nnL9yv9o8eHUn5bPh\n3CRhe9qVCbPpESwD/qqsfLgA+M+2n5H0fWCXpE3AEeCjALbHJO0CxoDTwObKJ/tm4AFgCfDk1CQQ\nEXG29kkhZq7nHsFCS49guJy9NDI9gvQI5q9uPhvOrZseQS4xEfMoSyQjhkESQUREzSURRMTQy5na\ns5OLzsWs5EJgMRgm5oKiF+kRxBzIXEDEMEuPIKJGMnwS7SQRxIzlw2SY9b72Pv/3xStDQ9GjDAfV\nT/7ni1V6BBGxaFR7LVm40L0kguhKhgViOOSyE73I0FDMQIYGIhaj9AgiYlHLuS6dJRHEOWU4KBaP\nDBlNJ4kgOsgb6FySKIdTJpTPlkQQ0bMkyW4NVtLM/22qJILIGGosgMH88E3voCWrhqLIiqCoo7zu\nIT2C2hqsrnpE/9W5d5BEUCPTf31kRN2d/X6oy7DpwAwNSVor6ZCkw5L+db/bs1hMfGHH5As6XeGI\nmVn875mBSASSzgf+E7AWWA3cIena/raqs2az2e8mnFH9wG9/5L+4X8iL5xuqmh23WPh9bS7gcw22\nc7/HBuvzYKYGIhEA7wFetn3E9ingvwDr+tymjvr9j29/tH9XP5vUR4sl2TW72Gah97W5gM816Cb/\n9lMPvm688cahPRgZlESwAjhauT9eYjFFhnoiBkX1vXf2QdjURDFdb6LfBiURDPwn2SuvvMKFF154\n5h+5fv2tPf+u6V4g7V4sC/3hP5sXa7/qRgymqcmifW9iuvf8QrwnNAiz4JLeC4zaXlvubwXesH1P\nZZv+NzQiYgjZnjabDEoiuAD478BNwE+B/cAdtn/U14ZFRNTAQJxHYPu0pH8JPA2cD2xPEoiIWBgD\n0SOIiIj+GZTJ4q5IGpU0LulAua3td5smSPpTSW9IWjoAbfmSpBckPS9pr6SVA9Cm/yDpR6Vd/1XS\nZQPQpn8s6UVJ/0/S7/a5LQN3QqWkb0g6Ielgv9syQdJKSc+W/9sPJX1qANp0iaTnyvttTNKX+92m\nCZLOL5+VT0y33VAlAlpT7l+1fX257e53g6D14gQ+BPyk320p/r3t37F9HfAog3FywTPAO23/DvAS\nsLXP7QE4CNwK/G0/GzHAJ1R+k1abBskp4DO23wm8F/hEv/9Wtn8N3Fjeb+8GbpT0/n62qeLTwBgd\nlhkOWyKAwbwwzleBf9XvRkyw/YvK3bcA/6tfbZlge4/tN8rd54Ar+9keANuHbL/U73YwoCdU2v4e\n8PN+t6PK9nHbz5fyL4EfAVf0t1Vg+1eleBGtec6f9bE5AEi6EvgI8HU6fG4OYyL4ZBle2C7p8n43\nRtI6YNz2D/rdlipJ/07S/wA2Anf3uz1T/AnwZL8bMUByQmUPJF0NXE/rwKKvJJ0n6XngBPCs7bF+\ntwm4F/gs8EanDQdi1VCVpD3A29s89HlgG/DFcv9LwFeATX1u01bg5urm892eDm36nO0nbH8e+Lyk\nLbReEP+s320q23we+L+2vzXf7em2TQMgKzZmSNJbgL8EPl16Bn1VervXlbmvpyU1bDf71R5Jfwi8\navuApEan7QcuEdj+UDfbSfo6sCBv5HO1SdJvA6uAF8rZf1cCfyfpPbZf7Ueb2vgWC3T03alNkv4p\nra7qTQvRHpjR36mfjgHVCf2VtHoF0YakC4FHgL+w/Wi/21Nl+zVJ3wZ+j/5epOn3gVskfQS4BPgN\nSQ/a/iftNh6qoSFJyyt3b6U12dc3tn9oe5ntVbZX0Xrz/u58J4FOJI1U7q4DDvSrLRPKCq/PAuvK\n5Nqg6efc0/eBEUlXS7oIuB14vI/tGVhqHXFtB8Zsf63f7QGQ9NaJYWpJS2gtHOnre87252yvLJ9L\nG4DvnisJwJAlAuAeST+Q9ALwQeAz/W7QFIPSxf+ypINlzLIB/Gmf2wPwH2lNXO8py9nu73eDJN0q\n6Sit1SfflvRUP9ph+zQwcULlGPDwIJxQKWkn8N+AayQdlTTvw4tdeB/wx7RW5gzKMvLlwHfL++05\n4Anbe/vcpqmm/WzKCWURETU3bD2CiIiYY0kEERE1l0QQEVFzSQQRETWXRBARUXNJBBERNZdEEBFR\nc0kEERE19/8B70hTCeluwM4AAAAASUVORK5CYII=\n", 66 | "text/plain": [ 67 | "" 68 | ] 69 | }, 70 | "metadata": {}, 71 | "output_type": "display_data" 72 | } 73 | ], 74 | "source": [ 75 | "plt.hist(xs, normed=False, bins=100)\n", 76 | "plt.show()" 77 | ] 78 | } 79 | ], 80 | "metadata": { 81 | "kernelspec": { 82 | "display_name": "Python 3", 83 | "language": "python", 84 | "name": "python3" 85 | }, 86 | "language_info": { 87 | "codemirror_mode": { 88 | "name": "ipython", 89 | "version": 3 90 | }, 91 | "file_extension": ".py", 92 | "mimetype": "text/x-python", 93 | "name": "python", 94 | "nbconvert_exporter": "python", 95 | "pygments_lexer": "ipython3", 96 | "version": "3.4.3" 97 | } 98 | }, 99 | "nbformat": 4, 100 | "nbformat_minor": 0 101 | } 102 | -------------------------------------------------------------------------------- /Mixture_of_Gaussians.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [ 10 | { 11 | "data": { 12 | "text/html": [ 13 | "\n", 220 | "\n" 239 | ], 240 | "text/plain": [ 241 | "" 242 | ] 243 | }, 244 | "execution_count": 1, 245 | "metadata": {}, 246 | "output_type": "execute_result" 247 | } 248 | ], 249 | "source": [ 250 | "# style the notebook\n", 251 | "from IPython.core.display import HTML\n", 252 | "import urllib.request\n", 253 | "response = urllib.request.urlopen('http://bit.ly/1LC7EI7')\n", 254 | "HTML(response.read().decode(\"utf-8\"))" 255 | ] 256 | }, 257 | { 258 | "cell_type": "markdown", 259 | "metadata": {}, 260 | "source": [ 261 | "# Mixture of Gaussians\n", 262 | "\n", 263 | "This is a simple concept - we create a mixture of Gaussians as a weighted sum of a set of Gaussians:\n", 264 | "\n", 265 | "$$p(x) = \\sum\\limits_{k=1}^N w_k \\mathcal{N}(\\mu_k, \\sigma_k^2)$$\n", 266 | "\n", 267 | "with the constraint that the weights are all positive and sum to one: $\\sum\\limits_{k=1}^N w_k=1$.\n", 268 | "\n", 269 | "Here is some code to visualize this for 1D Gaussians" 270 | ] 271 | }, 272 | { 273 | "cell_type": "code", 274 | "execution_count": 45, 275 | "metadata": { 276 | "collapsed": false 277 | }, 278 | "outputs": [ 279 | { 280 | "data": { 281 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAEACAYAAABS29YJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xm4nfO5//H3LTHPc4iYRUYijshAbRI9oUhPe1QVP0MN\nbcWhNbdFXFSP9tAKVWNPqWhOzSlBYtitKiESiSEjURmIiIogken+/XGvXdu2917Ts9azhs/runIl\na61nuMnOZ3/39/kO5u6IiEhtWyvtAkREpPQU9iIidUBhLyJSBxT2IiJ1QGEvIlIHFPYiInUga9ib\n2VAzm25ms8zswlY+P87MppjZVDN71sz2avbZW5n3J5vZC0kXLyIiubH2xtmbWQdgBjAEmA+8CBzr\n7tOaHTMAeN3dl5jZUGCEu/fPfDYH2NfdPyjhf4OIiGSRrWXfD5jt7m+5+0pgNDCs+QHu/py7L8m8\nnADs0OIalkilIiJSsGxh3xmY2+z1vMx7bfkuMLbZaweeMLOJZnZaYSWKiEixOmb5POe1FMzsYOAU\nYFCztwe5+ztmtjUw3symu/szBdQpIiJFyBb284EuzV53IVr3X5B5KHsrMNTd/9n0vru/k/l9kZk9\nQHQLPdPiXC3OIyJSAHfPuZs8WzfORGAPM9vZzNYBjgHGND/AzHYE7geOd/fZzd7fwMw2zvx5Q+Cr\nwCttFFzxvy677LLUa1CdqrNaa1Sdyf/KV7ste3dfZWbDgceBDsDt7j7NzM7IfH4zcCmwOfBbMwNY\n6e79gE7A/Zn3OgKj3H1c3hWKiEjRsnXj4O6PAo+2eO/mZn8+FTi1lfPeBPokUKOIiBRJM2hz1NDQ\nkHYJOVGdyaqGOquhRlCdaWt3UlVZCjDztGsQEak2ZoYn+IBWRERqgMJeRKQOKOxFROqAwl5EpA4o\n7EVE6oDCXkSkDijsRUTqgMJeRKQOKOxFROqAwl5EpA4o7EVE6oDCXkSkDijsRUTqgMJeRKQOKOxF\nROqAwl5EpA4o7EVE6oDCXkSkDijsRUTqgMJeRKQOKOxFROqAwl5EpA4o7EVE6oDCXkSkDijsRUTq\ngMJeRKQOKOxFROqAwl5EpA4o7EVE6oDCXkSkDijsRUTqgMJeRKQOKOxFROqAwl5EpA5kDXszG2pm\n081slpld2Mrnx5nZFDObambPmtleuZ4rIiLlYe7e9odmHYAZwBBgPvAicKy7T2t2zADgdXdfYmZD\ngRHu3j+XczPne3s1iIjIl5kZ7m65Hp+tZd8PmO3ub7n7SmA0MKz5Ae7+nLsvybycAOyQ67kiIlIe\n2cK+MzC32et5mffa8l1gbIHnikgNuukm2G472GYb+PWvQT/Ip6Njls9z/msxs4OBU4BB+Z47YsSI\nf/25oaGBhoaGXE8VkQp2yy0R8OPHw9prw9FHw8qVcP75aVdWfRobG2lsbCz4/Gx99v2JPvihmdcX\nA2vc/eoWx+0F3A8MdffZeZ6rPnuRGjR7NgwYAH/9K3TvHu/NnQv77APPPw+7755ufdUu6T77icAe\nZrazma0DHAOMaXHDHYmgP74p6HM9V0Rq18UXw3nnfR70AF26wLnnwgUXpFdXvWq3ZQ9gZocBvwY6\nALe7+8/N7AwAd7/ZzG4D/gN4O3PKSnfv19a5rVxfLXuRGjNzJhxwALz5Jmy00Rc/W7YMdtstunZ6\n9kynvlqQb8s+a9iXmsJepPaccUY8lG32OO4LLr0UPvwQRo4sa1k1RWEvIqn69FPo3Bleew223771\nY95+O/ru334bNtywvPXViqT77EVE8jJmDOy3X9tBD7DjjtCvHzz8cPnqqncKexFJ1F13wQknZD/u\nm9+E++8vfT0S1I0jIon56CPYYQeYPx823rj9Y997D7p2hXffhfXWK099tUTdOCKSmnHjYODA7EEP\nMaO2T58YlSOlp7AXkcQ8/DAccUTux3/ta/D446WrRz6nsBeRRKxeDWPHwpFH5n7OkCFq2ZeLwl5E\nEjF1KmyxBey0U+7n7L03fPBBDMGU0lLYi0giGhvh4IPzO2ettWDwYHjyyZKUJM0o7EUkEY2NUMiC\ntYMHw1NPJV2NtKSwF5GirV4NzzwDBx2U/7kDB8JzzyVfk3yRwl5EijZlCnTqFL/y1b07LF4c4+6l\ndBT2IlK0QrtwIPrt+/dX677UFPYiUrRiwh5ik5O//z2paqQ1CnsRKYo7PPtsrF9fqIEDFfalprAX\nkaLMnh3LFLe3ymU2/frB5MmwYkVydckXKexFpCgTJkSfezE22SR2r3r55WRqki9T2ItIUSZMgP33\nL/46/fvHtaQ0FPYiUpTnn08m7Pv2ja4cKQ2FvYgUbPlyeP31COpi7bMPTJpU/HWkdQp7ESnY5MnQ\nrRtssEHx1+rdG2bOhM8+K/5a8mUKexEpWFJdOADrrw+77w6vvprM9eSLFPYiUrCJE2Nz8aSoK6d0\nFPYiUrCXX46ATkrfvgr7UlHYi0hBli2DN9+EHj2Su6ZG5JSOwl5ECvLaa7DnnrDOOsldc++94ZVX\nYNWq5K4pQWEvIgV5+eUI5yRtskksuzBjRrLXFYW9iBTo5ZehT5/kr9u7d/zUIMlS2ItIQUoV9r16\nafhlKSjsRSRva9bA1KnJd+OAwr5UFPYikrc5c2DzzWGLLZK/dq9e8ZBWkqWwF5G8leLhbJM99oB5\n8+DTT0tz/XqlsBeRvJWqvx5g7bWha1eYNq00169XCnsRyduUKaULe1C/fSko7EUkb6Vs2YPCvhSy\nhr2ZDTWz6WY2y8wubOXzbmb2nJktN7NzW3z2lplNNbPJZvZCkoWLSDoWL4YPP4Sddy7dPRT2yevY\n3odm1gG4ARgCzAdeNLMx7t68N20xcBbw9VYu4UCDu3+QUL0ikrIpU+Lh7Fol7BdQ2Ccv219XP2C2\nu7/l7iuB0cCw5ge4+yJ3nwisbOMaVnyZIlIpSt1fD7DTTvDPf8YvSUa2sO8MzG32el7mvVw58ISZ\nTTSz0/ItTkQqT6n76yF+aujZU8smJClb2HuR1x/k7vsAhwFnmtmBRV5PRFJWyjH2zfXurclVSWq3\nz57op+/S7HUXonWfE3d/J/P7IjN7gOgWeqblcSNGjPjXnxsaGmhoaMj1FiJSRp99FvvE9uxZ+nv1\n6KGx9s01NjbS2NhY8Pnm3nbj3cw6AjOAwcAC4AXg2BYPaJuOHQEsdfdrMq83ADq4+1Iz2xAYB1zu\n7uNanOft1SAilePll+H448vz8PTRR+Haa2H8+NLfqxqZGe6e8zPRdlv27r7KzIYDjwMdgNvdfZqZ\nnZH5/GYz6wS8CGwCrDGzs4EewDbA/WbWdJ9RLYNeRKpLOfrrm3TrBtOnl+de9aDdln1ZClDLXqRq\nnHMOdO4M559f+nutWQMbbQQLF8LGG5f+ftUm35a9ZtCKSM7K2bJfa61YI0e7ViVDYS8iOXH/fEJV\nuXTvroe0SVHYi0hO3n4bNtgAttmmfPdUv31yFPYikpNyja9vTi375CjsRSQn5eyvb6KWfXIU9iKS\nk3KsidNS167w5puwsq2VtyRnCnsRyUkaLfv11ouhnm++Wd771iKFvYhktWQJLFoEu+1W/nur3z4Z\nCnsRyWrq1FhjvkOH8t9b/fbJUNiLSFZpdOE0Ucs+GQp7EckqzbBXyz4ZCnsRySrtsJ82LWbwSuEU\n9iLSrpUrI2x79Urn/ltuGaNy3nknnfvXCoW9iLRrxgzYcUfYcMP0alC/ffEU9iLSrjS7cJqo3754\nCnsRaVclhL1a9sVT2ItIu9JYAK0lteyLp7AXkTa5w+TJ0LdvunWoZV88hb2ItOntt2HddWHbbdOt\no0sX+PBD+OijdOuoZgp7EWnT5Mmwzz5pVxFbFO65p7YoLEbHtAsQqXWvvgq33x5LBK+zDgwYACef\nHMMZK92kSel34TRpmly1335pV1Kd1LIXKZFly2D4cDj0UNhsM7j4YjjrLPjggwjQ//mfyp8VWikt\ne9BD2mKpZS9SAkuWwBFHwHbbRWt0s80+/+xrX4Mf/hD+8z/h9dfh1lvTWU0yF5Mnw8iRaVcRuneH\nP/4x7Sqql1r2Ign77DM48kjo3RtGj/5i0DfZeWf4y19gzhw4++zKbOG/9x588knUWgmaunGkMAp7\nkYQNHw5bbQU33BAPFtuy4Ybw4IMR+rfdVr76ctXUhWOWdiVhjz3im6O2KCyMunFEEvTgg/D00zER\nqb2gb7LppnDPPXDggTBwIPTsWfoac1VJ/fXwxS0K99wz7Wqqj1r2Igl57z34/vfhjjtgo41yP69b\nN7jiCjj9dFizpnT15WvSpMoKe4h+ez2kLYzCXiQhP/oRHH88DBqU/7mnnx799rfemnxdhaqEmbMt\nqd++cOrGEUnA889H902hk37WWgtuuQUOPhi++c3o80/TkiWxfnyldZd06wZ/+1vaVVQntexFiuQO\n55wDV12VX/dNS716wbe+BT//eXK1Feqll2Kly0obEqpunMIp7EWKdO+9sGoVnHBC8de65BL4/e9j\nTZo0vfAC9OuXbg2t0RaFhVPYixRhzRq4/HK48srcRt9k06lTPOQdMaL4axWjUsN+yy1jyYl33027\nkuqjsBcpwn33xXj5f//35K553nnw0EPw1lvJXTNflRr2oK6cQinsRQrU1KofMSLZiUebbQannQbX\nXJPcNfOxYAEsXw677JLO/bPRiJzCKOxFCjR2bKz1PnRo8tc+5xwYNSrG7pfbiy/GypKVMnO2JS2I\nVhiFvUiBrrkmulxKEYqdOsExx6SzCFkld+GAwr5QWcPezIaa2XQzm2VmF7byeTcze87MlpvZufmc\nK1KtJk2CN96IlStL5bzz4Kabyr87U6WHvbYoLEy7YW9mHYAbgKFAD+BYM+ve4rDFwFnA/xRwrkhV\nuvZa+K//grXXLt09dtsNDjkkhmKWy+rVEfb771++e+Zrxx1h8WJYujTtSqpLtpZ9P2C2u7/l7iuB\n0cCw5ge4+yJ3nwi0XIsu67ki1WjevOivP/XU0t/r7LPh+uvLt2bO1Kmx2FjaM3jb06FDrIA5c2ba\nlVSXbGHfGZjb7PW8zHu5KOZckYr129/GBKrW1qlP2sCBsMkm8Nhjpb8XwDPPwAEHlOdexdDwy/xl\nC/ti5qlpjpvUnBUr4He/gx/8oDz3M4vuonI9qP3b32K55Uqn4Zf5y7YQ2nygS7PXXYgWei5yPndE\ns+mCDQ0NNDQ05HgLkfJ66KEImnIuEHbMMXDBBdGS7datdPdxj5b91VeX7h5J6dYtlqmoJ42NjTQ2\nNhZ8vnk7i0yYWUdgBjAYWAC8ABzr7l/6nmpmI4Cl7n5NPueambdXg0glGTw4liM+5pjy3veSS+Cf\n/4zdr0rljTfgoINg7tzKHWPfZMoUOO44ePXVtCtJj5nh7jn/TbUb9pkLHgb8GugA3O7uPzezMwDc\n/WYz6wS8CGwCrAGWAj3c/ePWzm3l+gp7qQozZ0YXx9y5sT5LOc2fH3vazpkTu1uVwh13wKOPxr65\nle7TT2OdnKVLoWOdLtSeeNiXmsJeqsV550Ww/Pd/p3P/Y4+F/v1jhE4pnHIK7LsvnHlmaa6ftF12\ngXHjYmROPco37DWDViQHy5dHy/f009OroelB7erVyV/bHcaPhyFDkr92qWgmbX4U9iI5uPfeaPXu\numt6NfTvH10XjzyS/LWnT48lmrt2Tf7apaKZtPlR2Ivk4Kab4Iwz0q3BLBZIu+665K89bhx89auV\n/2C2ObXs86OwF8ni1VfjwegRR6RdSazFM316zHRNUlPYVxONtc+Pwl4ki5tvjqURSrkOTq7WWScm\ndCU5yeqzz2J8/eDByV2zHHr2hNde0xaFuVLYi7Tjk0/g7rvLsw5Ork4/PXbIWrQomes9+yz06AFb\nbJHM9cplyy1jl7C5c7MfKwp7kXaNHg2DBkGXLtmPLZett4ZvfhNuuSWZ6/35z3DYYclcq9x69YrW\nvWSnsBdpx803w/e+l3YVX3b22XDjjbFWTzHc4f7745tHNerVq75n0eZDYS/ShkmTYlvAJDcTT0rv\n3vGAstjZri+9FFsr9uyZTF3l1rOnwj5XCnuRNvz2t7Hxd4cOaVfSuh//GH7+8+ImWd13H3zjG9U1\n5LI5dePkTssliLTiww9jOv706bDttmlX0zr3WO/+hz+Eb30r//NXr4add46NWHr3Try8svjoI9hu\nu/i9Ur8pl4qWSxBJwJ13wtChlRv0EK3xSy6BK68sbCerp56K/75qDXqIjV222irmQUj7FPYiLbhH\nF873v592Jdkddhist15hffe/+x2cdFLiJZWdunJyo7CXf1mxIn6kP/98+MpXYKedYKONYvu9XXeN\nGZY//Sn85S+lWYyrUjQ2RpdANezYZAbXXAMXXwzLluV+3ty58PjjcPzxpautXDQiJzcKe2HBgli+\nt3Nn+NnP4kfjyy6L0HvnHXjrrZhOf8450eo955zo6x0xAhYvTrf2Umhq1VfLQ8sDD4xF2vJZM+e6\n66JVX459dEutaSattE8PaOvY0qVwxRVw221w4olw1lm5r+o4dSpcf32M0T7rLLjoouhOqHbvvBOz\nSf/xj/imVy1mzYIBA+DFF+PBcnsWLoyAfOml+Omt2k2aFN+4kl4vqNLpAa3k5PHHY4nY996D11+H\nX/0qv+V799oLbr01AuOVV+Ih3xNPlK7ecrnttthysJqCHmIDjwsugO9+N/vD2p/8JMKxFoIe4ut4\n1ixYuTLtSiqbWvZ1ZsWK+Mf+xz/GiJNDDknmug8/HAt0HX10jP0u97Z9SVi5Mr7hPfww7L132tXk\nb/Xq+Ps88MAYodOap5+G73wnhpSWanvDNHTtCg88UL2Twwqhlr206f33Y2XD6dPh5ZeTC3qI5X8n\nT/58n9a3307u2uVyzz2w++7VGfQQD5XvuQdGjWp9Vcw334ygv+uu2gp6gD59YhNyaZvCvk409ekO\nGgQPPRRjk5O25ZYwZkzMyBwwACZOTP4epeIeo1rOPTftSoqzzTbxYP03v4GTT46/91Wr4u9l0KB4\nqF5tSxnnok+faMBI2xT2dWDKlBhKef75sVn2WiX8WzeDCy+EG26IMeAPPVS6eyXpr3+Fjz+Gww9P\nu5Li7bRTPKjdems4+OBY++aKK6JFn/ZuW6WisM9OffY1bvLkCN2RIwubUl+MF1+Eo46CX/6y8sdz\nDxsW/58qcYXLYq1eXftLCSxYEIG/cGH1DJktVr599h1LWYyka+rUCLDf/CadJWz32w+efDImY336\naWy6UYlmzoTnnouH1rWo1oMeYn0ciKGz22+fbi2VSmFfo+bMiS6JkSPTXau8R4+YcTtkSAT+Oeek\nV0tbrroKzjwTNtgg7UqkUGafd+Uo7FunsK9BixbFGuwXXVT+rpvW7LZbBH5DQ+zjeuaZaVf0udmz\nY6jl7NlpVyLF6tMnui1r4blLKSjsa8wnn8QX+zHHwPDhaVfzuR13jC6dgw6Kmbbf/W7aFYWf/Sz+\nP9XCsgH1rk+fGGsvrdMD2hqyZk205DfeOFY0rMQHVTNnxgiRX/wCjjsu3VreeAP23z9a9Qr76vf6\n6/GgfdastCspDz2grWNXXBGjEp5+ujKDHmKm47hx0Ye/zjox4zYtV1wRXUoK+trQtWt8/S9dGg0e\n+SKFfY247z64/XZ44YUYV13JevaERx+N5wrrrhvDM8tt8uRYH2jGjPLfW0qjY8f42po6NSaQyRdp\nUlUNmDIlxoc/+CB06pR2Nbnp0ycejJ56aqyhX07u8KMfxWzSalvwTNrX9JBWvkxhX+WWLInlCa6/\nHvr2Tbua/Oy3X8ywPfFEGD++fPd98MEYsVQpD4klOfvuGyuxypcp7KuYO5x2WuyV+u1vp11NYQYM\niDXxv/OdeNZQakuWxPr7N9wQP/ZLbdlvv5i5LV+m0ThV7OabY1el55+v/o1Dnn46RhLdf39ptwP8\n3vdi1NItt5TuHpKelSvjgfu779b+Q1otcVwnpk6N/WD/9KfqD3qI4Zh33x1dUn/5S2nuMW5cPCf4\nxS9Kc31J39prx8Y6kyalXUnlUdhXoY8/jklTv/pVDDerFYceCqNHx3DMBx9M9trz5sWzgbvu0lDL\nWtevX4xKky9S2Feh4cOhf//KX0myEIMHx7DM738/hpImYfny+OZ41lmxZIPUNvXbty5r2JvZUDOb\nbmazzOzCNo4Zmfl8ipnt0+z9t8xsqplNNjN9r03AnXfChAnxgLFW7btvdOX87Gdw3nmx+UahVq6M\nh9c77BBrBUntU9i3rt0HtGbWAZgBDAHmAy8Cx7r7tGbHHA4Md/fDzWx/4Dp375/5bA6wr7t/0M49\n9IA2RzNmwAEHwFNPxQbftW7x4lhSYfny6N7Jdw7BihXRdbNkSXQLVeO+uJK/NWtgiy1i2YStt067\nmtJJ+gFtP2C2u7/l7iuB0cCwFsccBdwB4O4TgM3MbNvmNeVajLRt2bIYrXLllfUR9BDbHD7ySCye\nttde8Pvfx3DTXMyfH+voL18O996roK8na60VrfsJE9KupLJkC/vOwNxmr+dl3sv1GAeeMLOJZnZa\nMYXWu3PPhW7dKncDkFLp0AEuvzyWNrjhhujieeCB6J5pzbJlsYZ/376xofq992qd+no0aBA8+2za\nVVSWbNNKcu1faav1foC7LzCzrYHxZjbd3Z9pedCIESP+9eeGhgYa9BTtC+69N8Ju0qTKXeCs1PbZ\nJ0ZYjBkTQydPPz124dp7b9h0U/jww5gm/9hjMHBgzMjda6+0q5a0DBoUC93VksbGRhobGws+P1uf\nfX9ghLsPzby+GFjj7lc3O+YmoNHdR2deTwcOcveFLa51GfCxu1/T4n312bdjzpxYhveRR+JHUwn/\n+Ed8A5w2LfrkN90UevWK4Zs77ph2dZK2pUtjq8LFiyt/YcBCJb3E8URgDzPbGVgAHAMc2+KYMcBw\nYHTmm8OH7r7QzDYAOrj7UjPbEPgqcHmuhUk8YPz2t+HiixX0Le20U/11aUnuNt4Y9twz1skZODDt\naipDu2Hv7qvMbDjwONABuN3dp5nZGZnPb3b3sWZ2uJnNBj4BTs6c3gm436LfoSMwyt3Hleo/pBb9\n+MewzTaVuW+rSKVr6rdX2AetjVOhHnkkJhZNnhyjUkQkP3/6E4waFSur1qJ8u3EU9hVo3jz4t3+L\nB7MHHJB2NSLVaf78eIC/aFFtDmzQQmhVbtWqWO73rLMU9CLF6Nw5Hty/9lralVQGhX2FufRSWH/9\neCgrIsUZMgSeeCLtKiqDwr6CPPZYrH3zhz/ELEARKc7gwfDkk2lXURnUZ18hmvrp//Qn+MpX0q5G\npDa8/z7stlv8vvbaaVeTLPXZV6FVq2I8/dlnK+hFkrTVVrDrrloFExT2FeGnP41JIBe2uoC0iBRD\n/fZBYZ+yP/85xgLfeaf66UVKYfBghT2ozz5VM2bE5tpjxsTOUyKSvGXLYNttYz2lzTdPu5rkqM++\nSixZAsOGwVVXKehFSmn99WND+0cfTbuSdCnsU7BmDZxwQqy3fuqpaVcjUvuOOip+gq5n6sZJwYgR\nMfb3ySe1g5JIObz7LnTvDgsX1s6/OXXjVLgHHoDbb4d77qmdLzqRStepUyx5/MyXtk6qHwr7Mnrh\nhViD/YEH8t88W0SKc9RRsfF8vVI3Tpm8+Wasr33LLXDkkWlXI1J/Zs2K0W/z5kHHbNs2VQF141Sg\nDz6Aww+PyVMKepF07LFHbFn51FNpV5IOhX2JLVsGX/86fO1rcOaZaVcjUt+OOw7uvjvtKtKhbpwS\nWrEC/uM/YJNNYpasZsiKpOudd6BHD1iwIMbfVzN141SI1avh+OOjb1BLIYhUhu22i9Vla3WrwvYo\ngkpgzRo47bToq/+//6u9pVVFqtlpp8FNN6VdRfmpGydhq1bBKafEOhxjx8KGG6ZdkYg0t3Il7LQT\nPP449O6ddjWFUzdOilasgGOPjdl6jz6qoBepRGuvHfNdbrwx7UrKSy37hHzySWxAYha7Ta23XtoV\niUhbFiyAnj1hzhzYbLO0qymMWvYpWLgwVtXbfHO47z4FvUil2377mFF7/fVpV1I+atkXadq0mDB1\n0klw6aXRsheRyjdrFgwcCLNnw6abpl1N/tSyL6MHH4SDDoLLL4fLLlPQi1STPfaIhtrIkWlXUh5q\n2Rdg1Sr4yU9g9Ojon99//7QrEpFCzJoFAwbAa6/FblbVJN+WvcI+T3PmwIknxuy7UaNi93oRqV7n\nnQeLFsEdd6RdSX7UjVMi7rFiZb9+sZjZ2LEKepFacNllsTjaX/+adiWlVQMLfZbezJkwfHjMiG1s\njCFbIlIbNt4Yrrsutgh96aV4XYvUsm/Hxx/DRRfFE/uvfhWee05BL1KLvvGNGGxxxhnxU3wtUti3\nYvny+E7ftSvMnw+vvBL9elrjRqR2jRwJr74KN9yQdiWloW6cZpYuhd//Hq6+GvbdN/rl+/RJuyoR\nKYf114/h1F/5Cmy5JXznO2lXlCyFPTGp4sYb42n8wQfHHrH77Zd2VSJSbrvuCo89BkOGxOq1xx+f\ndkXJqduwf//9GCP/hz/E/rAnnQSTJ8e2ZSJSv3r1giefhCOOgOnTY9Jkhw5pV1W8rH32ZjbUzKab\n2Swzu7CNY0ZmPp9iZvvkc265uMfEiV/+Eg45BHbfHZ59NpY4mDcvum4U9CICMRBjwgT4+9/hgANi\nWZRq127Ym1kH4AZgKNADONbMurc45nBgd3ffAzgd+G2u55bSZ5/F6Jlrr4Wjj4Yddoh9YOfMgR/9\nKB68jhoFhx2W24PXxsbGktecBNWZrGqosxpqhOqrc5tt4Ikn4IQT4MADY2jmG2+kW1sxsrXs+wGz\n3f0td18JjAaGtTjmKOAOAHefAGxmZp1yPLco7rHi5AsvxI5QI0ZEsPfsGcuW/uAH0R8/bFhMmJgz\nJ/rmjzgi/7Xmq+0LtdKpzuRUQ41QnXWutVbkyKxZ0KkT9O8fPQO33RbZU02y9dl3BuY2ez0PaLkS\nTGvHdAa2z+Hcdk2cCDNmRP/6++/D4sWf/3nePJg7FzbaKLpfdt4ZunePDb5/+tMYNlntGwqLSGXY\nfHO48kqo8a/cAAAEm0lEQVS45BIYMyaWMv/oo+glqBbZwj7X6QUlWe/xiSdg6tRYlmCrraLFvtVW\nMSxqhx2gSxftBiUi5bPuutF7cPTRaVeSv3YXQjOz/sAIdx+aeX0xsMbdr252zE1Ao7uPzryeDhwE\n7JLt3Mz7NTpfTUSktPJZCC1by34isIeZ7QwsAI4Bjm1xzBhgODA6883hQ3dfaGaLczg3r2JFRKQw\n7Ya9u68ys+HA40AH4HZ3n2ZmZ2Q+v9ndx5rZ4WY2G/gEOLm9c0v5HyMiIq1LfT17EREpvYpaCM3M\nzjWzNWa2Rdq1tMbMfmlm0zKTx+43s4rZubKSJrC1xcy6mNnTZvaamb1qZv+Vdk3tMbMOZjbZzP6c\ndi1tMbPNzOzezNfl65mu1IpjZhdn/t5fMbO7zWzdtGsCMLPfmdlCM3ul2XtbmNl4M5tpZuPMbLM0\na8zU1FqdeeVRxYS9mXUBDgX+kXYt7RgH9HT3vYGZwMUp1wOkP4EtDyuBH7p7T6A/cGaF1tnkbOB1\nch+VlobrgLHu3h3YC6i4rtLMc7vTgL7u3pvo1v12mjU187/Ev5vmLgLGu3tX4MnM67S1VmdeeVQx\nYQ9cC1yQdhHtcffx7r4m83ICsEOa9TRT8glsSXD3d9395cyfPyaCaft0q2qdme0AHA7cRomGFhcr\n05I70N1/B/GczN2XpFxWaz4ivtFvYGYdgQ2A+emWFNz9GeCfLd7+10TRzO9fL2tRrWitznzzqCLC\n3syGAfPcfWrateThFGBs2kVktDWxrWJlWnv7EF+klehXwPnAmmwHpmgXYJGZ/a+ZTTKzW81sg7SL\nasndPwCuAd4mRuZ96O5PpFtVu7Z196b5sQuBatiKPGselS3sM31gr7Ty6yjix4/Lmh9errpaaqfO\nI5sd8xNghbvfnVadLVRyN8OXmNlGwL3A2ZkWfkUxsyOA99x9MhXaqs/oCPQFbnT3vsRouErocvgC\nM9sNOAfYmfhJbiMzOy7VonLkMYKlov995ZpHZVvi2N0Pbe19M+tFtFCmmBnEjyIvmVk/d3+vXPU1\naavOJmZ2EvHj/eCyFJSb+UCXZq+7EK37imNmawP3AXe5+4Np19OGgcBRmUX+1gM2MbM73f3/pVxX\nS/OIn4hfzLy+lwoMe+DfgL+7+2IAM7uf+H88KtWq2rbQzDq5+7tmth1Q9hzKVT55lHo3jru/6u7b\nuvsu7r4L8QXcN42gz8bMhhI/2g9z9+Vp19PMvya/mdk6xAS2MSnX9CUW381vB15391+nXU9b3P3H\n7t4l8/X4beCpCgx63P1dYK6Zdc28NQR4LcWS2jId6G9m62e+BoYQD74r1RjgxMyfTwQqslGSbx6l\nHvatqOQfma4HNgLGZ4bk3Zh2QRAP5ohZzI8T/4j+r0InsA0CjgcOzvz/m5z5gq10lfw1eRYwysym\nEKNxrkq5ni9x9ynAnUSjpOm53C3pVfQ5M/sj8HdgTzOba2YnA/8NHGpmM4FDMq9T1Uqdp5BnHmlS\nlYhIHajElr2IiCRMYS8iUgcU9iIidUBhLyJSBxT2IiJ1QGEvIlIHFPYiInVAYS8iUgf+PxI/FU8X\nDMonAAAAAElFTkSuQmCC\n", 282 | "text/plain": [ 283 | "" 284 | ] 285 | }, 286 | "metadata": {}, 287 | "output_type": "display_data" 288 | } 289 | ], 290 | "source": [ 291 | "%matplotlib inline\n", 292 | "\n", 293 | "import matplotlib.pyplot as plt\n", 294 | "import numpy as np\n", 295 | "from scipy.stats import norm\n", 296 | "\n", 297 | "def plot_mog(gaussians, weights):\n", 298 | " ys = []\n", 299 | " \n", 300 | " # find range for x\n", 301 | " min_x, max_x = 1e308, -1e308\n", 302 | " for g in gaussians:\n", 303 | " min_x = min(min_x, norm.ppf(.001, g[0], g[1]))\n", 304 | " max_x = max(max_x, norm.ppf(.999, g[0], g[1]))\n", 305 | " \n", 306 | " xs = np.linspace(min_x, max_x, 500)\n", 307 | " ys = np.zeros(len(xs))\n", 308 | " for g, w in zip(gaussians, weights):\n", 309 | " ys += w * norm.pdf(xs, loc=g[0], scale=g[1])\n", 310 | " \n", 311 | " plt.plot(xs, ys)\n", 312 | "\n", 313 | "gaussians = [[1, 1.5], \n", 314 | " [5, 1],\n", 315 | " [8, .8]]\n", 316 | "\n", 317 | "weights = [.3, .3, .4]\n", 318 | "\n", 319 | "plot_mog(gaussians, weights)" 320 | ] 321 | } 322 | ], 323 | "metadata": { 324 | "kernelspec": { 325 | "display_name": "Python 3", 326 | "language": "python", 327 | "name": "python3" 328 | }, 329 | "language_info": { 330 | "codemirror_mode": { 331 | "name": "ipython", 332 | "version": 3 333 | }, 334 | "file_extension": ".py", 335 | "mimetype": "text/x-python", 336 | "name": "python", 337 | "nbconvert_exporter": "python", 338 | "pygments_lexer": "ipython3", 339 | "version": "3.4.3" 340 | } 341 | }, 342 | "nbformat": 4, 343 | "nbformat_minor": 0 344 | } 345 | -------------------------------------------------------------------------------- /Poisson-Point-Processes.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [ 10 | { 11 | "data": { 12 | "text/html": [ 13 | "\n", 220 | "\n" 239 | ], 240 | "text/plain": [ 241 | "" 242 | ] 243 | }, 244 | "execution_count": 1, 245 | "metadata": {}, 246 | "output_type": "execute_result" 247 | } 248 | ], 249 | "source": [ 250 | "# style the notebook\n", 251 | "from IPython.core.display import HTML\n", 252 | "import urllib.request\n", 253 | "response = urllib.request.urlopen('http://bit.ly/1LC7EI7')\n", 254 | "HTML(response.read().decode(\"utf-8\"))" 255 | ] 256 | }, 257 | { 258 | "cell_type": "markdown", 259 | "metadata": {}, 260 | "source": [ 261 | "# Poisson Point Processes" 262 | ] 263 | }, 264 | { 265 | "cell_type": "code", 266 | "execution_count": 162, 267 | "metadata": { 268 | "collapsed": true 269 | }, 270 | "outputs": [], 271 | "source": [ 272 | "import numpy as np\n", 273 | "import scipy.stats\n", 274 | "\n", 275 | "def PPP(rt, Dx, Dy=None):\n", 276 | " '''\n", 277 | " Determines the number of events `N` for a rectangular region,\n", 278 | " given the rate `rt` and the dimensions, `Dx`, `Dy`.\n", 279 | " Returns a <2xN> NumPy array.\n", 280 | " \n", 281 | " source: http://connor-johnson.com/2014/02/25/spatial-point-processes/\n", 282 | " '''\n", 283 | " if Dy == None:\n", 284 | " Dy = Dx\n", 285 | " mu = rt*Dx*Dy\n", 286 | " N = scipy.stats.poisson(mu).rvs()\n", 287 | " x = scipy.stats.uniform.rvs(0, Dx, (N, 1))\n", 288 | " y = scipy.stats.uniform.rvs(0, Dy, (N, 1))\n", 289 | " P = np.hstack((x, y))\n", 290 | " return P" 291 | ] 292 | }, 293 | { 294 | "cell_type": "code", 295 | "execution_count": 163, 296 | "metadata": { 297 | "collapsed": false 298 | }, 299 | "outputs": [ 300 | { 301 | "data": { 302 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXEAAAEACAYAAABF+UbAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADdlJREFUeJzt3W+MHHd9x/HPpzbQddMSoUoxFEtOUdPSKq1wq8oCVdnS\n3J4V0QPrqkLEn1NE8BNaAt3SS/qguYfkwQmQKh6YkOhQIVVr95CpaOeuwIqekChpnBJip2lRUeNU\nuSDaBoH2QWi+fbCDvXHM3e3c7s5+d98v6aSd2dnZr8a+j373nfnNOCIEAMjpJ+ouAABQHSEOAIkR\n4gCQGCEOAIkR4gCQGCEOAIntGOK2H7C9bfuxvnWvsr1p+0nbG7avH32ZAIBr2W0k/qCkE1etu1vS\nZkTcJOmL5TIAoAbebbKP7aOSPh8RN5fLT0i6JSK2bR+W1ImIXxp1oQCAl6rSE78hIrbL19uSbhhi\nPQCAAezrxGb0hvHM2weAmhys8Jlt24cj4hnbr5b07LU2sk24A0AFEeG9bltlJH5O0lL5eknS53Yo\nhJ8I3XvvvbXXMCk/HAuOBcdi559B7XaJ4UOSvirpF20/ZfsOSR+RNGf7SUlvLpcBADXYsZ0SEbf/\nmLduHUEtAIABMWNzDJrNZt0lTAyOxRUciys4FtXtep145R3bMap9A8C0sq0Y8YlNAMCEIMQBIDFC\nHAASI8QBIDFCHAASI8QBIDFCHAASI8SBGVQUhVqtRbVaiyqKou5ysA+EOIaKcJh8RVHo5MklbW4u\naHNzQSdPLvFvlRgzNjE0PwqHbvc+SVKjsaz19TXNz8/XXBn6tVqL2txc0JWbka5pbu6cNjbO1lkW\nSoPO2KxyP3HgmlZXT5cB3guHbre3jhAHRocQB2ZMu31KW1tL6nZ7y43GstrttXqLQmW0UzA0tFPy\nKIpCq6unJfVCnX+jyTFoO4UQx1ARDsD+EOIAkBi3ogWAGUKIA0BihDgAJEaIA0BihDgAJEaIA0Bi\nhDgAJEaIA0BihDgAJEaIA0BihDgAJEaIA0BihDgAJEaIA0BihDgAJEaIA0BihDhGqigKtVqLarUW\nVRRF3eUAU4cn+2BkeOYmMDgez4aJ0WotanNzQdJSuWZNc3PntLFxts6ygIk2tsez2b7H9uO2H7P9\nWduvqLovAEA1lULc9lFJ75N0LCJulnRA0juGVxamQbt9So3GsqQ1SWtqNJbVbp+quyxgqhys+Lnv\nSXpe0iHb/yfpkKSnh1YVpsL8/LzW19e0unpaktRu0w8Hhq1yT9z2KUmrkrqSioh491Xv0xMHgAEN\n2hOvNBK3/TpJH5R0VNJzkv7a9jsj4jP9262srFx+3Ww21Ww2q3wdAEytTqejTqdT+fOVRuK23y5p\nLiLuLJffLel4RLy/bxtG4gAwoHFdnfKEpOO2G7Yt6VZJFyruCxg6JhlhVuynJ/4n6l0A/IKkRyTd\nGRHP973PSBy1YJIRMmOyD2Yek4yQ2dgm+wAA6lf1OnFgYrXbp7S1taRut7fcm2S0Vm9RwIjQTsFU\nKoqib5LRKfrhSIOeOAAkRk8cAGYIIQ4AiRHiAJAYIQ4AiRHiAJAYIQ4AiRHiwJhxcy4ME9eJA2PE\nzbmwG64Tx8yb5JHu6urpMsCXJPXC/EczS4EquHcKpsrVI92trSVGuphqhDimyotHulK321s3KSHO\nzbkwbIQ4MEbz8/NaX1/ruzkXfyVgfzixianCiUNkx10MMfO4DS0yI8QBIDEuMQSAGUKIA0BihDgA\nJEaIA0BihDgAJEaIA0BihDgAJEaIA0BihDgAJEaIA0BihDgAJEaIA0BihDgAJEaIA0BihDgAJEaI\nA0BilUPc9vW2z9i+aPuC7ePDLAwAsLv9PCj545K+EBG/Z/ugpJ8aUk0AgD2q9Hg226+UdD4ifn6H\nbXg8GwAMaFyPZ7tR0ndsP2j7EduftH2o4r4AABVVDfGDko5J+kREHJP0A0l3D60qAMCeVO2JX5J0\nKSK+Xi6f0TVCfGVl5fLrZrOpZrNZ8esAYDp1Oh11Op3Kn6/UE5ck21+RdGdEPGl7RVIjIpb73qcn\nDgADGrQnvp8Q/zVJ90t6uaRvSbojIp7re58QB4ABjS3E91AIIQ4AAxrX1SkAgAlAiANAYoQ4ACRG\niANAYoQ4ACRGiANAYoQ4ACRGiANAYoQ4ACRGiANAYoQ4ACRGiANAYoQ4ACRGiANAYoQ4ACRGiANA\nYoQ4ACRGiANAYoQ4ACRGiANAYoQ4ACRGiANAYoQ4ACRGiANAYoQ4ACRGiANAYoQ4ACRGiANAYoQ4\nACRGiANAYoQ4ACRGiGMgRVGo1VpUq7WooijqLgeYeY6I0ezYjlHtG/UoikInTy6p271PktRoLGt9\nfU3z8/M1VwZMD9uKCO95e0Ice9VqLWpzc0HSUrlmTXNz57SxcbbOsoCpMmiI004BgMQO1l0A8mi3\nT2lra0ndbm+50VhWu71Wb1HAjNtXO8X2AUkPS7oUEb971Xu0U6ZQURRaXT0tqRfq9MOB4RprT9z2\nH0n6dUk/HRELV71HiAPAgMbWE7f9Wkm3Sbpf0p6/EAAwPPs5sflRSR+W9MKQagEADKhSiNt+i6Rn\nI+K8GIUDQG2qXp3yRkkLtm+T9JOSfsb2pyPiPf0braysXH7dbDbVbDYrfh0ATKdOp6NOp1P58/ue\n7GP7Fkl/zNUpALB/dU32Ia0BoAZMuweACcK0ewCYIYQ4ACRGiANAYoQ4ACRGiANAYoQ4ACRGiANA\nYoQ4ACRGiANAYoQ4ACRGiANAYoQ4ACRGiANAYoQ4ACRGiANAYoQ4ACRGiANAYoQ4ACRGiANAYoQ4\nACRGiANAYoQ4ACRGiANAYoQ4ACRGiANAYoQ4ACRGiANAYoQ4ACRGiANAYoQ4ACRGiANAYoQ4ACRG\niANAYoQ4ACRGiANAYoQ4ACRWKcRtH7H9ZduP2/6m7Q8MuzAAwO4cEYN/yD4s6XBEPGr7Okn/LOlt\nEXGxb5uosm8AmGW2FRHe6/aVRuIR8UxEPFq+/r6ki5JeU2VfAIDq9t0Tt31U0hskfW2/+wIADObg\nfj5ctlLOSLqrHJG/yMrKyuXXzWZTzWZzP18HAFOn0+mo0+lU/nylnrgk2X6ZpL+V9HcR8bFrvE9P\nHAAGNGhPvOqJTUtak/TdiPjQj9mGEAeAAY3lxKakN0l6l6Tftn2+/DlRcV8AZkRRFGq1FtVqLaoo\nirrLmQqV2ym77piROIA+RVHo5Mkldbv3SZIajWWtr69pfn6+5somy1jaKXsshBAHcFmrtajNzQVJ\nS+WaNc3NndPGxtk6y5o442qnAAAmwL4uMQSAvWq3T2lra0ndbm+50VhWu71Wb1FTgHYKgLEpikKr\nq6cl9UKdfvhL0RMHgMToiQPADCHEASAxQnwXTE4AMMnoie+AyQkAxo0Tm0PE5AQA48aJTQCYIUz2\n2QGTEwBMOtopu2ByAoBxoicOAInREweAGUKIA0BihDgAJEaIA0BihHgy3AYAQD+uTkmE2wAA049L\nDKcYtwEAph+XGALADGHafSLcBgDA1WinJMNtAIDpRk8cABKjJw4AM4QQB4DECHEASIwQB4DECHEA\nSIwQB4DECHEASIwQB4DECHEASIwQB4DEKoe47RO2n7D9b7aXh1kUAGBvKoW47QOS/lzSCUm/LOl2\n268fZmHTpNPp1F3CxOBYXMGxuIJjUV3VkfhvSvr3iPh2RDwv6S8lvXV4ZU0X/oNewbG4gmNxBcei\nuqoh/nOSnupbvlSuAwCMUdUQ5x6zADABKt1P3PZxSSsRcaJcvkfSCxFxX982BD0AVDDyh0LYPijp\nXyX9jqT/kvRPkm6PiIsD7wwAUFmlZ2xGxA9t/4GkQtIBSZ8iwAFg/Eb2eDYAwOiNZMYmE4F6bB+x\n/WXbj9v+pu0P1F1TnWwfsH3e9ufrrqVOtq+3fcb2RdsXynNMM8n2PeXvx2O2P2v7FXXXNC62H7C9\nbfuxvnWvsr1p+0nbG7av320/Qw9xJgK9yPOSPhQRvyLpuKT3z/CxkKS7JF0QVzd9XNIXIuL1kn5V\n0ky2Im0flfQ+Scci4mb1WrPvqLOmMXtQvZzsd7ekzYi4SdIXy+UdjWIkzkSgUkQ8ExGPlq+/r94v\n62vqraoetl8r6TZJ90va85n3aWP7lZJ+KyIekHrnlyLiuZrLqsv31BvoHCovljgk6el6SxqfiPhH\nSf9z1eoFSWvl6zVJb9ttP6MIcSYCXUM56niDpK/VW0ltPirpw5JeqLuQmt0o6Tu2H7T9iO1P2j5U\nd1F1iIj/lrQq6T/Vu8rtfyPiH+qtqnY3RMR2+Xpb0g27fWAUIT7rfyq/hO3rJJ2RdFc5Ip8ptt8i\n6dmIOK8ZHoWXDko6JukTEXFM0g+0hz+Zp5Ht10n6oKSj6v2Fep3td9Za1ASJ3lUnu+bpKEL8aUlH\n+paPqDcan0m2XybprKS/iIjP1V1PTd4oacH2f0h6SNKbbX+65prqcknSpYj4erl8Rr1Qn0W/Iemr\nEfHdiPihpL9R7//KLNu2fViSbL9a0rO7fWAUIf6wpF+wfdT2yyW9XdK5EXzPxLNtSZ+SdCEiPlZ3\nPXWJiD+NiCMRcaN6J66+FBHvqbuuOkTEM5Kesn1TuepWSY/XWFKdnpB03Haj/F25Vb0T37PsnKSl\n8vWSpF0HfpUm++yEiUAv8iZJ75L0Ddvny3X3RMTf11jTJJj1ltsfSvpMOcj5lqQ7aq6nFhHxL+Vf\nZA+rd67kEUmn661qfGw/JOkWST9r+ylJfybpI5L+yvZ7JX1b0u/vuh8m+wBAXjyeDQASI8QBIDFC\nHAASI8QBIDFCHAASI8QBIDFCHAASI8QBILH/Bw8tGE7rpNLYAAAAAElFTkSuQmCC\n", 303 | "text/plain": [ 304 | "" 305 | ] 306 | }, 307 | "metadata": {}, 308 | "output_type": "display_data" 309 | } 310 | ], 311 | "source": [ 312 | "%matplotlib inline\n", 313 | "import matplotlib.pyplot as plt\n", 314 | "\n", 315 | "pts = PPP(.2, 10)\n", 316 | "plt.scatter(pts[:, 0], pts[:, 1]);\n", 317 | "plt.xlim(0,10); plt.ylim(0, 10);" 318 | ] 319 | }, 320 | { 321 | "cell_type": "code", 322 | "execution_count": 111, 323 | "metadata": { 324 | "collapsed": false 325 | }, 326 | "outputs": [ 327 | { 328 | "data": { 329 | "text/plain": [ 330 | "100" 331 | ] 332 | }, 333 | "execution_count": 111, 334 | "metadata": {}, 335 | "output_type": "execute_result" 336 | } 337 | ], 338 | "source": [ 339 | "# \n", 340 | "rt = 1\n", 341 | "DX, DY = 10, 10\n", 342 | "mu = rt*DX*DY\n", 343 | "scipy.stats.poisson(mu).rvs()" 344 | ] 345 | }, 346 | { 347 | "cell_type": "code", 348 | "execution_count": 154, 349 | "metadata": { 350 | "collapsed": false 351 | }, 352 | "outputs": [ 353 | { 354 | "data": { 355 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEACAYAAACkvpHUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHEJJREFUeJzt3X+QXeV93/H3B2TJIlYlhBuhn5hJljaq3QHLRm7d1NeD\nEbKnleSWgtwalEbT8WRrQzPTxCgZw27kxpCMjfF00D/IINSioIaEH2NZ7CK4rjsdsZiRsMyiSmqs\nRruylnSFBB7XjmS+/eM8iw5Xd3fvufvj/tjPa+bMPvd7nnP2PFrtfu9znuc5VxGBmZlZrS5p9AWY\nmVlrceIwM7NCnDjMzKwQJw4zMyvEicPMzApx4jAzs0JqShySLpV0QNIz6fVCSb2SjkjqkbQgV3eL\npKOSDktak4uvknQo7XsgF58j6fEU3y/pqty+Tel7HJF0++Q02czMJqLWHsedQD8wsujjLqA3Iq4B\n9qXXSFoJ3AqsBNYCD0pSOmYbsDkiOoAOSWtTfDMwnOL3A/elcy0E7gauT9s9+QRlZmaNMW7ikLQM\n+AzwEDCSBNYBO1J5B7AhldcDuyLiXEQcB44BqyUtBuZFRF+q92jumPy5ngBuSOWbgJ6IOBMRZ4Be\nsmRkZmYNVEuP437g94C3c7FFETGUykPAolReAgzk6g0AS6vEB1Oc9PUEQEScB85KumKMc5mZWQON\nmTgk/TPg9Yg4wIXexrtE9swSP7fEzGyGmDXO/n8MrJP0GeC9wN+RtBMYknRlRJxKt6FeT/UHgeW5\n45eR9RQGU7kyPnLMCuCkpFnA/IgYljQIlHLHLAeer7xASU5aZmZ1iIiqHYJaDqxpAz4BPJPKfwJ8\nOZXvAu5N5ZXAQWA2cDXwvwGlfS8Cq8l6LnuAtSneCWxL5Y3An6XyQuCvgAXA5SPlKtcV7eyee+5p\n9CVMqVZuH1lPu+at3bTyz64W7d6+9H+y5hyQ38brcVyUZ9LXe4HdkjYDx4Fb0l/wfkm7yWZgnQc6\n0wWOJIhHgLnAnojYm+LbgZ2SjgLDKXkQEaclbQVeSvW6IxskN5t2FyYHVqrW4VUu3pW2+t7YmTWj\nmhNHRHwP+F4qnwY+NUq9Pwb+uEr8ZeBDVeK/ICWeKvseBh6u9RrNplZlkiiWDEZLPhfeW5m1hqI9\nDptmpVKp0Zcwpdq9fe8ephutd9Ka2v1n1+7tmwi1+rsdSdHqbbDml/UWqvU4xrtVNX5d//+1RpBU\n9+C4n1VlZmaF+FaVWc7og+DT+z3dC7Fm5sRhdpHpHouY2KC72XTzrSozMyvEicPMzApx4jAzs0Kc\nOMzMrBAPjtuM1YgZVLXyKnNrZk4cNsM164ym9lplbu3Ft6rMzKwQJw4zMyvEicPMzApx4jAzs0Kc\nOMzMrBAnDjMzK8TTcc1aiJ+ka83AicPaXjMv9CuuWded2Ewy5q0qSe+V9KKkg5L6JX0txbskDUg6\nkLZP547ZIumopMOS1uTiqyQdSvseyMXnSHo8xfdLuiq3b5OkI2m7fXKbbjNLVNnMrB7jfnSspMsi\n4meSZgH/A/iPwA3AWxHxjYq6K4HHgI8CS4HngI6ICEl9wBcjok/SHuBbEbFXUifwwYjolHQr8NmI\n2ChpIfASsCqd/mVgVUScqfie/uhYG1P1j32Foh/x2qx1/f/f6jGlHx0bET9LxdnApcAbI9+3SvX1\nwK6IOBcRx4FjwGpJi4F5EdGX6j0KbEjldcCOVH6CLCkB3AT0RMSZlCx6gbW1NszMzKbGuIlD0iWS\nDgJDwAsR8Wra9SVJr0jaLmlBii0BBnKHD5D1PCrjgylO+noCICLOA2clXTHGuczMrIFq6XG8HRHX\nAsuAfyqpBGwDrgauBX4CfH0qL9LMzJpHzbOqIuKspO8AH4mI8khc0kPAM+nlILA8d9gysp7CYCpX\nxkeOWQGcTOMo8yNiWNIgUModsxx4vtq1dXV1vVMulUqUSqVq1czMZqxyuUy5XJ6Uc405OC7p/cD5\niDgjaS7wLNANvBoRp1Kd3wU+GhH/Ojc4fj0XBsd/PQ2OvwjcAfQB3+Hdg+MfiojfkbQR2JAbHP8B\n8GGy8ZSXgQ97cNyK8uC42cUmMjg+Xo9jMbBD0iVkt7V2RsQ+SY9Kupbsf/GPgS8ARES/pN1AP3Ae\n6Mz9Ve8EHgHmAnsiYm+Kbwd2SjoKDAMb07lOS9pKNrMKoLsyaZiZ2fQbdzpus3OPw8bT7j2O0fj3\nwsYylT0Os5bSXqvEa+VPC7Tp5cRhbciP5TCbSn46rpmZFeLEYWZmhThxmJlZIU4cZmZWiBOHmZkV\n4sRhZmaFOHGYmVkhXsdh1qb8+eQ2VZw4zNqWF0La1PCtKjMzK8Q9DmtJM/OZVGbNwYnDWpgf7mfW\nCL5VZWZmhThxmJlZIU4cZmZWiBOHmZkV4sRhZmaFjJk4JL1X0ouSDkrql/S1FF8oqVfSEUk9khbk\njtki6aikw5LW5OKrJB1K+x7IxedIejzF90u6KrdvU/oeRyTdPrlNNzOzeoyZOCLi58AnI+Ja4B8C\nn5T0T4C7gN6IuAbYl14jaSVwK7ASWAs8qAsT7rcBmyOiA+iQtDbFNwPDKX4/cF8610LgbuD6tN2T\nT1BmZtYY496qioifpeJs4FLgDWAdsCPFdwAbUnk9sCsizkXEceAYsFrSYmBeRPSleo/mjsmf6wng\nhlS+CeiJiDMRcQboJUtGZmbWQOMmDkmXSDoIDAEvRMSrwKKIGEpVhoBFqbwEGMgdPgAsrRIfTHHS\n1xMAEXEeOCvpijHOZWZ1klR1Myti3JXjEfE2cK2k+cCzkj5ZsT8kNfSRm11dXe+US6USpVKpYddi\n1ty82n6mKpfLlMvlSTlXzY8ciYizkr4DrAKGJF0ZEafSbajXU7VBYHnusGVkPYXBVK6MjxyzAjgp\naRYwPyKGJQ0Cpdwxy4Hnq11bPnFY+/E7YrOJq3xT3d3dXfe5xptV9f6RAWlJc4EbgQPA08CmVG0T\n8GQqPw1slDRb0tVAB9AXEaeANyWtToPltwFP5Y4ZOdfNZIPtAD3AGkkLJF2evvezdbfUWlxUbGbW\nKOP1OBYDOyRdQpZkdkbEPkkHgN2SNgPHgVsAIqJf0m6gHzgPdMaFT47pBB4B5gJ7ImJvim8Hdko6\nCgwDG9O5TkvaCryU6nWnQXIzM2sgtfongkmKVm+DjS3rpFb7UKLR7te7btG6/h2aeSQREXXdB/bK\ncTMzK8SJw8zMCnHiMDOzQpw4zMysECcOMzMrxInDzMwKqXnluJm1r2qr8z1F10bjxGFmVF/zYVad\nb1WZmVkh7nFY0/DDDM1agxOHNRk/9tus2flWlZmZFeLEYWZmhThxmJlZIU4cZmZWiBOHmZkV4sRh\nZmaFOHGYmVkhThxmZlbIuIlD0nJJL0h6VdKPJN2R4l2SBiQdSNunc8dskXRU0mFJa3LxVZIOpX0P\n5OJzJD2e4vslXZXbt0nSkbTdPnlNN7OxSKq6mWm8J2BKuhK4MiIOSnof8DKwAbgFeCsivlFRfyXw\nGPBRYCnwHNARESGpD/hiRPRJ2gN8KyL2SuoEPhgRnZJuBT4bERslLQReAlal078MrIqIM7nvF36K\nZ3vI/iiNtnK82kP4XLcRdf371h4kERF1vRMYt8cREaci4mAq/xR4jSwhQPVnQawHdkXEuYg4DhwD\nVktaDMyLiL5U71GyBASwDtiRyk8AN6TyTUBPRJxJyaIXWFugfWZmNskKjXFI+gBwHbA/hb4k6RVJ\n2yUtSLElwEDusAGyRFMZH+RCAloKnACIiPPAWUlXjHEuMzNrkJofcphuU/05cGdE/FTSNuCP0u6t\nwNeBzZN/iePr6up6p1wqlSiVSo24DCvA98rNple5XKZcLk/KuWpKHJLeQ3YL6b9ExJMAEfF6bv9D\nwDPp5SCwPHf4MrKewmAqV8ZHjlkBnJQ0C5gfEcOSBoFS7pjlwPOV15dPHNZK/OFBZtOl8k11d3d3\n3eeqZVaVgO1Af0R8MxdfnKv2WeBQKj8NbJQ0W9LVQAfQFxGngDclrU7nvA14KnfMplS+GdiXyj3A\nGkkLJF0O3Ag8W0c7zcxsktTS4/g48Hngh5IOpNgfAJ+TdC3Z28YfA18AiIh+SbuBfuA80Jmb9tQJ\nPALMBfZExN4U3w7slHQUGAY2pnOdlrSVbGYVQHd+RpWZmU2/cafjNjtPx21N1afeNvc0VNfN4v59\naw9TOh3XzMwsz4nDzMwKceIwM7NCnDjMzKyQmhcAmplB9cWbHjCfWZw4zKwgL9yc6XyryszMCnHi\nMDOzQpw4zMysEI9x2JTzk3DN2osTh00TD6iatQvfqjIzs0KcOMzMrBAnDjMzK8SJw8zMCnHiMDOz\nQpw4zMysEE/HNbMJG22tjh9+2J6cOMxsEoz2kbTWjsa9VSVpuaQXJL0q6UeS7kjxhZJ6JR2R1CNp\nQe6YLZKOSjosaU0uvkrSobTvgVx8jqTHU3y/pKty+zal73FE0u2T13QzM6tHLWMc54DfjYh/AHwM\n+PeSfgO4C+iNiGuAfek1klYCtwIrgbXAg7rQj90GbI6IDqBD0toU3wwMp/j9wH3pXAuBu4Hr03ZP\nPkGZmdn0GzdxRMSpiDiYyj8FXgOWAuuAHanaDmBDKq8HdkXEuYg4DhwDVktaDMyLiL5U79HcMflz\nPQHckMo3AT0RcSYizgC9ZMnIzMwapNAYh6QPANcBLwKLImIo7RoCFqXyEmB/7rABskRzLpVHDKY4\n6esJgIg4L+mspCvSuQaqnMuakB9maDYz1Jw4JL2PrDdwZ0S8lf8jEREhqWHTJ7q6ut4pl0olSqVS\noy7FPEhq1pTK5TLlcnlSzlVT4pD0HrKksTMinkzhIUlXRsSpdBvq9RQfBJbnDl9G1lMYTOXK+Mgx\nK4CTkmYB8yNiWNIgUModsxx4vvL68onDzMwuVvmmuru7u+5z1TKrSsB2oD8ivpnb9TSwKZU3AU/m\n4hslzZZ0NdAB9EXEKeBNSavTOW8DnqpyrpvJBtsBeoA1khZIuhy4EXi2jnaamdkkqaXH8XHg88AP\nJR1IsS3AvcBuSZuB48AtABHRL2k30A+cBzrjwiqgTuARYC6wJyL2pvh2YKeko8AwsDGd67SkrcBL\nqV53GiQ3M7MGUauv7JQUrd6GdpF1JEcb46j2QU61xFy3lev6d7N5SSIi6hqA9LOqzMysECcOMzMr\nxM+qMrMpU21tj29ftT4nDjObQtXGQ6zV+VaVmZkV4sRhZmaFOHGYmVkhThxmZlaIB8etLn4SrtnM\n5cRhE+AZM2YzkW9VmZlZIU4cZmZWiBOHmZkV4sRhZmaFeHDczKbVaDPy/Ayr1uHEYWbTzJ9L3+p8\nq8rMzApx4jAzs0KcOMzMrJBxE4ekb0saknQoF+uSNCDpQNo+ndu3RdJRSYclrcnFV0k6lPY9kIvP\nkfR4iu+XdFVu3yZJR9J2++Q02czMJqKWHsfDwNqKWADfiIjr0vZdAEkrgVuBlemYB3VhCsU2YHNE\ndAAdkkbOuRkYTvH7gfvSuRYCdwPXp+0eSQvqbKeZmU2ScRNHRHwfeKPKrmrTINYDuyLiXEQcB44B\nqyUtBuZFRF+q9yiwIZXXATtS+QnghlS+CeiJiDMRcQbo5eIEZlNMUtXNzGauiYxxfEnSK5K253oC\nS4CBXJ0BYGmV+GCKk76eAIiI88BZSVeMcS6bdlFlM7OZqt51HNuAP0rlrcDXyW45NURXV9c75VKp\nRKlUatSlmJk1pXK5TLlcnpRz1ZU4IuL1kbKkh4Bn0stBYHmu6jKynsJgKlfGR45ZAZyUNAuYHxHD\nkgaBUu6Y5cDz1a4nnzjMzOxilW+qu7u76z5XXbeq0pjFiM8CIzOungY2Spot6WqgA+iLiFPAm5JW\np8Hy24CncsdsSuWbgX2p3AOskbRA0uXAjcCz9VyvmTU/j6W1jnF7HJJ2AZ8A3i/pBHAPUJJ0LdnN\n7h8DXwCIiH5Ju4F+4DzQGRceQNMJPALMBfZExN4U3w7slHQUGAY2pnOdlrQVeCnV606D5GbWlvzB\nYK1Crf5gMUnR6m1oZtm7vtGeLVTtF30idafze7luK9T17/bUkURE1JWdvXLczMwKceIwM7NCnDjM\nzKwQJw4zMyvEicPMzApx4jAzs0L80bH2Di+4MrNaOHFYBS/CMrOxOXGYWdMarRfshYGN5cRhZk1s\ntFXm1kgeHDczs0KcOMzMrBAnDjMzK8SJw8zMCnHiMDOzQpw4zMysEE/HnYG8QtzMJsKJY8by/Hgz\nq48Th5m1nGq9Zq8mnz7jjnFI+rakIUmHcrGFknolHZHUI2lBbt8WSUclHZa0JhdfJelQ2vdALj5H\n0uMpvl/SVbl9m9L3OCLp9slpspm1vqjYbDrVMjj+MLC2InYX0BsR1wD70mskrQRuBVamYx7UhbcG\n24DNEdEBdEgaOedmYDjF7wfuS+daCNwNXJ+2e/IJyszMGmPcxBER3wfeqAivA3ak8g5gQyqvB3ZF\nxLmIOA4cA1ZLWgzMi4i+VO/R3DH5cz0B3JDKNwE9EXEmIs4AvVycwMzMbJrVOx13UUQMpfIQsCiV\nlwADuXoDwNIq8cEUJ309ARAR54Gzkq4Y41xmZtZAEx4cj4iQ1NCbjF1dXe+US6USpVKpYddiZtaM\nyuUy5XJ5Us5Vb+IYknRlRJxKt6FeT/FBYHmu3jKynsJgKlfGR45ZAZyUNAuYHxHDkgaBUu6Y5cDz\n1S4mnzjMzOxilW+qu7u76z5XvbeqngY2pfIm4MlcfKOk2ZKuBjqAvog4BbwpaXUaLL8NeKrKuW4m\nG2wH6AHWSFog6XLgRuDZOq/XzMwmybg9Dkm7gE8A75d0gmym073AbkmbgePALQAR0S9pN9APnAc6\n48Lk6k7gEWAusCci9qb4dmCnpKPAMLAxneu0pK3AS6ledxoktwK8StzMJptafdGMpGj1NkylLHFU\n+xzx0VaON7Jus16X67ZG3er896E6SUREXe8svXLczNqEH6MzXfx0XDMzK8SJw8zMCnHiMDOzQpw4\nzMysECcOMzMrxInDzMwK8XTcNuGFfmbV+UOfJp8TR1vxPHazi9W+WNBq41tVZmZWiBOHmZkV4sRh\nZmaFOHGYmVkhThxmZlaIZ1WZ2Ywz2vR1T9OtjRNHC/KaDbOJ8tT1iXDiaFmem25mjeExDjMzK2RC\niUPScUk/lHRAUl+KLZTUK+mIpB5JC3L1t0g6KumwpDW5+CpJh9K+B3LxOZIeT/H9kq6ayPWamdnE\nTbTHEUApIq6LiOtT7C6gNyKuAfal10haCdwKrATWAg/qws36bcDmiOgAOiStTfHNwHCK3w/cN8Hr\nNTOzCZqMW1WVN9fXATtSeQewIZXXA7si4lxEHAeOAaslLQbmRURfqvdo7pj8uZ4AbpiE6zUzq0rS\nRZtdbDJ6HM9J+oGkf5diiyJiKJWHgEWpvAQYyB07ACytEh9McdLXEwARcR44K2nhBK/ZzGwUUbFZ\nNROdVfXxiPiJpL8L9Eo6nN8ZESHJ//p18rsdM2tGE0ocEfGT9PVvJP0lcD0wJOnKiDiVbkO9nqoP\nAstzhy8j62kMpnJlfOSYFcBJSbOA+RFxuvI6urq63imXSiVKpdJEmtVkPN/czCauXC5TLpcn5Vyq\nd6WkpMuASyPiLUm/AvQA3cCnyAa075N0F7AgIu5Kg+OPkSWXpcBzwK+nXsmLwB1AH/Ad4FsRsVdS\nJ/ChiPgdSRuBDRGxseI6ol1Xe2Y9jtESR7V1HK1et1mvy3Vnct12/vsSEXW9C51Ij2MR8Jfpdsos\n4L9GRI+kHwC7JW0GjgO3AEREv6TdQD9wHujM/cXvBB4B5gJ7ImJvim8Hdko6CgwD70oaZmZTzY8n\nuVjdPY5m4R7HWLFWq9us1+W6rntx3Vb/u9OoHodNIg+Em1mrcOJoKn7+lJk1Pz+ryszMCnGPw8ys\nDtVuL7f6uEetnDjMzOoyc28tO3FMMw+Cm1mrc+JoCK8GN7PW5cRhZjZJZspiQScOM7NJMzPuJjhx\nTCGPZ5hZO3LimHIzd+aFmWXabequE4eZ2ZRrrzeQXjluZmaFuMcxCTyWYWZFtfIMLCeOSTMzZlOY\n2WRp3b8ZThxmZk2kFQbSnTgK8m0pM5tazT+Q7sRRl+b/wZpZ+2i28RAnjlG4Z2FmzaP6eEijbms1\n/XRcSWslHZZ0VNKXp+h7XLRlospmZtYsGvP3qakTh6RLgf8MrAVWAp+T9BsTOF+BJNEsyo2+gClW\nbvQFTLFyoy9gCpUbfQFTrNzoC6jL6H/jJk+z36q6HjgWEccBJP0ZsB54LV/pq1/96kUHfuUrXxnl\nlK02Ba4MlBp8DVOpjNvXqsq0b9ugVRPHdIzBNnviWAqcyL0eAFZXVrr77p+/6/Xs2TtTyYPYZmaT\nrdkTR033jObNe+Vdr3/xi59OycWYmRmo2RaW5En6GNAVEWvT6y3A2xFxX65O8zbAzKyJRURdt2Ga\nPXHMAv4XcANwEugDPhcRr415oJmZTZmmvlUVEeclfRF4FrgU2O6kYWbWWE3d4zAzs+bT1Os4Kkn6\ne5IO5Lazku6QtFBSr6QjknokLWj0tdZD0hZJr0o6JOkxSXPapW0Aku5MbfuRpDtTrGXbJ+nbkoYk\nHcrFRm1P+vkeTQta1zTmqms3Svv+Vfo/+ktJH66o3w7t+1NJr0l6RdJfSJqf29cy7RulbVtTuw5K\n2idpeW5fsbZFREtuZEnvJ8By4E+A30/xLwP3Nvr66mjPB4C/Auak148Dm9qhbenaPwgcAt5Ldtux\nF/i1Vm4f8JvAdcChXKxqe8gWsB4E3pN+1seASxrdhjra9/eBa4AXgA/n4u3SvhtHrhu4t1V/fqO0\nbV6u/CXgoXrb1lI9jgqfIlsceAJYB+xI8R3AhoZdVf3eBM4Bl6VJAZeRTQhoh7ZB9gfnxYj4eUT8\nEvge8C9p4fZFxPeBNyrCo7VnPbArIs5FtqD1GNkC16ZVrX0RcTgijlSp3i7t642It9PLF4FlqdxS\n7RulbW/lXr4P+L+pXLhtrZw4NgK7UnlRRAyl8hCwqDGXVL+IOA18HfhrsoRxJiJ6aYO2JT8CfjPd\nyrkM+AzZL2W7tG/EaO1ZQraAdcQA2QLXdtGO7fttYE8qt0X7JP0nSX8N/BbwtRQu3LaWTBySZgP/\nHPhvlfsi63u13Ii/pF8D/gNZV3EJ8D5Jn8/XadW2QfZOFbgP6AG+S9Y1/mVFnZZtXzU1tKdt2jqK\nlm2fpD8E/jYiHhujWsu1LyL+MCJWAA8D3xyr6ljnacnEAXwaeDki/ia9HpJ0JYCkxcDrDbuy+n0E\n+J8RMRwR54G/AP4RcKoN2gZARHw7Ij4SEZ8g60YfoT1+dnmjtWeQbDxuxLIUaxdt0z5Jv0XWI/43\nuXDbtC95DPhoKhduW6smjs9x4TYVwNNkA8mkr09O+xVN3GHgY5LmKnuc5aeAfuAZWr9tAEj61fR1\nBfAvyP7ztsPPLm+09jwNbJQ0W9LVQAfZgtZWll913Bbtk7QW+D1gfUTkH4LX8u2T1JF7uR44kMrF\n29bo0f86Zgv8CtmgTn6GwELgObJ3sD3AgkZfZ51t+33gVbLZRzvIZjm0RdtS+/57at9B4JOt/rMj\ne/NyEvhbsodx/tux2gP8AdnA42HgpkZffx3t+22ywf4TwP8DTgHfbbP2HQX+T/qjegB4sBXbN0rb\n/jz9bTkIPAH8ar1t8wJAMzMrpFVvVZmZWYM4cZiZWSFOHGZmVogTh5mZFeLEYWZmhThxmJlZIU4c\nZmZWiBOHmZkV8v8B39nxHAflLHEAAAAASUVORK5CYII=\n", 356 | "text/plain": [ 357 | "" 358 | ] 359 | }, 360 | "metadata": {}, 361 | "output_type": "display_data" 362 | } 363 | ], 364 | "source": [ 365 | "r = scipy.stats.poisson.rvs(mu, size=10000000)\n", 366 | "plt.hist(r, range=(70, 129), bins=60);" 367 | ] 368 | }, 369 | { 370 | "cell_type": "code", 371 | "execution_count": 161, 372 | "metadata": { 373 | "collapsed": false 374 | }, 375 | "outputs": [ 376 | { 377 | "data": { 378 | "text/plain": [ 379 | "array([[ 8.65767291],\n", 380 | " [ 9.93155374],\n", 381 | " [ 8.05602327],\n", 382 | " [ 4.65563379],\n", 383 | " [ 9.91815902],\n", 384 | " [ 6.67549771],\n", 385 | " [ 5.63729741],\n", 386 | " [ 6.94743734],\n", 387 | " [ 8.19205175],\n", 388 | " [ 4.92907641],\n", 389 | " [ 7.26686247],\n", 390 | " [ 9.04216567],\n", 391 | " [ 5.73742076],\n", 392 | " [ 2.37046826],\n", 393 | " [ 8.8779433 ],\n", 394 | " [ 3.66722037],\n", 395 | " [ 6.21479708],\n", 396 | " [ 0.35393213],\n", 397 | " [ 9.19963022],\n", 398 | " [ 3.70105519],\n", 399 | " [ 8.79884274],\n", 400 | " [ 8.54001686],\n", 401 | " [ 2.61756854],\n", 402 | " [ 3.54812033],\n", 403 | " [ 0.84817879],\n", 404 | " [ 3.66985662],\n", 405 | " [ 9.30266099],\n", 406 | " [ 9.68064496],\n", 407 | " [ 7.09215211],\n", 408 | " [ 6.39424195],\n", 409 | " [ 1.20508923],\n", 410 | " [ 9.81181468],\n", 411 | " [ 7.67728576],\n", 412 | " [ 9.75189981],\n", 413 | " [ 7.48424559],\n", 414 | " [ 9.43891582],\n", 415 | " [ 1.11173315],\n", 416 | " [ 2.51951983],\n", 417 | " [ 1.34964893],\n", 418 | " [ 5.13973376],\n", 419 | " [ 3.11963892],\n", 420 | " [ 4.91247467],\n", 421 | " [ 6.89985512],\n", 422 | " [ 1.60551467],\n", 423 | " [ 0.62415829],\n", 424 | " [ 1.43144539],\n", 425 | " [ 5.62317732],\n", 426 | " [ 2.29765186],\n", 427 | " [ 8.69596333],\n", 428 | " [ 7.03364533],\n", 429 | " [ 8.46430813],\n", 430 | " [ 8.45238233],\n", 431 | " [ 5.62715734],\n", 432 | " [ 0.32914981],\n", 433 | " [ 7.96619529],\n", 434 | " [ 6.24609853],\n", 435 | " [ 3.1059534 ],\n", 436 | " [ 3.16184206],\n", 437 | " [ 7.59913242],\n", 438 | " [ 8.02637774],\n", 439 | " [ 8.40811318],\n", 440 | " [ 4.43229319],\n", 441 | " [ 5.44577462],\n", 442 | " [ 9.37307882],\n", 443 | " [ 5.66335961],\n", 444 | " [ 5.77963214],\n", 445 | " [ 9.03951836],\n", 446 | " [ 5.8654611 ],\n", 447 | " [ 5.2729127 ],\n", 448 | " [ 0.76436223],\n", 449 | " [ 3.37652697],\n", 450 | " [ 3.22517353],\n", 451 | " [ 1.10891106],\n", 452 | " [ 2.01600544],\n", 453 | " [ 3.41546483],\n", 454 | " [ 6.1128611 ],\n", 455 | " [ 4.54594137],\n", 456 | " [ 2.35748124],\n", 457 | " [ 6.06883529],\n", 458 | " [ 5.47178252],\n", 459 | " [ 8.37787816],\n", 460 | " [ 1.12974238],\n", 461 | " [ 6.53433623],\n", 462 | " [ 5.93882657],\n", 463 | " [ 8.17360821],\n", 464 | " [ 7.65432636],\n", 465 | " [ 6.7644442 ],\n", 466 | " [ 5.16266629],\n", 467 | " [ 6.39352962],\n", 468 | " [ 0.99015929],\n", 469 | " [ 1.9083384 ],\n", 470 | " [ 1.48662298],\n", 471 | " [ 7.10067227],\n", 472 | " [ 5.07503881],\n", 473 | " [ 2.239591 ],\n", 474 | " [ 5.60478646],\n", 475 | " [ 7.98470634],\n", 476 | " [ 0.69679701],\n", 477 | " [ 0.80884546],\n", 478 | " [ 6.6620451 ]])" 479 | ] 480 | }, 481 | "execution_count": 161, 482 | "metadata": {}, 483 | "output_type": "execute_result" 484 | } 485 | ], 486 | "source": [ 487 | "N=100\n", 488 | "scipy.stats.uniform.rvs(0, DX, ((N,1)))" 489 | ] 490 | }, 491 | { 492 | "cell_type": "code", 493 | "execution_count": null, 494 | "metadata": { 495 | "collapsed": true 496 | }, 497 | "outputs": [], 498 | "source": [] 499 | }, 500 | { 501 | "cell_type": "code", 502 | "execution_count": null, 503 | "metadata": { 504 | "collapsed": true 505 | }, 506 | "outputs": [], 507 | "source": [] 508 | }, 509 | { 510 | "cell_type": "code", 511 | "execution_count": null, 512 | "metadata": { 513 | "collapsed": true 514 | }, 515 | "outputs": [], 516 | "source": [] 517 | } 518 | ], 519 | "metadata": { 520 | "kernelspec": { 521 | "display_name": "Python 3", 522 | "language": "python", 523 | "name": "python3" 524 | }, 525 | "language_info": { 526 | "codemirror_mode": { 527 | "name": "ipython", 528 | "version": 3 529 | }, 530 | "file_extension": ".py", 531 | "mimetype": "text/x-python", 532 | "name": "python", 533 | "nbconvert_exporter": "python", 534 | "pygments_lexer": "ipython3", 535 | "version": "3.5.1" 536 | } 537 | }, 538 | "nbformat": 4, 539 | "nbformat_minor": 0 540 | } 541 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Tutorials 2 | 3 | Just a collection of tutorials I am working on. They may or may be turned into my next book. I'm wavering between linear algebra, machine learning, computer vision, and/or some type of bayesian inference book. Maybe multi-track tracking? 4 | 5 | 6 | Most of these are probably very incomplete and not ready for public consumption. They are me working out ideas, not a complete attempt to provide tutorials that are readable by others. As things develop I'll probably change this notice and/or list the ones that are ready for public consumption. I think it is worth making public so early - it is often hard to go from abstract math to working code, and seeing that is worthwhile even if the explanatory text is bad or missing. 7 | 8 | 9 | ## Licence 10 | 11 | Creative Commons License
tutorials by Roger R. Labbe Jr is licensed under a Creative Commons Attribution 4.0 International License. -------------------------------------------------------------------------------- /em.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Fri Aug 28 16:16:07 2015 4 | 5 | @author: rlabbe 6 | """ 7 | 8 | import numpy as np 9 | import math 10 | from scipy.misc import comb 11 | 12 | 13 | def coin_likelihood(N, d, bias): 14 | return comb(N, d, exact=True) * bias**d * (1-bias)**(N-d) 15 | 16 | #### E-M Coin Toss Example as given in the EM tutorial paper by Do and Batzoglou* #### 17 | 18 | # represent the experiments 19 | experiments = np.array([5,9,8,4,7]) 20 | num_flips = 10 21 | # initialise the pA(heads) and pB(heads) 22 | theta = np.array([.6, .5]) 23 | 24 | # E-M begins! 25 | delta = 0.0001 26 | improvement = float('inf') 27 | likelihoods = np.zeros(2) 28 | expectation = np.zeros((5,2,2)) 29 | EA = np.zeros((5,2)) 30 | EB = np.zeros((5,2)) 31 | 32 | while improvement > delta: 33 | for i, num_heads in enumerate(experiments): 34 | for coin in range(2): 35 | # likelihood of e given coin bias 36 | likelihoods[coin] = coin_likelihood(10, num_heads, theta[coin]) 37 | 38 | likelihoods /= sum(likelihoods) # normalize 39 | 40 | e = [num_heads, num_flips-num_heads] 41 | EA = np.dot(likelihoods[0], e) 42 | EB = np.dot(likelihoods[1], e) 43 | 44 | for coin in range(2): 45 | f = np.dot(likelihoods[coin], e) 46 | expectation[i,:,coin] = np.dot(likelihoods[coin], e) 47 | 48 | expectation[i] = np.outer(likelihoods, [num_heads, num_flips-num_heads]) 49 | 50 | old_theta = theta.copy() 51 | for coin in range(2): 52 | theta[coin] = sum(expectation[:, coin, 0]) / np.sum(expectation[:, coin,:]) 53 | 54 | 55 | 56 | improvement = max(abs(theta - old_theta)) 57 | 58 | print(theta) 59 | -------------------------------------------------------------------------------- /empty-closeup.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rlabbe/tutorials/a338c1319d0c5b6a07dfa9da01bc025c1ffa1446/empty-closeup.png -------------------------------------------------------------------------------- /empty-table.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rlabbe/tutorials/a338c1319d0c5b6a07dfa9da01bc025c1ffa1446/empty-table.jpg -------------------------------------------------------------------------------- /license.html: -------------------------------------------------------------------------------- 1 | Creative Commons License
tutorials by Roger R. Labbe Jr is licensed under a Creative Commons Attribution 4.0 International License. -------------------------------------------------------------------------------- /table1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rlabbe/tutorials/a338c1319d0c5b6a07dfa9da01bc025c1ffa1446/table1.jpg -------------------------------------------------------------------------------- /test.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 3, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "data": { 10 | "application/vnd.jupyter.widget-view+json": { 11 | "model_id": "2eebbda7e22e4c04b26f0379c785c6f4", 12 | "version_major": 2, 13 | "version_minor": 0 14 | }, 15 | "text/plain": [ 16 | "interactive(children=(IntSlider(value=1, description='x', max=3, min=-1), Output()), _dom_classes=('widget-int…" 17 | ] 18 | }, 19 | "metadata": {}, 20 | "output_type": "display_data" 21 | } 22 | ], 23 | "source": [ 24 | "from ipywidgets import interact\n", 25 | "\n", 26 | "def ff(x):\n", 27 | " print(x)\n", 28 | " \n", 29 | "interact(ff, x=1);" 30 | ] 31 | } 32 | ], 33 | "metadata": { 34 | "kernelspec": { 35 | "display_name": "Python 3", 36 | "language": "python", 37 | "name": "python3" 38 | }, 39 | "language_info": { 40 | "codemirror_mode": { 41 | "name": "ipython", 42 | "version": 3 43 | }, 44 | "file_extension": ".py", 45 | "mimetype": "text/x-python", 46 | "name": "python", 47 | "nbconvert_exporter": "python", 48 | "pygments_lexer": "ipython3", 49 | "version": "3.7.6" 50 | }, 51 | "widgets": { 52 | "application/vnd.jupyter.widget-state+json": { 53 | "state": { 54 | "002a74adfc66410388863e0fe6b5fa11": { 55 | "model_module": "@jupyter-widgets/controls", 56 | "model_module_version": "1.5.0", 57 | "model_name": "IntSliderModel", 58 | "state": { 59 | "description": "x", 60 | "layout": "IPY_MODEL_f3264372c12a4ee49eeef57b4b8f41b6", 61 | "max": 3, 62 | "min": -1, 63 | "style": "IPY_MODEL_752bcaf02718496c97a2f28c4061e1d1", 64 | "value": 1 65 | } 66 | }, 67 | "092d2623f7b244afa434a6f13601cf62": { 68 | "model_module": "@jupyter-widgets/base", 69 | "model_module_version": "1.2.0", 70 | "model_name": "LayoutModel", 71 | "state": {} 72 | }, 73 | "0a0e91bfc2da463981e4e2956cc045f7": { 74 | "model_module": "@jupyter-widgets/controls", 75 | "model_module_version": "1.5.0", 76 | "model_name": "IntSliderModel", 77 | "state": { 78 | "description": "x", 79 | "layout": "IPY_MODEL_9d7b69bcc6ef4f13a61f1fe78e37ab8c", 80 | "max": 3, 81 | "min": -1, 82 | "style": "IPY_MODEL_a82d694eb3ff4e85a77c623a6631fc4d", 83 | "value": 1 84 | } 85 | }, 86 | "18931e48b7b7441bb8a508e08dec51e2": { 87 | "model_module": "@jupyter-widgets/controls", 88 | "model_module_version": "1.5.0", 89 | "model_name": "VBoxModel", 90 | "state": { 91 | "_dom_classes": [ 92 | "widget-interact" 93 | ], 94 | "children": [ 95 | "IPY_MODEL_0a0e91bfc2da463981e4e2956cc045f7", 96 | "IPY_MODEL_fa7390fd555f434d8bc7843a3d94adda" 97 | ], 98 | "layout": "IPY_MODEL_97f417472936493281bab3026777c6da" 99 | } 100 | }, 101 | "2eebbda7e22e4c04b26f0379c785c6f4": { 102 | "model_module": "@jupyter-widgets/controls", 103 | "model_module_version": "1.5.0", 104 | "model_name": "VBoxModel", 105 | "state": { 106 | "_dom_classes": [ 107 | "widget-interact" 108 | ], 109 | "children": [ 110 | "IPY_MODEL_002a74adfc66410388863e0fe6b5fa11", 111 | "IPY_MODEL_7f7baaea389a4e09b6f9b986e4bf4341" 112 | ], 113 | "layout": "IPY_MODEL_f188eedb5c00405d9a0d72239acede0d" 114 | } 115 | }, 116 | "34584a2108474439849b8faf8f757adf": { 117 | "model_module": "@jupyter-widgets/base", 118 | "model_module_version": "1.2.0", 119 | "model_name": "LayoutModel", 120 | "state": {} 121 | }, 122 | "5356b50364c04a10844432a3d434a383": { 123 | "model_module": "@jupyter-widgets/base", 124 | "model_module_version": "1.2.0", 125 | "model_name": "LayoutModel", 126 | "state": {} 127 | }, 128 | "6dff5ef3dca84e6f83258145bd3c2ac2": { 129 | "model_module": "@jupyter-widgets/base", 130 | "model_module_version": "1.2.0", 131 | "model_name": "LayoutModel", 132 | "state": {} 133 | }, 134 | "752bcaf02718496c97a2f28c4061e1d1": { 135 | "model_module": "@jupyter-widgets/controls", 136 | "model_module_version": "1.5.0", 137 | "model_name": "SliderStyleModel", 138 | "state": { 139 | "description_width": "" 140 | } 141 | }, 142 | "7f7baaea389a4e09b6f9b986e4bf4341": { 143 | "model_module": "@jupyter-widgets/output", 144 | "model_module_version": "1.0.0", 145 | "model_name": "OutputModel", 146 | "state": { 147 | "layout": "IPY_MODEL_9dbefca623a146f48ef214bea270ffd5", 148 | "outputs": [ 149 | { 150 | "name": "stdout", 151 | "output_type": "stream", 152 | "text": "1\n" 153 | } 154 | ] 155 | } 156 | }, 157 | "8ac4478171df4278bf4f20e23d4593b6": { 158 | "model_module": "@jupyter-widgets/controls", 159 | "model_module_version": "1.5.0", 160 | "model_name": "SliderStyleModel", 161 | "state": { 162 | "description_width": "" 163 | } 164 | }, 165 | "97f417472936493281bab3026777c6da": { 166 | "model_module": "@jupyter-widgets/base", 167 | "model_module_version": "1.2.0", 168 | "model_name": "LayoutModel", 169 | "state": {} 170 | }, 171 | "9d7b69bcc6ef4f13a61f1fe78e37ab8c": { 172 | "model_module": "@jupyter-widgets/base", 173 | "model_module_version": "1.2.0", 174 | "model_name": "LayoutModel", 175 | "state": {} 176 | }, 177 | "9dbefca623a146f48ef214bea270ffd5": { 178 | "model_module": "@jupyter-widgets/base", 179 | "model_module_version": "1.2.0", 180 | "model_name": "LayoutModel", 181 | "state": {} 182 | }, 183 | "a181a813d53640b7a09b0fdf6182e18c": { 184 | "model_module": "@jupyter-widgets/controls", 185 | "model_module_version": "1.5.0", 186 | "model_name": "IntSliderModel", 187 | "state": { 188 | "description": "x", 189 | "layout": "IPY_MODEL_092d2623f7b244afa434a6f13601cf62", 190 | "max": 3, 191 | "min": -1, 192 | "style": "IPY_MODEL_8ac4478171df4278bf4f20e23d4593b6", 193 | "value": -1 194 | } 195 | }, 196 | "a82d694eb3ff4e85a77c623a6631fc4d": { 197 | "model_module": "@jupyter-widgets/controls", 198 | "model_module_version": "1.5.0", 199 | "model_name": "SliderStyleModel", 200 | "state": { 201 | "description_width": "" 202 | } 203 | }, 204 | "f188eedb5c00405d9a0d72239acede0d": { 205 | "model_module": "@jupyter-widgets/base", 206 | "model_module_version": "1.2.0", 207 | "model_name": "LayoutModel", 208 | "state": {} 209 | }, 210 | "f3264372c12a4ee49eeef57b4b8f41b6": { 211 | "model_module": "@jupyter-widgets/base", 212 | "model_module_version": "1.2.0", 213 | "model_name": "LayoutModel", 214 | "state": {} 215 | }, 216 | "f54d4c4d8e794d1994d0858f2b7e7b5f": { 217 | "model_module": "@jupyter-widgets/output", 218 | "model_module_version": "1.0.0", 219 | "model_name": "OutputModel", 220 | "state": { 221 | "layout": "IPY_MODEL_34584a2108474439849b8faf8f757adf", 222 | "outputs": [ 223 | { 224 | "name": "stdout", 225 | "output_type": "stream", 226 | "text": "-1\n" 227 | } 228 | ] 229 | } 230 | }, 231 | "f7329a984d684ee79528d60303915d3b": { 232 | "model_module": "@jupyter-widgets/controls", 233 | "model_module_version": "1.5.0", 234 | "model_name": "VBoxModel", 235 | "state": { 236 | "_dom_classes": [ 237 | "widget-interact" 238 | ], 239 | "children": [ 240 | "IPY_MODEL_a181a813d53640b7a09b0fdf6182e18c", 241 | "IPY_MODEL_f54d4c4d8e794d1994d0858f2b7e7b5f" 242 | ], 243 | "layout": "IPY_MODEL_5356b50364c04a10844432a3d434a383" 244 | } 245 | }, 246 | "fa7390fd555f434d8bc7843a3d94adda": { 247 | "model_module": "@jupyter-widgets/output", 248 | "model_module_version": "1.0.0", 249 | "model_name": "OutputModel", 250 | "state": { 251 | "layout": "IPY_MODEL_6dff5ef3dca84e6f83258145bd3c2ac2", 252 | "outputs": [ 253 | { 254 | "name": "stdout", 255 | "output_type": "stream", 256 | "text": "1\n" 257 | } 258 | ] 259 | } 260 | } 261 | }, 262 | "version_major": 2, 263 | "version_minor": 0 264 | } 265 | } 266 | }, 267 | "nbformat": 4, 268 | "nbformat_minor": 2 269 | } 270 | --------------------------------------------------------------------------------