├── .gitattributes ├── 010-understanding-DL-models.odp ├── 020-dl-pca-ica.ipynb ├── 030-2d-layers.ipynb ├── 040-bayesian-decision-theory.ipynb ├── 050-parameter-estimation.ipynb ├── PDF ├── 010-understanding-DL-models.pdf ├── 020-dl-pca-ica.pdf ├── 030-2d-layers.pdf ├── 040-bayesian-decision-theory.pdf ├── 050-parameter-estimation.pdf └── 100-Large Scale DL.pdf ├── chapter.png ├── flex.py ├── helpers.py ├── imgclass.py ├── imgimg.py ├── init.py ├── layers.py ├── nb.png ├── note.png ├── roadmodel.py ├── sicon.png ├── summary.png ├── texture.png ├── texture3.png ├── texture4.png └── trainers.py /.gitattributes: -------------------------------------------------------------------------------- 1 | *.ipynb linguist-language=Python 2 | *.tex linguist-detectable=false 3 | *.html linguist-detectable=false 4 | *.pdf linguist-detectable=false 5 | *.md linguist-detectable=false 6 | html/* linguist-vendored 7 | Figures/* linguist-vendored 8 | pdf/* linguist-vendored 9 | -------------------------------------------------------------------------------- /010-understanding-DL-models.odp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmbdev-tutorials/dl-2018/7b42601e90a9cad1d0fb8d399c55c178561313ab/010-understanding-DL-models.odp -------------------------------------------------------------------------------- /050-parameter-estimation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Bayesian vs Maximum Likelihood Parameter Estimation\n", 8 | "\n", 9 | "Thomas Breuel\n", 10 | "\n", 11 | "Deep Learning Summer School 2018, Genoa, Italy\n", 12 | "\n", 13 | "_This is a low-quality PDF conversion; to see the original notebook, please go to: [github.com/tmbdev/dl-2018](github.com/tmbdev/dl-2018)_\n", 14 | "\n", 15 | "The notebooks are directly executable." 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": 1, 21 | "metadata": { 22 | "slideshow": { 23 | "slide_type": "skip" 24 | } 25 | }, 26 | "outputs": [ 27 | { 28 | "name": "stdout", 29 | "output_type": "stream", 30 | "text": [ 31 | "Populating the interactive namespace from numpy and matplotlib\n" 32 | ] 33 | } 34 | ], 35 | "source": [ 36 | "%pylab inline\n", 37 | "from scipy import stats\n", 38 | "from scipy.stats import norm" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": { 44 | "slideshow": { 45 | "slide_type": "slide" 46 | } 47 | }, 48 | "source": [ 49 | "# Tiny Review" 50 | ] 51 | }, 52 | { 53 | "cell_type": "markdown", 54 | "metadata": { 55 | "slideshow": { 56 | "slide_type": "slide" 57 | } 58 | }, 59 | "source": [ 60 | "## Notation for Classification Problems\n", 61 | "\n", 62 | "- $\\omega$: class\n", 63 | "- $p(\\omega, x)$: joint distribution defining the classification problem\n", 64 | "- $D=\\{(\\omega_1, x_1),...\\}$: i.i.d. sample from $p$\n", 65 | "- $\\arg\\max_\\omega P(\\omega | x)$: optimal zero-one loss function classifier\n", 66 | "- $\\hat{\\theta}$: a maximum likelihood estimate\n" 67 | ] 68 | }, 69 | { 70 | "cell_type": "markdown", 71 | "metadata": { 72 | "slideshow": { 73 | "slide_type": "slide" 74 | } 75 | }, 76 | "source": [ 77 | "## Bayes Rule \n", 78 | "\n", 79 | "$$ P( \\omega | x ) = \\frac{ p(x | \\omega) P(\\omega) } { p(x) } $$\n", 80 | "\n", 81 | "- $P(\\omega | x)$ posterior distribution at input $x$\n", 82 | "- $p(x | \\omega)$ class conditional density (\"generative model\")\n", 83 | "- $p(x)$ sample distribution or \"evidence\"\n", 84 | "- $P(\\omega)$ prior class probabilities\n" 85 | ] 86 | }, 87 | { 88 | "cell_type": "markdown", 89 | "metadata": { 90 | "slideshow": { 91 | "slide_type": "slide" 92 | } 93 | }, 94 | "source": [ 95 | "# Maximum Likelihood Estimation\n", 96 | "\n" 97 | ] 98 | }, 99 | { 100 | "cell_type": "markdown", 101 | "metadata": { 102 | "slideshow": { 103 | "slide_type": "slide" 104 | } 105 | }, 106 | "source": [ 107 | "## Neural Networks and Maximum Likelihood Estimation\n", 108 | "\n", 109 | "A DNN is ultimate a parameterized function $f_\\theta(x)$, with $\\theta\\in\\mathbb{R}^{1000...}$.\n", 110 | "\n", 111 | "We're training the DNN on some random training sample $D$ drawn according to some joint distribution $p(y, x)$, and we try to maximize $\\theta$ so that the model matches the dataset best under some loss function." 112 | ] 113 | }, 114 | { 115 | "cell_type": "markdown", 116 | "metadata": { 117 | "slideshow": { 118 | "slide_type": "slide" 119 | } 120 | }, 121 | "source": [ 122 | "## \"One Parameter Generative Network\"\n", 123 | "\n", 124 | "Let's drop all the complexity of classification etc. and just focus on parameter estimation. Let's use a one-parameter model:\n", 125 | "\n", 126 | "$$p_\\theta(x) = (2\\pi)^{-1/2} e^{-\\frac{(x-\\theta)^2}{2}}$$\n", 127 | "\n", 128 | "Our training set $D$ consists of $N$ samples from $p_\\theta(x)$ and we need to find $\\theta$.\n", 129 | "\n", 130 | "You might call this a \"one parameter generative network\"." 131 | ] 132 | }, 133 | { 134 | "cell_type": "markdown", 135 | "metadata": { 136 | "slideshow": { 137 | "slide_type": "slide" 138 | } 139 | }, 140 | "source": [ 141 | "## Maximum Likelihood Estimate\n", 142 | "\n", 143 | "How do we find $\\theta$? You already know a formula for finding $\\theta$:\n", 144 | "\n", 145 | "$$ \\hat{\\theta} = \\frac{1}{N} \\sum_{x\\in D} x $$\n", 146 | "\n", 147 | "This is the _maximum likelihood estimate_ of $\\theta$.\n", 148 | "\n", 149 | "Let's derive this." 150 | ] 151 | }, 152 | { 153 | "cell_type": "markdown", 154 | "metadata": { 155 | "slideshow": { 156 | "slide_type": "slide" 157 | } 158 | }, 159 | "source": [ 160 | "## Univariate Normal Density\n", 161 | "\n", 162 | "For the univariate normal density, $\\theta = (\\mu,\\sigma)$, but we assume $\\sigma=1$ is known. We write:\n", 163 | "\n", 164 | "$$p(x|\\theta) = p_\\theta(x) = (2\\pi)^{-1/2} e^{-\\frac{(x-\\theta)^2}{2}}$$" 165 | ] 166 | }, 167 | { 168 | "cell_type": "markdown", 169 | "metadata": { 170 | "slideshow": { 171 | "slide_type": "slide" 172 | } 173 | }, 174 | "source": [ 175 | "## Univariate Normal Density (2)\n", 176 | "\n", 177 | "Now we can write a density for the entire dataset:\n", 178 | "\n", 179 | "$$ p(D|\\theta) = p(x_1,\\ldots,x_n|\\theta) $$\n", 180 | "$$ = p(x_1|\\theta)\\cdot\\ldots\\cdot p(x_n|\\theta)$$\n", 181 | "$$ = \\prod_{i=1}^n p(x_i|\\theta)$$\n", 182 | "\n", 183 | "We call this the _likelihood_ of the data.\n", 184 | "\n", 185 | "Note that likelihoods are parameterized densities viewed as functions of the parameters." 186 | ] 187 | }, 188 | { 189 | "cell_type": "markdown", 190 | "metadata": { 191 | "slideshow": { 192 | "slide_type": "slide" 193 | } 194 | }, 195 | "source": [ 196 | "## Maximum Likelihood Estimate\n", 197 | "\n", 198 | "The _maximum likelihood estimate_ is given by:\n", 199 | "\n", 200 | "$$\\hat{\\theta} = \\arg\\max_\\theta ~ p(D|\\theta)$$\n", 201 | "\n", 202 | "This seems like a reasonable thing to do: choose the parameter that was\n", 203 | "most likely to produce the data set." 204 | ] 205 | }, 206 | { 207 | "cell_type": "markdown", 208 | "metadata": { 209 | "slideshow": { 210 | "slide_type": "slide" 211 | } 212 | }, 213 | "source": [ 214 | "## Maximum Likelihood Instability\n", 215 | "\n", 216 | "However, in general, there is little reason why the maximum of the likelihood function\n", 217 | "should mean anything.\n", 218 | "\n", 219 | "For example, we can easily modify the likelihood function to put a tiny spike in it\n", 220 | "that moves the maximum somewhere arbitrary without actually changing the problem much at all.\n", 221 | "\n", 222 | "For bimodal densities, the Maximum Likelihood solution can jump between the two peaks with tiny changes in the data." 223 | ] 224 | }, 225 | { 226 | "cell_type": "markdown", 227 | "metadata": { 228 | "slideshow": { 229 | "slide_type": "slide" 230 | } 231 | }, 232 | "source": [ 233 | "## Maximum Likelihood Estimation of the Mean of a Normal Density\n", 234 | "\n", 235 | "Let's derive the ML estimator explicitly.\n", 236 | "\n", 237 | "We're trying to maximize:\n", 238 | "\n", 239 | "$$p(D|\\theta) = \\prod_i (2\\pi)^{-1/2} e^{-\\frac{(x_i-\\theta)^2}{2}}$$\n", 240 | "\n", 241 | "Let's take logarithms\n", 242 | "\n", 243 | "$$l(\\mu) = \\sum_i \\log p(x_i|\\theta) = \\hbox{const} - {1\\over 2} (x_i-\\theta)\\cdot(x_i-\\theta)$$" 244 | ] 245 | }, 246 | { 247 | "cell_type": "markdown", 248 | "metadata": { 249 | "slideshow": { 250 | "slide_type": "slide" 251 | } 252 | }, 253 | "source": [ 254 | "## Maximizing the Likelihood\n", 255 | "\n", 256 | "If $l$ is sufficiently well behaved (it is), then a necessary condition for a local maximum is that the gradient in the parameter to be estimated is 0. The gradient is:\n", 257 | "\n", 258 | "$$\\nabla_\\theta l(\\theta) = \\nabla_\\theta(\\hbox{const} - {1\\over 2} (x_i-\\theta)\\cdot(x_i-\\theta)) = \\sum_i (x_i-\\theta)$$\n", 259 | "\n", 260 | "$$ \\nabla_\\theta l(\\theta)= \\sum_i (x_i-\\theta) = 0 \\Rightarrow \\sum_i x_i = n \\theta $$\n", 261 | "\n", 262 | "$$ \\hat{\\theta} = \\frac{1}{n}\\sum x_i $$\n", 263 | "\n", 264 | "Therefore: the arithmetic mean is the maximum likelihood estimator for the parameter of the normal density with known variance." 265 | ] 266 | }, 267 | { 268 | "cell_type": "markdown", 269 | "metadata": { 270 | "slideshow": { 271 | "slide_type": "slide" 272 | } 273 | }, 274 | "source": [ 275 | "## Connection to Classification\n", 276 | "\n", 277 | "Let's say you have a classification problem in which you know that both classes are distributed according to $p_\\theta(x)$ for two different parameters, $\\theta_1$ and $\\theta_2$.\n", 278 | "\n", 279 | "Recipe for classification:\n", 280 | "- compute $\\theta_1$ and $\\theta_2$ from the training samples\n", 281 | "- apply Bayes formula to derive $P(\\omega | x; \\theta_1, \\theta_2)$ using the parametric densities\n", 282 | "\n", 283 | "Other recipe:\n", 284 | "- write down $P(\\omega | x; \\theta_1, \\theta_2)$\n", 285 | "- use gradient descent to optimize training set error" 286 | ] 287 | }, 288 | { 289 | "cell_type": "markdown", 290 | "metadata": { 291 | "slideshow": { 292 | "slide_type": "slide" 293 | } 294 | }, 295 | "source": [ 296 | "# Bayesian vs Maximum Likelihood\n", 297 | "\n", 298 | "Let's say that the $\\theta$ are the parameters of a DNN classifier $P_\\theta(\\omega|x)$. In typical DL, we estimate a single parameter vector $\\hat{\\theta}$ and use that for classification:\n", 299 | "\n", 300 | "$$\\hat{\\theta} = \\arg\\max_\\theta p(D | \\theta)$$\n", 301 | "\n", 302 | "In Bayesian methods, we average out our classifications over all possible parameter vectors:\n", 303 | "\n", 304 | "$$P(\\omega | x) \\propto \\int P_\\theta(\\omega | x) p(\\theta | D) d\\theta$$" 305 | ] 306 | }, 307 | { 308 | "cell_type": "code", 309 | "execution_count": 2, 310 | "metadata": { 311 | "slideshow": { 312 | "slide_type": "slide" 313 | } 314 | }, 315 | "outputs": [], 316 | "source": [ 317 | "# parameter estimate for nornmal density, unknown mean, sigma=1\n", 318 | "\n", 319 | "def p_x_theta(x, theta, sigma=1):\n", 320 | " return exp(-(x-theta)**2/(2*sigma**2))/sqrt(2*pi*sigma**2)\n", 321 | "def p_theta_D(theta, D):\n", 322 | " return prod([p_x_theta(x, theta) for x in D], axis=0)\n", 323 | "\n", 324 | "D = [0.7, 3.3]" 325 | ] 326 | }, 327 | { 328 | "cell_type": "code", 329 | "execution_count": 3, 330 | "metadata": { 331 | "code_folding": [], 332 | "slideshow": { 333 | "slide_type": "slide" 334 | } 335 | }, 336 | "outputs": [ 337 | { 338 | "data": { 339 | "text/plain": [ 340 | "[]" 341 | ] 342 | }, 343 | "execution_count": 3, 344 | "metadata": {}, 345 | "output_type": "execute_result" 346 | }, 347 | { 348 | "data": { 349 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEACAYAAABVtcpZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYVOWZ9/HvDQhqVOIKDgguKLgGzARBX7UcojZIxF1wwXVCosRkRhPQZAIak4hJnKDEEAUFlE3FBYwKKnSIY0QioKAg4AiKCCMqoqCI3ff7x1MtZdtLdXdVPbX8Ptd1LvpUPafPXXbb93l2c3dERKQ0NYsdgIiIxKMkICJSwpQERERKmJKAiEgJUxIQESlhSgIiIiUsrSRgZmVmtszMlpvZkFrK3G5mK8xskZl1Tb7WyszmmdlCM1tsZsNSyg8zszVmtiB5lGXmI4mISLpa1FfAzJoBo4BewFpgvpk95u7LUsr0Bg5y94PN7BhgNNDD3bea2UnuvsXMmgP/Y2ZPuvuLyUtvc/fbMv6pREQkLenUBLoDK9x9tbtvA6YA/aqV6QdMAHD3eUBrM2uTPN+SLNOKkHRSZ6dZE2IXEZEmSicJtAPeTjlfk3ytrjLvVJUxs2ZmthBYBzzt7vNTyg1ONh+NMbPWDY5eRESaJOsdw+5e6e7dgPbAMWZ2WPKtO4ED3b0rIUGoWUhEJMfq7RMgPNV3SDlvn3ytepn96irj7pvMbA5QBrzm7u+lvH03MKOmm5uZFjcSEWkEd6+3yT2dmsB8oJOZdTSzlkB/YHq1MtOBgQBm1gPY6O7rzWyvqmYeM9sJOBlYljxvm3L9WcCSOj5I0R7Dhg2LHoM+nz6bPl/xHemqtybg7hVmNhiYRUgaY919qZkNCm/7Xe7+hJn1MbOVwGbgsuTl+wLjkyOMmgFT3f2J5Hu3JoeSVgKrgEFpRy0iIhmRTnMQ7v4U0Lnaa3+pdj64husWA0fX8j0Hph+miIhkg2YMR5ZIJGKHkFXF/PmK+bOBPl+psIa0HcVgZp7vMYqI5BszwzPUMSwiIkVKSUBEpIQpCYiIlDAlARGREqYkICJSwpQERERKmJKAiEgJUxIQESlhSgIiIiVMSUBEpIQpCYiIlDAlARGREqYkICJSwpQERERKmJKAiEgJUxIQESlhSgIiIiVMSUBEpIQpCYiIlDAlARGREqYkICJSwtJKAmZWZmbLzGy5mQ2ppcztZrbCzBaZWdfka63MbJ6ZLTSzxWY2LKX87mY2y8xeN7OZZtY6Mx9JRETSVW8SMLNmwCjgVOBwYICZdalWpjdwkLsfDAwCRgO4+1bgJHfvBnQFeptZ9+RlQ4Fn3L0zMBu4PjMfSURE0pVOTaA7sMLdV7v7NmAK0K9amX7ABAB3nwe0NrM2yfMtyTKtgBaAp1wzPvn1eOCMxn4IERFpnHSSQDvg7ZTzNcnX6irzTlUZM2tmZguBdcDT7j4/WWYfd18P4O7rgH0aHr6IiDRFi2zfwN0rgW5mthvwqJkd5u6v1VS0tu8xfPjwL79OJBIkEolMhynScGbgtf7aiuRUeXk55eXlDb7OvJ5fYjPrAQx397Lk+VDA3X1ESpnRwBx3n5o8XwacWPWkn1Luv4DN7n6bmS0FEu6+3szaJq8/tIb7e30xikShJCB5zMxwd6uvXDrNQfOBTmbW0cxaAv2B6dXKTAcGJm/cA9iY/OO+V9WoHzPbCTgZWJZyzaXJry8BHksjFhERyaB6m4PcvcLMBgOzCEljrLsvNbNB4W2/y92fMLM+ZrYS2Axclrx8X2B8coRRM2Cquz+RfG8E8ICZXQ6sBs7L7EcTEZH61NscFJuagyRfbNkCU6fCfffBhx/CwkVGt65O587w/e/DSSeFFiKRfJDJ5iCRkvbee3DNNbDffvDww/DjH8M994T3xo6FE04Ir3XpAnfcARUVceMVaQjVBETqsGABnHUWnH46XHstdOyY8mZKx7A7/OMfMHQo7LorTJwI3/xmnJhFQDUBkSa7/3449VT4/e/h9turJYBqzODYY+HZZ+Ggg6B7d3itpoHQInlGNQGRGtx4Y2j7f/RROOKIWgrVMUR0/Hi47jp48EHQtBaJId2agJKASDX33gu//nVo3tl77zoK1jNP4Nln4YIL4O9/h0MOyXycInVREhBphPJyOP98mDsXOneup3Aak8XGjIERI+CFF2DPPTMWpki9lAREGmj5cjj+eJg8Gf7t39K4IM0Zwz/9KcyfD7NmQcuWTY9TJB1KAiIN8MkncPTR8LOfwZVXpnlRmkmgogLOOQfatIHRo5sWp0i6lAREGuCaa2DTJhg3rgEXNWDtoI8/hiOPhLvvhpNPblSIIg2iJCCSpv/5Hzj3XFiyBPbYowEXNnABuaeegh/+EBYvhl12aXicIg2heQIiafjsM7jiijDTt0EJoBHKysLs4htuyO59RBpCNQEpaTfcEDqEH3qoERc3YinpDz4I8w4efBCOO64R9xRJk5qDROqxaFGYEfzyy9C2bSO+QSP3E5g2DX7+83DfVq0acV+RNKg5SKQO7mEtoBtvbGQCaIKzzw5LS2ikkOQD1QSkJM2aBT/6Ebz6KrRo7CarTdhZbPFi+O53Q1NU69aNvL9IHVQTEKlFZSUMGQK//W0TEkATHXkk9O4dFqcTiUk1ASk5kyaFVUH/8Y8mbgLTxD2G33oLunULQ1P33bcJcYjUQB3DIjXYujVs/jJuHJx4YhO/WQY2mr/uujBbWf0DkmlKAiI1GDkSnn4aHn88A98sA0ng/fdDUnruuTQWrBNpACUBkWo++wwOOABmzoSjjsrAN8xAEoCwbPXrr8OECRmISSQp3SQQqVtMJPfGj4d//dcMJYAMuvrqMGR09eq6dy8TyQbVBKQkfPFFaG6ZMCGDM3UzVBOAMFrp009Dh7VIJmiIqEiKadPgX/4lf5dq+MlPwp7G770XOxIpNWklATMrM7NlZrbczIbUUuZ2M1thZovMrGvytfZmNtvMXjWzxWZ2TUr5YWa2xswWJI+yzHwkka9yh1tugaFDY0dSu333DSuZ3nFH7Eik1NTbHGRmzYDlQC9gLTAf6O/uy1LK9AYGu/tpZnYMMNLde5hZW6Ctuy8ys12Al4B+7r7MzIYBH7v7bfXcX81B0iQzZ4ahmK+80sR5AdVlsDkIYMUKOPZYePNNLTUtTZfJ5qDuwAp3X+3u24ApQL9qZfoBEwDcfR7Q2szauPs6d1+UfP0TYCnQLjXONO4v0iS33BLa3DOaALLg4IPhpJPCxjMiuZJOEmgHvJ1yvoav/iGvqcw71cuY2f5AV2BeysuDk81HY8xMK6hIxv3zn/C//xs2jy8EQ4bAf/936MgWyYWcDBFNNgU9BPw4WSMAuBO4yd3dzG4GbgOuqOn64cOHf/l1IpEgkUhkNV4pHn/6E1x1FeywQ+xI0vPtb8N++8GMGXDmmbGjkUJSXl5OeXl5g69Lp0+gBzDc3cuS50MBd/cRKWVGA3PcfWryfBlworuvN7MWwOPAk+4+spZ7dARmuPvXRnCrT0Aaa8OG0MSyYgXstVcWbpDhPoEqkyfDmDHw7LMZ/9ZSQjLZJzAf6GRmHc2sJdAfmF6tzHRgYPLGPYCN7r4++d49wGvVE0Cy07jKWcCSNGIRSdvYsXDGGVlKAFl09tnw2mvhEMm2tCaLJYdvjiQkjbHufouZDSLUCO5KlhkFlAGbgUvdfaGZHQfMBRYDnjxucPenzGwCoY+gElgFDEpJHKn3Vk1AGqyiIszCnTYtNLFkRZZqAgDDhoV1hUaNysq3lxKgtYOkpE2fDr/5DbzwQhZvksUk8M47Yc+BVatgt92ycgspcpoxLCVt1CgYPDh2FI3Xrl3YeUyLykm2qSYgRef118NeAatXZ3kj9yzWBADmzoVBg0LfQL7PcZD8o5qAlKzRo+Hyy7OcAHLg+OPD9pdz5sSORIqZagJSVLZuhfbtYd48OPDALN8syzUBCKuKzpsHEydm9TZShFQTkJL06KPwrW/lIAHkyEUXwV//Ch9+GDsSKVZKAlJUxo6FK2qcd16Y9tgDevdWTUCyR81BUjRWrQo7h61ZAzvumIMb5qA5CMLM4WuvhYUL1UEs6VNzkJSce++FCy/MUQLIoZNOgk2bYMGC2JFIMVISkKJQUQH33FNcTUFVmjULo53GjIkdiRQjJQEpCrNmhd258m0T+Uy59FKYOhW2bIkdiRQbJQEpCsXWIVxd+/bQsyc8+GDsSKTYqGNYCt6GDdCpU5gh3DqXWxPlqGO4ysMPhz2INXlM0qGOYSkZU6fCaaflOAFEcNppsHhxSHYimaIkIAVvwgQYODB2FNnXqlXYJvO++2JHIsVESUAK2rJl8Pbb0KtX7EhyY+DAkPTUQiqZoiQgBW3ChDA3oEVOdsuOr3v30BUxb17sSKRYKAlIwaqsDE0jl1wSO5LcMQufd/z42JFIsVASkIJVXg577w1HHBE7kty66CJ44IGwYqpIUykJSMEqlQ7h6jp0CCulPv547EikGCgJSEHavBkeewwGDIgdSRxVHcQiTaUkIAXpkUfg2GOhTZvYkcRx9tnwt7/Be+/FjkQKnZKAFKSJE0PbeKnaddewz4CWkZCmUhKQgvN//wcvvACnnx47krguvFCbzUjTpZUEzKzMzJaZ2XIzG1JLmdvNbIWZLTKzrsnX2pvZbDN71cwWm9k1KeV3N7NZZva6mc00syKf9C+ZMnUq9O0L3/hG7EjiOvVUWL4c3nwzdiRSyOpNAmbWDBgFnAocDgwwsy7VyvQGDnL3g4FBwOjkW18A/+nuhwM9gatTrh0KPOPunYHZwPUZ+DxSAiZODE/BpW6HHeDcc2HSpNiRSCFLpybQHVjh7qvdfRswBehXrUw/YAKAu88DWptZG3df5+6Lkq9/AiwF2qVcUzXlZTxwRpM+iZSEN94IT77f/W7sSPJDVZOQlpGQxkonCbQD3k45X8P2P+S1lXmnehkz2x/oCryQfGkfd18P4O7rgH3SDVpK16RJcN55pbNMRH169gwbzbz8cuxIpFDl5H8lM9sFeAj4sbtvrqVYrc8yw4cP//LrRCJBIpHIZHhSINzDU++4cbEjyR/NmsEFF4T/Ll27xo5GYiovL6e8vLzB19W7qYyZ9QCGu3tZ8nwo4O4+IqXMaGCOu09Nni8DTnT39WbWAngceNLdR6ZcsxRIJMu0TV5/aA3316YyAsBLL4WllFesCGvoRJfjTWVq8+qroZN49Wpo3jx2NJIvMrmpzHygk5l1NLOWQH9gerUy04GByRv3ADZWNfUA9wCvpSaAlGsuTX59CfBYGrFICZs4MTz15kUCyCOHHw577QVz58aORApRWttLmlkZMJKQNMa6+y1mNohQI7grWWYUUAZsBi5194VmdhwwF1hMaO5x4AZ3f8rM9gAeAPYDVgPnufvGGu6tmoBQUQH77QezZ0OXLvWXz4k8qQkA3HprqCHdfXfsSCRfpFsT0B7DUhDmzIFrr4UFC2JHkiKPksBbb0G3bvDuu9CyZexoJB9oj2EpKpMnl+5iceno0AEOOwxmzowdiRQaJQHJe59/Dg8/HDqFpXYDBoRkKdIQSgKS92bODE+5HTrEjiS/nXsuPPFEWGZbJF1KApL31BSUnr33DpPHplcfuydSByUByWubN4en23POiR1JYVCTkDSUkoDktenTw+Yxe+8dO5LCcMYZYbOZDz6IHYkUCiUByWtqCmqY3XaDU06BadNiRyKFQklA8taHH4an2jO0vmyDqElIGkJJQPLWtGnhqXbXXWNHUlj69IGFC2Ht2tiRSCFQEpC8paagxtlxR+jXDx54IHYkUgiUBCQvrV0blojo0yd2JIVJTUKSLiUByUsPPBCeZnfcMXYkhalXL1i1KuzEJlIXJQHJS2oKapoWLcLciilTYkci+U5JQPLOG2+Ep9hevWJHUtguuEBNQlI/JQHJO1OmhKdY7SPcND17wscfw+LFsSORfKYkIHlHTUGZ0awZ9O+v2oDUTUlA8srixbBpU1gqQppuwIBQs8qTvW8kDykJSF6ZPDk8vTbTb2ZGfOtb0KoVvPBC7EgkX+l/Nckb7jBpUujQlMwwUwex1E1JQPLG88/DzjuHp1fJnAEDYOpU+OKL2JFIPlISkLwxaRJceGF4epXM6dQJDjgAnn02diSSj5QEJC9s2wYPPqhRQdlywQUhyYpUpyQgeeGZZ+Cgg+DAA2NHUpzOOy9s0PPpp7EjkXyTVhIwszIzW2Zmy81sSC1lbjezFWa2yMy6pbw+1szWm9kr1coPM7M1ZrYgeZQ17aNIIVOHcHa1bQvf+Q48/njsSCTf1JsEzKwZMAo4FTgcGGBmXaqV6Q0c5O4HA4OAP6e8fW/y2prc5u5HJ4+nGvMBpPBt2QIzZoSnVckeNQlJTdKpCXQHVrj7anffBkwB+lUr0w+YAODu84DWZtYmef4c8GEt31tdgMKMGXDMMdCmTexIituZZ8Ls2WHHNpEq6SSBdsDbKedrkq/VVeadGsrUZHCy+WiMmbVOo7wUoYkT1RSUC61bw8kna/9h+aqYS3TdCdzk7m5mNwO3AVfUVHD48OFffp1IJEgkErmIT3Jgw4awj/D998eOpDRceCHcfjtceWXsSCTTysvLKS8vb/B15vUsKmJmPYDh7l6WPB8KuLuPSCkzGpjj7lOT58uAE919ffK8IzDD3Y+q5R61vm9mXl+MUrjuvBPmzi3Qde/NCm5Rnq1boV27sGtbhw6xo5FsMjPcvd4m93Sag+YDncyso5m1BPoD06uVmQ4MTN64B7CxKgFUxUO19n8za5tyehawJI1YpMjcfz9cfHHsKEpHq1ZhmW51EEuVepOAu1cAg4FZwKvAFHdfamaDzOz7yTJPAG+a2UrgL8BVVdeb2STgeeAQM3vLzC5LvnWrmb1iZouAE4H/yOQHk/z3xhuwciWcckrsSErLRRfBffcVXCVGsqTe5qDY1BxUvG68Ed5/P7RRF6QCbA4CqKwME/Mefhi6dau/vBSmTDYHiWSce2gKuuii2JGUnmbNwn93dcYLKAlIJC++GP4Yfec7sSMpTRddFJaXrqiIHYnEpiQgUdx3X/hDpBVD4+jcGdq318qioiQgEWzbFta3v/DC2JGUNjUJCSgJSARPPhmeRLViaFz9+4eVRT/+OHYkEpOSgOTcuHFw6aWxo5B99oETTtAyEqVOSUByasOGsIjZuefGjkQgJOPx42NHITEpCUhOTZ4MffuGxcwkvr59YckSePPN2JFILEoCklNqCsovLVuGLT0nTIgdicSiJCA588or8N57cNJJsSORVFVNQpWVsSORGJQEJGfGj4eBA6F589iRSKpu3WCXXeDvf48dicSgJCA5sW1b2DzmkktiRyLVmYXawLhxsSORGJQEJCeeego6dYKDD44didTkwgvh0Ufhk09iRyK5piQgOXHPPeoQzmdt2sDxx8ODD8aORHJNS0lL1r37Lhx2GLz1Fuy6a+xoMqhAl5KuzfTpcMst8PzzsSORTNBS0pI3xo8Pu1kVVQIoQn36wOrV8OqrsSORXFISkKyqrIQxY+Df/z12JFKfFi3gssvCz0tKh5KAZFV5Oey8s/YNKBSXXx5WFv3ss9iRSK4oCUhW3X13qAVo34DCcOCB0LUrPPJI7EgkV9QxLFmzYUMYFvrmm7D77rGjyYIi6xiu8sADMHp0WOhPCpc6hiW6+++H732vSBNAEevXLywqt3Jl7EgkF5QEJCvcQ1PQlVfGjkQaqlUruPhidRCXCiUByYq5c8PIoBNOiB2JNMagQXDvvbB1a+xIJNvSSgJmVmZmy8xsuZkNqaXM7Wa2wswWmVm3lNfHmtl6M3ulWvndzWyWmb1uZjPNTCvMF5E//QmuukodwoXqkEPgW9/SDOJSUG8SMLNmwCjgVOBwYICZdalWpjdwkLsfDAwC/pzy9r3Ja6sbCjzj7p2B2cD1jfoEknfWroWnnw4rhkrhuvrqkMyluKVTE+gOrHD31e6+DZgC9KtWph8wAcDd5wGtzaxN8vw54MMavm8/oGpju/HAGQ0PX/LRXXeFTcy1e1hh69s3JPQFC2JHItmUThJoB7ydcr4m+VpdZd6poUx1+7j7egB3Xwfsk0Yskue2bQtJ4OqrY0ciTdW8OfzgB6oNFLsWsQNIUeuA6+HDh3/5dSKRIJFI5CAcaYxHHgnLRR9xROxIJBOuvDL0D/zud7DHHrGjkbqUl5dTXl7e4OvqnSxmZj2A4e5eljwfCri7j0gpMxqY4+5Tk+fLgBOrnvTNrCMww92PSrlmKZBw9/Vm1jZ5/aE13F+TxQrIiSeGWsB558WOJAeKdLJYdRdfHDqJr7sudiTSEJmcLDYf6GRmHc2sJdAfmF6tzHRgYPLGPYCNVQmgKp7kUf2aS5NfXwI8lkYskseWLIEVK+DMM2NHIpl09dXw5z9DRUXsSCQb6k0C7l4BDAZmAa8CU9x9qZkNMrPvJ8s8AbxpZiuBvwBXVV1vZpOA54FDzOwtM7ss+dYI4GQzex3oBdySwc8lEfzxj/DDH8IOO8SORDLpmGNgzz3h8cdjRyLZoLWDJCPWr4dDD4Xly2GvvWJHkyMl0hwEMGUK3HlnmAQohUFrB0lO/elPcP75JZQASsw554QNZ+bPjx2JZJpqAtJkW7bA/vvDc8+FkSQlo4RqAgC33QYvvhhqBZL/VBOQnJkwAXr2LLEEUIKuvDLMBF+9OnYkkkmqCUiTVFZCly5hxcmSWyyuxGoCEIaJVlaGWoHkN9UEJCdmzAjLQxx/fOxIJBeuuQbGjYOPPoodiWSKkoA0mnuYSXrttVottFR06AC9e8Nf/hI7EskUNQdJo5WXw/e/D0uXhnVmSk4JNgcBLF4Mp5wCb7wBO+8cOxqpjZqDJOtuvhmuv75EE0AJO/JI6NEj7BwnhU81AWmUf/wDBgwIy0SU7AzhEq0JALz0UtiLeOVK2HHH2NFITVQTkKz61a9g6NASTgAl7tvfDovKjRsXOxJpKtUEpMGqngLfeCNsSl6ySrgmAKoN5jvVBCRrbr4ZfvazEk8AQs+e0KkT3Hdf7EikKVQTkAZZuBD69NHIEKDkawIQFpS77LIwQqxly9jRSCrVBCQrrr8efvELJQAJTjgh7CQ3ZkzsSKSxVBOQtM2ZE9aP0VNfkmoCQKgdnnZaWEZ8l11iRyNVVBOQjHIPo4F+9SslAPmqbt3CtqJ//GPsSKQxVBOQtDz8cEgAL70EzfToEKgm8KWVK8MEsmXLtKdEvki3JqAkIPX64gs44ojwpFdWFjuaPKIk8BVXXQU77QR/+EPsSASUBCSD7r4bJk2C2bO1UNxXKAl8xbp1cPjhoba4//6xoxElAcmIDz8Mewf/9a9hlqikUBL4mptugpdfhmnTYkciSgKSEddcA59/DqNHx44kDykJfM2nn4bawF13wXe/Gzua0qYkIE22eDH06hWGhO65Z+xo8pCSQI0eeyyMJHv5ZY0ki0lDRKVJ3OFHP4Lhw5UApGFOPz30CdxxR+xIJB1pJQEzKzOzZWa23MyG1FLmdjNbYWaLzKxrfdea2TAzW2NmC5KHxp3kkQcegI0bYdCg2JFIoTGDkSPht7+Fd9+NHY3Up97mIDNrBiwHegFrgflAf3dfllKmNzDY3U8zs2OAke7eo65rzWwY8LG717lltZqDcu+jj8KQ0EmTtHdwndQcVKehQ2H1apg8OXYkpSmTzUHdgRXuvtrdtwFTgH7VyvQDJgC4+zygtZm1SeNaDTjMQ9deC337KgFI0/zyl2G46KOPxo5E6pJOEmgHvJ1yvib5Wjpl6rt2cLL5aIyZtU47asmamTPhmWfg1ltjRyKFbuedYexYuPpq+OCD2NFIbVpk6fum84R/J3CTu7uZ3QzcBlxRU8Hhw4d/+XUikSCRSGQgRKlu06awcfyYMbDrrrGjkWJw/PFw9tnwk5/AhAmxoylu5eXllJeXN/i6dPoEegDD3b0seT4UcHcfkVJmNDDH3acmz5cBJwIH1Hdt8vWOwAx3P6qG+6tPIEd+8AOoqNAG4mlTn0BaNm8Om9PfcUdYbVRyI5N9AvOBTmbW0cxaAv2B6dXKTAcGJm/cA9jo7uvrutbM2qZcfxawJI1YJEtmzoQnnoDf/z52JFJsvvGN0Cw0aBBs2BA7Gqmu3uYgd68ws8HALELSGOvuS81sUHjb73L3J8ysj5mtBDYDl9V1bfJb35ocSloJrAI0GDGStWvh0kthyhRorZ4ZyYKTToILLoCBA+Hxx7USbT7RjOES98UXYXp/r17wX/8VO5oCo+agBtm2Lew7cOaZ8NOfxo6m+GnZCEnLL38Jzz8fmoOaN48dTYFREmiwt96C73wnDBvt2TN2NMVNy0ZIvZ55JrTVTpyoBCC50aFDGHjQvz+8/37saASUBErWihVw0UVw333Qpk3saKSUnH46nHcenHtuWKFW4lJzUAn64INQFb/22jAvQBpJzUGNVlER+gb22SfUDLRZUeapOUhq9PnnYfJO375KABJP8+ZhbaqXXoLf/S52NKVNNYES4g5XXhnGaj/8sPoBmkw1gSZbsybUSkeOhLPOih1NcUm3JpCtZSMkz7jDddfBK6/AnDlKAJIf2rcPm9CUlYVJZaeeGjui0qPmoBLgDjfcEDaKnzkTdtkldkQi2x19dBgyevHF4XdUcktJoATceGOYpfn007DHHrGjEfm6Y4+Fhx4KQ0fnzo0dTWlREihi7vDzn4ddwp59FvbaK3ZEIrU74YSwAc0554QHFskNJYEi9fnnYT2gZ56Bv/0tDMUTyXe9esG0aWEOi5aezg11DBehTZvCMNCddgqdwDvvHDsikfQdfzyUl0OfPvD226E/S/MIskc1gSLz+utw3HHQqVMYBqoEIIXo0EPDmlbTpoUa7ebNsSMqXkoCRWTyZPh//w8GD4Y774QWqudJAdt3X/j730PfVvfusHRp/ddIw2myWBHYsiXMAXj66dAJ3K1b7IhKhCaL5YQ73HMPDB0Kf/hDGEqq5qH6admIEvHss3DUUfDRR/DPfyoBSPExgyuuCL/rt94K3/teWJJaMkNJoEB98AFcfnk4Ro4My0FrVzApZkcdBQvRubM3AAAHPElEQVQWQI8eYYLZqFFhITppGiWBAvPpp2HBrc6dw8zfJUu0ebeUjpYt4Re/gOeegwcfhK5dw0RItco1nvoECsTWrWHt/xtvDJ1kv/41dOkSO6oSpz6BqNxDAhg6NMyE//Wvw/BS9RcE2l6ySHzwAYweHaq+RxwBN90UqsOSB5QE8kJFRZhY9pvfwO67h30yzj5bo+OUBApYZWVYP2X8+LDCYr9+8B//EdpEJY8oCeSViopQM/jDH2DVqjC/YODAMGemFCkJFJjKSnjxRZg+PWy20bo1XHIJXHABtG0bOzqpkZJA3lq0KDxETZoUksD554dRRQccEDuy3MloEjCzMuCPhI7kse4+ooYytwO9gc3Ape6+qK5rzWx3YCrQEVgFnOfuH9XwfYs2CaxZE9b1mT0b/vrXsMDb974XfmG7do0dndRLSSDvbdsWlk9/5JFQS9hnnzCQIpEIM+t33TV2hNmTsSRgZs2A5UAvYC0wH+jv7stSyvQGBrv7aWZ2DDDS3XvUda2ZjQDed/dbzWwIsLu7D63h/kWRBDZtCk8nCxaELfWefz6M7T/00HLOPTdB375w4IGxo8y88vJyEolE7DCyotyMRBH8btam2H52FRWhtv3kk+Hh68UXyznyyATdu8O3vx2GnR52GOywQ+xIMyOTO4t1B1a4++rkN54C9AOWpZTpB0wAcPd5ZtbazNoAB9RxbT/gxOT144Fy4GtJoFC4w4cfhgWvVq0KxxtvhKnuS5eG9448MvyyJRIwZEj4hbvppnKuuSYRN/gsKrY/JKnKgUTkGLKp2H52zZuHrSx79gznv/hFOaeckuCf/9w+Ee3NN0OTUZcu4TjgANh//3C0axd2Pys26SSBdsDbKedrCImhvjLt6rm2jbuvB3D3dWYWbbFj97D08tatYRx+1fHJJ9uPTZtg48bw9L5xY9ind8MGeO89WLcO3n0XWrWCDh22/9IceCD07h0Ww+rQAZppVoZI3mjRIuxhcMIJ21/77DNYuTI8uL3+eqg5TJ0aksPateH/8X33Df10e+0Vjj33hG9+M/TjtW4Nu+0W5vDssktIGjvuGFb03WmncH2LFvk1jDVbg6ga8xFrrVeffHL4Q13bUVm5/d+Kiq//W1EBX3yx/di2bfuxdWtIADvsEH5AVT+snXb66g+y6gf8zW+GH3rnztt/Cdq2Db8YWrFTpLDtuGMYin3EEV9/zz08AL77bnjwe//98CD4/vuwfj0sXx7e//jjsOpp1QPkp5+G5PLpp+HvTWVl+FvTsmX4u1N1tGix/WjePBzNmm3/N/Uw2/4vhH+rH2lz9zoPoAfwVMr5UGBItTKjgfNTzpcBbeq6FlhKqA0AtAWW1nJ/16FDhw4dDT/q+/vu7mnVBOYDncysI/Au0B8YUK3MdOBqYKqZ9QA2uvt6M9tQx7XTgUuBEcAlwGM13Tydjg0REWmcepOAu1eY2WBgFtuHeS41s0Hhbb/L3Z8wsz5mtpIwRPSyuq5NfusRwANmdjmwGjgv459ORETqlPeTxUREJHsKYryKmZ1jZkvMrMLMjo4dTyaYWZmZLTOz5cl5EkXDzMaa2XozeyV2LNlgZu3NbLaZvWpmi83smtgxZZKZtTKzeWa2MPn5hsWOKdPMrJmZLTCz6bFjyTQzW2VmLyd/fi/WV74gkgCwGDgT+FvsQDIhOYluFHAqcDgwwMyKaU3QewmfrVh9Afynux8O9ASuLqafn7tvBU5y925AV6C3mVUfFl7ofgy8FjuILKkEEu7ezd3r/bkVRBJw99fdfQWNG3qaj76cgOfu24CqSXRFwd2fAz6MHUe2uPu6qmVR3P0Twki3dnGjyix335L8shWh77Bo2o3NrD3QBxgTO5YsMRrwt70gkkARqm1ynRQYM9uf8LQ8L24kmZVsLlkIrAOedvf5sWPKoP8GfkoRJbZqHHjazOab2b/XVzhvVtw2s6cJcwu+fInwYX7u7jPiRCVSOzPbBXgI+HGyRlA03L0S6GZmuwGPmtlh7l7wzSdmdhqw3t0XmVmC4mldSHWcu79rZnsTksHSZO28RnmTBNz95Ngx5NA7QIeU8/bJ16RAmFkLQgK4z91rnONSDNx9k5nNAcoojjb044DTzawPsBOwq5lNcPeBkePKGHd/N/nve2b2CKH5udYkUIjNQcWQub+cgGdmLQmT6IptlIJRHD+r2twDvObuI2MHkmlmtpeZtU5+vRNwMl9dMLJgufsN7t7B3Q8k/H83u5gSgJntnKyhYmbfAE4BltR1TUEkATM7w8zeJixD8biZPRk7pqZw9wqgahLdq8CUlEl0Bc/MJgHPA4eY2VtmdlnsmDLJzI4DLgT+LTkMb0Fy34xisS8wx8wWEfo6Zrr7E5FjkvS0AZ5L9ue8AMxw91l1XaDJYiIiJawgagIiIpIdSgIiIiVMSUBEpIQpCYiIlDAlARGREqYkICJSwpQERERKmJKAiEgJ+/9gsOy5WEP6dQAAAABJRU5ErkJggg==\n", 350 | "text/plain": [ 351 | "" 352 | ] 353 | }, 354 | "metadata": {}, 355 | "output_type": "display_data" 356 | } 357 | ], 358 | "source": [ 359 | "# plot\n", 360 | "xs = linspace(-2, 6, 1000)\n", 361 | "thetas = linspace(-1, 5, 100)\n", 362 | "# p(theta|D)\n", 363 | "plot(thetas, p_theta_D(thetas, D), color=\"blue\")\n", 364 | "# arg max p(theta|D)... or mean(D)\n", 365 | "plot([mean(D)]*2, [0, 0.03], color=\"red\")" 366 | ] 367 | }, 368 | { 369 | "cell_type": "code", 370 | "execution_count": 4, 371 | "metadata": { 372 | "code_folding": [], 373 | "slideshow": { 374 | "slide_type": "slide" 375 | } 376 | }, 377 | "outputs": [ 378 | { 379 | "data": { 380 | "text/plain": [ 381 | "[]" 382 | ] 383 | }, 384 | "execution_count": 4, 385 | "metadata": {}, 386 | "output_type": "execute_result" 387 | }, 388 | { 389 | "data": { 390 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEACAYAAAC9Gb03AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xuc1GP7wPHPVdtBR5WejkophyKERNQqshUVQoWQ04N+Tg8poSSe8hCRkCflUCLpQEmRRUjRdj5sB9JJ8nSgw6Zt798f12xNNbs7s83Mdw7X+/Xa1+7MfL8zV+xec8/1ve/rFuccxhhjElcRrwMwxhgTWZbojTEmwVmiN8aYBGeJ3hhjEpwlemOMSXCW6I0xJsEFlehFJE1ElotIpog8ks9x54rIPhG5KtRzjTHGRIYUNI9eRIoAmUArYCMwF+jsnFse4LgZwB7gTefcR8Gea4wxJnKCGdE3AVY659Y65/YBY4EOAY77P+BD4PdCnGuMMSZCgkn0NYB1frfX++47QESqAx2dc68CEsq5xhhjIitcF2NfBKz+bowxMSgliGM2ALX8btf03efvHGCsiAhwHNBGRLKDPBcAEbGmO8YYEyLnnARzUL5fQFFgFVAbKA7MB07N5/iRwFWhnquhxLa+fft6HUJQLM5C2LfPuf79natY0blevZxbterAQwfiXLHCuQce0GOefVbPiSEx9d8zHxZn+PjyZoF5vMARvXNuv4j0AKajpZ4RzrllInKn70WGH35KQecW+O5jTDT9/jtcfTUULw4LFkDNmoGPO+kkGDwYevSAW2+FL76A99+H8uWjG68xIQqmdINzbhpw8mH3vZ7Hsd0LOteYmLF2LbRuDZ06wVNPQZEgLlvVrQszZsADD8AFF2jCr1o18rEaU0i2MjYEqampXocQFIszSJs2wcUXw113wdNP55nkA8aZkgIvvwydO8Mll8Aff0Q21iB4/t8zSBZn9BW4YCpaRMTFSiwmCfz1F7RoAVddBY89VvjncQ5694Yvv4SvvoKSJcMXozEFEJGgLsZaojfJxzmtyVeqBMOHgxQ8aaHA57v2WihTBt588+ifz5ggBZvorXRjks+QIbBuHQwdGp6kLAIjR8JPP8Ebbxz98xkTZjaiN8klIwMuuwx++AHq1Anvcy9bBs2bw7ff6gwdYyLMRvTGHG7fPujeHZ57LvxJHuDUU6FvX7jxRsjODv/zG1NIluhN8nj2WZ0GeeONkXuNe+6B0qVh2LDIvYYxIbLSjUkOK1fC+efDvHlQq1bBxx+N5cvhoot08VX16pF9LZPUbNaNMf7at9fk+/DD0Xm9Pn1gzRp4773ovJ5JSpbojck1Y4YuilqyBEqUiM5r7t4NDRrAu+/ChRdG5zVN0rGLscaAXhR94AG9AButJA9QqpS2VOjZU+fZG+MhS/Qmsb37ri6M6uDBxmZdu8KuXTBpUvRf2xg/VroxiWvfPjj5ZBg1Sue3e2HaNLj/fli8WPvjGBNGVroxZtQoOPFE75I86OKsf/xD2xkb4xEb0ZvEtHevrk597z1tJeylGTPgvvt0VB9MG2RjgmQjepPcRo7UWS9eJ3nQNsZly8KECV5HYpKUjehN4tm//2BtPlamNn78MTzxhC7Ysu6WJkxsRG+S14QJULkyNGvmdSQHXX65TrOcOtXrSEwSskRvEotz8J//6ArYWBo5i+ic+uef9zoSk4SCSvQikiYiy0UkU0QeCfB4exFZICIZIjJHRJr5PfaL/2PhDN6YI3zzDWzd6s28+YJ06gSZmdoDx5goKrBGLyJFgEygFbARmAt0ds4t9zumlHNut+/n04EPnHOn+m6vAc52zm0r4HWsRm+OXvv20LYt/POfXkcS2L//rcl+5EivIzEJIJw1+ibASufcWufcPmAscMhwKTfJ+5QBcvxjCfJ1jDk6P/8M330H3bp5HUne7rgDJk6EzZu9jsQkkWAScA1gnd/t9b77DiEiHUVkGfAx0N3vIQfMEJG5InL70QRrTL5eew1uukn7zMSqSpXguuvg1Ve9jsQkkbCNtJ1zE33lmo7AAL+HmjnnGgNtgXtEJEbmu5mEkpWl5ZC77vI6koLdey+8/rq2aDAmCoJpvrEB8N+poabvvoCcc7NEpK6IVHTObXXObfLdv0VEJqCloFmBzu3Xr9+Bn1NTU0lNTQ0iPGOADz6Axo2hXj2vIylYgwZQvz5MngxXX+11NCaOpKenk56eHvJ5wVyMLQqsQC/GbgLmAF2cc8v8jjnRObfa93NjYJJz7ngRKQUUcc7tFJHSwHTgSefc9ACvYxdjTeGddx489hhccYXXkQRn9Gh46y2YfsSfgjFBC9vFWOfcfqAHmqSXAGOdc8tE5E4RucN32NUislhE5gEvA9f67q8CzBKRDGA28HGgJG/MUfnxR7242bat15EE7+qrISMDVq/2OhKTBKwFgol/d9wBJ5wAjz7qdSSh+de/oFgxGDjQ60hMnLKtBE1y2LULatbUzpA1jpgMFttWrIAWLeDXX6F4ca+jMXHIet2Y5DB+vHaojLckD9p4rUEDnVdvTARZojfxbeRI6N694ONi1a232ipZE3FWujHxa/VqaNoUNmyI39LH7t36aWTJEqhe3etoTJyx0o1JfKNGwfXXx2+SB13Fe/XVOt3SmAixEb2JT/v360ybTz6BM87wOpqj8803uqJ30aLYaq1sYp6N6E1i+/xz3XQ73pM86C5Ye/bATz95HYlJUJboTXx6+224+WavowgPEe24+dZbXkdiEpSVbkz82blTL2CuXKmj+jD76y/d2nXtWt3DBLTp5AknaDud0qXD/pLaYrlJE1i/HkqUiMALmEQUbOkmmKZmxsSWyZN17nwYk/yWLfDee/q1aBE0agQnnggVK+rjc+fCqlW6LqtxY+jaVbsN5z5+1OrUgYYNYcoUuOqqMD2pMcpG9Cb+tGsHXbrADTcc9VNt3AiDBsE77+j+3ddfD6mpeQ+qs7L08sDo0dqP7NZb4aGHwvSe89//wrRp8OGHYXgykwzsYqxJTFu2wKxZ0LHjUT1NdjYMHqwj95QUWLpUy/6XXZZ/5aRkSX1DeO893fp1zx4diL/6KuTk5H1eUK6+GmbMgD//PMonMuZQNqI38eWVVzTRv/deoZ/i55+17FK2rG5KVb/+0YW0ZIn2VStaVMM6qm4M7dvrJuKxvB2iiRk2ojeJacwYra8U0scfa+v6rl21BHO0SR50RP/NN5CWBueco89baF26HNWbmDGB2IjexI/cmSkbN2p73xC98go884z2QWvaNALxAenp0Lmzvk6hWvDs2nVwRlHlyuEOzyQYG9GbxDNmjJY1QkzyzkGfPvDSS1r1iVSSB72Q+9VX8PTTMGBAgYcfqXRpaNNG342MCRNL9CZ+vPdeyGUb56BnT50h8+23Oosx0k4+Gb77TsMtVLK38o0JM0v0Jj4sXQrbt+v8+RD07asTWT77DI47LkKxBVClCnzxBbz7biE2kLrsMp2wv359RGIzyccSvYkP48Zp2aZI8L+ygwfrlPTp08O4sCkEVavCzJnwxhs6RT5oJUrAlVfC++9HLDaTXIL6qxGRNBFZLiKZIvJIgMfbi8gCEckQkTki0izYc40JyrhxcM01QR8+eTI8/7yO5CPQJSFo1avDp5/CY4/pJ4ugde4MY8dGLC6TXAqcdSMiRYBMoBWwEZgLdHbOLfc7ppRzbrfv59OBD5xzpwZzrt9z2KwbE9iyZXDppbq3ahAj+vnz9fApU3SSTiz45htdDzVzJpx2WhAnZGfru8ScOdpkx5gAwjnrpgmw0jm31jm3DxgLdPA/IDfJ+5QBcoI915gCjRunWTKIJL95s645euWV2EnyABddpJ8wrroKduwI4oSUFF39a7NvTBgEk+hrAOv8bq/33XcIEekoIsuAj4HuoZxrTL6CLNvs36/tb268Ea69NgpxhejGG6FVK51fH9SH106drO+NCYuwda90zk0EJorIhcAA4NJQn6Nfv34Hfk5NTSU1NTVc4Zl4tXy59goOYrbNM8/A33/Dk09GIa5CevFF3WfkhRfgwQcLOPjii3UJ77p1cPzxUYnPxLb09HTS09NDPi+YGn1ToJ9zLs13uxfgnHOD8jlnNXAucFKw51qN3gT01FPayOyll/I97MsvNSf+9FPs77G9dq2WlSZP1nYM+ereXXfRuu++qMRm4ks4a/RzgXoiUltEigOdgcmHvdiJfj83Boo757YGc64x+QqibLN1q5ZFRo2K/SQPULu2XkO48UbteJAvK9+YMAiq142IpAFD0DeGEc65gSJyJzo6Hy4iPYFuwN/AHuAh59z3eZ2bx2vYiN4casUKLV+sX5/vhdgbboAKFeDll6MYWxh06wZlysCwYfkc9PffOiF/8eL4eBczURXsiN6ampnYNWCATqPJJ4NPmKAtDubPj9AWfxG0Y4dWZYYNg7Zt8znwppu01nPPPVGLzcQHa2pm4l8BZZstW+Duu7VkE29JHqB8eY399tu1u0OerHxjjpKN6E1sWrkSmjfXsk3RogEPufZarXf/5z9Rji3M7rxTK1OvvprHAVlZUK2azkCqUiWqsZnYZiN6E98mTNAFQ3kk+SlTICMD+vePclwRMGiQzsCZNSuPA0qW1NrOxIlRjcskDkv0JjZNmKCNvQLYvRt69NCZK8ccE+W4IuDYY3X26O23w969eRzUqZOWsowpBCvdmNizaRM0aKAXYosXP+LhRx/VzaYSqWW7c/q+duaZ4Ldu8KDdu7V88/PP3rTiNDHJSjcmfk2apKWKAEl+6VJt+zt4sAdxRZAIDB2qX2vWBDigVClo2VJrVsaEyBK9iT0TJ2p9/jDOwV136WYi1ap5EFeE1awJ//qXfgV05ZVa0jImRFa6MbFlxw7t67JhA5Qte8hDY8ZoB8g5c/K8Rhv3srK0jfGwYdC69WEPbt2qeyFu2qQjfJP0rHRj4tPUqTqt8rAkv2cP9O6tTcESNcmDTrB54QVtbbNv32EPVqwI55wT4g4mxliiN7Emj7LN4MG6OPSiizyIKcouv1zXBwwdGuDBjh1tmqUJmZVuTOzIytK+LpmZh+z/t2kTnH66lmzq1vUwvihavlzbGS9dethWiL/+Co0bw2+/6eYkJqlZ6cbEn5kzNaMftsnrY49pt95kSfIAp5wC11+vXZoPUauWbi2Y5+oqY45kid7EjgCLpDIydEZhnz4exeShxx7TtQIrVx72QMeONvvGhMRKNyY27N+vbXi///7A0N053Xrv2mvhn//0OD6PPP00LFgAH3zgd+eSJbrO4JdfdAK+SVpWujHxZfZsrc/71WemT4eNG+G22zyMy2MPPADffQc//OB3Z4MGupgsI8OzuEx8sURvYkNuEzOfnBxtdTBgQHJfcyxVSlsi9Ozpt6G4iJa4bPaNCZIleuM9546YVjl+vOazq6/2MK4YcfPN2nv/kO4HVqc3IbBEb7y3eDFkZ2tHL/THxx+HZ56xEjToJ5qBA6FXL72UAUDTpvDHH7Bqlaexmfhgid54L3c078vqb72l5fpLL/U4rhhyxRW6v+yBi7JFikD79toAzpgCBJXoRSRNRJaLSKaIPBLg8a4issD3NUtEGvk99ovv/gwRmRPO4E2CmDjxwLTKrCx48kn4979tNO9PROfU9+unn3gAK9+YoBWY6EWkCDAUuAxoCHQRkVMOO2wN0Nw5dwYwABju91gOkOqcO8s51yQ8YZuEsXatrvZs1gzQ7fTOPBPOP9/juGLQJZfoToKjR/vuaNlSy16bN3sal4l9wYzomwArnXNrnXP7gLFAB/8DnHOznXM7fDdnAzX8HpYgX8cko0mTtC6RksKuXbqt3oABXgcVm3JH9f37+xqelSgBaWm6D6Ex+QgmAdcA1vndXs+hifxwtwGf+t12wAwRmSsit4ceokloftMqhw3TxpWNGhVwThJr0UI7FY8a5bvDetSbIIR1hrKIXAzcAlzod3cz59wmEamMJvxlzrmAjTr6+e2hlpqaSmpqajjDM7Hmf/+DefPg0kvZtUt7zVsH3oI99RR07gzdukGJNm10s9k//4Ry5bwOzURYeno66enpIZ9XYAsEEWkK9HPOpflu9wKcc27QYcc1AsYDac651Xk8V1/gL+fcERvBWQuEJPTWW1p2GD+e55/X7gcffuh1UPGhbVto1w7uuQdo0wZuuUV7RZikEs4WCHOBeiJSW0SKA52BQ4qCIlILTfI3+id5ESklImV8P5cGWgOLg/9nmITmK9vs3g3PPQdPPOF1QPGjf39dZ5CVha2SNQUKqqmZiKQBQ9A3hhHOuYEicic6sh8uIm8AVwFr0Yuv+5xzTUSkDjABrdOnAKOdcwPzeA0b0SeT3bt1svzatbwwqgLffAMffeR1UPGlXTvdpOSuK3+DU0/V2TcBNlQ3iSvYEb11rzTemDgRhg5lz8efc+KJuoOgb2GsCdL332utftUqKNbiAt01/bLLvA7LRJF1rzSxzbdIavhw3SLQknzozj8f6teHd97ByjcmXzaiN9GXnQ1Vq5L1fQYnph7Pxx/r7ngmdF99pW2cl01eSUqrFrB+vbZHMEnBRvQmds2aBXXq8Ma04zn7bEvyR6N5c73U8f68+lChgm6sa8xhLNGb6Js4kezLOzJokM20OVoi2unz6afBdeho5RsTkCV6E12+3vMT6chpp8E553gdUPy79FLtbDmzvNXpTWCW6E10LViAK1aMR99tQO/eXgeTGHJH9f8aczZu505YvtzrkEyMsURvomvCBFac2pHjKgvNm3sdTOK4/HJAhF/OsNbF5kiW6E1UuYkTGbS8I717W7/5cBKBxx6D51Z1xFn5xhzGplea6Fmzhr1nn8+51Tcyf1FRmwUYZjk50OjUfWRsqkqxZQuhRn5NZk0isOmVJvZMmsSMku3p2duSfCQUKQIPPlKMr8q0tR715hD252ai5s+3J/JRTkc6d/Y6ksR1/fXwwd6O/PmW1enNQZboTXRs2ULRxfM579FWpIR1FwTjr0QJOP2hy0j5aTZs3+51OCZGWKI3UbH+tU+YWbQ13e4o6XUoCe/mHmX4WlL57c2pXodiYoQlehMVm4dPxHXoyDHHeB1J4itbFvakdWT9UJt9Y5TNujER98uSXVQ6rRo5v6ylfO0KXoeTFP5YtoViDeuze/VvVKtjn6ISlc26MTFj+kPT+e2E8yzJR9Fxp1ZmS7VGTHv4C69DMTHAEr2JqM2boezMiVS5o6PXoSSdit2vpOgnE+2arLFEbyLrpcHZXCGfUO6G9l6HknQq3tKBDjKZ117Z73UoxmOW6E3E7NgBi1/9hmIn1YXjj/c6nORTty7Fa1Xlu+e/Z88er4MxXgoq0YtImogsF5FMEXkkwONdRWSB72uWiDQK9lyTuF59Fe6uPpES11nZxivHdO7ILRUmMmqU15EYLxU460ZEigCZQCtgIzAX6OycW+53TFNgmXNuh4ikAf2cc02DOdfvOWzWTQLZswfq1nGsLXICxWdMhYYNvQ4pOc2fz57LO9Gw2EoyV4otVksw4Zx10wRY6Zxb65zbB4wFOvgf4Jyb7Zzb4bs5G6gR7LkmMY0cCdedPJ/ipYtDgwZeh5O8zjiDY1KyaV5xMePGeR2M8Uowib4GsM7v9noOJvJAbgM+LeS5JgFkZ8N//gP/qjMerrzS+hF7SQQ6duTRBhMZOFA3+DLJJ6wf5ETkYuAW4MLCnN+vX78DP6emppKamhqWuEx0ffCBXns9/ofxWHE4Blx5JfUffBDnHmfaNGjTxuuATGGlp6eTnp4e8nnB1OibojX3NN/tXoBzzg067LhGwHggzTm3OpRzfY9ZjT4BOAdnngkv37WU5s+kwdq1NqL3WnY2VKvGxMd/YvCHtfj6a68DMuESzhr9XKCeiNQWkeJAZ+CQZtciUgtN8jfmJvlgzzWJZdo0TfYX/T4errrKknwsSEmByy/niv0TWb8evv3W64BMtBWY6J1z+4EewHRgCTDWObdMRO4UkTt8hz0OVASGiUiGiMzJ79wI/DtMjBg4EHr1AvloPFx9tdfhmFwdO1J08gQefhgGHfF52iQ6a2pmwub776FrV1j56SpSLr4I1q+HokW9DsuAznetVo09C1dS97zKzJgBp53mdVDmaFlTMxN1gwbBQw9ByqTx0LGjJflYcswxcNllHPPZRO67z0b1ycYSvQmLpUt1RH/LLcB4K9vEpGuugXHjuOsumDoVfv7Z64BMtFjpxoTFzTdD/frQ58ZfoXFj2LQJihXzOizjb9cuqF4d1qyh93OV+OsvGDrU66DM0Qi2dGOJ3hy1devgjDNg9Wqo8NaLsHAhvPmm12GZQK65BtLS+K3drTRoAMuXwz/+4XVQprCsRm+iZvBg6N4dKlTAyjaxrlMnGDeOqlXhuutgyBCvAzLRYCN6c1T+9z8t2SxaBDWKbNK+Nr/9BiVKeB2aCWTnTqhRA37+mTXbK9KkCaxZA+XKeR2YKQwb0ZuoeOUVbWdTowYwYQK0a2dJPpaVKQOXXAKTJlG3LrRuDa+95nVQJtIs0ZtC27VLL+Y9/LDvDivbxIdOneDDDwFd3Pbii5CV5XFMJqIs0ZtCGzECLroITjkF+P13+OknuOwyr8MyBbn8cvjmG9i+nUaN4Kyz4O23vQ7KRJIlelMo+/bB88/DI7l7ho0fr20RS5XyNC4ThLJloWVLmDQJgN694dlntfeZSUyW6E2hjB0LJ54ITZr47nj/fejc2dOYTAiuueZA+ebCC6FqVX2vNonJZt2YkOXkQKNGOq2ydWtg40ZtnLJpk12IjRd//gk1a+oiiPLlmTIF+vSBjAxrOBpPbNaNiZgpU6B4cbj0Ut8d48bBFVdYko8n5cpBaip8/DEAbdvqG/hnn3kblokMS/QmJM7B00/Do4/6jfysbBOfrr1W/9+h/y979dI20ybxWOnGhOTzz+Hee2HxYihSBN1B6uyzrbdNPPrrLy3frFkDlSqRna2L38aMgfPP9zo4Ewwr3ZiIGDBAR/NFcn9zPvhAd5KyJB9/ypbV6bC+q7ApKbomwkb1iccSvQnaN9/otbtDqjRjx2rTFBOfunaF9947cPOWW+CHH/QTm0kcVroxQUtL04Wvt9/uu2PVKmjWDDZs0OGgiT9790K1ar5mRTUA+Pe/dX+Bd97xODZTICvdmLCaOxeWLIFu3fzufP99nY9tST5+lSihzYp8F2UB7rlHN3lftcrDuExYBZXoRSRNRJaLSKaIPBLg8ZNF5DsRyRKRBw977BcRWeC/abiJP08/DT17HjaD0so2iaFLF70C61OuHPzf/8Ezz3gYkwmrAks3IlIEyARaARuBuUBn59xyv2OOA2oDHYFtzrnBfo+tAc52zm0r4HWsdBOjFi3ShVFr1ujWo4BuLtKunc66KWIfDOPa/v06++arr+CkkwDYvh3q1dNPcnXqeByfyVM4SzdNgJXOubXOuX3AWKCD/wHOuT+ccz8BgbplSJCvY2LUM8/AAw/4JXmAd9+FG26wJJ8IihbVOfV+F2WPPRbuvlvr9Sb+BfNXWgNY53d7ve++YDlghojMFZHbCzzaxJQVK3Tu/F13+d25fz+MHq2J3iSGrl21fOP3qfr++3Xm5dq1HsZlwiIaV9GaOec2iUhlNOEvc87NCnRgv379DvycmppKampqFMIz+Rk4UOu1Zcv63fnll1ClCjRs6FlcJsyaNNH2lRkZurk7ULEi3HknDBoEw4Z5HJ8BID09nfT09JDPC6ZG3xTo55xL893uBTjn3KAAx/YF/vKv0Qf7uNXoY8+aNXDuuTr7okIFvwduvll3A3/gAa9CM5Hw2GOwZ4/2n/b54w84+WRYsEDL+Ca2hLNGPxeoJyK1RaQ40BmYnN9r+wVRSkTK+H4uDbQGbClGnBgwQKfaHZLkd+/WPuZdungWl4mQG27Q8o1fY/rjjoNbb9VRvYlfQS2YEpE0YAj6xjDCOTdQRO5ER/bDRaQK8CNQFsgBdgINgMrABLROnwKMds4FXGBtI/rYsmoVNG0KK1celujHjNGVNJ9+6llsJoKaNoUnntB2lj6bN8Opp+pq2erVPYzNHCHYEb2tjDUB3XQT1K0Lffse9kCbNjryu/56T+IyEfbaa3oNxm8BFcCDD+p12hde8CguE5AlelNoK1borkOrVkH58n4P/PabbhC7YQOULu1ZfCaCtm3TifM//3zIR7lNm3RvmcWLtWOCiQ3WAsEUWv/+OrXukCQPuhK2QwdL8omsQgXtaHnYiL5aNb0Gb6tl45ON6M0hli2DFi10NF+unN8DzsFZZ8Fzz8Ell3gWn4mCqVP13X727EPu/v13rdXPmwe1a3sUmzmEjehNoTz5pNZjD0nyoH/dO3ZAy5aexGWiqHVrXSW1fPkhd//jH7pw7qmnPIrLFJqN6M0BixdDq1awejWUKXPYg3ffDVWr6owMk/h69tTWCIf1QNi2TdvhfPed7kZlvGUXY03IOnXSBZI9ex72wJ49ulomIwNq1fIkNhNlS5ZorX7tWk34fp5+WvvVjx7tUWzmACvdmJDMnQvffw89egR48KOPdImsJfnk0bChXoH9/PMjHrr3XvjiC9uFKp5YojcA9O6tVZlSpQI8OGKELo80yeW22+CNN464u2xZ/dRnVbz4YaUbw4wZ2upgyZIAe3yvXg3nn6+bxR6y64hJeH/+qdNrli3T6zN+9uzRGv3EiXDOOR7FZ6x0Y4KTkwO9emnd9YgkDzBypK6CtSSffMqV060iR4484qFjjoE+feDRRz2Iy4TMEn2S+/BDENFNv4+QnQ2jRlnZJpndcYeWb3Jyjnjottvgl19g+vToh2VCY4k+ie3bp6OygQPz2ChqyhQ4/nhd+26S09ln63ZTAS7KFiumvzs9e+peNCZ2WaJPYiNGwAkn5LPQddgwLd6b5CWiu48MHx7w4Suv1I4Y774b5bhMSOxibJLatUsXvkyerIO2I2RmamezX3+FkiWjHp+JIflclAVdPNW5szbDO2RfYRNxdjHW5Os//9GeNgGTPGi72u7dLcmbgxdl33wz4MMXXKDLLIYMiXJcJmg2ok9C69frToB5LnTdvVsf+PFHre0YM2+e1mlWr4ZvUy2NAAAWoklEQVSUI7eazszUhL9sGVSu7EF8ScpG9CZPvXtrc6o8F7qOHatz5y3Jm1yNG+svzMSJAR8+6SQt3wwYEOW4TFBsRJ9k5szRgdmKFQEal4G2Iz7nHP2LbdMm6vGZGDZuHLz8Mnz9dcCHt2zRNsbffqsbipvIsxG9OYJz2oL4qafySPKg7wTbt2tDK2P8XXmlTpzPyAj4cOXK+mnx/vv1d83EjqASvYikichyEckUkUcCPH6yiHwnIlki8mAo55roGTdOy+833ZTPQS++qFMqA06sN0ktJUXbVedz1fX//k93IZwyJYpxmQIVWLoRkSJAJtAK2AjMBTo755b7HXMcUBvoCGxzzg0O9ly/57DSTQTt2aMfq996S2fbBPTrr7qL1M8/B9h5xBjgf/+DevW09vePfwQ8ZNo07YK6ZIl1zoi0cJZumgArnXNrnXP7gLFAB/8DnHN/OOd+ArJDPddEx6BBWnrPM8mDjtRuvtmSvMlbpUq6ccHrr+d5SFoaNGigHw5NbAgm0dcA1vndXu+7LxhHc64Jk9WrYehQeOGFfA7asUP72tx3X7TCMvHq3nvh1Vdh7948Dxk8WNdqbNwYxbhMno6cEOuhfv36Hfg5NTWV1NRUz2JJFM5p3bRnT21bk6cRI3SvUNtcxBTk9NP1a/RoXVQXQL162g/tkUfgnXeiHF8CS09PJz09PeTzgqnRNwX6OefSfLd7Ac45NyjAsX2Bv/xq9KGcazX6CJgwQRuXzZ8PxYvncVB2Npx4Iowfb83FTXBmztQLs0uX5nnhfudOvS707rsFlAxNoYWzRj8XqCcitUWkONAZmJzfax/FuSaMdu3SqW7DhuWT5EGn49SubUneBO/ii/VazqRJeR5Spgy89JL2RMunymOiIKgFUyKSBgxB3xhGOOcGisid6Oh8uIhUAX4EygI5wE6ggXNuZ6Bz83gNG9GHWe/eOpEm302cc3K0H8KgQdC2bdRiMwngo4/092b2bO1ymYeOHXUyV9++UYwtSQQ7oreVsQlqwQJtP7xwoe7xnKdJk6B/f+1rk88fqzFHyMnR6TWvvqoj/DysW6eJftYsOOWUKMaXBGxlbBLLztZNoQYOLCDJO6fLZB97zJK8CV2RInqVf9ARl9wOcfzx8Pjj8M9/2opZr1iiT0Avvgjly+c5IeKgzz6DrCzoYEsbTCFdf72ujJozJ9/DevTQi7OjRkUnLHMoK90kmFWroGlT/burWzefA53TjUV69IAuXaIWn0lAr72mJcBPP833sIwMbaE0fz5Urx6l2BKclW6SUE6Obtj86KMFJHmA9HRtN3jttdEIzSSy7t21Ef133+V72FlnaXvsO+6wEk60WaJPIMOHa0+bAhe3Oqd1+ccfh6JFoxKbSWDFi+vvUxDTavr0gQ0brIQTbVa6SRC5JZuvv9aJEPn65BPo1Uun5liiN+Gwb582oR81Cpo3z/fQhQuhVSvdtCrf1dqmQFa6SSLZ2XDjjfDEE0Ek+Zwcre0884wleRM+xYrpL+ATTxRYl2nUSBfy3XablXCixRJ9Ahg4UFch9ugRxMHvvacHX3FFxOMySeaGG2DTJp3NVYBHHoFt2/Q6rok8K93EuR9/hHbt4KefoGbNAg7++29tPjJiBFjDOBMJkyYdbK4UYBNxf5mZ0KwZfPklnHZalOJLMFa6SQI7d+ogasiQIJI86NXaevUsyZvIad8ejjsO3nyzwENPOklbGV93ne58ZiLHRvRxyjno1k0nPIwYEcQJW7fq+vMvvtAWs8ZEyo8/amkwMxPKls33UOd0sFKmTL57mZg82Ig+wb35pi5AefnlIE/o1093BrIkbyLtnHO00dKzzxZ4qIi2yvn8c22iaiLDRvRxaNEiaNlSp1KeemoQJyxZouWaZcv0Y7UxkbZuHZx5ps6hrF27wMPnzIHLL9dGmAUu9jMH2Ig+Qf31F1xzDTz/fJBJ3jl44AFdHGVJ3kTL8cfrHMr77w/q8CZNdM3VlVfqPgomvGxEH0dycrT6UqkSvPFGkCd9+KGuWJw/X+c6GxMtWVk6aX7wYB2uFyD3utP+/bqHgjVULZiN6BPQU0/B5s260XdQtm/Xfgivv25J3kRfyZL6y3rvvUFNqxHRX9Vly7QDqwkfG9HHiQkTNGfPmQNVqwZ50l136TDJVqUYL113HdSvDwMGBHX4L7/Aeefp2r6WLSMbWryzHaYSyOLF+gs/dWoI27p++612plyyBI49NqLxGZOvjRv1wuz06fo9CF98AV27wldf2a5U+bHSTYLYuFFXvr74YghJfs8euP12eOEFS/LGe9Wr68qom27S1dlBaNUK/v1v/d3fsiXC8SWBoBK9iKSJyHIRyRSRR/I45iURWSki80XkLL/7fxGRBSKSISL5b0NjDvHnn9CmjVZgunYN4cRHH9X58tdcE7HYjAlJt246E+fpp4M+pXt33ROnfXsdu5jCK7B0IyJFgEygFbARmAt0ds4t9zumDdDDOddORM4DhjjnmvoeWwOc7ZzbVsDrWOnGz99/Q9u22vl16NAQZiDMnKl/VAsXQsWKEY3RmJDklnCmTYPGjYM6xTndrfDvv+H9963h6uHCWbppAqx0zq11zu0DxgKHbzLaAXgbwDn3A1BeRKrkxhLk6xifnBwdzZQpAy+9FEKS374dbrkF/vtfS/Im9lSvrjXILl20UVMQRHQV+Pbttrn40QgmAdcA1vndXu+7L79jNvgd44AZIjJXRG4vbKDJwjkt1fz6K4wZE8IIxjm4806dr5yWFtEYjSm0rl3hgguC7KmtSpaEiRN1UsK//mXJvjDy7yMaHs2cc5tEpDKa8Jc552YFOrBfv34Hfk5NTSU1ybosOqdTKBcu1AkKpUqFcPIrr8DKlfDWWxGLz5iwGDoUzj4b3nlHd8wJQpkyOussNRX69w9q18KElJ6eTnp6esjnBVOjbwr0c86l+W73ApxzbpDfMa8BXzrn3vfdXg60cM5tPuy5+gJ/OecGB3idpK7ROwc9e2pv7s8/D3GyzA8/aLfA77+HE0+MWIzGhE3ufoJffRXEtmgHbd6sOxV266Zt75NdOGv0c4F6IlJbRIoDnYHJhx0zGejme+GmwHbn3GYRKSUiZXz3lwZaA4tD+HckBefgoYd0FP/ZZyEm+T/+0AUpw4dbkjfxo1EjeO456NBBW2gHqUoVSE/XsmafPlbGCVaBpRvn3H4R6QFMR98YRjjnlonInfqwG+6cmyoibUVkFbALuMV3ehVggog432uNds5Nj8w/JT7t3w933KHrmr78MsRrqHv3aheoLl2gY8eIxWhMRNx0k25Q37mz1mUK2JEqV7Vqmuxbt9Zpl88/b31xCmIrYz20d69uurBtm15sKlMmhJOd0z+U3bvhgw+giE1sMnEoO1vnETdsqAv8QrBtm847OP107WmfjO2cbGVsjNu2TVf9ZWfDJ5+EmOQBnnlGuz+9/bYleRO/UlJ0gvxnn+nQPAQVKuj1rA0b9BLVX39FKMYEYBnCA6tXw/nn64bI48bp9LGQDB+u+wdOmhTi1BxjYlCFCprohwyBd98N6dSyZeHjj3Vvk+bNNembI1mij7Kvv4ZmzXQa5YsvBl2WPGjMGJ1fNmOGLkAxJhEcfzx8+qlOlJ86NaRTU1K0Qet110HTprpLlTmUJfoocU4HLJ066fThu+4qxJNMmAAPPqijH5thYxJNw4b6KfXmm2HKlJBOFYFevXSKfvv2mviT7JJfvuxibBTs2KEtDdau1eumhdoTc/RoHe1MmaKLTYxJVLnrQkaM0O8hysyEq66Cc8/VdYSJXN20i7ExYu5czctVq8KsWYVM8q+/Do88ok26LcmbRHfeeTqgue02HeCE6KSTtHzz99/65zJvXgRijDM2oo+Qv//WUvobb8DLL+seICFzDvr105k1n39u5RqTXBYv1qlp//yn1mUKMVl+zBjdn/yBB3TleaJ1v7Qdpjw0f75Oca9dWyfIBL31n789e7RW+euvOsm+SpUCTzEm4WzYoPPsmzbVEVPx4iE/xa+/6p9SVpZ+OD799PCH6RUr3Xhgxw4dPbRurSOISZMKmeR/+QVatNDpBF9+aUneJK8aNeCbb+C33/RvYv36kJ+iVi39QHzzzdpep3fvoPYqTyiW6MMgJ0en/556KuzaBUuX6i9VoZZlf/QRNGmiy8LffbcQk+yNSTDlyumMs/bt9Qrr9NC7qBQpoq1GFi7USRENG8LYsckzM8dKN0fBOf2d691ba38vv6yfMAvlzz+1iDh9uv4GNmkS1liNSQgzZ+oo6oor4NlnoXTpQj1Nero2EixSRHurNW8e1iijxko3Efbtt9CypS586tMH5sw5iiT/ySe6THb/fp0iYEnemMBattRh+c6dcMYZmrELITVV/2bvv19bHrdrB999F9ZIY4qN6EOQk6M5+dlnYdMm3YP7ppsKsbo115o1Om1y3jydntOyZVjjNSahTZoE996rO1Y9+6yuri2ErCwYOVKf4oQT9O/6kkvioyOmjejDaMcOGDZM63r9++vv1ooVcOuthUzy27bp58Zzz9VRyaJFluSNCVWHDnpBrH593XS8b1/dXDZEJUvqSvXMTF3YeN99+mf52muJ0yjNEn0enNMNm7p313f59HRN9nPn6pz4QiX433/X4UL9+lqTX7IEHnsssZfuGRNJpUvr6OvHH3UeZb168OSThUr4xYrpzoZLlmjH5BkzdIr0PffoAqwYLzjkyxK9H+e0itKrl65NuuUWnUmzYoW2Lrj44kJ+nFu4EO6+G045RX8Bf/zxKCbYG2OOUKeO1l9mz9aSaJ06+je3dGnITyWi0zDHj9c/3SpVtERbrx48/rh2B483SV+j37NHO0pOm6b195wc7YJ37bX68a3Qdbrt23Wq5PDhuujjttt0fle1amGN3xgTwMaNujpq+HAdrd14ozbAKV++UE+XOwgcM0YnxZUurRdwL78cLrqoUOu4wsJWxuZh71746SedNTNzpvafOfNM3ammTRs466yjSO7/+x9MngwffqiLPFq21EJ+mzZHccXWGFNof/+tF23HjNE/+Esv1e03L7sMjjuuUE/pHGRkaDueTz7RT/wXXaRTNJs3h8aNo7fbVVgTvYikAS9ycM/YQQGOeQlog+4Ze7Nzbn6w5/qOC3uiz86GlSt1W8qMDE3uGRlw8snaE75FC726HtJm3P5279Yn/eILXXqXmalPeM01+lZftmxY/z3GmKOwbZt+yv7kE036DRroCO+ii7SRWiHn5P/+O3z1lVYGvv5aK0fnnqsN1Ro31q/69SOzEVzYEr2IFAEygVbARmAu0Nk5t9zvmDZAD+dcOxE5DxjinGsazLl+z1HoRL9rl+7atGqVfq1Yocl92TKtlJxxhn5dcIH+/yxU/t25k/TRo0l1TmvsP/6oif2sszS5X3KJPrlXn+H8pKenk5qa6nUYBbI4w8viDMHevfpxfto0HawtWKAlnvPP12Rx+umkb91Kaps2IT/11q16qSAjQ8s98+bBH39Ao0Y6yDz5ZO2wefLJei2wRInC/zOCTfTB1BOaACudc2t9TzwW6AD4J+sOwNsAzrkfRKS8iFQB6gRxbp5ycnRq45YtWnLL/dqw4eD3NWu0HF63rv5Hq1dP301vvVWbFwWd1Pft034aGzbo17p1+o6Rmanft24lvWxZUtu1g3PO0Xp7o0Yx2aIgJv6QgmBxhpfFGYISJfSKa6tWejsrS2u6s2dr2XXYMNIXLSK1Vi3NynXqaJKpU0e/ataESpUCtsOsWFH7sLVte/C+rVt1FvWKFfo1a5Z+X7sWKlfWJQCHf1WpotWl447T5zyaTwTBJPoawDq/2+vR5F/QMTWCPPeANm30P0ju144dmqgrVdJd82rU0O81a+ri0Ro1oG4dR/WKWRTJ2q1XVnf7fZ/rd3v7dv3otnXrkd83bdL6euXK+qQ1auiLNGig9byTTtLOSP37a9tgY0xiKVlS67nNmh2874knoGtXLRP8/LN+zZql3zdu1JxSqZJm5NyvY4/V3jzly+uX7+eK5crRomwpWlxYElqV0NcrWZJ9RUuyaVtJ1m0syrp1Or5cs0ZLQL//rp8EtmzR2djHHqtJv3Jl/TmUknOkrhAW6nLm2JWNKUY2RckmpcJ+ihybjWRnQ9Z+yMyGpdlaeN+/X7/n/ly8uM5FP+aYvL8fe6xuQlyxok6Mz/25QgWd5li1ql0wNcYcVKSITok+5ZTAj2dnaxbevPng144dB8sQq1Zphs69LyvriK9iWVnUysqiVtGimvxTUvRTQu73okWhfAquYlH2U5Ts7BSyNxYle0NRsl0K7wT5TwmmRt8U6OecS/Pd7gU4/4uqIvIa8KVz7n3f7eVAC7R0k++5fs8RG9N/jDEmjoSrRj8XqCcitYFNQGegy2HHTAbuAd73vTFsd85tFpE/gjg36GCNMcaErsBE75zbLyI9gOkcnCK5TETu1IfdcOfcVBFpKyKr0OmVt+R3bsT+NcYYY44QMwumjDHGREbM9LoRkWdFZJmIzBeR8SJSzuuYAhGRTiKyWET2i0hjr+M5nIikichyEckUkUe8jicQERkhIptFZKHXseRHRGqKyEwRWSIii0TkXq9jOpyIlBCRH0QkwxdjX69jyo+IFBGReSIy2etY8iIiv4jIAt9/0zlex5MX3zT2cb68ucS3himgmEn0aHmnoXPuTGAl0NvjePKyCLgS+MrrQA7nW6A2FLgMaAh0EZE8pgx4aiQaY6zLBh50zjUEzgfuibX/ns65vcDFzrmzgDOBNiISyzvX3AeE3mksunKAVOfcWc65WP5vOQSY6pw7FTgDyLMsHjOJ3jn3uXMux3dzNlDTy3jy4pxb4ZxbSSGnkEbYgcVtzrl9QO4CtZjinJsFbPM6joI4537LbeXhnNuJ/iHV8DaqIznncre6LoFed4vJeqyI1ATaAv/1OpYCCDGUGwPxVTwucs6NBHDOZTvn/szr+Fj9x3QHPvU6iDiU18I1c5RE5AR0xPyDt5EcyVcOyQB+A2Y45+Z6HVMeXgAeJkbfiPw4YIaIzBWR270OJg91gD9EZKSvFDZcRI7J6+CoJnoRmSEiC/2+Fvm+X+F3TB9gn3NuTDRjCzVOkzxEpAzwIXCfb2QfU5xzOb7STU3gPBFp4HVMhxORdsBm3yckITY/Eedq5pxrjH76uEdELvQ6oABSgMbAK75YdwO98js4apxzl+b3uIjcjP7H9XRfvYLijGEbgFp+t2v67jOFJCIpaJJ/xzk3yet48uOc+1NEvgTSiL06eDOgvYi0BY4ByorI2865bh7HdQTn3Cbf9y0iMgEtic7yNqojrAfWOed+9N3+EMhz8kXMlG587YwfBtr7LjDFg1gblRxY3CYixdEFarE6uyHWR3W53gSWOueGeB1IICJynIiU9/18DHApQTYNjCbn3KPOuVrOubro7+XMWEzyIlLK9wkOESkNtAYWexvVkZxzm4F1InKS765W5PPmHjOJHngZKIPWxuaJyDCvAwpERDqKyDqgKfCJiMTMtQTn3H4gd4HaEmBsLC5QE5ExwHfASSLyq4jc4nVMgYhIM+B6oKVvqt0834AkllQDvhSR+ej1g8+cc1M9jimeVQFm+a55zAY+ds5N9zimvNwLjPb9vz8DeCavA23BlDHGJLhYGtEbY4yJAEv0xhiT4CzRG2NMgrNEb4wxCc4SvTHGJDhL9MYYk+As0RtjTIKzRG+MMQnu/wH1nhj43pNDygAAAABJRU5ErkJggg==\n", 391 | "text/plain": [ 392 | "" 393 | ] 394 | }, 395 | "metadata": {}, 396 | "output_type": "display_data" 397 | } 398 | ], 399 | "source": [ 400 | "# p(x|D) using Bayesian vs ML\n", 401 | "\n", 402 | "def bayesian_integral(x, D):\n", 403 | " thetas = linspace(-5, 10, 1000)\n", 404 | " return sum(p_x_theta(x, thetas) * p_theta_D(thetas, D)) / sum(p_theta_D(thetas, D)) \n", 405 | "\n", 406 | "ps = [(x, bayesian_integral(x, D)) for x in xs]\n", 407 | "\n", 408 | "ps = array(ps); plot(ps[:,0], ps[:,1])\n", 409 | "plot(xs, p_x_theta(xs, mean(D)), color=\"red\")" 410 | ] 411 | }, 412 | { 413 | "cell_type": "code", 414 | "execution_count": 5, 415 | "metadata": { 416 | "slideshow": { 417 | "slide_type": "slide" 418 | } 419 | }, 420 | "outputs": [ 421 | { 422 | "data": { 423 | "text/plain": [ 424 | "[]" 425 | ] 426 | }, 427 | "execution_count": 5, 428 | "metadata": {}, 429 | "output_type": "execute_result" 430 | }, 431 | { 432 | "data": { 433 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEACAYAAAC9Gb03AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd4lFX2wPHvSUJIQkJNL4QaSqQJUhQ161pQ92dfy7r2ruCqq4KggoJ1LdgV61rR1ZVVV12wRAWkKEV6qCEdQkJ6mWTu748bMISUSUgyk+R8nmceMu/cd+YkZE7u3Pfec8UYg1JKqfbLy90BKKWUalma6JVSqp3TRK+UUu2cJnqllGrnNNErpVQ7p4leKaXaOZcSvYhMEpHNIpIkIlPraXeMiDhE5LzGnquUUqplSEPz6EXEC0gC/gikAyuBi40xm2tptwgoAd4wxvzb1XOVUkq1HFd69GOBrcaYZGOMA5gPnF1LuynAx8CeJpyrlFKqhbiS6KOAlGr3U6uOHSQikcA5xpiXAGnMuUoppVpWc12MnQvo+LtSSnkgHxfapAG9q92PrjpW3RhgvogIEAycLiIVLp4LgIho0R2llGokY4y40qjeG+ANbANiAV9gDTCknvZvAuc19lwbimebOXOmu0NwicbZeE6n02zZssV89dVXZuPGjaawsPDgYwfiLCwsNOvXrzdfffWV2bZtm3E6nW6Ktnae9POsj8bZfKryZoN5vMEevTGmUkQmAwuxQz2vG2M2icgNVS8yr+YpDZ3b4F8fpVpRWVkZv/zyC15eXiQkJODn51druy5duhAfH0/fvn1Zs2YNe/fuZfTo0XTq1KmVI1aqcVwZusEY8zUwqMaxV+poe3VD5yrlKUpKSvj555+JjIxk0KBB2NHH+gUEBDBhwgQ2bNjAkiVLmDBhAp07d26FaJVqGl0Z2wgJCQnuDsElGqdrSktLWbp0KX369GHw4MF1Jvna4hQRjjrqKCIjI/n5558pLy9v4Wgb5u6fp6s0ztbX4IKp1iIixlNiUe1fRUUFS5cuJSIigoEDBx7Rc23atIns7GyOO+44vLy076Raj4i4dDFWfytVh7R69Wq6det2xEkeYMiQIfj7+/Pbb781Q2RKNT9N9KrD2bFjB6WlpQwbNqzZnnPkyJHk5eWRnJzcbM+pVHPRRK86lLy8PLZu3cro0aObdZjFx8eH0aNHs3nzZoqKiprteZVqDproVYfhdDpZs2YN8fHxBAQENPvzBwYGEhcXx6pVq9DrTcqTaKJXHcb27dvx8/MjOjq6xV6jb9+++Pj4sGvXrhZ7DaUaSxO96hCKiorYsWMHw4cPb/HXGjZsGElJSZSWlrb4aynlCk30qkPYsGEDAwYMwN/fv8VfKzAwkNjYWDZu3Njir6WUKzTRq3Zv7969FBYW0rdv31Z7zYEDB5KTk0NOTk6rvaZSddFEr9o1YwwbNmxg6NChrbqYydvbm8GDB2uvXnkETfSqXUtNTcXX15fw8PBWf+2oqCgqKyvJzMxs9ddWqjpN9KrdcjqdJCUlMWiQe2rqiQhDhgxh06ZNOt1SuZUmetVupaam0qVLF3r16uW2GEJDQ/H19SU9Pd1tMSiliV61Swd683Fxce4Ohbi4OJKSkrRXr9xGE71ql1JSUggKCqJnz57uDoWQkBB8fHx0rF65jSZ61e4YY9i+fXuzVKZsLgd69Uq5gyZ61e5kZmbi6+vrEb35A8LCwgDIyspycySqI9JEr9qdbdu20b9/f3eHcZgBAwawY8cOd4ehOiCXEr2ITBKRzSKSJCJTa3n8LBFZKyKrRWSFiBxX7bFd1R9rzuCVqmnfvn04HA63zJtvSEREBIWFheTn57s7FNXBNLiVoIh4AUnAH4F0YCVwsTFmc7U2AcaY4qqvhwEfGWOGVN3fAYw2xuQ28Dq6laA6YitWrCAsLIzY2Fh3h1KrrVu3UlRUxMiRI90dimoHmnMrwbHAVmNMsjHGAcwHzq7e4ECSrxIIOKvH4uLrKHVEiouLyc3NbdEyxEcqNjaWzMxMysrK3B2K6kBcScBRQEq1+6lVxw4hIueIyCbgc+Dqag8ZYJGIrBSR644kWKXqk5ycTHR0NN7e3u4OpU6+vr5ERkZqvXrVqpqtp22MWVA1XHMOMKfaQ8cZY44GzgBuEZGJzfWaSh3gdDrZvXs3ffr0cXcoDerbty/Jyck4nc6GGyvVDHxcaJMG9K52P7rqWK2MMYtFpJ+I9DTG5BhjMqqO7xWRT7FDQYtrO3fWrFkHv05ISCAhIcGF8JSC9PR0unXrRpcuXdwdSoOCgoLo0qULWVlZREREuDsc1YYkJiaSmJjY6PNcuRjrDWzBXozNAFYAlxhjNlVr098Ys73q66OB/xhjYkQkAPAyxhSKSBdgIfCAMWZhLa+jF2NVk/3000/ExcUdnK/u6dLS0khJSWH8+PHuDkW1Ya5ejG2wR2+MqRSRydgk7QW8bozZJCI32IfNPOB8EbkcKAdKgAurTg8DPhURU/Va79WW5JU6Evv376esrIzQ0FB3h+KyiIgI1q9fT1FRUZv4FKLatgZ79K1Fe/SqqdauXUtAQIBHlTxwxYYNG/Dy8mLIkCHuDkW1Uc05vVIpj1VZWUlGRgYxMTHuDqXRYmNjSUlJ0YuyqsVpoldtWkZGBj169MDPz8/doTRaYGAgQUFBWtVStThN9KpN2717N7179264oYfq3bs3KSkpDTdU6ghooldtVlFREQUFBW1mpk1twsPDyc3NpbS01N2hqHZME71qs1JSUoiOjsbLq+3+Gnt7exMREUFaWp1LU5Q6Ym33HaI6NGMMKSkpbfIibE0xMTE6fKNalCZ61SZlZ2fTuXNnunbt6u5QjljPnj2prKxk//797g5FtVOa6FWb1F568wfExMSQmprq7jBUO+VKrRulPEpFRQVZWVnEx8e3yPMXFMCqVZCcDDk59livXtCnDxx9NLTEQtbo6Gh++uknhg4d2qavOSjPpIletTlZWVn07NmTzp07N9tz7t0LH3xgb+vWwfDh0L8/HNh2duVK2LYN1q+3yf4vf4GLLvr98SMVEBBAUFCQFjpTLUITvWpzUlNTiYo6bEuEJklPh8ceg3fegT/9CWbNgoQEqOtvSGkpfPMNvPce3HsvXHMN3HknNEeZnejoaNLS0jTRq2annxFVm1JeXk5OTs4R7wlbUQFPPWV77j4+sHEjvP02nHZa3UkewM/P/kH44ANYuxZKSiA+Hl56CY60kkFERAR79+6loqLiyJ5IqRq0qJlqU3bt2kVOTg5HH310k59j50477BIUBC+/DEdaC23DBrj+evD2tn8AjuTDxooVK4iMjPTo7RCV59CiZqpdSktLO6Jhm88/h3Hj7Bj7N98ceZIH26P/6SeYNAnGjLHP21RRUVG6eEo1O030qs0oLi6msLCQkJCQJp3/wgtw443w2Wdw220gDfaDXOflBdOn2x79X/8Kb7zRtOc5UBKhvLy8+YJTHZ5ejFVtxoELlY2dfmiMvXD68ceweDH07dtCAWIv5P7wA5xxhr3Qe++9jTvf29ub0NBQMjIyiI2NbZEYVcejPXrVZqSlpTV67NoYuPtuWLgQlixp2SR/wKBBsHSp7d3PmdP48yMjI3X4RjUrTfSqTSgoKMDhcNCjR49GnTdzJixaBP/7HwQHt1BwtQgLg2+/hXffhUcfbdy5oaGh5Ofna0VL1Ww00as2ISMjg8jISKQRA+tPPWWHaxYubL6FTY0RHg7ffQevvgqvveb6eV5eXlrRUjUrlxK9iEwSkc0ikiQiU2t5/CwRWSsiq0VkhYgc5+q5SrkiPT29UQuJPvsMnnzS9uRdXsy0Z49dOTVlCpxwAsTF2Y8BEREwYACcfDJMngwffWSX0rogMhK++sqO1S9a5HL4REZGkp6e7voJStWjwYuxIuIFPA/8EUgHVorIf4wxm6s1+8YY81lV+2HAR8AQF89Vql6FhYWNGrZZs8auWP3vf6HBumfl5bbbP2+ePfHkk+HYY+Hcc+2E+B49oLISCgthxw5bA+Gdd+zE+fHj4aqr4Pzz7aqrOsTFwb/+ZZt99x0cdVTD30NwcDDFxcUUFxcTEBDg0vetVJ2MMfXegPHAV9XuTwOm1tN+ArChsefaUJQ63JYtW8y6detcapuZaUxMjDEffthAQ4fDmFdeMSYqypg//MGYTz4xpqTE9aCKiox57z1jjj/emH79jHntNWMqKuo95e23jRk40Jj9+117iTVr1pht27a5HpPqcKryZoN53JWhmyig+q4IqVXHDiEi54jIJuBz4OrGnKtUfdLT04mMjGywXWWlncN+2WVw4YX1NPz+exgxwk6LWbDAdrPPO8/WN3BVQIBddfXjj/DWW7Z+wpgxdrpNHS67DP74R7j6ajsbqCGRkZFkZGS4HpNSdWi2efTGmAXAAhGZCMwBTmnsc8yaNevg1wkJCSQkJDRXeKqNasywzcMP25GYBx6oo0FxMUybBv/+t109ddZZzbNq6vjjITER5s+HP/8ZLr3Uzqv09T2s6dy5MHEiPP003HFH/U/bq1cvioqKKCkpwd/f/8jjVG1eYmIiiYmJjT6vwVo3IjIemGWMmVR1fxr248Jj9ZyzHTgGiHP1XK11o2qTlJREeXk5RzUwsP3997aD/euv9gLoYbZsgXPOgdGj4bnn7Nh7S8jOthcIUlPhww/tRdwakpNh7Fh7wXjcuPqfbs2aNXTt2pV+/fq1TLyqTWvOWjcrgQEiEisivsDFwGc1Xqx/ta+PBnyNMTmunKtUfQ5Mq6xPTo4dFnnrrTqS/Bdf2F733/9uJ7a3VJIHO0tnwQI7PjNxou3p1xAbaz9QXHYZFBXV/3Q6fKOaQ4OJ3hhTCUwGFgIbgPnGmE0icoOIXF/V7HwRWS8iq4DngAvrO7cFvg/VDhUWFlJeXt7gsM2tt9pJMqedVsuDL74IN9wA//kPXHttywRakwjccgu8/74tk1lL4ZsLLrCTdu66q/6nCg4OpqCgQBdPqSOiZYqVx9q6dStlZWX1Dtt8+qktcbBmTY0t/oyBhx6CN9+0E9jdNfSxZYv9C/S3v8Httx/yUF6evSb84ou2Nk5dVq9eTY8ePejTp0/LxqraHC1TrNq8hhZJ7d0LN99sh2wOS/JTp9ox8sWL3ZfkwRa++fFHm81rFL7p1s3Gft11sH9/3U+hi6fUkdJErzxSUVERZWVl9KyndsEtt9jplMcdV+OBWbNs3YMffrCrWt2td2+b7N9/Hx5//JCHEhLsjlX33FP36SEhIeTn51NWVtaycap2SxO98kgZGRmEh4fXWdvmv/+F1avhwQdrPPDUU3aao7sK3NQlIsIOIb300mGFbx57zM7AWby49lO9vLwIDQ0lMzOzFQJV7ZEmeuWRMjMz6xy2KS62JWdeeAEOmV7+1lvw7LN2i6fm2K27uUVF2T9A999vyy5U6d7dhn3ddVBXp12Hb9SR0ESvPE5paSmFhYX06tWr1sfnzLEzVk49tdrBH36w4/Jff+1CgRs3GjjQfhy56SZYvvzg4fPOs8P5jzxS+2khISHs378fh8PRSoGq9kQTvfI4WVlZhIaG1rqT1MaNtuzvU09VO7h9u53G+N57MHhw6wXaVKNGweuv2+yeYiuEiMDzz9vbjh2Hn+Lt7U1wcDBZWVmtHKxqDzTRK49zYHy+JmNsR3jmzGrXWPfvt1czZ860lSfbirPOslMuzzrr4Kqp6Gi7puvvf6/9lIiICF08pZpEE73yKA6Hg9zcXEJrGWP/4ANbLfimm6oOOJ2/Vwo7eLANuesuGD7cDs5XrSG5/XZYt84O5dcUFhZGdnY2lZWVrRyoaus00SuPsmfPHnr16oVPjfruJSV2CuLcueDtXXXwySdtbZlDxnHaEBF4+WXYsMHWw8cW0Hz6advZrzkc36lTJ7p3785eFzc9UeoATfTKo2RmZtY6bPPUU7YQ2PHHVx1YvNgm+g8/rLVKZJvh7293Jbn3Xli1CrAjUbGxdry+pvDwcJ1mqRpNE73yGE6nk7179xIWFnbI8YwM28t97EDN07174ZJLbA2Z3r1bP9DmFhdns/qFF0JeHiL2k8tDD9ndDasLDw8nKysLLReiGkMTvfIY2dnZBAUF0blz50OO33uvLQbZrx92LPuqq2zN9/oKxLQ1F11ka+LceCNgJw9deinMnn1oM39/f/z9/cnJyXFDkKqt0kSvPEZGRsZhi6RWr7bTzmfMqDrw6quQnl7Lkth24B//sN/wBx8A9g/cBx/A1q2HNtPZN6qxNNErj2CMISsr65DxeWPsVMNZs2wBMLZvtxn/3Xfb9rh8XQIC7Pf2t79BaiohIXYWzsE/clV0nF41liZ65RFyc3Pp3LkzAQEBB48tXGg779deC1RU2KmUM2bA0KHuC7SljRlj6ztcfTU4ndx+u92GttoiWoKCgvDy8iIvL899cao2RRO98gg1Z9s4nTB9ui134OODrfro7293GWnvpk+3xepfeIGAAPuJ5u67D91QXHv1qjE00SuPUDPRf/KJnWZ+/vnYugdPP203EamlLEK74+MD77xjdznfsYMrr7QTjf7739+bhIeH6zi9clkHeNcoT5efn4/T6aRbt26AHaW57z54+GEQ47RjNw8+2D6mUroqLs6unL3xRny8DY8+CtOmwYFFsT169KC8vJyihjadVQpN9MoD1CxJ/M9/Qng4nHIKtn67l5fd97WjueMO25V/5x3+7/8gMBA++sg+JCI6fKNc5lKiF5FJIrJZRJJEZGotj/9FRNZW3RaLyPBqj+2qOr5aRFY0Z/Cqfag+bFNaakcsHnkEJDXFDlC/+mrHGLKpqVMnu0nJXXche/cwe7b9cVRU2Ic10StXNfjuEREv4HngNCAeuEREataC3QGcYIwZAcwB5lV7zAkkGGNGGWPGNk/Yqr0oKSmhpKTk4JaBL70EI0fChPHGbgh7660wZIibo3Sj0aPh8svh9ts5+WQIC7PVmAGCg4MpKCjQLQZVg1zpJo0Fthpjko0xDmA+cHb1BsaYZcaYA3O9lgFR1R4WF19HdUCZmZmEhYUhIhQV2TIHc+Zg67/s3Gk3E+noHngAli1DvvqS2bPt5QqHw24xGBISojXqVYNcScBRQEq1+6kcmshruhb4qtp9AywSkZUicl3jQ1TtWfXa8y++CCecAMP7Ftjx6Xnz2ufCqMYKCLAfdaZM4cRxpfTta3dNBF0lq1zj03AT14nIH4CrgInVDh9njMkQkRBswt9kjKl1G+RZs2Yd/DohIYGEhITmDE95mPLycvLy8ggJCaGoyBajXLQI22U95RQ49lh3h+g5Tj3Vjmk9/jizZ9/PxRfbEZ3Q0FDWrl1LRUXFYaWdVfuTmJhIYmJio8+Thqrgich4YJYxZlLV/WmAMcY8VqPdcOATYJIxZnsdzzUTKDDGHFZAXESMVuTrWFJSUsjKymLMmDE8+ST8/DN8/OBGOPFEW6PdEzf4dqfdu+Hoo2HlSs64pS9nngm33ALLly8nJiaGyMhId0eoWpmIYIyRhtq5MnSzEhggIrEi4gtcDHxW48V6Y5P8ZdWTvIgEiEhg1dddgFOB9a5/G6o9OzDbprgYnngC7r/PwJQpcP/9muRr07u3LX5z++08+KBdZ1BaqrNvVMMaTPTGmEpgMrAQ2ADMN8ZsEpEbROT6qmb3AT2BF2tMowwDFovIauxF2s+NMbVskqY6msrKSrKzswkLC+OVV2DCBBi++SO7Y1Rb3Bawtdx5J2zYwJi9XzFypF0sHB4ezp49e3A6ne6OTnmoBoduWosO3XQsmZmZ7Ny5k5EjJ9C/P3z9cSHDLxpi6/JOnNjwE3RkX30Ft97KstfWc9Hlndm2DZYvX8ygQYMICQlxd3SqFTXn0I1Sze7Aath58+wWgcM/fwgSEjTJu+L00yE+nvFLnmTgQFsWR2ffqProZXrV6g7Uno+NHcTjj8P/Xt4JV86DdevcHVrb8eSTMG4cD867miumhnP++eHs2LEUYwwiDXbwVAejPXrV6nJycggICODdd/0ZPRqOeu8eu9mGzhpxXf/+cNVVTPjyPsLD4YsvutCpUyf279/v7siUB9IxetXqNmzYAPhy2mkDWfTgzwyZeSFs3gxdurg7tLZl/34YNIilsxZy7XMj+PjjzYgYhnTkkhEdjI7RK4+VkZHB99+Hc1S8Yci82+08QU3yjde9O8ycyYSP7yCwi2HlSp1mqWqniV61qvz8fIzx4tlng3hq3HxbivHSS90dVtt1/fVIRgbPT/qCp5/ujsNRQWFhobujUh5GE71qVRkZGaxbF05kjxKG/HOa3TmqI5Ygbi4+PvDEExzz0Z34GAdbtujOU+pw+g5TrSojI5O33grnpUFzkWOOgeOPd3dIbd/ppyN9+vDa6Jd4441wMjJ0+EYdShO9ajXFxcUsXVpG97JyBn/5pK1JrI6cCDzxBCO+mEOnAmHp0iJKS0vdHZXyIJroVavJzMzk3/8O47XoWcgVV9gpgqp5DBuGnHMOrw18nAULwvSirDqEJnrVar75JpOAtGIG/vYJzJjh7nDan5kzGbb8NRw7KklM1ESvfqeJXrWK8vJyXn89j9ci5yK33w5VWweqZhQVhdx4I2/FPs/rr+ficDjcHZHyEJroVav48ccs/DfnMDD9J7sKVrWMu+8mfteXlK/OY9myPe6ORnkITfSqVTz5ZCav9HoNue9eXRzVkrp3x+ueabwW+gZPPaXDN8rSRK9a3LZtlZgfvmeA2Q3XXuvucNq/m29mSPkWiv73PWlpWqNeaaJXreChOXt4LvAdvOc8qJt9twY/P3wens2zAW/zyMM6fKM00asWlpUFxR99SFS3CrjkEneH03Fceikx3crZ9+Z8tKCl0kSvWtTcJyt43Gcu3o89pKUOWpO3N97/eJhH5Wleek5n33R0+s5TLSYvD3JfmEdAZFc6n3uuu8PpcPzOPRffmB5kPf4SJSXujka5k0uJXkQmichmEUkSkam1PP4XEVlbdVssIsNdPVe1X/OeK2OWeZjSmffYZfqqdYlQ/uB0ZpQ/zLvzit0djXKjBhO9iHgBzwOnAfHAJSIyuEazHcAJxpgRwBxgXiPOVe1QSQnsf3we5QOi6f6nP7k7nA6rx6RJ5A2JY98Dz1NR4e5olLu40qMfC2w1xiQbYxzAfODs6g2MMcuMMXlVd5cBUa6eq9qnd14q5O9lc8iccg1BQUHuDqfD6tq1K3tuvZIbCh5nwZu57g5HuYkriT4KSKl2P5XfE3ltrgW+auK5qh2oqICC2XPJHzOWwOOOc3c4HV63cePIGv8H8mY8ju7W2TH5NOeTicgfgKuAiU05f9asWQe/TkhIICEhoVniUq3rP69nc03hXDbdNJfBERHuDqfDCw8PZ+utF3HBJdfz/XtTOOmvugl7W5WYmEhiYmKjz2twc3ARGQ/MMsZMqro/DTDGmMdqtBsOfAJMMsZsb8y5VY/p5uDtgDHwduidTDx6PzvvupiTTz7Z3SF1eMYYFi5cSNTTX7FtbRnnZLzk7pBUM2nOzcFXAgNEJFZEfIGLgc9qvFhvbJK/7ECSd/Vc1b4kvpvK2blvYu69kgjtzXsEESEsLAz/By/nhD3/4tf5W90dkmplDSZ6Y0wlMBlYCGwA5htjNonIDSJyfVWz+4CewIsislpEVtR3bgt8H8pDFN39AGln3kCas1ITvQcJDw8nq6KUrWfeTtHf73N3OKqVNTh001p06KbtW/3BZnr/9Xh8dqxm+ZaNnHLKKYjOn/cIlZWVLFq0iAnDx1ESE0/hB18w4MKj3R2WOkLNOXSjlEuK77yPLf93J3sdpYSHh2uS9yDe3t6EhISQV1HA2j/dS/Ft090dkmpFmuhVs9jx0S/0y1zKyNemkJGRocM2HigyMpL09HTGv3YtQVlbyXj/e3eHpFqJJnrVLIpvm86aP92HdBGKi4vp1auXu0NSNYSGhpKbm4t/N1h+5mzKbp+GTqzvGDTRqyO2Z/53BGTtYPyr15CRkUFYWBheWqnS4xwYvsnMzCTh5Ysp2FfG/n8ucHdYqhXou1EdGWMoue0elp4+hx6hnXTYxsMdGL4Jj/Tih9MeofzOGWgRnPZPE706IgXvLCAvu5w/vHQhpaWlFBQUEBIS4u6wVB0ODN84HA7OeHYS2/JCKJn3jrvDUi1ME71quooKSu+cwbcnPUxUjBeZmZmEhobqsI0H8/HxOTh806+/8L+ER3BMnwmlpe4OTbUgfUeqJit77R227g/h9GcmAZCRkUFkpNZR8XQRERGkp6cDcP6Tx/Jz6Sgcz7zo5qhUS9JEr5qmtJSy6bP44thHGDxEKCsrIy8vT4dt2oCwsDBycnJwOBwMHw4Lxj5MxUOP2i3BVLukiV41SeULL7O8dATnPH4sYHvzoaGheHt7uzky1RAfHx+Cg4PJzMwE4NKH4/mSM3A+/oSbI1MtRRO9arz8fMoffIT5Rz3E2LH2UHp6ug7btCGRkZFkZGQAMHEivB/3AI5nX4SsLDdHplqCJnrVaOaJJ1kkp3HRnGEAlJaWkp+fT2hoqJsjU64KCwtj3759OBwOAK5+IJZ/+V2OmT3HzZGplqBFzVTjZGVRPnAo58b8yhfr+yACO3bsIC8vj1GjRrk7OtUIK1asIDIykujoaIyBhPi9LEodjO+aldCvn7vDUy7QomaqRZg5D/HvLpdx1QM2yYMdtomK0h0i25oDi6cAROCGe0N4r+etcP/9bo5MNTdN9Mp1O3bg+Od7PB80nfPOs4dKSkooKioiODjYvbGpRgsPD2ffvn2Ul5cDcOGF8KS5g/KvvoG1a90cnWpOmuiV62bO5L2et3Lj/aEcWBOVnp5OeHi4LpJqg3x8fAgNDT14UdbHB26eGsQ/I6fDjBlujk41J313Ktf89hvlXy7iabmDiy/+/XBaWpoO27RhUVFRpKWlHbx/1VUwe88NlK9eDz/95MbIVHPSRK9cM306r4dOZ8r0IHx87KGioiJKS0u1JHEbFhoaSn5+PqVVJRD8/eGm2zrzRp8HYZqWMW4vNNGrhv30E2WrNvBEwQ1cfvnvh9PT04mIiNCdpNowLy+vQ0oiANxyC8zccill2flqtP+EAAAgAElEQVTwxRdujE41F5cSvYhMEpHNIpIkIlNreXyQiCwVkVIRuaPGY7tEZG31TcNVG2IMTJ3KSxEPctvUznTu/PtDOmzTPkRGRpKamnrwfteucMut3rwc8zBMnw6VlW6MTjWHBhO9iHgBzwOnAfHAJSIyuEazfcAU4B+1PIUTSDDGjDLGjD3CeFVr+/xzSrIL+UfaX7j22t8P5+fn43A46NGjh/tiU80iODiY0tJSioqKDh679VaYvfpPlHbuCu+/78boVHNwpUc/FthqjEk2xjiA+cDZ1RsYY7KNMb8Cte1gIC6+jvI0lZUwfTrPhT/M3+7wxt//94dSU1OJjo7WYZt2QESIjIw85KJs9+5w8y3CsxGP2nn1ZWVujFAdKVcScBSQUu1+atUxVxlgkYisFJHrGhOccrN336XYrwf/2HgmN930+2FjDGlpaURHR7svNtWsoqKiDhm+AbjtNnhs6fEU94uHV15xU2SqOfi0wmscZ4zJEJEQbMLfZIxZXFvDWbNmHfw6ISGBhISEVghP1aqkBO6/n6fj32fKWUJQ0O8PZWdn07lzZ4KqH1RtWo8ePTDGkJeXR7du3QDo2RNuuAGe3v4wMx4+1c691P9zt0pMTCQxMbHR5zVY60ZExgOzjDGTqu5PA4wx5rFa2s4ECowxT9XxXHU+rrVuPMyjj1L4wy/ErviYbdug+lD8mjVr6Nq1K/20Hkq7snnzZiorK4mPjz94LDsbBg2C3Sf+lS4jBsLMmW6MUNXUnLVuVgIDRCRWRHyBi4HP6nvtakEEiEhg1dddgFOB9S68pnKnvXvhiSeY0+VRbrnl0CRfWVlJZmamzrZph6Kjo0lLS6N6hys4GK65Bv4R+CA8+6z93VBtjkvVK0VkEvAM9g/D68aYR0XkBmzPfp6IhAG/AEHYWTaFwFAgBPgUO07vA7xnjHm0jtfQHr2nmDyZ/YU+DPhiLlu3Hpro09LSSE1NZdy4ce6LT7WYxYsXExcXd0jJ6awsGDIEdp8zhcBuPvD0026MUFXnao9eyxSrQ23ZAhMncvNJmwkb2uuwT+rLly8nOjpae/TtVHJyMtnZ2YwePfqQ43fcAV0Ks5j9yVBYtQpiY90UoapOyxSrppk6lT1XTeVf3/XittsOfaisrIzc3FzCw8PdE5tqcZGRkezdu/fghiQH3HUXvPhJGAWX36zj9G2QJnr1ux9+gLVruXv3ZG67DaomXxyUlpZGWFiY7gvbjnXq1ImQkJBDSiIARETAlVfCg0V3wldfwZo17glQNYkmemU5nXDnnaRNfoQvv/NjypTDm6SkpOjc+Q4gJiaGlJSUw45PnQpvfNKNfVNmwe23a8GzNkQTvbLmzwcvL/6+4iLuuMPWO6kuLy+PiooK3WCkAwgJCaG4uJjCwsJDjoeGwk03wT07rrPzLhcscFOEqrH0Yqyyi6OGDGHHA+8w4e7j2b4dAgMPbbJu3To6d+5MXFyce2JUrWrjxo2ICEOGDDnkeG4uxMXBmn8sImr2jbBxI4dUulOtSi/GKtc9+SSMGcPdnx/P3/9+eJKvrKwkLS2NmJgY98SnWl1MTAypqanU7Hz16GFLI9y96BSIj4dnnnFThKoxtEff0aWmwsiRrH39F864uQ9bt0JAwKFN0tLSSElJYfz48e6JUbnFTz/9xODBgwkJCTnkeEEBDBwIP76WRNyVx8KGDRAW5qYoOzbt0SvX3H033Hwzf3+uD/fff3iSB9i9eze9e/du/diUW/Xu3Zvk5OTDjgcF2V+baW/EwRVXwH33uSE61Ria6DuyxYth8WK+HTOV3bvh6qsPb1JUVER+fr7One+AoqKiyM7OpqyWEsU33QQrVsDqP90Hn32m0y09nCb6jqqyEm69Feejj3P3A1146CHo1OnwZgemVHp56a9KR+Pj40NERAS7d+8+7DF/f5gxA6Y+0h1mzbID9zr06rH03dtRvf46BAbysfdFiMD55x/exBhDSkqKDtt0YLGxsezevfuwi7IA114Lu3bBothrYd8++Pe/Wz9A5RJN9B1Rbi7cfz+OJ55hxr3Co49CbR32rKws/P39te58B9a9e3c6depEdnb2YY916gSPPgp33eND5dznbEGcatsRKs+hib4jeuABOOssXl81ij594OSTa2+2a9cu+vTp05qRKQ8UGxtb60VZgHPPhS5d4N3UBDj+eJg9u3WDUy5pjR2mlCdZuxbef5+iFRuYfZy9jlaboqIi8vLyGDtW93Pv6KKioti0aRNlZWV0rrE4SgT+8Q+4+GK48Icn8B87zM7EqbHQSrmXzqPvSJxOOO44uPpqZqVdR1ISvP9+7U03bNiAl5fXYSsjVce0du1aAgICGDhwYK2Pn38+HHMMTAt41pZG+PZb+1dAtSidR68O99prIELqadfw3HN2fLU2lZWVpKamEqs1x1WVPn36kJycXOtFWYBHHoEnnoC9f74ZcnJs7STlMbRH31Hs2QNHHQWLFnHZEyOIjYU5c2pvunv3bjIzM3XYRh1iyZIl9OvXj4iIiFofnzwZvL3hmYuWwp//DJs2HV4dTzUr7dGrQ919N1x2GSvKRvDddzBtWt1N9SKsqk3fvn3ZsWNHnY/PnAnvvQdbeh0Lp52mG5R4EE30HcEPP8C332JmzuKOO+zEiJqFyw7Izc3F4XAcVt9EqYiICEpKSsjLy6v18ZAQuOeeqrVTjz5ms/6qVa0cpaqNS4leRCaJyGYRSRKRqbU8PkhElopIqYjc0ZhzVQsrL7fr1efO5V9fB1FcbCdF1GXnzp307dsX0QtpqgYRoU+fPuzcubPONlOmwM6d8N8VIfD443ZVVUVFK0apatNgohcRL+B54DQgHrhERAbXaLYPmAL8ownnqpb0yCPQrx8lp5/H3XfD00/bcdTalJSUsGfPHl0Jq+rUu3dvMjMza61/A+DrC3Pn2l592cVXQHAwPPVUK0epanKlRz8W2GqMSTbGOID5wNnVGxhjso0xvwI1/3Q3eK5qQevWwfPPw8sv89jjwpgxcOKJdTffsWMHMTEx+Pjo8gpVO19fXyIiIupcQAUwaRIMHQpznxF45RXbs9+6tRWjVDW5kuijgOobSKZWHXPFkZyrjkRFhS1H+dBDbC+L5vnnbW++Lg6Hg5SUFPr169d6Mao2qW/fviQnJ+N0Outs89RTdiFVeue+tvrZ9ddr0TM38qiu26xZsw5+nZCQQEJCgttiafOefhqCgjDXXseUP9lJN/VtEJWSkkJoaCj+/v6tF6Nqk7p27UpQUBCpqal1DvMNGGBz+9Sp8M5bt9p59a+9Btdd18rRti+JiYkkJiY2+rwG59GLyHhgljFmUtX9aYAxxjxWS9uZQIEx5qkmnKvz6JtLUhIceywsX86nv/VnxgxbLtzXt/bmxhi+/fZbxowZQ/fu3Vs3VtUmZWdns27dOhISEuq8cF9YaCshvPsunNhzHZx0ki3BERnZytG2X805j34lMEBEYkXEF7gYqKNCin3tIzhXHSmn0/aa7r2XovD+3HYbvPhi3UkeID09HX9/f03yymXBwcH4+PiQlZVVZ5vAQHj2WbjhBiiLGwY332x/N7VD1+oaTPTGmEpgMrAQ2ADMN8ZsEpEbROR6ABEJE5EU4HZghojsFpHAus5tqW9GAc89Bw4HTJnCnDkwcSLUNwJmjGHr1q111jBRqi4DBgxgawMXWc89FwYPriq3MWMGZGbaIRzVqrQEQnuyYYPN6suWsbawPyefDL/9BnWsWAcgMzOTpKQkTjjhhFYLU7UPxhgSExMZNmwYwcHBdbZLSYFRo+zOlYOdG+3Ur+XLQS/8HzEtgdDRlJfDX/8KjzxCRWx/rrnG9qLqS/IASUlJxMXFtU6Mql0REfr378+2bdvqbRcTY/cPv/FGMEOGwvTpcPnldjtL1So00bcXs2bZd9Q11zB3LnTrVvtm39Xt2bMHp9NJWFhYq4So2p/o6GgKCgrYv39/ve0mT7YXZ996C/jb38DHB558slViVDp00z4sXmyrBa5Zw7aCMMaPhxUrGv5kvGTJEvr06UNUlC5tUE2XnJxMZmYm48aNq7fd6tW21tmaNRDpSIYxY2zd+uHDWynS9keHbjqK/Hz7MfiVV3CGhHHttfaTcUNJPjs7m7KyMiJ1qps6QjExMRQUFJCTk1Nvu1GjbNml668H0zvW9ugvvlj3mW0FmujbMmPsu+bUU+Gss5g3D0pK7CfjhmzZsoW4uDgtXqaOmJeXF3FxcSQlJTXYdsYMSEurGsK5/HK7LdWUKS0eY0enib4tmzfPbu7w9NNs2wb33gtvvll30bIDsrKycDgcOmSjmk10dDRFRUXs27ev3na+vvDPf9qV2ikpwAsvwNKltqSxajGa6NuqtWttZv/oIyo6+XPZZXD//baYVH2MMWzevJnBgwdrb141mwO9+i1btjTYdvhwW93y2mvBdAmEDz+0B1z4RKCaRhN9W1RQABdeaOvBDhrEo4/aVYiTJzd8anp6Ot7e3oSHh7d8nKpDiY6OprS0lD179jTYdupUyM2Fl18GRoyABx+Eiy6C0tKWD7QD0lk3bY0xcOml0KULvPoqv/wCZ54Jv/4K0dH1n+p0Ovn+++8ZOXIkvXr1ap14VYeSmZnJ5s2bOfHEExv8xJiUBMcdB99/D0fFG9t56dnTljZWLtFZN+3VM8/YcflnnqGw0K6ReuaZhpM82GlwXbp00SSvWkx4eDi+vr7s3r27wbZxcbaU8UUXQXGJwOuvw48/wquvtkKkHYsm+rbk22/hscdgwQKMfwA33WR7RBdf3PCpDoeDpKQkhjY0iK/UERo6dChbtmyhwoUtBK+4AkaOhNtvB7p2hQUL7NScZctaPtAORBN9W7Frlx2yef99iI3ljTfsApTnnnPt9C1bthAZGUnXrl1bNEylunfvTkhISIOlEQBE4KWX4Jtv4F//AgYNsj37Cy6AjIyWD7aD0ETfFhQXwznnwD33wB/+wLp1MG2afWMEBDR8ekFBAWlpaQwaNKjlY1UKGDx4MLt27aKkpKTBtl27wgcfwC23wI4dwP/9ny1nfMEFtoaTOmJ6MdbTOZ12EDMgAN56i4JC4Zhjfq8L5Yply5YRFhZG3759WzZWpapJSkoiLy+PY445xqX2zz5rO/NLl0IXf6dN9IGBduK9TgWulV6MbS/uvhuysmDePJxGuOIKOP5415N8RkYGpaWlxMbGtmycStUwYMAACgoK6t2cpLopU+wc++uuAyNe8M47sHmznXqpjogmek/2wgvwxRf2AlXnzsyebXP+88+7drrD4WD9+vUMHz4cLy/9r1aty8vLi2HDhrF+/XoqXShJLGJnVm7aZJeI0KULfP65rZfwzjstHm975lGbg6tqPv8cHnrIVqbs2ZNPP7Ufa1esgM6dXXuKTZs2ERYWRs+ePVs2VqXqEBISQvfu3dm6dSuDBw9usH1AAHz6KYwbZ9dRnXRSGPz3v3ZDnZiY+rdLU3XSbp4nWr7cFpNfsAD69WP9ervv5r//Da4uaM3JySErK4shQ4a0bKxKNSA+Pp7k5GTy8/Ndat+nj51cdsklduSGoUNh/ny7oGrNmhaNtb3SRO9p1q6Fs86yH1fHjiU93a58nTvXlu92RWVlJWvXriU+Pp5OnTq1aLhKNcTPz4+hQ4eyevVqnE6nS+f88Y/wyCP2d3/vXuCkk+wu92ecAS7U01GHcinRi8gkEdksIkkiMrWONs+KyFYRWSMio6od3yUia0VktYisaK7A26UtW+D00+0g/Jlnkp9v7950E/zlL64/zebNm+natavWmlceIyYmBn9//wY3E6/u6qttr/6ss2z5bS64wA5nnnoquLDyVv2uwUQvIl7A88BpQDxwiYgMrtHmdKC/MWYgcAPwUrWHnUCCMWaUMWZss0Xe3uzaBaecYn+R//xnysvhvPNg4kRbAMpV2dnZpKenM1x37VEeZvjw4ezatYu8vDyXz5k9G/r2hcsuq9pi9qqr4I474OSTITOz5YJtZ1zp0Y8Fthpjko0xDmA+cHaNNmcDbwMYY5YD3UTkwEak4uLrdFzJyfYX96674KqrcDptbyYw0M4tdnUKscPhYM2aNYwYMUKHbJTH8fPz46ijjuLXX391qTwC2N/9N96A/furNhc32J11Lr/cDufo6lmXuJKAo4CUavdTq47V1yatWhsDLBKRlSJyXVMDbbe2b4cTT7TLAqdMwRg7VLN7t70g1dAmItX99ttvhIWFERoa2nLxKnUEoqKi6NmzJ+vXr3f5HD8/Oy9h/Xr4+9+rkv2999qKfieeCKmpLRdwO9Ea0yuPM8ZkiEgINuFvMsYsrq3hrFmzDn6dkJBAQnufSrVli+3JT58ON92EMbaz8ttvsHCha+UNDti5cydFRUWMGjWq4cZKudGwYcP48ccfSU1NJdqVsqvYT7dffmlnVz74IMyciX3f+PnBCSfAd9/Z6TrtXGJiIomJiY0/0RhT7w0YD3xd7f40YGqNNi8DF1W7vxkIq+W5ZgJ31PE6pkNZu9aYiAhj3njDGGOM02nMnXcaM3q0Mbm5jXuqnJwc8/XXX5vCwsIWCFSp5peXl2e+/vprk5+f36jzMjONiYszZs6cageff96Y3r2N2by5eYNsA6ryZoN53JWhm5XAABGJFRFf4GLgsxptPgMuBxCR8cB+Y0yWiASISGDV8S7AqYDrn9naq2+/tT35uXPhqqswBu680/bi//c/6N7d9acqLy/n119/ZcSIEXTp0qXlYlaqGXXt2pWhQ4eycuVKHA6Hy+eFhUFioh3WnDGjahjnllvggQfsMM6SJS0Wc1vWYKI3xlQCk4GFwAZgvjFmk4jcICLXV7X5EtgpItuAV4Cbq04PAxaLyGpgGfC5MWZhC3wfbce779q5kv/6F1x4IZWVdu/MJUvsTjuN2RPE6XSycuVKoqKidGtA1ebExMQQFhbGr7/+euBTvUsiImyy//LLamP2V15pi5+dcw588klLhdxmafXK1mIMPPwwzJtnf0Pj4ykrs9eTcnPtxabAwMY95erVq6msrGT06NG60bdqk4wxLF++nKCgIOLj4xt1bm4uTJoEw4bZmvadOgGrVtkyx3feaTccb+fvC61e6UmKimyp4c8+szVY4+PJzbWr/ioqbN2yxib5rVu3UlhYyKhRozTJqzZLRBg9ejR79uxh+/btjTq3Rw+7YUlams3tBQXA0Ufbj8dvvGHnKOtm44Am+pa3cycce6ytxPfDDxAVxfbtMGECHHWUHcHx82vcUyYnJ7N7926OOeYYvBsz/1IpD9SpUyfGjx/Pzp07SW3kVMmgIFv/LzbWTr5JS8POvvn5Z7thz/HHQ0pKQ0/T7mmib0mffWYz+rXX2h6Gnx8//mj3ef3b3+y1WJ9GTnBNS0sjKSmJ8ePH49fYvxBKeSh/f3/GjRvHxo0b2bNnT6PO9fGBl1+2H5rHj6/abjYw8PdCaGPH2umXHZkrU3Na40Z7ml5ZUmLM5MnG9OljzJIlxhg7fXLuXGNCQoxZuLBpT5uRkWH+97//NXpKmlJtxYGpwpmZmU06f8EC+x576SX7njPGGLNokZ3KPHWqMWVlzResB8DF6ZV6Mba5bdxoKzENGmQvvHbvTl6eHS5MToaPPoJ+/Rr/tGlpaWzYsIFx48bRrVu35o9bKQ+Rm5vLihUrGDlyJGFhYQ2fUENSkq0Tdcwxdu+egABgzx645hpbMuG99+z7sx3Qi7GtzeGws2pOOAEmT4YPP4Tu3Vm5EkaPtnXkFy9uWpJPTk5m48aNTJgwQZO8avd69OjBuHHjWLt2LWlpaY0+Py7ODt+Ul9v33qpVQGioHUq96io7dvrcc1VV0joG7dE3h9WrbZc9NNT24mNjKS+3S7VffdX+Tl14YdOeesuWLaSmpjJ+/HhdEKU6lIKCApYvX05sbCwDBw5s0nO8/76dZXn77Xb7ZW9v7G4m111nO2evvmrnZ7ZR2qNvDfv32xUbp51mr65+/TXExrJmjf3Y+Ntvdh+RpiT5yspKfv31V/bu3cvEiRM1yasOJygoiIkTJ5Kens5vv/3m8qYl1f3lL/DLL7BokZ2As24dMHiwnQF31VW2Aua999op0O2YJvqmqKy0PYHBgyE/3/72XHklefnCbbfZfRFuvx3+8x/Xt/6rrri4mKVLlyIiHHvssXR2dZNYpdoZPz8/jjvuOMrKyli6dCmlTZgX37u3nW9/5ZV256p77oHiUi+7P+fatbaC7ODB8Pbb0IQ/Jm2BJvrGMAa++sru6ff223aF66uv4gwJ4913YcgQ2zHYuNH+UjVlHVNGRgY//fQTUVFRHH300Xh56X+R6th8fHwYM2YMYWFh/Pjjj+zdu7fRz+HlBddfbz9lJydDfLydfWkiIuGDD+wsiRdftFMxm1Id0sPpGL0rjLFdgvvvt8vvHngAzjsPg7Bwoe0heHvbsfjx45v2EhUVFWzcuJG9e/cyevRoujemsplSHUR2djZr1qwhLCyMoUOHNnnBYGKirZLg5QVPPGHnUGCMzf733QcxMTBrli2U5sFcHaPXRF+fykp7pf6JJ2DfPvsff+GF4OXFkiV2aC8jw+7+d955TS+rkZWVxbp16wgJCWHo0KG6O5RS9XA4HKxfv57c3FxGjBhBr8ZUAqzG6bR5ffp028OfMcMuYqeiwk7BnD3bJvxp0+x4rAeWGtFEfyQKC+Gtt+zS1eBge8H13HNxevnwxRfw+OM2wU+fDldc0fjVrQcUFxezceNG8vLyGDFiBMHBwc36bSjVnmVmZrJ+/Xp69uzJkCFD8Pf3b9LzlJbCm2/a93WfPvZ9ffLJIJUVdtrOU09BWZmdcHHZZbaciYfQRN9YxsDKlfD667YAzUkn2U2Ijz2WvDz7B/655+z/8d132x58UxO8w+Fg69at7N69m/79+9OvXz+tWaNUE1RWVrJt2zZ27txJ37596devX5M/ETsctof/yCP2vX3zzXDppRAUaOwsnWeegZ9+siVnr7wSRo5s3m+mCTTRuyo5GT7+2PbgS0rsfPgrrsBERrFsmZ1c8+mncMopdi/XhISmf4IrKytj586dJCcnEx4ezuDBg3VGjVLNoLi4mKSkJLKysujbty99+/ZtcsI3xpbGefFFu0fEJZfYjvy4cSC7dtpc8c9/2h2CrrzSFtmJiGjW78dVmujrs22b3Zzgk09sdcmzz4bLLsMcfwKr1wgffWQvwvv62lXTV1xh10I1VX5+PsnJyaSlpREVFUX//v0JaMyGsEoplxQVFR1M+FFRUfTp04egoKAmP19qqq1H+N57duj+L3+xtyGDnLaX/9Zb9jre0KH2Y/655zZt+XsTaaKvrqDAXmZfuNDe8vLsf8gFF1Ay9kR+XOrD11/buvBOp/0DfeGFMGJE03vvDoeDjIwMdu/eTUlJCbGxsfTu3VsrTirVCkpLS0lOTiY5OZmgoCCio6MJDw8/ol7+qlV2yH7+fDuEe+aZ8Kc/wfHjyvFd/J396L9gge0VnnKKHeg/4YTGbzbRCB070Wdl2XrUP/9sN/pYs8Z+7jr1VMoTTuWX8uEs+dmL776z9WdGjrQ71Zx+Oowa1fTkXl5eTlZWFunp6eTk5BAcHEzv3r0JDQ3VzUGUcgOn00lmZiZpaWlkZ2cTEhJCREQEISEh+Pr6Nuk5jbFVT/77X9s53LLFrro94QQ44bhKRjtX4pP4jZ2S/csvtuDOCSfYudfjxtkJHs2kWRO9iEwC5mIXWL1ujHmsljbPAqcDRcCVxpg1rp5b1a7xid4Yu9PAb7/9flu2zO4xNn48leMmkBYzgeXex/LLpi4sWWL/gwYNsnWNTjzR/tFt6pT1yspKcnJyyM7OZu/evRQVFREcHExkZCRhYWH4NPVqrVKq2R34lJ2VlUV2djZBQUGEhobSs2dPevTo0eQJEXv22FGcH3+0tx07bAmU0aPhmKFFTKj8iehdS5Dly+yEj+Bgm/BHj7a7Dx11lB3jb0JnsNkSvYh4AUnAH4F0YCVwsTFmc7U2pwOTjTFnisg44BljzHhXzq32HHUn+rw8u0y5+i0pySb2zp2pjB9OTtQwkrsNZ43vWJblDmLNb15s2mR/fiNG2Nuxx9qfb1OG7CoqKvj6668ZNmwY+/fvZ//+/RQVFdGtWzeCg4MJCQmhe/fuHrGSNTExkYSEBHeH0SCNs3lpnK5zOp3k5OSwZ88ecnJyyM/PJzAwkJ49e9K1a1eCgoJYtWoVf/zjHxv93Dk5tr+5erUd7lm1CrKzYfhwGBznZEKPzRxdvozYvLX0SFmH14Z1ds3OUUfZsf7+/e2tXz9769q1ztdyNdG70uUcC2w1xiRXPfF84GygerI+G3gbwBizXES6iUgY0NeFc3/3yiuQnn7wZtLTMbtToLSU4vD+5PbsT2aX/uzyHs1muZjlA4exKjWU/T/bn0f//jBgAIwZC1dfa4vSuZrUnU4nZWVllJaWUlpaSklJCYWFhRQVFVFYWIjD4eDjjz8mOjqa7t27ExsbS9euXT0isdfkCW8kV2iczUvjdJ2XlxfBwcEH1644nU72799Pbm4u+/btY9euXbzxxhsABAYGEhAQcMjNz88PX1/fWodke/aEM86wtwNycmxJrC1bvNi0ZSj/SR7Kli120l9ICAwP38M4r/UMS9pIn807CC9eQrd9O/DP3AF+fkjfPkhkpO25Vv/XRa4k+iig+qaLqdjk31CbKBfPPeirOb+S6owk2XEM20si2VYcSV5gFM7wUCKjhKgo+71FRsKASDgxyib4sLBKjKnE6XRSWVl58FZWVklxsf26oqKC8vJyHA4HDofjkK9LS0txOBz4+vri5+eHn58f/v7+BAUFERERQZcuXfD39z+4GYJSqn3x8vKiZ8+e9OzZ8+Cxb7/9lnHjxlFUVERxcTHFxcXk5ORQXFx8SM7o3LnzwVunTp3w8fHBx8fn4NcH/v9edAcAAAU5SURBVB050pujj/bCy8sLb29vvLy8cDq9ycryIjU1lJSUk9iechKJKXa2zx5fyPYxmD17CV6zk/5JGfQPyCDWN4NIWUFoZYbL319LDSI36crjuil/JSjIMCjQyehAQ5cuuXh55dS6NZbT6SQ/37B6tTn4g6t5q368U6dOB28BAQGH3Pfz86Nz5856wVQpdZCIEBgYSGAds2aMMZSVlVFeXk5ZWRllZWU4HI6DncqioiIqKiqoqKjA4XAc7Ig6nc5Dvq6srERE6NbNm+7dheHDBZFDb06nUFDQjfz87uTlDSW9SCgsFHjkS9e+FxfG6McDs4wxk6ruT7Pf4+8XVUXkZeB7Y8yHVfc3Aydih27qPbfac3jG9B+llGpDmmuMfiUwQERigQzgYuCSGm0+A24BPqz6w7DfGJMlItkunOtysEoppRqvwURvjKkUkcnAQn6fIrlJRG6wD5t5xpgvReQMEdmGnV55VX3ntth3o5RS6jAes2BKKaVUy/CYuYEi8riIbBKRNSLyiYjUPXnUjUTkAhFZLyKVInK0u+OpSUQmichmEUkSkanujqc2IvK6iGSJyG/ujqU+IhItIt+JyAYRWScit7o7pppEpLOILBeR1VUxznR3TPURES8RWSUin7k7lrqIyC4RWVv1M13h7njqUjWN/V9VeXND1RqmWnlMoscO78QbY0YCW4F73BxPXdYB5wI/uDuQmqoWqD0PnAbEA5eIyGD3RlWrN7ExeroK4A5jTDwwAbjF036expgy4A/GmFHASOB0EalzCrMH+Buw0d1BNMAJJBhjRhljPPln+QzwpTFmCDACqHNY3GMSvTHmG2PMgZ15lwHR7oynLsaYLcaYrTRxCmkLO7i4zRjjAA4sUPMoxpjFQK6742iIMSbzQCkPY0wh9o0U5d6oDmeMKa76sjP2uptHjseKSDRwBvCau2NpgOBBubE2VSMexxtj3gQwxlQYY/Lrau+p38zVwFfuDqINqmvhmjpCItIH22Ne7t5IDlc1HLIayAQWGWNWujumOjwN3IWH/iGqxgCLRGSliFzn7mDq0BfIFpE3q4bC5olInVtstWqiF5FFIvJbtdu6qn//r1qbGYDDGPN+a8bW2DhVxyEigcDHwN+qevYexRjjrBq6iQbGichQd8dUk4icCWRVfUISPPMT8QHHGWOOxn76uEVEJro7oFr4AEcDL1TFWgxMq69xqzHGnFLf4yJyJfaHe1KrBFSHhuL0YGlA72r3o6uOqSYSER9skn/HGPMfd8dTH2NMvoh8D0zC88bBjwPOEpEzAH8gSETeNsZc7ua4DmOMyaj6d6+IfIodEl3s3qgOkwqkGGN+qbr/MVDn5AuPGbqpKmd8F3BW1QWmtsDTeiUHF7eJiC//3879qlQQxFEc/57oH7AIYjEYbrUKNkWz3eQL+AA+gF0Eo02xa9GgRcR0MagP4E0+hBzDjsm9YpEdhvNJy7LhB7ucnd2Z33QNarWubqh9VPftDHizfTx0IX0kLUpaKMczwDbTNg0ckO1D2yu2V+mey7saQ17SbPmCQ9IcsAO8DFvVT7Y/gImkUTm1xS8v92qCHjgB5un+jY0lnQ5dUB9Ju5ImwDpwLamauQTbn8B3g9orcFljg5qkC+ARGEl6l7Q/dE19JG0Ae8BmWWo3LgOSmiwD95Ke6eYPbmz/bQOU6LMEPJQ5jyfgyvbtwDVNcwCcl3u/BhxNuzANUxERjatpRB8REf8gQR8R0bgEfURE4xL0ERGNS9BHRDQuQR8R0bgEfURE4xL0ERGN+wItI5mYh1zesQAAAABJRU5ErkJggg==\n", 434 | "text/plain": [ 435 | "" 436 | ] 437 | }, 438 | "metadata": {}, 439 | "output_type": "display_data" 440 | } 441 | ], 442 | "source": [ 443 | "# overparameterizing the distribution\n", 444 | "plot(ps[:,0], ps[:,1])\n", 445 | "plot(xs, p_x_theta(xs, mean(D)), color=\"black\", alpha=0.3)\n", 446 | "plot(xs, p_x_theta(xs, mean(D), var(D)**.5), color=\"red\")" 447 | ] 448 | }, 449 | { 450 | "cell_type": "markdown", 451 | "metadata": { 452 | "slideshow": { 453 | "slide_type": "slide" 454 | } 455 | }, 456 | "source": [ 457 | "## Maximum Likelihood vs Bayesian Estimates\n", 458 | "\n", 459 | "The Bayesian estimate $p(x|D)$ usually is quite different from the ML estimate $p(x|\\hat{\\theta}(D))$. It often doesn't even have the same form.\n", 460 | "\n", 461 | "ML estimates tend to be \"overtrained\" / underestimate variance. Bayesian estimates can never be \"overtrained\".\n", 462 | "\n", 463 | "Using an overparameterization accidentally can compensate for this to some degree (that probably makes DNNs work better)." 464 | ] 465 | }, 466 | { 467 | "cell_type": "markdown", 468 | "metadata": { 469 | "slideshow": { 470 | "slide_type": "slide" 471 | } 472 | }, 473 | "source": [ 474 | "# Maximum A-Posteriori Estimate" 475 | ] 476 | }, 477 | { 478 | "cell_type": "markdown", 479 | "metadata": { 480 | "slideshow": { 481 | "slide_type": "slide" 482 | } 483 | }, 484 | "source": [ 485 | "## Maximum A Posteriori Estimate\n", 486 | "\n", 487 | "By analogy to Bayesian methods, we can also ``multiply in'' a prior:\n", 488 | "\n", 489 | "$$\\hat{\\theta} = \\arg\\max_\\theta ~ p(D|\\theta) p(\\theta)$$\n", 490 | "\n", 491 | "This is called the _maximum a-posterior estimate_ (MAP) for the parameter $\\theta$." 492 | ] 493 | }, 494 | { 495 | "cell_type": "markdown", 496 | "metadata": { 497 | "slideshow": { 498 | "slide_type": "slide" 499 | } 500 | }, 501 | "source": [ 502 | "## MAP vs Bayesian Methods\n", 503 | "\n", 504 | "Although MAP looks like the derivation may have involved _Bayes rule_, it is\n", 505 | "not a Bayesian method at all.\n", 506 | "Bayesian methods are *not* methods that involve Bayes rule somewhere.\n", 507 | "\n", 508 | "**Bayesian methods are methods that result in decisions that minimize expected loss.**\n", 509 | "\n", 510 | "**Bayesian methods are NOT methods that happen to use Bayes rule somewhere.**" 511 | ] 512 | }, 513 | { 514 | "cell_type": "markdown", 515 | "metadata": { 516 | "slideshow": { 517 | "slide_type": "slide" 518 | } 519 | }, 520 | "source": [ 521 | "# Summary\n", 522 | "\n", 523 | "We know how to do classification / estimation optimally and correctly: Bayesian methods.\n", 524 | "\n", 525 | "Why aren't we using them with DNNs? Because they are computationally prohibitive.\n", 526 | "\n", 527 | "However, several methods attempt to approximate Bayesian solutions:\n", 528 | "- dropout methods\n", 529 | "- ensemble methods\n", 530 | "- some forms of variational methods" 531 | ] 532 | }, 533 | { 534 | "cell_type": "markdown", 535 | "metadata": { 536 | "slideshow": { 537 | "slide_type": "slide" 538 | } 539 | }, 540 | "source": [ 541 | "# Uniform Distribution Example" 542 | ] 543 | }, 544 | { 545 | "cell_type": "markdown", 546 | "metadata": { 547 | "slideshow": { 548 | "slide_type": "slide" 549 | } 550 | }, 551 | "source": [ 552 | "Bayesian vs Maximum Likelihood Example\n", 553 | "==============================\n", 554 | "\n", 555 | "Let's apply this to another, simple example and compare maximum likelihood and bayesian approaches\n", 556 | "~\n", 557 | "Assume that the samples $x$ come from a uniform density over the interval $[0,\\theta]$\n", 558 | "\n", 559 | "$$p(x|\\theta) = U(x;0,\\theta) = 1/\\theta \\cdot \\lfloor x\\in[0,\\theta]\\rfloor$$\n", 560 | "\n", 561 | "We also assume a prior\n", 562 | "\n", 563 | "$$p(\\theta) = U(\\theta;0,10)$$\n", 564 | "\n", 565 | "That is, $\\theta$ is distributed uniformly over the interval $[0,10]$\n", 566 | "\n", 567 | "Let's assume we see a sequence of training examples $D = \\\\{4,7,2\\\\}$" 568 | ] 569 | }, 570 | { 571 | "cell_type": "code", 572 | "execution_count": 8, 573 | "metadata": { 574 | "slideshow": { 575 | "slide_type": "slide" 576 | } 577 | }, 578 | "outputs": [ 579 | { 580 | "data": { 581 | "text/plain": [ 582 | "[]" 583 | ] 584 | }, 585 | "execution_count": 8, 586 | "metadata": {}, 587 | "output_type": "execute_result" 588 | }, 589 | { 590 | "data": { 591 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEACAYAAABfxaZOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAEpBJREFUeJzt3W2MHeV5xvHr2jhWeKsbKuIIvzUN1GlIE5cPjlta5bS0iUMFjlRR2ZGahirIamNKQtWSUlWsP0FbtRURtMGKi0IFIgJKcSRCDaLbFjmAIUCA2Ng0jWMb1wkECCRAjffuhzO2x0frPTNnZnfm2fn/JMt7zpk5exueufZ57pk564gQAKAbxpouAAAwewh9AOgQQh8AOoTQB4AOIfQBoEMIfQDokEKhb3u17Z22d9m+corXP2n7yezPg7Y/WHRfAMDs8bDr9G2PSdol6XxJz0vaLmltROzMbbNK0o6IeMX2aknjEbGqyL4AgNlTZKa/UtLuiNgTEYck3SZpTX6DiHgoIl7JHj4kaVHRfQEAs6dI6C+StDf3eJ+OhfpUPiPp6yPuCwCYQfPqfDPbvy7pEkm/Wuf7AgDqUST090tamnu8OHvuONnJ202SVkfES2X2zfbnQ4AAoKSIcJnti7R3tks6y/Yy2/MlrZW0Jb+B7aWS7pT0exHx32X2HSg+yT9XX3114zVQf/N1UH+af1KufxRDZ/oRcdj2Bklb1f8hsTkidthe3385Nkn6S0mnS/oH25Z0KCJWnmjfkSoFAFRWqKcfEfdKWj7w3I25ry+VdGnRfQEAzeCO3Br0er2mS6iE+ptF/c1Kvf6yht6cNVtsR1tqAYAU2FbMwIlcAMAcQegDQIcQ+gDQIYQ+AHQIoQ8AHULoA0CHEPoA0CGEPgB0CKEPAB1C6ANAhxD6ANAhhD4AdAihDwAdUuvvyO2qa6+Vnnuu6SrQFp/7nPSBDzRdBTA1Qr8GN9wgXXaZdPrpTVeCpt18s/SNbxD6aC9Cvybr1klLljRdBZr2yCNNVwBMj55+DfjdL8hjPKDNCH0A6BBCvyYu9QvLMFcxDtB2hH4NWM4jj/GANiP0a8IMDxLjAO1H6ANAhxD6NWA5jzzGA9qM0K8Jy3pIjAO0H6FfA2Z2yGM8oM0IfQDoEEK/JizrITEO0H6Efg1YziOP8YA2I/QBoEMI/ZqwrIfEOED7Efo1YDmPPMYD2ozQB4AOIfRrwrIeEuMA7Ufo14DlPPIYD2gzQr8mzPAgMQ7QfoQ+UDNm+mgzQr8GHOQAUkHo14RlPSTGAdqP0K8BM33kMR7QZoQ+AHQIoV8TlvWQGAdoP0K/Biznkcd4QJsR+gDQIYR+TVjWQ+qPA2b6aDNCvwYc5ABSUSj0ba+2vdP2LttXTvH6ctvbbL9h+4qB175r+0nbj9t+pK7C24aZPiTGAdpv3rANbI9Jul7S+ZKel7Td9t0RsTO32YuSLpP0iSneYlJSLyJeqqFeoPVY+aHNisz0V0raHRF7IuKQpNskrclvEBEvRMRjkt6aYn8X/D7J4iAHkIoiYbxI0t7c433Zc0WFpPtsb7d9aZniUsKyHhLjAO03tL1Tg/Mi4oDtM9QP/x0R8eBUG46Pjx/9utfrqdfrzUJ51THTRx7jATNlYmJCExMTld6jSOjvl7Q093hx9lwhEXEg+/sHtu9Sv100NPQBAMcbnAxv3Lix9HsUae9sl3SW7WW250taK2nLNNsfXeDaPtn2qdnXp0j6qKSnS1eZAJb1kBgHaL+hM/2IOGx7g6St6v+Q2BwRO2yv778cm2wvlPSopNMkTdq+XNL7JZ0h6S7bkX2vWyJi60z9Y5rCch55jAe0WaGefkTcK2n5wHM35r4+KGnJFLu+JmlFlQIBAPWZ05dSziaW9ZD4GAa0H6FfAw5yAKkg9GvCTB8S4wDtR+gDNWPlhzYj9GvAQQ4gFYR+TVjWQ2IcoP0IfaBmrPzQZoR+DTjIAaSC0K8Jy3pIjAO0H6FfA2b6yGM8oM0IfQDoEEK/JizrIfExDGg/Qr8GHOQAUkHoA0CHEPo1ob0DifYO2o/QrwEHOYBUEPo1YaYPiXGA9iP0gZqx8kObEfo14CAHkApCvyYs6yExDtB+hH4NmOkjj/GANiP0AaBDCP2asKyHxHX6aD9CvwYc5ABSQegDQIcQ+jWhvQOJ9g7aj9CvAQc5gFQQ+jVhpg+JcYD2I/SBmrHyQ5sR+jXgIAeQCkK/JizrITEO0H6Efg2Y6SOP8YA2I/QBoEMI/ZqwrIfEdfpoP0K/BhzkAFJB6ANAhxD6NaG9A4n2DtqP0K8BBzmAVBD6ANAhhH5NaO9Aor2D9iP0AaBDCP2aMNOHxEwf7UfoA0CHEPoVMasDkBJCH6gR7R20HaFfEQc4gJQQ+gDQIYVC3/Zq2ztt77J95RSvL7e9zfYbtq8os+9cwJU7OIL2DtpuaOjbHpN0vaSPSTpH0jrb7xvY7EVJl0n6mxH2TRoHOICUFJnpr5S0OyL2RMQhSbdJWpPfICJeiIjHJL1Vdl8AwOyZV2CbRZL25h7vUz/Mi6iybzrOfFRfe/ZA01WgBXaG9Ialrz3bdCUoYsmCJVrx7hVNlzGrioT+rBkfHz/6da/XU6/Xa6yWoiKkuPh3dN3DZ+ukt5/UdDlo2O5J6fCY9MY3m64Ew7z65qt68fUX9dQfPtV0KYVNTExoYmKi0nsUCf39kpbmHi/Oniui1L750E/K2GHdtOYmLVmwpOlK0LBrrpF+9Lp0zbqmK8Ewz3z/GV18+8VNl1HK4GR448aNpd+jSE9/u6SzbC+zPV/SWklbptk+fy1L2X0BADNo6Ew/Ig7b3iBpq/o/JDZHxA7b6/svxybbCyU9Kuk0SZO2L5f0/oh4bap9Z+xf04D+1Tshc90mkBTbCnXv8rtCPf2IuFfS8oHnbsx9fVDSlL2NqfadcxyyCH1wnX5KLCs6+D+LO3Ir6uCYAZAwQr8WtHeA1HS1vUPo14H2DjK0d9JBewcj6eCYAZAwQr8WtHeA1NDeweho7yBDeycdtHcwkg6OGQAJI/TrYNo76GOmnw7aO6iA9g6QGto7GEkHxwyAhBH6daC9gwztnXTQ3kEFtHeA1NDewUg6OGYAJIzQrwPtHWRo76SD9g5GcvTz9GnvAEmhvQMAmPMI/TrQ3kGG9k46aO9gJBzgAFJC6NeCnj6OYSKQBnr6GB3tHWQYBumgvYORdHCiACBhhH4taO+gjxO56aC9g9HR3gGSQ3sHI+ngRAFAwgj9WtDeQR/tnXTQ3sFIjn4MA+0dICm0dwDUooOTRySE0K+Dae+gjwVfOmjvYCS0d4A00d4BUIsOTh6REEK/DrR3kGHBlw7aOxgJ7R0gTbR3AFTGdfpoO0K/DrR3gOTQ3sFIaO8AaaK9A6Ay2jtoO0K/ogjR3gESRHsHo+OjlZHTwRxJEu0dAJXxsx9tR+hXdGRWR3sHSAvtHYzkyKChvYMjOpgjSaK9A6Ayfvaj7Qj9iiaZ1gFJor2DkUSEFEzvcEwHcyRJtHcAVEZ7B21H6Fc0yUwfSBLtHYykP2gIffTxMQzpoL0zDdurbe+0vcv2lSfY5ou2d9t+wvYv5Z7/ru0nbT9u+5G6CgcAlDdv2Aa2xyRdL+l8Sc9L2m777ojYmdvm45LeGxFn2/6wpH+UtCp7eVJSLyJeqr36FqC9g0HM9NNAe+fEVkraHRF7IuKQpNskrRnYZo2kmyUpIh6WtMD2wuw1F/w+SeovDwl99HEiNx20d05skaS9ucf7suem22Z/bpuQdJ/t7bYvHbXQturgRAFAwoa2d2pwXkQcsH2G+uG/IyIenGrD8fHxo1/3ej31er1ZKK8aTuRiEBOBNKTY3pmYmNDExESl9ygS+vslLc09Xpw9N7jNkqm2iYgD2d8/sH2X+u2ioaEPpIj2DmbS4GR448aNpd+jSHtnu6SzbC+zPV/SWklbBrbZIulTkmR7laSXI+Kg7ZNtn5o9f4qkj0p6unSVLcaJXCBNXe3pD53pR8Rh2xskbVX/h8TmiNhhe33/5dgUEffYvsD2c5J+LOmSbPeFku6yHdn3uiUits7MP6UZnMjFoMQ6Bp2VYnunDoV6+hFxr6TlA8/dOPB4wxT7/Y+kFVUKBFJCewdtN2cvpZwtMUl7B0hRV9s7hH5FIX4pOo7hYxjS0dX2DqEPAB1C6Fc0OcmJXByvg5PHJNHewUhC9PRxDCdy09HVtiyhXxGzOgApIfQr4jp9DGIikAZny7Kuncwl9Cvis3eQR3snPV3r6xP6FXVskgAgcYR+RZzIxSAmAuno4rX6hH5ltHdwDO2dtHTxsk1Cv6KOTRIAJI7QryiCj2HAMXwMQ1po76A0evpAumjvoLSOTRJQAGMCbUboV8TNWcjjRG5aaO+gtODXJQLJor2D0jo2SUABjAm0GaFfEe0d5NHeSQvtHZTGZ+8A6aK9A6Cyjk0ckRhCvyJm+sijvZMW2jsobTJC5uod5HQsQ5JGewcAMKcR+hVx9Q7yaO+khfYOSpucJPRxvI5lSNJo7wCohJk+2o7Qr4hP2QTSRXsHpXHJJgZ1LEOSRnsHQCW0d9B2hH5Fk/zmLCBZtHcAVNaxDEFiCP2K+Dx95NHeSQs9fZQ2yYlcDGCmnw7aOwCAOY3Qr4yZPo6hvZMW2jsobXKSnj6O17FuQdJo72AkzO5wBGMBbUfoV8THMADpor2D0rh6B4M61i1IGu0dAJXQ3kHbEfoV8YFrQLpo76A07sjFoI51C5JGewdAJbR30HaEfkV8yiYGdWzimDTaOyiNX4wOpIv2zgnYXm17p+1dtq88wTZftL3b9hO2V5TZF5graO+g7YaGvu0xSddL+pikcySts/2+gW0+Lum9EXG2pPWSvlR039RFhCb3/qTpMiqZmJhouoRK2lZ/2Ylj2+ovK+X6bWvbf21ruoxZVWSmv1LS7ojYExGHJN0mac3ANmsk3SxJEfGwpAW2FxbcN2n90H+96TIqSfmglai/aSnXbxH6U1kkaW/u8b7suSLbFNkXmDNo76Dt5s3Q+7Zm6P/focNa8mefmLH3fzNe5eodHDU2Jm3bJl14YfF9nn1WeuyxmatppqVc/w9/8W268T/u1q2ff25G3n+e52v/3905I+89Kg87c217laTxiFidPf6CpIiIv8pt8yVJ/x4RX80e75T0EUnvGbZv7j26dQodAGoQUe7u0CIz/e2SzrK9TNIBSWslrRvYZoukz0r6avZD4uWIOGj7hQL7jlQ4AKC8oaEfEYdtb5C0Vf1zAJsjYoft9f2XY1NE3GP7AtvPSfqxpEum23fG/jUAgGkNbe8AAOaO1tyRa/uvbe/Ibu660/ZPNV1TESnffGZ7se0HbD9j+ynbf9x0TWXZHrP9Tdtbmq6lLNsLbN+ejftnbH+46ZrKsP1520/b/pbtW2zPb7qm6djebPug7W/lnnun7a22n7X9b7YXNFnjdE5Qf+ncbE3oq98COiciVkjaLenPG65nqDlw89lbkq6IiHMk/bKkzyZWvyRdLunbTRcxousk3RMRvyDpQ5KSaX3aPlPSZZLOjYgPqt8qXttsVUPdpP6xmvcFSfdHxHJJD6jduTNV/aVzszWhHxH3R8Rk9vAhSYubrKegpG8+i4j/jYgnsq9fUz90krmPwvZiSRdI+nLTtZSVzch+LSJukqSIeCsiftRwWWW9TdIptudJOlnS8w3XM62IeFDSSwNPr5H0lezrr0iaueu7K5qq/lFyszWhP+APJH296SIKmDM3n9n+WUkrJD3cbCWl/L2kP5WS/JjE90h6wfZNWXtqk+2Tmi6qqIh4XtLfSvqepP3qX7F3f7NVjeRdEXFQ6k+CJL2r4XqqKJSbsxr6tu/L+n9H/jyV/X1hbpu/kHQoIm6dzdq6zPapku6QdHk24289278t6WC2UrFadENgQfMknSvphog4V9JP1G81JMH2T6s/S14m6UxJp9r+ZLNV1SLFCUSp3JypO3KnFBG/Nd3rtj+t/nL9N2aloOr2S1qae7w4ey4Z2dL8Dkn/HBF3N11PCedJusj2BZJOknSa7Zsj4lMN11XUPkl7I+LR7PEdklK6EOA3JX0nIn4oSbb/RdKvSEptsnbQ9sLsvqJ3S/p+0wWVVTY3W9Pesb1a/aX6RRHxZtP1FHT0xrXsyoW16t+olpJ/kvTtiLiu6ULKiIirImJpRPyc+v/dH0go8JW1FPba/vnsqfOV1gnp70laZfsdtq1+/SmciB5cFW6R9Ons69+X1PaJz3H1j5KbrblO3/ZuSfMlvZg99VBE/FGDJRWS/Ue/TsduPru24ZIKs32epP+U9JT6y9qQdFVE3NtoYSXZ/oikP4mIi5qupQzbH1L/JPTbJX1H0iUR8UqzVRVn+2r1f+AekvS4pM9kFzS0ku1bJfUk/Yykg5KulvSvkm6XtETSHkm/GxEvN1XjdE5Q/1UqmZutCX0AwMxrTXsHADDzCH0A6BBCHwA6hNAHgA4h9AGgQwh9AOgQQh8AOoTQB4AO+X/yyD6I7NWHhAAAAABJRU5ErkJggg==\n", 592 | "text/plain": [ 593 | "" 594 | ] 595 | }, 596 | "metadata": {}, 597 | "output_type": "display_data" 598 | } 599 | ], 600 | "source": [ 601 | "xs = linspace(-1,11,1000)\n", 602 | "mus = xs\n", 603 | "C = (amax(xs)-amin(xs))/len(xs)\n", 604 | "def pxt(x,mu): return (xs>=0)*(x<=mu)*1.0/maximum(mu,1e-6)\n", 605 | "plot(xs,pxt(xs,5.5))\n", 606 | "pmu = (xs>=0)*(xs<=10)*0.1\n", 607 | "plot(xs,pmu)" 608 | ] 609 | }, 610 | { 611 | "cell_type": "markdown", 612 | "metadata": { 613 | "slideshow": { 614 | "slide_type": "slide" 615 | } 616 | }, 617 | "source": [ 618 | "Now assume we draw the sample $x_1=4$. How should we update our estimate?\n", 619 | "\n", 620 | "$$p(\\mu|x) = \\frac{p(x|\\mu) p(\\mu)}{p(x)}$$" 621 | ] 622 | }, 623 | { 624 | "cell_type": "code", 625 | "execution_count": 9, 626 | "metadata": { 627 | "slideshow": { 628 | "slide_type": "slide" 629 | } 630 | }, 631 | "outputs": [ 632 | { 633 | "data": { 634 | "text/plain": [ 635 | "[]" 636 | ] 637 | }, 638 | "execution_count": 9, 639 | "metadata": {}, 640 | "output_type": "execute_result" 641 | }, 642 | { 643 | "data": { 644 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEACAYAAABfxaZOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAGwFJREFUeJzt3XuUVPWZ7vHv0yBeSLwlwpmAeE0wMhEkM+gRNaUxSnQQczGCMTOaJToKxkRzJF6izUQTxYm3GCcyMiY4ZqHG4GV5Cd56lHEUVLymEaJREdGjJhqN8UjwPX/sQsu26a6qru5dv97PZ61eVu3au+pt7P3Ur9767b0VEZiZWTG05F2AmZn1HYe+mVmBOPTNzArEoW9mViAOfTOzAnHom5kVSFWhL2mCpKWSlkma0cnjB0p6RNISSYskja92WzMz6zvqbp6+pBZgGfB54AVgMTA5IpZWrLNRRLxVvv0Z4OqI+HQ125qZWd+pZqQ/DlgeEc9GxGpgHjCpcoW1gV/2EeDdarc1M7O+U03oDwNWVNx/vrzsAyQdJKkduBH4Zi3bmplZ32jYF7kRcV1EfBo4CDizUc9rZmaNM7CKdVYCIyruDy8v61RELJS0raTNa9lWkk8CZGZWo4hQLetXM9JfDGwvaStJg4DJwA2VK0jaruL2WGBQRPyhmm07FJ/kzxlnnJF7Da4//zpcf5o/Kddfj25H+hGxRtJ0YAHZm8SciGiXdHT2cMwGviLpH4F3gL8AX+tq27oqNTOzHqumvUNE3AqM7LDs0orbs4BZ1W5rZmb58BG5DVAqlfIuoUdcf75cf75Sr79W3R6c1VckRbPUYmaWAklEL3yRa2Zm/YRD38ysQBz6ZmYF4tA3MysQh76ZWYE49M3MCsShb2ZWIA59M7MCceibmRWIQ9/MrEAc+mZmBeLQNzMrEIe+mVmBOPQtaS+9BOeem3cVZulw6FvSnnoKTjoJHn8870rM0uDQt37hrLPyrsAsDQ59S1oEfOYzcMcd8OSTeVdj1vwc+pa8j34UjjsOfvjDvCsxa35VXRjdrNkddxxsvz08/TRsu23e1Zg1L4/0LWkRIMGmm8Kxx8KPfpR3RWbNzaFv/ca3vw2//jU8+2zelZg1L4e+9Rubbw7HHAP/8i95V2LWvBz6lrS17Z21vvtduPFGz+QxWxeHvvUrm24KJ54I3/9+3pWYNSeHviUt4sPLpk+HhQvhoYf6vh6zZufQt+RVtncABg+GU0+F007Lpx6zZlZV6EuaIGmppGWSZnTy+KGSHin/LJS0U8Vjz5SXL5G0qJHFm63L1KnQ3g733JN3JWbNpdvQl9QCXAzsB4wCpkjaocNqTwN7RsRo4ExgdsVj7wKliNg5IsY1pmyzTMcvctcaNAhmzoQZMzpvAZkVVTUj/XHA8oh4NiJWA/OASZUrRMR9EfF6+e59wLCKh1Xl65g11GGHwdtvwzXX5F2JWfOoJoyHASsq7j/PB0O9oyOBWyruB3CbpMWSptZeoll9Wlrgxz/ORvtvv513NWbNoaEjcEl7AUcAlX3/8RExFtgfmCZp90a+phXbuto7a+21F4weDRdd1Hc1mTWzak64thIYUXF/eHnZB5S/vJ0NTIiIP65dHhGryv99WdJ8snbRws5eqLW19b3bpVKJUqlURXlmXZs1C3bbDQ4/HIYMybsas/q1tbXR1tbWo+dQdPMtl6QBwJPA54FVwCJgSkS0V6wzArgD+EZE3FexfCOgJSLelDQYWADMjIgFnbxOdFeLWUd33ZV9YdvdfvDtb8M778All/RJWWZ9QhIR0cVn3Q/rdqQfEWskTScL7BZgTkS0Szo6ezhmA98HNgcukSRgdXmmzlBgvqQov9aVnQW+WU901d5Z6/TTYeRImDYNRo3q/ZrMmlW3I/2+4pG+1eOuu7ITrN11V/frXnQRXH893H57dW8UZs2unpG+p1Ja0moZJxx7LLzyClx9de/VY9bsHPqWvGpH7QMHZj39E0+EN97o3ZrMmpVD3wpl/Hj4wheyL3/Nisihb0nrbp5+Z845B+bOhccf752azJqZQ98KZ8gQaG3NZvJ47oAVjUPfCunoo+HPf4af/zzvSsz6lkPfklZPewdgwAC47LLsvDwvvtj4usyalUPfCmvMGDjyyOxKW2ZF4dC3pPW0J3/66fDYYzB/fmPqMWt2Dn1LXk+Ort1gg6zNM306vPZa42oya1YOfSu8PfaASZPgu9/NuxKz3ufQt6Q1asrl2Wdn5+S5+ebGPJ9Zs3LoW/IacfK0jTfOpm9OnZqdn8esv3Lom5WVSjB5MhxzjA/asv7LoW9Jq3ee/rqcdRa0t8OVVzbuOc2aiUPfrMIGG8AVV8AJJ8CKFXlXY9Z4Dn2zDnbeObu84uGHw7vv5l2NWWM59C1pjW7vrHXSSbB6dTarx6w/ceibdWLgQPjlL7NLLC5cmHc1Zo3j0Lek9eYsm+HDYc4cOPRQePXV3nsds77k0Lfk9eZFzg84AA4+GI44wtM4rX9w6Jt140c/yk6/fOGFeVdi1nMD8y7ArCf6YvQ9aBDMmwe77grjxsFuu/X+a5r1Fo/0LXm92d5Za9tt4fLL4Wtfg1Wrev/1zHqLQ9+sSgccAEcdlfX433kn72rM6uPQt6T11jz9dTntNPjYx7Ijds1S5NA3q0FLC8ydCwsWwC9+kXc1ZrXzF7lmNdpkk+zyiqUS7Lgj/P3f512RWfU80rek9XV7Z61Ro7LLLB50EDz3XN+/vlm9qgp9SRMkLZW0TNKMTh4/VNIj5Z+FknaqdluzVE2aBN/5DkycCG+8kXc1ZtXpNvQltQAXA/sBo4ApknbosNrTwJ4RMRo4E5hdw7Zmdcv7KNkTT4RddoEpU2DNmnxrMatGNSP9ccDyiHg2IlYD84BJlStExH0R8Xr57n3AsGq3NeupPNo7la/905/C22/7wuqWhmpCfxhQeTmJ53k/1DtzJHBLnduaJWe99eCaa+CWW7I3ALNm1tDZO5L2Ao4Adq9n+9bW1vdul0olSqVSQ+qy/ivv9s5am20GN98Me+wBQ4fCV7+ad0XWH7W1tdHW1taj56gm9FcCIyruDy8v+4Dyl7ezgQkR8cdatl2rMvTNqpVne6fSttvCTTfBvvtmB3DttVfeFVl/03EwPHPmzJqfo5r2zmJge0lbSRoETAZuqFxB0gjgWuAbEfFULdua9SdjxsBVV8Ehh8CSJXlXY/Zh3YZ+RKwBpgMLgCeAeRHRLuloSUeVV/s+sDlwiaQlkhZ1tW0v/B5WUHnN0+/KXnvBJZfAP/wDPPVU9+ub9aWqevoRcSswssOySytuTwWmVrutWX/31a/Cyy/DfvvBPffA3/xN3hWZZXwaBrNecswx2WUW99kH2tpgiy3yrsjMoW+Ja8b2TqXTTsvm8O+zD9x1F2y+ed4VWdH53DtmvewHP8hm9Oy7L7z2Wt7VWNE59C1pzTJPvysSzJoF48fDF7/o8/RYvhz6lrxmbu+sJcEFF8Do0Vnw/+lPeVdkReXQN+sjUjaVc6edsh7/H/6Qd0VWRA59S1oK7Z1KLS3Z+Xn22AP23jub1mnWlxz6lrwU2juVJPjXf83Ow/+5z8ELL+RdkRWJp2ya5UDKZvVsuCHsuSfccQdstVXeVVkROPQtaam1dzo65RQYPDhr99xyS3YZRrPe5NC35KXW3uno+OPh4x/PevzXXgu713VicrPquKdvSUt9pL/W178OV1wBX/4yXHdd3tVYf+aRviUv9ZH+Wvvum7V4Jk6EF1+Ef/7nvCuy/sihb9ZEPvtZuPtumDABVqzIvuxt8edxayD/OVnS+kt7p9L228O992Zn5jzkEHjrrbwrsv7EoW/J6y/tnUpDhmTTODfYwHP5rbEc+mZNaoMNYO5c+NKXYJdd4KGH8q7I+gOHviWtP7Z3KknZXP4LLsiuwnXttXlXZKnzF7mWvP7Y3unoK1+BrbfOpnQuXgxnngkDvfdaHTzSN0vEZz8LDzyQ/UyY4JO1WX0c+pa0/t7e6WiLLeDWW+Hv/i77Wbw474osNQ59S14R2juVBg6Es8+G88+H/feHyy7LuyJLiUPfkla0kX6lL38Z7rknC/9vfMOXYbTqOPQteUUb6VfaYQdYtAjWXx/GjvW0TuueQ98scYMHZy2eH/wgm9Z54YXF/gRkXXPoW9Icbu+bPBnuuw+uvBIOPBBeeSXviqwZOfQteUVu73S03XawcGHW9hk9Gm66Ke+KrNk49M36mUGD4NxzsxH/tGkwdaq/5LX3VRX6kiZIWippmaQZnTw+UtK9kt6WdEKHx56R9IikJZIWNapwM3B7pyulEjz6KLz7Luy0E/zXf+VdkTWDbkNfUgtwMbAfMAqYImmHDqu9ChwHnNvJU7wLlCJi54gY18N6zT7E7Z1123hjmDMHLroIpkyBE0+Ev/wl76osT9WM9McByyPi2YhYDcwDJlWuEBGvRMSDwF872V5Vvo6Z9ZKJE7NR//PPZ71+j/qLq5owHgasqLj/fHlZtQK4TdJiSVNrKc6sO27vVO/jH4errsr6/YcdBkcdBa+9lndV1tf64jx94yNilaQtyMK/PSIWdrZia2vre7dLpRKlUqkPyrPUub1Tm0mTsn7/ySfDqFHwk59kR/da82tra6Otra1Hz6HoZqgkaVegNSImlO9/D4iIOKeTdc8A3oiI89bxXOt8XFJ0V4tZR//5n9nFxK+8Mu9K0rRwYTa7Z4cdsvAfPjzviqwWkoiImoY91bR3FgPbS9pK0iBgMnBDV3VUFLSRpI+Ubw8G9gUer6VAs+54pF+/3XeHhx/OZveMGQPnnAPvvJN3Vdabug39iFgDTAcWAE8A8yKiXdLRko4CkDRU0grgO8Cpkp4rh/1QYKGkJcB9wI0RsaC3fhkzq93668PMmdnRvPfck70B3HZb3lVZb+m2vdNX3N6xelxxBfzmN1mbxxrjxhvh+OOzE7iddx6MGJF3RbYuvdXeMWtqbu801sSJ8MQT2Yh/7Njs0oxvvZV3VdYoDn0z+5ANN4TTT8+uzPXoozByJMydmx3da2lz6FvS3BHsXdtsA1dfnc3v/9nPsuv03nln3lVZTzj0LXlu7/S+3XaD//5vOOWUbIrnxInQ3p53VVYPh76ZVUWCgw+G3/4W9toL9twTjjwSnnsu78qsFg59S5rbO31v/fXhhBNg2TIYOhR23hm+9S148cW8K7NqOPQteW7v5GOzzeCss7I2z4ABsOOOMGMGvPpq3pVZVxz6ljSP9PM3ZAicf342y+f11+FTn4LWVp/MrVk59C15Huk3h+HDsxk+ixbBM89kl2489VR4+eW8K7NKDn0za6jttoOf/xweeCBr9YwcmX0H8MILeVdm4NC3xLm907y22SYb+T/2WPb/6W//Fo49NvsUYPlx6Fvy3N5pbsOGZT3/pUth002zA7wOOwyWLMm7smJy6JtZnxgyBH74Q3jqqey8PhMnwt57w003+fQOfcmhb0lzeyc9m24KJ50ETz8N3/wmnHZa1vr593+Ht9/Ou7r+z6FvyXN7J02DBmVtnocegosvhuuug623zqZ7rlqVd3X9l0PfzHIlvd/mufPO7MjeHXeEQw6Bu+/2p7lGc+hb0hwI/cuOO2Yzfp55BsaPh6OOgtGj4dJL4c03866uf3DoW/Lc3ul/NtkkO59Pe3t29a5bb4Wttnp/mdXPoW9J80i/f5Ngn31g/vzsAu4bb5y1gsaPh8sv9+i/Hg59S55H+sWw5ZbZpRufey47sdv8+dmyo46C++/3AKBaDn0zS8p668GBB8INN2TX8t1mG/j617O5/xdcAK+8kneFzc2hb0nz6K7YPvEJOPnk7Nz+P/kJPPhgdu6fgw6Ca6/1vP/OOPQteW7vWEsLlEpwxRVZ+2fSJLjkkuxNYerUbOqnj/rNOPTNrF/ZZBM44gi44w545BH45Cdh2jTYdtvsVM9Fn/3j0Lekub1jXdlyy+yUD489ln0H8M472WygsWNh1ixYuTLvCvueQ9+S5/aOVWOnneDcc7P2z3nnZVf6+tKX8q6q7w3MuwCznvBI32o1YEDW/99iCzj44Lyr6Xse6ZtZIUnFHDRUFfqSJkhaKmmZpBmdPD5S0r2S3pZ0Qi3bmvWU2ztWD4f+OkhqAS4G9gNGAVMk7dBhtVeB44Bz69jWrG5F3GmtcYr491PNSH8csDwino2I1cA8YFLlChHxSkQ8CPy11m3NesojfauHR/rrNgxYUXH/+fKyavRkWzOzXlPUwUJTzd5pbW1973apVKJUKuVWi6WhiCM1a4wUR/ptbW20tbX16DmqCf2VwIiK+8PLy6pR07aVoW9WraKO2KxnUgz9joPhmTNn1vwc1bR3FgPbS9pK0iBgMnBDF+tX7oK1bmtm1idSDP1G6HakHxFrJE0HFpC9ScyJiHZJR2cPx2xJQ4EHgI8C70o6HtgxIt7sbNte+22scIq401pjOPS7EBG3AiM7LLu04vZLwJbVbmvWSG7vWD2KGvo+IteSVsSd1hrDoW9mViAOfbNEub1j9XLomyWmiDutNYZH+maJ8kjf6uHQNzMrkKIOFhz6lrQijtSsMTzSN0tUUUds1jMOfTOzAnHomyWoiDutNYZD3yxRbu9YPRz6Zgkq4k5rjeHQNzMrGIe+WYLc3rF6eKRvlqAi7rTWGA59s0R5pG/1KOrfjUPfzArJI32zBBVxp7XGcOibJaqoH9OtZxz6ZmYF4tA3S1ARd1prDIe+WaLc3rF6OPTNElTEndYap4h/Pw59Myskj/TNEuX2jtXDoW+WoCLutNYYRR0sOPQteUXdea1nPNI3MysQh34XJE2QtFTSMkkz1rHORZKWS3pY0s4Vy5+R9IikJZIWNapwMyjmTmuNUdTQH9jdCpJagIuBzwMvAIslXR8RSyvW+SKwXUR8UtIuwL8Bu5YffhcoRcQfG169GW7vWH2KGvrVjPTHAcsj4tmIWA3MAyZ1WGcSMBcgIu4HNpE0tPyYqnwds5oVcae1xnDor9swYEXF/efLy7paZ2XFOgHcJmmxpKn1Fmpm1khFDf1u2zsNMD4iVknagiz82yNiYR+8rhWE2ztWL4d+51YCIyruDy8v67jOlp2tExGryv99WdJ8snZRp6Hf2tr63u1SqUSpVKqiPCuyIu601hgpjvTb2tpoa2vr0XMouvmtJQ0AniT7IncVsAiYEhHtFevsD0yLiAMk7QpcEBG7StoIaImINyUNBhYAMyNiQSevE93VYtbROefAq6/CrFl5V2KpWbMGBg5ML/grSSIiavqs2+1IPyLWSJpOFtgtwJyIaJd0dPZwzI6ImyXtL+l3wJ+BI8qbDwXmS4rya13ZWeCb9YTbO1aPov7dVNXTj4hbgZEdll3a4f70Trb7PTCmJwWadSXlUZrlq6ih76mUlryi7rzWM2v/boo2cHDom1mhOfTNElK0HdYaK8UZPD3l0Lfkub1j9XLomyWmaDusNZZD38ysYBz6Zolxe8fq5ZG+WWKKtsNaYzn0zcwKpIifEh36lrwi7rjWGB7pmyWmaDusNZZD3yxBHulbvRz6ZmYF4tA3S0zRdlhrLIe+WYLc3rF6OfTNElO0HdYar2h/Qw59Myssj/TNEuT2jtXLoW+WmKLtsNZYDn0zswIp4qdEh74lr4g7rjWGR/pmiSnaDmuN5dA3S5BH+lYvh76ZWYE49M0SU7Qd1hrLoW+WILd3rCcc+mYJKdoOa43lkb6ZWYE49NdB0gRJSyUtkzRjHetcJGm5pIcljallW7OecHvH6uXQ74SkFuBiYD9gFDBF0g4d1vkisF1EfBI4GvhZtdv2B21tbXmX0CMp1x8Bv/99W95l9EjK//6Qdv0S3HtvW85V9K1qRvrjgOUR8WxErAbmAZM6rDMJmAsQEfcDm0gaWuW2yUv5jx7Sr/+ZZ9ryLqFHUv/3T7l+Cf7nf9ryLqNPVRP6w4AVFfefLy+rZp1qtjUzy0UR2zsDe+l5m6bLumYNHHRQ777Gk0/Cgw/27mv0ppTrX7oUttwy7yosVQMGwPXXw+9+1zvPP2gQXHtt7zx3vRTdvM1J2hVojYgJ5fvfAyIizqlY52fAXRFxVfn+UuBzwDbdbVvxHAV7vzUz67mIqGmQXc1IfzGwvaStgFXAZGBKh3VuAKYBV5XfJF6LiJckvVLFtnUVbmZmtes29CNijaTpwAKy7wDmRES7pKOzh2N2RNwsaX9JvwP+DBzR1ba99tuYmVmXum3vmJlZ/9E0R+RKmiWpvXxw17WSNs67pmqkfPCZpOGS7pT0hKTHJH0r75pqJalF0kOSbsi7llpJ2kTSNeW/+yck7ZJ3TbWQ9B1Jj0t6VNKVkgblXVNXJM2R9JKkRyuWbSZpgaQnJf1G0iZ51tiVddRfc242TeiTtYBGRcQYYDlwcs71dKsfHHz2V+CEiBgF/G9gWmL1AxwP/DbvIup0IXBzRHwaGA0k0/qU9AngOGBsROxE1iqenG9V3bqcbF+t9D3g9ogYCdxJc+dOZ/XXnJtNE/oRcXtEvFu+ex8wPM96qpT0wWcR8WJEPFy+/SZZ6CRzHIWk4cD+wGV511Kr8ohsj4i4HCAi/hoRf8q5rFoNAAZLGghsBLyQcz1dioiFwB87LJ4E/KJ8+xdAL0/wrl9n9deTm00T+h18E7gl7yKq0G8OPpO0NTAGuD/fSmpyPvB/gBS/mNoGeEXS5eX21GxJG+ZdVLUi4gXgx8BzwEqyGXu351tVXYZExEuQDYKAITnX0xNV5Wafhr6k28r9v7U/j5X/O7FinVOB1RHxy76srcgkfQT4FXB8ecTf9CQdALxU/qQimuiAwCoNBMYCP42IscBbZK2GJEjalGyUvBXwCeAjkg7Nt6qGSHEAUVNu9tYRuZ2KiC909bikw8k+ru/dJwX13EpgRMX94eVlySh/NP8VcEVEXJ93PTUYDxwoaX9gQ+CjkuZGxD/mXFe1ngdWRMQD5fu/AlKaCLAP8HRE/AFA0q+B3YDUBmsvSRpaPq7ofwH/N++CalVrbjZNe0fSBLKP6gdGxP/Lu54qvXfgWnnmwmSyA9VS8h/AbyPiwrwLqUVEnBIRIyJiW7J/9zsTCnzKLYUVkj5VXvR50vpC+jlgV0kbSBJZ/Sl8Ed3xU+ENwOHl2/8ENPvA5wP115ObTTNPX9JyYBDwannRfRFxbI4lVaX8j34h7x98dnbOJVVN0njgbuAxso+1AZwSEbfmWliNJH0OODEiDsy7llpIGk32JfR6wNPAERHxer5VVU/SGWRvuKuBJcCR5QkNTUnSL4ES8DHgJeAM4DrgGmBL4FngaxHxWl41dmUd9Z9CjbnZNKFvZma9r2naO2Zm1vsc+mZmBeLQNzMrEIe+mVmBOPTNzArEoW9mViAOfTOzAnHom5kVyP8H3slfhZZAVgkAAAAASUVORK5CYII=\n", 645 | "text/plain": [ 646 | "" 647 | ] 648 | }, 649 | "metadata": {}, 650 | "output_type": "display_data" 651 | } 652 | ], 653 | "source": [ 654 | "pmu1 = pxt(4,mus)*pmu\n", 655 | "pmu1 /= C*sum(pmu1)\n", 656 | "plot(xs,pmu1)" 657 | ] 658 | }, 659 | { 660 | "cell_type": "markdown", 661 | "metadata": { 662 | "slideshow": { 663 | "slide_type": "slide" 664 | } 665 | }, 666 | "source": [ 667 | "The maximum likelihood estimate is clearly at $\\mu=4$. If we now plug this\n", 668 | "into $p(x|\\mu)$ we get..." 669 | ] 670 | }, 671 | { 672 | "cell_type": "code", 673 | "execution_count": 10, 674 | "metadata": { 675 | "slideshow": { 676 | "slide_type": "slide" 677 | } 678 | }, 679 | "outputs": [ 680 | { 681 | "data": { 682 | "text/plain": [ 683 | "[]" 684 | ] 685 | }, 686 | "execution_count": 10, 687 | "metadata": {}, 688 | "output_type": "execute_result" 689 | }, 690 | { 691 | "data": { 692 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEACAYAAABI5zaHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAD+pJREFUeJzt3H+s3Xddx/Hnq5QJ8mMKhBlaNt34MV1ky6JzisqRoSszUOIf2i0BmYE0yJAo0QHG7JKQACZGgaFY7SYYyQiDaDX8KAgnBGU/+DE2WLsWB6XtRgmMYcBgSvf2j3s27y7tvd/v6bn99nz6fCQ3u9/v+fR7XrntXvdz3+d8b6oKSVJb1g0dQJI0e5a7JDXIcpekBlnuktQgy12SGmS5S1KDVi33JNuTHEpy+wpr3pZkb5Lbklww24iSpL667NyvBy491oNJng+cU1VPB7YC75xRNknSlFYt96r6FPDtFZZsBt49WXszcHqSM2YTT5I0jVnM3DcA+5ccH5yckyQNxBdUJalB62dwjYPAU5ccb5yc+yFJ/EU2kjSFqkqf9V137pl8HM0O4CUASS4G7q+qQysEnNuPa665ZvAM5h8+x6mYf56zt5B/Gqvu3JO8BxgBT0zyNeAa4LTFnq5tVfXBJJcl+TLwPeDKqZJIkmZm1XKvqis6rLlqNnEkSbPgC6o9jEajoSMcF/MPa57zz3N2mP/808i085ypniypE/l8ktSCJNQavaAqSZojlrskNchyl6QGWe6S1CDLXZIaZLlLUoMsd0lqkOUuSQ2y3CWpQZa7JDXIcpekBlnuktQgy12SGmS5S1KDLHdJapDlLkkNstwlqUGWuyQ1yHKXpAZZ7pLUIMtdkhpkuUtSgyx3SWqQ5S5JDbLcJalBlrskNchyl6QGWe6S1CDLXZIaZLlLUoMsd0lqkOUuSQ2y3CWpQZa7JDWoU7kn2ZRkd5I9Sa4+yuOPT7IjyW1J7kjy0pknlSR1lqpaeUGyDtgDXALcA9wKbKmq3UvWvA54fFW9LsmTgLuAM6rqB8uuVas9nyTp4ZJQVenzZ7rs3C8C9lbVvqo6DNwAbF62poDHTT5/HPCt5cUuSTpxupT7BmD/kuMDk3NLXQv8TJJ7gC8Ar55NPEnSNNbP6DqXAp+vqucmOQf4aJJnVdV3ly9cWFh46PPRaMRoNJpRBElqw3g8ZjweH9c1uszcLwYWqmrT5Pi1QFXVW5as+TfgTVX1H5PjfweurqrPLLuWM3dJ6mmtZu63Ak9LclaS04AtwI5la/YBz5uEOAN4BnB3nyCSpNlZdSxTVUeSXAXsZPGbwfaq2pVk6+LDtQ14I/APSW6f/LE/qar71iy1JGlFq45lZvpkjmUkqbe1GstIkuaM5S5JDbLcJalBlrskNchyl6QGWe6S1CDLXZIaZLlLUoMsd0lqkOUuSQ2y3CWpQZa7JDXIcpekBlnuktQgy12SGmS5S1KDLHdJapDlLkkNstwlqUGWuyQ1yHKXpAZZ7pLUIMtdkhpkuUtSgyx3SWqQ5S5JDbLcJalBlrskNchyl6QGWe6S1CDLXZIaZLlLUoMsd0lqkOUuSQ2y3CWpQZ3KPcmmJLuT7Ely9THWjJJ8PskXk3xitjElSX2kqlZekKwD9gCXAPcAtwJbqmr3kjWnA/8J/EZVHUzypKr65lGuVas9nyTp4ZJQVenzZ7rs3C8C9lbVvqo6DNwAbF625grg/VV1EOBoxS5JOnG6lPsGYP+S4wOTc0s9A3hCkk8kuTXJi2cVUJLU3/oZXudC4LnAY4BPJ/l0VX15RteXJPXQpdwPAmcuOd44ObfUAeCbVfV94PtJPgmcD/xQuS8sLDz0+Wg0YjQa9UssSY0bj8eMx+PjukaXF1QfAdzF4guq9wK3AJdX1a4la84F3g5sAn4EuBn4naq6c9m1fEFVknqa5gXVVXfuVXUkyVXAThZn9NuraleSrYsP17aq2p3kI8DtwBFg2/JilySdOKvu3Gf6ZO7cJam3tXorpCRpzljuktQgy12SGmS5S1KDLHdJapDlLkkNstwlqUGWuyQ1yHKXpAZZ7pLUIMtdkhpkuUtSgyx3SWqQ5S5JDbLcJalBlrskNchyl6QGWe6S1CDLXZIaZLlLUoMsd0lqkOUuSQ2y3CWpQZa7JDXIcpekBlnuktQgy12SGmS5S1KDLHdJapDlLkkNstwlqUGWuyQ1yHKXpAZZ7pLUIMtdkhpkuUtSgzqVe5JNSXYn2ZPk6hXW/XySw0l+a3YRJUl9rVruSdYB1wKXAucBlyc59xjr3gx8ZNYhJUn9dNm5XwTsrap9VXUYuAHYfJR1rwJuBL4xw3ySpCl0KfcNwP4lxwcm5x6S5CnAi6rqb4DMLp4kaRqzekH1r4Cls3gLXpIGtL7DmoPAmUuON07OLfVzwA1JAjwJeH6Sw1W1Y/nFFhYWHvp8NBoxGo16Rpakto3HY8bj8XFdI1W18oLkEcBdwCXAvcAtwOVVtesY668H/rWqPnCUx2q155MkPVwSqqrXRGTVnXtVHUlyFbCTxTHO9qralWTr4sO1bfkf6RNAkjR7q+7cZ/pk7twlqbdpdu7eoSpJDbLcJalBlrskNchyl6QGWe6S1CDLXZIaZLlLUoMsd0lqkOUuSQ2y3CWpQZa7JDXIcpekBlnuktQgy12SGmS5S1KDLHdJapDlLkkNstwlqUGWuyQ1yHKXpAZZ7pLUIMtdkhpkuUtSgyx3SWqQ5S5JDbLcJalBlrskNchyl6QGrR86wLy47z54xzvgyJGhk5yaHvUoeM1r4JGPHDqJNB/cuXf0uc/BddcNneLU9aY3wb33Dp1Cmh/u3DuqgrPPhoWFoZOcmq67bvHvQFI37tw7qoJ1frUGs26d5S71YV11VAXJ0ClOXYnlLvVhuXf0wAOW+5CSxb8DSd1Y7h05lhmWYxmpH+uqI8cyw3IsI/XTqdyTbEqyO8meJFcf5fErknxh8vGpJD87+6jDciwzLMcyUj+rlnuSdcC1wKXAecDlSc5dtuxu4Fer6nzgjcDfzTro0BzLDMuxjNRPl7q6CNhbVfuq6jBwA7B56YKquqmqvjM5vAnYMNuYw3MsMyzHMlI/Xcp9A7B/yfEBVi7vlwEfOp5QJyPHMsNyLCP1M9M7VJP8GnAl8MvHWrOw5BbP0WjEaDSaZYQ141hmWI5ldCoZj8eMx+PjukaXcj8InLnkeOPk3MMkeRawDdhUVd8+1sUW5vT+fccyw3Iso1PJ8o3vG97wht7X6LIXvRV4WpKzkpwGbAF2LF2Q5Ezg/cCLq+q/eqeYA5b7sBzLSP2sunOvqiNJrgJ2svjNYHtV7UqydfHh2gb8GfAE4K+TBDhcVRetZfAT7YEHHMsMybGM1E+nmXtVfRh45rJzf7vk85cDL59ttJOLO/dhOZaR+nEv2pHlPizHMlI/lntHjmWG5VhG6se66sid+7Acy0j9WO4dWe7Dciwj9WO5d+RYZliOZaR+rKuO3LkPy7GM1I/l3pHlPizHMlI/lntH/m6ZYTmWkfqxrjryt0IOy7GM1I/l3pFjmWE5lpH6sdw7styH5c5d6sdy78i3Qg7LmbvUj3XVkTv3Yblzl/qx3Duy3IflzF3qx3LvyLHMsBzLSP1YVx25cx+WYxmpH8u9I8t9WI5lpH4s9468Q3VYjmWkfqyrjrxDdViOZaR+LPeOHMsMy7GM1I/l3pFjmWE5lpH6sa46ciwzLMcyUj+We0eOZYblWEbqx3LvyLHMsBzLSP1YVx05lhmWYxmpH8u9I8cyw3IsI/VjuXfkWGZYjmWkfqyrjty5D8uxjNSP5d6RM/dhOZaR+rHcO3IsMyzHMlI/1lVHjmWG5VhG6sdy78ixzLAcy0j9WO4dOZYZlmMZqR/rqiPHMsNyLCP1Y7l35FhmWI5lpH46lXuSTUl2J9mT5OpjrHlbkr1JbktywWxjDs+d+7DcuUv9rFruSdYB1wKXAucBlyc5d9ma5wPnVNXTga3AO9cg66Cq4KtfHQ8d47iMx+OhI0xt3TrYtWs8dIzjMs9f/3nODvOffxpddu4XAXural9VHQZuADYvW7MZeDdAVd0MnJ7kjJkmHdgDD8BXvjIeOsZxmed/4Ans3j0eOsZxmeev/zxnh/nPP40u5b4B2L/k+MDk3EprDh5lzVxzJDAsxzJSP+uHDjArO3fC29++dte/804455y1u75Wtn493HILvOAFQyeZ3l13wWc/O3SK6cxzdlj7/JddBq94xdpdfxqpVbZDSS4GFqpq0+T4tUBV1VuWrHkn8Imqeu/keDfwnKo6tOxa7r0kaQpV1estHV127rcCT0tyFnAvsAW4fNmaHcArgfdOvhncv7zYpwknSZrOquVeVUeSXAXsZHFGv72qdiXZuvhwbauqDya5LMmXge8BV65tbEnSSlYdy0iS5s8Jv0M1yZ8n2TW52en9SR5/ojP01eUmrpNVko1JPp7kS0nuSPIHQ2eaRpJ1ST6XZMfQWfpKcnqS903+3X8pyS8MnamPJH+Y5ItJbk/yT0lOGzrTSpJsT3Ioye1Lzv14kp1J7krykSSnD5lxJcfI37s3h/j1AzuB86rqAmAv8LoBMnTW5Sauk9wPgD+qqvOAXwReOWf5H/Rq4M6hQ0zprcAHq+qngfOBXQPn6SzJU4BXARdW1bNYHOVuGTbVqq5n8f/XpV4LfKyqngl8nJO7d46Wv3dvnvByr6qPVdWDvyXkJmDjic7QU5ebuE5aVfX1qrpt8vl3WSyWuboHIclG4DLg74fO0tdkh/UrVXU9QFX9oKr+e+BYfT0CeEyS9cCPAvcMnGdFVfUp4NvLTm8G3jX5/F3Ai05oqB6Oln+a3hz6F4f9HvChgTOspstNXHMhyU8CFwA3D5ukt78E/hiYxxeIfgr4ZpLrJ2OlbUkePXSorqrqHuAvgK+xeHPi/VX1sWFTTeXJD76Dr6q+Djx54DzHo1Nvrkm5J/noZD734Mcdk/++YMmaPwUOV9V71iKDHi7JY4EbgVdPdvBzIclvAocmP31k8jFP1gMXAu+oqguB/2FxRDAXkvwYi7ves4CnAI9NcsWwqWZiHjcKvXpzTe5QrapfX+nxJC9l8cfs567F88/YQeDMJccbJ+fmxuTH6RuBf6yqfxk6T0/PBl6Y5DLg0cDjkry7ql4ycK6uDgD7q+ozk+MbgXl6Uf55wN1VdR9Akg8AvwTM26bsUJIzqupQkp8AvjF0oL769uYQ75bZxOKP2C+sqv890c8/hYdu4pq8S2ALizdtzZPrgDur6q1DB+mrql5fVWdW1dksfu0/PkfFzmQUsD/JMyanLmG+Xhj+GnBxkkclCYv55+EF4eU/5e0AXjr5/HeBk32T87D80/TmCX+fe5K9wGnAtyanbqqq3z+hIXqafGHfyv/fxPXmgSN1luTZwCeBO1j8UbSA11fVhwcNNoUkzwFeU1UvHDpLH0nOZ/HF4EcCdwNXVtV3hk3VXZJrWPzGehj4PPCyyZsLTkpJ3gOMgCcCh4BrgH8G3gc8FdgH/HZV3T9UxpUcI//r6dmb3sQkSQ0a+t0ykqQ1YLlLUoMsd0lqkOUuSQ2y3CWpQZa7JDXIcpekBlnuktSg/wNC/JuqhprR7wAAAABJRU5ErkJggg==\n", 693 | "text/plain": [ 694 | "" 695 | ] 696 | }, 697 | "metadata": {}, 698 | "output_type": "display_data" 699 | } 700 | ], 701 | "source": [ 702 | "ylim(0,1)\n", 703 | "plot(xs,pxt(xs,4))" 704 | ] 705 | }, 706 | { 707 | "cell_type": "markdown", 708 | "metadata": { 709 | "slideshow": { 710 | "slide_type": "slide" 711 | } 712 | }, 713 | "source": [ 714 | "This is odd because it predicts that only values between 0 and 4 can occur.\n", 715 | "But the training sample $x_1=4$ only excludes that $\\mu\\lt4$; it doesn't \n", 716 | "exclude any values greater than $4$.\n", 717 | "\n", 718 | "What's the Bayesian estimate?\n", 719 | "\n", 720 | "$$p(x|D) = \\int p(x|\\theta) p(\\theta|D) d\\theta \\propto \\int_0^{10} 1/\\theta \\cdot \\lfloor x\\in[0,\\theta]\\rfloor \\cdot 1/\\theta \\cdot \\lfloor \\theta\\in[4,10]\\rfloor d\\theta\n", 721 | "= \\int_{x_1}^{10} 1/\\theta^2 \\cdot \\lfloor x\\in[0,\\theta]\\rfloor d\\theta$$\n", 722 | "\n", 723 | "You can either think about it, or we can simply perform this integral numerically." 724 | ] 725 | }, 726 | { 727 | "cell_type": "code", 728 | "execution_count": 11, 729 | "metadata": { 730 | "slideshow": { 731 | "slide_type": "slide" 732 | } 733 | }, 734 | "outputs": [ 735 | { 736 | "data": { 737 | "text/plain": [ 738 | "[]" 739 | ] 740 | }, 741 | "execution_count": 11, 742 | "metadata": {}, 743 | "output_type": "execute_result" 744 | }, 745 | { 746 | "data": { 747 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEACAYAAABfxaZOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAHO9JREFUeJzt3Xu0FOWZ7/Hvs9mgETxGJ5EMIAgSFcmomBE0mNgcUYgZLxMZBxINMkowioF4ohhNwsao0WQUr6MyR1GzvI5IZI3G4G3HpS4VjUZEQIgJIhLUHHWCRoLs5/xRjWk6+1K9q7rfqu7fZ6292N1d1f3Q4q/ffuupt8zdERGRxtAUugAREakdhb6ISANR6IuINBCFvohIA1Hoi4g0EIW+iEgDiRX6ZjbezFaY2StmNqudx79mZr8p/jxuZvuWPPb74v3Pm9kzaRYvIiKVsa769M2sCXgFOAx4A1gCTHT3FSXbHAQsd/f3zGw80OLuBxUfexX4vLu/U6W/g4iIxBRnpD8SWOXua9x9M3AHcEzpBu7+lLu/V7z5FNC/5GGL+ToiIlJlccK4P7C25PbrbBvq5U4BflFy24EHzWyJmU2tvEQREUlLc5pPZmZjgCnAISV3j3b39Wb2aaLwX+7uj6f5uiIiEk+c0F8HDCy5PaB43zaKB2/nAeNL5+/dfX3xz7fMbCHRdNHfhL6ZaREgEZEKubtVsn2c6Z0lwFAzG2RmvYCJwKLSDcxsILAAONHdf1ty/w5m1qf4e2/gCOClTorP5c+OR+zIa+++FryO7v7Mnj07eA3d/fn+w9+nMLkQvI5Gff9Vf9if7uhypO/uW8xsOrCY6EPiBndfbmbTood9HvADYBfgP8zMgM3uPhLoCywsjuKbgVvdfXG3KhURkcRizem7+wPAXmX3XV/y+1Tgbw7SuvvvgP0T1igiIilRK2UKtttjO6IvOPlUKBRCl9BtZsbu++8euoxE8vz+g+rPG4V+CrYbul3oEhLJ+z/6wSMGhy4hkby//6o/XxT6KXDUeBRSdw9oiTQihX5KjPxO7+SZ3neRyij0RUQaiEI/BZpeCEvTayLxKfRTkufunTzT+y5SGYW+iEgDUeinQNMLYWl6TSQ+hX5K1EUSht53kcoo9EVEGohCPwWaXghL02si8Sn0U6IukjD0votURqEvItJAFPop0PRCWJpeE4lPoZ8SdZGEofddpDIKfRGRBqLQT4GmF8LS9JpIfAr9lKiLJAy97yKVUeiLiDQQhX4KNL0QlqbXROJT6KdEXSRh6H0XqYxCX0SkgSj0U6DphbA0vSYSn0I/JeoiCUPvu0hlFPoiIg1EoZ8CTS+Epek1kfgU+ilRF0kYet9FKqPQFxFpIAr9FGh6ISxNr4nEp9BPibpIwtD7LlIZhb6ISANR6KdA0wthaXpNJD6FfkrURRKG3neRyij0RUQaiEI/BZpeCEvTayLxxQp9MxtvZivM7BUzm9XO418zs98Ufx43s33j7lsv1EUSht53kcp0Gfpm1gRcDYwDhgOTzGzvss1eBb7k7vsBFwDzKthXRERqJM5IfySwyt3XuPtm4A7gmNIN3P0pd3+vePMpoH/cfeuBphfC0vSaSHxxQr8/sLbk9uv8NdTbcwrwi27um1vqIglD77tIZZrTfDIzGwNMAQ7pzv4tLS0f/14oFCgUCqnUJSJSD1pbW2ltbU30HHFCfx0wsOT2gOJ92ygevJ0HjHf3dyrZd6vS0M8TTS+Epek1aRTlg+E5c+ZU/BxxpneWAEPNbJCZ9QImAotKNzCzgcAC4ER3/20l+9YLdZGEofddpDJdjvTdfYuZTQcWE31I3ODuy81sWvSwzwN+AOwC/IdF/xdudveRHe1btb+NiIh0Ktacvrs/AOxVdt/1Jb9PBabG3bfeaHohLE2vicSnM3JToi6SMPS+i1RGoS8i0kAU+inQ9EJYml4TiU+hnxJ1kYSh912kMgp9EZEGotBPgaYXwtL0mkh8Cn3JNXXviFRGoZ8ShY+I5IFCPwWaXghL02si8Sn0JdfUvSNSGYV+ShQ+IpIHCv0UaHohLE2vicSn0Jdc0wF0kcoo9FOi8BGRPFDop0DTC2Fpek0kPoW+5JoOoItURqGfEoWPiOSBQj8Fml4IS9NrIvEp9CXXdABdpDIK/ZQofEQkDxT6knuaXhOJT6GfAs0ph6MD6CKVUeinROEjInmg0Jfc0zctkfgU+inQnHI4OoAuUhmFfkoUPiKSBwp9yT190xKJrzl0AfVgyxZnyBAwZU/N/WmY8alhwFdCVyKSDwr9FLjDbbcZg3YLXUnjuflluGQ+LF0K//APoasRyT6FfkoGDICBA0NX0Xj+bgPsPcy57jq45prQ1Yhkn+b0U6F5nVAMY9gwuP122LgxdDUi2afQT0mTTs4Kpk9vOPTQKPhFpHMKfck9xzn1VLj22uj4ioh0TKGfBnM00A9j6/IXhx8O774Lzz4buCCRjFPop0Rr74TV1ATTpsF114WuRCTbYoW+mY03sxVm9oqZzWrn8b3M7Ekz+9DMzix77Pdm9hsze97Mnkmr8KxR5oezde2dKVPgnnvgnXcCFySSYV2Gvpk1AVcD44DhwCQz27tssz8CZwA/becp2oCCu49w95EJ680mnZUVTOnyF7vuCl/+MsyfH7AgkYyLM9IfCaxy9zXuvhm4AzimdAN3f9vdnwM+amd/i/k6uabunWyYOROuugq2bAldiUg2xQnj/sDaktuvF++Ly4EHzWyJmU2tpDiROErX3hk5Ev7+72HRooAFiWRYLUbgo939AOBI4HQzO6QGr1lzGuiH0d4B9Bkz4PLLAxQjkgNxlmFYB5QuMDCgeF8s7r6++OdbZraQaLro8fa2bWlp+fj3QqFAoVCI+zLBqXsnO776Vfjud+H552HEiNDViKSntbWV1tbWRM8RJ/SXAEPNbBCwHpgITOpk+4/Tz8x2AJrcfaOZ9QaOAOZ0tGNp6OeNMj+c8itn9ewJ06fDFVfATTeFqUmkGsoHw3PmdBinHeoy9N19i5lNBxYTTQfd4O7LzWxa9LDPM7O+wLPAjkCbmc0A9gE+DSw0My++1q3uvrjiKjNMl+oLq6OL10ydCnvsARs2QN++NS5KJMNirbLp7g8Ae5Xdd33J7xuA9hYW3gjsn6TAvNBIP1t22QX+9V+jpRly/AVSJHV130op9a+jK2fNnBmF/vvv17ggkQxT6Ce0NXA00g+jswPoe+8No0fDjTfWsCCRjFPop8GV+Fk1axZceils3hy6EpFsUOinRCP9cDo7mD5qFAweDHfeWcOCRDJMoZ+QunfC6qh7p9Q558DFF0NbWw0KEsk4hX4a3DTSz7Ajjoh69++/P3QlIuEp9CX3Oure2cosmtu/5JIaFSSSYQr9hNS9E1bc5S8mTIA33oDH210ARKRxKPRTocTPuubmaLT/ox+FrkQkLIV+SjTSD6er6Z2tTjoJVq6EJ5+sbj0iWabQT0jdO2HF6d7ZqlcvOO88LcsgjU2hnwZ17+TG5MmwahU88UToSkTCUOhL7lXybUujfWl0Cv2E1L0TVncuXjN5MqxerU4eaUwK/VQo8fOkZ0/4/vehG9efEMk9hX5KNNIPJ273TqlvfANefRV+9asqFCSSYQr9hNS9E1Yl3TulevaE88+Pevf1n1AaiUI/DereyaVJk+DDD2HhwtCViNSOQl9yr7vftpqaotU3zz0XPvoo5aJEMkqhn5C6d8LqTvdOqXHjoF8/mD8/pYJEMk6hnwolfl6ZRatvtrTABx+Erkak+hT6KdFIP5zudO+UOvBA+MIX4MorUypIJMMU+gmpeyes7nbvlLvwQvj3f4e33krl6UQyS6GfEo30823PPeGEE+AHPwhdiUh1KfTT4Er8kNL6tjV7dtS++eKLqTydSCYp9BNS905YSbt3Su28M/zwhzBzpk7Ykvql0BcpMW0avPkm3Htv6EpEqkOhnwqdkRtS0u6dUs3NMHcufPe7sGlTak8rkhkK/YTUvRNWWt07pQ4/HIYNg8svT/2pRYJT6KdEI/36Mncu/PSnsHZt6EpE0qXQT4O6d4KqxretoUNh+nSYMSP1pxYJSqGfkLp3wkqze6fcOefA0qVw331VewmRmlPoi3Rg++3hmmvgjDO0Lo/UD4V+KtS9E1Ka3TvljjgiWpvnoouq9hIiNaXQT0jdO2FVo3un3Ny5cP31sGJF1V9KpOoU+inRSL9+9esXXUj91FOhrS10NSLJxAp9MxtvZivM7BUzm9XO43uZ2ZNm9qGZnVnJvnVB3TtB1eLb1vTp0aUV582r+kuJVFWXoW9mTcDVwDhgODDJzPYu2+yPwBnAT7uxb66peyesanbvlOrRA268MVqF87XXavKSIlURZ6Q/Eljl7mvcfTNwB3BM6Qbu/ra7PweUX2m0y31F8mKffaLF2KZN04Jskl9xQr8/UHpe4uvF++JIsm+OqHsnpGp275Q7+2z4wx/glltq9pIiqdKB3IS2zicr9MOoRfdOqZ49o2mes86C9etr+tIiqWiOsc06YGDJ7QHF++KoaN+WlpaPfy8UChQKhZgvI1I7I0bAN78ZdfP8/Of6wJfaaW1tpbW1NdFzWFedD2bWA1gJHAasB54BJrn78na2nQ1sdPdLu7Gv57Hn/b0P3+OTcwbiP34vdCkN6c6X7mTB8gXc9S931fR1N22CUaOis3VPPrmmLy3yMTPDvbL2wS5H+u6+xcymA4uJpoNucPflZjYtetjnmVlf4FlgR6DNzGYA+7j7xvb2rfDvlWk5/JyqK7Xq3im33XZw661QKEQ/e+wRpAyRisWZ3sHdHwD2Krvv+pLfNwC7xd23ntTyIKJky/DhcN55cOKJ8Nhj0QVYRLJOB3ITikb6mtQNKeQH77e/DTvsABdfHKwEkYoo9CXXat29U66pCW66Ca68Ep59NmgpIrEo9BNqa9P0TqMbMCAK/RNOgI0bQ1cj0jmFfhq09k5QWej6mjgRDj4YTj89dCUinVPoJ5SBvGloobp32nP11dEUz003ha5EpGMK/YTUvSNb9e4Nd90Vna27bFnoakTap9BPSN074WXpg3f4cLjkEjj+eHj//dDViPwthb7kWujunfZMmQIHHBCdrSuSNQr9hNS9I+XM4Npr4cknYf780NWIbEvnEKZB3TtBZaF7p1yfPnDPPXDoofC5z0UXVxfJAo30E8pe3DSWLHXvlNtnn+jyiscdB2++GboakYhCP6EsjjIlO/75n2Hy5OjA7ubNoasRUegnpu6d8LLUvdOelpaonfOss0JXIqLQl5zLYvdOuR49omWY77tPl1mU8HQgN6E2Te9IDJ/8ZHSVrTFjYMgQOOSQ0BVJo9JIPxXZH23Ws7wcVxk+HH72M5gwAX7729DVSKNS6CeUk7ypW1nu3mnPuHEwezb80z/BO++ErkYakUI/obyMMiU7vvWtKPwnTFBHj9SeQj8hd3RyVmBZ795pz6WXwic+Aaedpm+LUlsKfcm1PHTvtKdHD7j9dliyBC66KHQ10kjUvZOQpneku3bcEe6/H0aPhr594ZRTQlckjUChn4p8jjbrRZ4/ePv1g1/+MlqjZ9dd4eijQ1ck9U7TOwnlOG/qQt66d9qz556waFE00n/88dDVSL1T6CfkuMb5ktiBB0Zn7R53HLz0UuhqpJ4p9BPS2jvh5bF7pz2HHw6XXw7jx8Pq1aGrkXqlOX3Jtbx273Rk0iT4059g7Fj41a9g0KDQFUm9UegnpLV3JG3f/CZs2gSHHRYFf//+oSuSeqLQT4NOzgoqz907HTnjDPjzn/8a/H37hq5I6oVCP6E6zJtcqYfunY6cfXYU/GPHwqOPwqc+FboiqQc6kJuQu+s4rlTND38YLc42diy8/XboaqQeaKSfkNbeCa9eunfaYxYt09DUBIUCPPywpnokGYW+5Fq9de+0xwwuuAC22y46c/fhh3VwV7pPoZ+QunekFsyiqZ6twf/IIzBwYOiqJI8U+qmo/9FmltVj905HZs3adsQ/ZEjoiiRvFPpJNU7eZFI9d+90ZOZM2H57+NKXolU69903dEWSJwr9hOr5IKJk16mnwi67RF09d98dfQCIxBGrZdPMxpvZCjN7xcxmdbDNlWa2ysxeMLMRJff/3sx+Y2bPm9kzaRWeFVp7J7xG/eA9/ni47bbosov33hu6GsmLLkf6ZtYEXA0cBrwBLDGze919Rck2Xwb2cPfPmtko4FrgoOLDbUDB3ev2MtCK/HAaoXunM2PHRlM8Rx0V9fGffHLoiiTr4oz0RwKr3H2Nu28G7gCOKdvmGOAWAHd/GtjJzLZ2E1vM18mltrbGHGVKdvzjP0ZLNVxwAVx4oc4Sl87FCeP+wNqS268X7+tsm3Ul2zjwoJktMbOp3S1UpCON1L3TkT33hCeegAULotH+X/4SuiLJqlocyB3t7uvN7NNE4b/c3du9PlBLS8vHvxcKBQqFQg3KS8ZBZ+QG1IjdOx3p1w8eewy+/vVoTf4FC2DnnUNXJWlqbW2ltbU10XPECf11QOlpIAOK95Vvs1t727j7+uKfb5nZQqLpoi5DPy80ypQs6dMH7rkHzjoLDj4Y7rsP9tgjdFWSlvLB8Jw5cyp+jjjTO0uAoWY2yMx6AROBRWXbLAK+AWBmBwHvuvsGM9vBzPoU7+8NHAHU1cXglPnhNWr3Tkd69IDLLoMZM2D0aF13V7bV5Ujf3beY2XRgMdGHxA3uvtzMpkUP+zx3v9/MjjSz1cD7wJTi7n2BhWbmxde61d0XV+evEpKmGEJp9O6dznzrWzB4MHz1q3DJJTBlStf7SP2LNafv7g8Ae5Xdd33Z7ent7Pc7YP8kBWad1t6RLBs/PursOfZYeO45mDsXevYMXZWEVLetlNI4dFylc8OGwTPPwJo10ZW4NmwIXZGEpNBPSmfkBqXunXh22ik6a7dQgAMPjD4EpDEp9BPSQUTJi6YmOP98uPJK+MpX4IYb1IjQiBT6Cel/mvD0wVuZY4+N+vkvuwwmT4aNG0NXJLWk0E+BOkjC0XvfPVvn+Xv0iKZ7XqqrRmrpjEI/IXXvSF717g3z50cXZhkzBm68Ud9cG4FCX3JP3TvJnHQStLbCpZdquqcRKPQTckdr7wSk7p10DB8eTfc0N8OIEfD006ErkmpR6Cfk7urYlLrQu3c0xfPjH8PRR0edPh99FLoqSZtCPyHNLISn7p10TZgAv/51tGbPF78Iq1eHrkjSpNBPg6Z3glH3TnX07w8PPAATJ8JBB6mnv54o9BPSQUSpV01N0Uqdra1w1VXRlM+68kXVJXcU+pJ7+uCtrs99LjrI+/nPRwd51dqZbwr9hFxr7wSl7p3a6NULWlrgwQfh6quj1TvXrAldlXSHQj8hxxX50jD22y9q5ywUoguyX3sttLWFrkoqodBPSF9zw1P3Tm317Anf+160Tv/NN8Ohh8KyZaGrkrgU+qnQWD8Ude+Es88+8MQTMGlSNPI/5xx4//3QVUlXFPoJae0daWQ9esBpp8HSpbB2bXTQ9777QlclnVHoS+6peye8z3wGbr0V/vM/4Tvfia7Lu3Zt6KqkPQr9hNS9E5a6d7Jl7Fh48UXYd1/Yf3+44AL4859DVyWlFPoikqrtt4/aO599Fl54IZr7v/tuNT1khUI/IU0thKfunWwaPDgK+xtvhB/9KFqz/4UXQlclCv0UqIMkHL332TdmDDz3XLSOz7hxMG0avPlm6Koal0I/IQ30RbrW3AynngorVsAOO0SXazz/fF2wJQSFfkKaWghPU2z5sfPOMHcuLFkCK1fCnntGZ/Vu3hy6ssah0E9IV84KS907+TRkSNTi+d//DffcE125Swd7a0Ohnwbljki3HHBAtIjbNdfARRdFa/c/9JDCv5oU+gnpjNzwNMWWf4cfHrV4zpwJp58erefT2hq6qvqk0E+DpneCUfdO/WhqitbxWbYMpk6FU06JOn8eeyx0ZfVFoZ+UBpkiqWpuhhNPjDp9Jk+Gk06KzvR94onQldUHhX5C6hwJT/8N6lNzcxT4K1dG3wC+/vUo/B9+WHP+SSj0E3I0xRCSunfqX8+ecPLJ8MorcMIJ0Zz/qFGwcKEu4NIdCn0RyYVevaKR/7JlMGsWXHhhtJTzzTerz78SCv2E2tr0PTM0de80lh494LjjohO8rrgiCv2hQ+Gqq3SGbxwK/VRoiiEUTa01LrOo1fORR+Cuu+DRR2H33aNvAVrLv2OxQt/MxpvZCjN7xcxmdbDNlWa2ysxeMLP9K9k3zzTGFAlv1KjozN5nnoFNm6ILuE+aFN2WbXUZ+mbWBFwNjAOGA5PMbO+ybb4M7OHunwWmAdfF3Tf33Glbm+8Lg7bm/CyYd5e/G7qERPL+/mep/iFD4PLL4Xe/gwMPhOOPh0MOgQULYMuW9vfJUv21EGekPxJY5e5r3H0zcAdwTNk2xwC3ALj708BOZtY35r655g5tr38QuoxE8vyP3sz4n5X/E7qMRPL8/kM2699pJzjzTFi9OjrL97LLonn/n/wE3n57222zWH81xQn9/kDpDNnrxfvibBNn39zTrLJINjU3w4QJ0Yldd90FL78Mn/1sdNJXo079NFfpeTOTg3/ZvIXdzj62as+/yf9Ehv66DafJmnjrg7c46vajQpfSbSuXruS5258LXUa35ar+cTByDCx5DQrzoOd8+Oj5lVz7XnXqb7ZerLtsQVWeu7usq7MZzewgoMXdxxdvnwO4u19Sss11wKPufmfx9grgUGBwV/uWPIeOiYqIVMi9ssW/4oz0lwBDzWwQsB6YCEwq22YRcDpwZ/FD4l1332Bmb8fYt1uFi4hI5boMfXffYmbTgcVExwBucPflZjYtetjnufv9Znakma0G3gemdLZv1f42IiLSqS6nd0REpH5k5oxcM/uJmS0vnty1wMz+V+ia4sjzyWdmNsDMHjGzZWa21My+HbqmSplZk5n92swWha6lUma2k5n9V/Hf/TIzGxW6pkqY2XfM7CUze9HMbjWzXqFr6oyZ3WBmG8zsxZL7djazxWa20sx+aWY7hayxMx3UX3FuZib0iaaAhrv7/sAq4HuB6+lSHZx89hFwprsPBw4GTs9Z/QAzgJdDF9FNVwD3u/swYD8gN1OfZtYPOAM4wN33JZoqnhi2qi7NJ/p/tdQ5wEPuvhfwCNnOnfbqrzg3MxP67v6Qu29dKPUpYEDIemLK9cln7v4Hd3+h+PtGotDJzXkUZjYAOBL4v6FrqVRxRPZFd58P4O4fuXvezjLrAfQ2s2ZgB+CNwPV0yt0fB94pu/sY4Obi7zcD1evvTqi9+ruTm5kJ/TL/BvwidBEx1M3JZ2a2O7A/8HTYSioyFziLfC6BNBh428zmF6en5pnZJ0IXFZe7vwFcCrwGrCPq2HsobFXdsqu7b4BoEATsGrieJGLlZk1D38weLM7/bf1ZWvzzqJJtzgM2u/tttaytkZlZH+BuYEZxxJ95ZvYVYEPxm4qRvzPkmoEDgGvc/QDgA6Kphlwws08SjZIHAf2APmb2tbBVpSKPA4iKcrNaZ+S2y90P7+xxMzuJ6Ov6/65JQcmtAwaW3B5QvC83il/N7wZ+5u73hq6nAqOBo83sSOATwI5mdou7fyNwXXG9Dqx192eLt+8G8tQIMBZ41d3/H4CZ3QN8AcjbYG2DmfUtnlf0GeDN0AVVqtLczMz0jpmNJ/qqfrS7bwpdT0wfn7hW7FyYSHSiWp7cCLzs7leELqQS7n6uuw909yFE7/sjOQp8ilMKa81sz+Jdh5GvA9KvAQeZ2fYWXbPyMPJxILr8W+Ei4KTi75OBrA98tqm/O7mZmT59M1sF9AL+WLzrKXc/LWBJsRTf9Cv468lnFwcuKTYzGw08Biwl+lrrwLnu/kDQwipkZocC/8fdjw5dSyXMbD+ig9A9gVeBKe7+Xtiq4jOz2UQfuJuB54FTig0NmWRmtwEF4O+ADcBs4OfAfwG7AWuA4909k2t1d1D/uVSYm5kJfRERqb7MTO+IiEj1KfRFRBqIQl9EpIEo9EVEGohCX0SkgSj0RUQaiEJfRKSBKPRFRBrI/wdSTIcap/2x7gAAAABJRU5ErkJggg==\n", 748 | "text/plain": [ 749 | "" 750 | ] 751 | }, 752 | "metadata": {}, 753 | "output_type": "display_data" 754 | } 755 | ], 756 | "source": [ 757 | "result = zeros(xs.shape)\n", 758 | "for i,mu in enumerate(mus):\n", 759 | " weight = pmu1[i]\n", 760 | " result += weight * pxt(xs,mu)\n", 761 | "result /= C*sum(result)\n", 762 | "plot(xs,result)\n", 763 | "plot(xs,pxt(xs,4))" 764 | ] 765 | }, 766 | { 767 | "cell_type": "markdown", 768 | "metadata": { 769 | "slideshow": { 770 | "slide_type": "slide" 771 | } 772 | }, 773 | "source": [ 774 | "Now assume we get another sample, $x_1=7$" 775 | ] 776 | }, 777 | { 778 | "cell_type": "code", 779 | "execution_count": 12, 780 | "metadata": { 781 | "slideshow": { 782 | "slide_type": "slide" 783 | } 784 | }, 785 | "outputs": [ 786 | { 787 | "data": { 788 | "text/plain": [ 789 | "[]" 790 | ] 791 | }, 792 | "execution_count": 12, 793 | "metadata": {}, 794 | "output_type": "execute_result" 795 | }, 796 | { 797 | "data": { 798 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEACAYAAABI5zaHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAFnJJREFUeJzt3X20VHW9x/H398BlXTXNuhUpiNcn1Kg0S8qbD4fwAalEi7xg6TVFScGjeS3UfDhULrVVJlc0Jb2uvGlgelMsH4irZ6mZhVhYBkK6JEBFzfC5QvzeP/YcHKYzZ2bP2TO/+e39ea111pmH3cyH4+lzfvOdvfeYuyMiIvnSETqAiIhkT+UuIpJDKncRkRxSuYuI5JDKXUQkh1TuIiI5VFe5m9k4M1tmZsvNbEYf9x9gZuvM7OHS1znZRxURkXoNrrWBmXUAs4GxwFPAIjO71d2XVWx6r7sf1oSMIiKSUj0r99HACndf6e7rgbnAhD62s0yTiYhIw+op92HAqrLrq0u3VdrHzH5rZj8zs/dlkk5ERBpScyxTp8XACHd/zcwOBW4BRmb02CIiklI95b4GGFF2fXjpto3c/ZWyy3eY2RVm9k53f6F8OzPTiWxERBrg7qlG3/WMZRYBO5vZ9mY2BJgEzC/fwMyGll0eDVhlsZcFjPbr/PPPD55B+cPnKGL+mLPnIX8jaq7c3X2DmU0HFpD8MbjG3Zea2dTkbp8DTDSzk4D1wOvAvzeURkREMlHXzN3d7wR2rbjtqrLLlwOXZxtNREQapSNUU+js7AwdYUCUP6yY88ecHeLP3whrdJ7T0JOZeSufT0QkD8wMb8IbqiIiEhmVu4hIDqncRURySOUukgN6K0sqqdxFIvfKK7DjjvDqq6GTSDtRuYtE7m9/gyefhHnzQieRdqJyF8mJ730vdAJpJyp3kci5w9Zbw/PPw0MPhU4j7ULlLpIDgwbBiSfClVeGTiLtQuUukhPHHQc33wzr1oVOIu1A5S4SOXcwg6FD4eCD4Yc/DJ1I2oHKXSRHTjopeWNV+72Lyl0kRw44ADZsgPvvD51EQlO5i0SudywDyfcvfQmuuCJsJglP5S6SM8ceC3fdBWvW1NxUckzlLhK5yvn61lvDUUfpoKaiU7mL5IBVfIxDVxfMmQOvvx4mj4SnchfJoZEjYe+94YYbQieRUFTuIpGrttvjaafBrFnaLbKoVO4iOVA5lgE48MBkt8h77ml9HglP5S6SU2bJ7H3WrNBJJASVu0jk+hu7HH00PPAAPP546/JIe1C5i+RAX2MZgM03hylT4NJLW5tHwlO5i+RcVxdcfz0891zoJNJKKneRyNXaG2abbWDiRJg9uzV5pD2o3EVyoNpYptcZZyRHrOpDtItD5S5SACNHwn77wTXXhE4iraJyF4lcvQcpzZgB3/kOrF/f3DzSHlTuIjlQaywDMHo07LgjzJvX/DwSnspdpEC++lX41rd0SoIiULmLRC5NUY8bBx0d8NOfNi+PtAeVu0gO1DOW6d3u3HPh61/X6j3vVO4iBXPEEfDXv8Idd4ROIs2kcheJXPlnqNajoyNZvc+cqdV7ntVV7mY2zsyWmdlyM5vRz3Z7m9l6M/tMdhFFJGsTJ8IrrySftSr5VLPczawDmA0cAowCJpvZblW2uwjQr4tICzWy+tbqPf/qWbmPBla4+0p3Xw/MBSb0sd0pwE3AsxnmE5E6pBnL9Prc52DdOli4MPs8El495T4MWFV2fXXpto3MbFvgcHf/HtDAr5mItNqgQXDOOVq959XgjB7nUqB8Fl+14Lu7uzde7uzspLOzM6MIIsU0kGKeNAm+8Q34+c/h4IOzyyQD09PTQ09Pz4Aew7zGb4aZfQzodvdxpetnAu7uF5dt80TvReBdwKvAie4+v+KxvNbziUg6K1fC/vsn3xtx443JUauLFjU23pHmMzPcPdV/nXrGMouAnc1sezMbAkwCNiltd9+x9LUDydz95MpiF5H2NHEivPkm3Hxz6CSSpZrl7u4bgOnAAuBRYK67LzWzqWZ2Yl//k4wzikg/BvpiuKMDLrwwmb+/8UY2mSS8mmOZTJ9MYxmRzD35JHR2Jt8b5Q5jxiQfqH388RkFk8w0aywjIjlnlqzeZ85MTk0g8VO5i0QuqxfD++wDH/pQ8nF8Ej+Vu0gOZLWXywUXwEUXwUsvZfN4Eo7KXUQ2ev/74dBDkxGNxE3lLhK5rPdRuOACmDNnYG/QSngqd5EcyPLgo2HDoKsLzjwzu8eU1lO5i8g/OOMM+MUv4Je/DJ1EGqVyF4lcMw4d2WKLZDzz5S8nR69KfFTuIjnQjHPCfOELyRGr8+Zl/9jSfCp3EelTRwd897vJ7P3110OnkbRU7iKRS/sZqmnstx+MHg0XX1x7W2kvKncR6dcll8Ds2fDEE7W3lfahcheJXLPPxbfddsneM6ee2tznkWyp3EVyoNkfsnH66bBiBdx2W3OfR7KjcheRmoYMgcsuSw5u0purcVC5i0SuVR+RcNBBsPfeyYnFpP2p3EVyoFWffXrJJXD55cmIRtqbyl1E6jZ8OJx9Nkyd2rpXDNIYlbtI5Fpdsl1d8PLLcO21rX1eSUflLpIDrRrLAAweDFdfnRy5+swzrXteSUflLiKp7bEHTJkCp5wSOolUo3IXiVyo2fe558KSJXDLLWGeX/qnchfJgVaOZXptthl8//swfTq8+GLrn1/6p3IXkYYdcACMHw9f+UroJFJJ5S4SudC7JH7727BgAdx+e9gcsimVu0gOhBjL9Npqq2S3yBNOgBdeCJdDNqVyF5EBGzMGJk6EadNCJ5FeKneRyIUey/S68EJ4+GG48cbQSQRU7iK5EHIs02vzzeG665J9359+OnQaUbmLSGY++tFk9j5lSvu8oigqlbtI5NqtRM87D559NvloPglncOgAIjJw7TCW6TVkCPzoR7DPPskHbO+5Z+hExaSVu4hkbued4dJLYdIkePXV0GmKSeUuEjn39lq59/r855MZfFdX6CTFpHIXkaaZPRvuuw/mzg2dpHjqKnczG2dmy8xsuZnN6OP+w8xsiZn9xsx+bWYfzz6qiPSl3d5QLbfllsn8vasLHn88dJpiqVnuZtYBzAYOAUYBk81st4rNFrr7Hu7+IeB44OrMk4pIVe04lun14Q8ne9B89rPw2muh0xRHPSv30cAKd1/p7uuBucCE8g3cvfw/2duAN7OLKCKxmzYNRo2Ck09u71caeVJPuQ8DVpVdX126bRNmdriZLQVuA47LJp6I1BJDWZrBnDmweHHyXZovszdU3f0Wd98dOBz4ZlaPKyK1tfNYptcWW8DNNyef4LRoUeg0+VfPQUxrgBFl14eXbuuTu99vZjua2Tvd/R9OANrd3b3xcmdnJ52dnXWHFZG4jRwJV12VnEFy8WJ417tCJ2pPPT099PT0DOgxzGu8pjOzQcBjwFjgaeDXwGR3X1q2zU7u/njp8l7Are6+XR+P5bWeT0TSWbIEjj4aHnkkdJL6zZiRlPudd8JgHSdfk5nh7qlen9Ucy7j7BmA6sAB4FJjr7kvNbKqZnVja7LNm9nszexi4DDgyZXYRGYAYxjLlLrggKfXTTw+dJL9qrtwzfTKt3EUyt2QJHHNM8j0m69Yl55857TSYOjV0mvbWyMpdL4hEIhfremnrrWH+fNh332QWP2ZM6ET5otMPiORAbGOZXrvsAjfcAJMnwx//GDpNvqjcRSSosWOTI1gPOwxefDF0mvxQuYtELtaxTLmTT4ZPfCLZRfLvfw+dJh9U7iI5EOtYptysWcnnsOoj+rKhcheRtjBoUHIGyeXLk6NYZWBU7iKRy9Mqd/PN4bbbYN685EhWaZx2hRTJgTyMZXq9+93Jkav77gvDhsGnPhU6UZy0cheRtrPTTnDrrXDccXD//aHTxEnlLhK5PI1lyo0eDddfD5/5DDz8cOg08VG5i+RAnsYy5Q46KJm9f/KTsHRp7e3lLZq5i0Quryv3XkccAS+/DAcfDPfeCzvsEDpRHFTuIjmQ15V7r2OOSQr+wAPhvvtg221DJ2p/KncRicK0afDSS0nB33MPDB0aOlF7U7mLRC7vY5lyZ52VnJ5gzBi4+25473tDJ2pfKneRHMj7WKbc+ecn/97egt9mm9CJ2pPKXUSic955bxX8Pfeo4PuicheJXJHGMuXOPRc6OlTw1ajcRXKgSGOZcl/7WlLw++8PCxfC9tuHTtQ+VO4iErWzzoIttoD99oO77oLddw+dqD2o3EUiV9SxTLmuLnjHO5IP/LjtNvjIR0InCk/lLpIDRR3LlDv6aNhqKxg/Hm68ETo7QycKS+eWEZHcmDAhORf8kUfC/Pmh04SllbtI5DSW2dSYMfCznyUfuP3cc3D88aEThaFyF8kBjWU2tffeyUnGxo2DVaveOvCpSDSWEZFc2mUXeOAB+MlP4LrrQqdpPZW7SOQ0lqlu6NDkVMFr14ZO0noqd5EcKNrIQWpTuYtIrpkV89WNyl0kckUsrjRU7iISLY1lpJLKXSRyRVyVpqGVu4hIDqncRSRaGstUV9SfjcpdJHJFXJWmVcSfkcpdJAeKujqth8Yy/TCzcWa2zMyWm9mMPu4/ysyWlL7uN7MPZB9VRCQ9lXsVZtYBzAYOAUYBk81st4rNngD2d/c9gG8C3886qIj0rYjFJbXVs3IfDaxw95Xuvh6YC0wo38DdH3T3F0tXHwSGZRtTRPqjsUx1WrlXNwxYVXZ9Nf2X9xTgjoGEEhHJSlHLPdPzuZvZGOCLwL5ZPq6IVFfE4pLa6in3NcCIsuvDS7dtwsw+CMwBxrn7X6o9WHd398bLnZ2ddBb9gw5FMqCxTHUxrtx7enro6ekZ0GOY1/hXm9kg4DFgLPA08GtgsrsvLdtmBPB/wNHu/mA/j+W1nk9E0rn3XjjnnOS7/KPu7qTcZ84MnaRxZoa7p/oTXnPl7u4bzGw6sIBkRn+Nuy81s6nJ3T4HOBd4J3CFmRmw3t1Hp/8niEhaWi/1L8aVexbqmrm7+53ArhW3XVV2+QTghGyjiUi9NJbpXxHLXUeoikiuFXXlrnIXiVwRiysNlbuIREtjGamkcheRXNPKXUSiVMTiSkPlLiLR0lhGKqncRSTXtHIXkSgVsbjSULmLSLQ0lqlO5S4iUSpicUltKncRyTWt3EUkWhrLVKdyF5EoFbG4pDaVu0gOaOVenVbuIiI5pHIXkSgVsbjSKOqrGpW7SA4UtcDqVcQ/gCp3Eck1jWVEJEpFLK40VO4iEi2NZaSSyl1Eck0rdxGJUhGLKw2Vu4hES2MZqaRyF5Fc08pdRKJUxOJKQ+UuItHSWKa6ov5sVO4ikntauYtIdIpYXGloLCMi0Srq6KEeKncRiVIRi0tqU7mLSK5p5S4i0dJYpjqVu4hEqYjFJbWp3EUk17RyF5FoaSxTncpdRKJUxOJKQ+XeDzMbZ2bLzGy5mc3o4/5dzewBM/urmZ2efUwR6Y9W7lJpcK0NzKwDmA2MBZ4CFpnZre6+rGyzPwOnAIc3JaWISIO0cq9uNLDC3Ve6+3pgLjChfAN3f97dFwNvNCGjiPSjiMWVhsq9umHAqrLrq0u3iUib0FhGKtUcy2Stu7t74+XOzk46OztbHUFECiTGlXtPTw89PT0Deox6yn0NMKLs+vDSbQ0pL3cRGbjYiqvVYiz3yoXvzJkzUz9GPWOZRcDOZra9mQ0BJgHz+9leLxBFWkxjGalUc+Xu7hvMbDqwgOSPwTXuvtTMpiZ3+xwzGwo8BGwJvGlmpwLvc/dXmhleRKSWGFfuWahr5u7udwK7Vtx2VdnltcB22UYTkXoUsbjSKGq56whVkRzQWKY6lbuIiOSGyl0kckVclaahlbuIREtjmepU7iISpSIWl9SmcheRXNPKXUSipbFMdSp3EYlSEYtLalO5i0iuaeUuItHSWKY6lbuIRKmIxZWGyl1ERHJD5S6SAxrLVKeVu4hEqYjFlYbKXUSipZW7VFK5i0iuaeUuIlEqYnGloXIXkWhpLNM/lbuISM4U9Q+fyl0kckVclaahsYyIRKuoq9N6qNxFRCQ3VO4ikSviqjQNrdxFJFoay1SncheRKBWxuKQ2lbuI5JpW7iISLY1lqlO5i0iUilhcaajcRUQkN1TuIjmgsUx1WrmLSJSKWFxpqNxFRCQ3VO4iOaCxTHVauYtIlIpYXGmo3EVEJDfqKnczG2dmy8xsuZnNqLLNf5nZCjP7rZntmW1MEemPxjLVaeVehZl1ALOBQ4BRwGQz261im0OBndx9F2AqcGUTsgbX09MTOsKAKH9YzcrfiuKK+WdvBi+80BM6RsvVs3IfDaxw95Xuvh6YC0yo2GYCcB2Au/8KeLuZDc00aRuI+RcclD+0ZuZv9so95p+9yr26YcCqsuurS7f1t82aPrYREZEWGRw6QFYWLIDLLmvuczz2GCxe3NznaCblD6tZ+Vevhg98IPvHzYtBg2DtWvj0p5v3HOPHw0knNe/xG2FeY2BnZh8Dut19XOn6mYC7+8Vl21wJ3OPu80rXlwEHuPvaiscq4NsaIiID5+6phm/1rNwXATub2fbA08AkYHLFNvOBacC80h+DdZXF3kg4ERFpTM1yd/cNZjYdWEAyo7/G3Zea2dTkbp/j7reb2Xgz+yPwKvDF5sYWEZH+1BzLiIhIfFp+hKqZfcvMlpYOdrrZzLZqdYa06jmIq12Z2XAzu9vMHjWz35lZV+hMjTCzDjN72Mzmh86Slpm93cx+XPq9f9TMPho6Uxpm9mUz+72ZPWJm15vZkNCZ+mNm15jZWjN7pOy2d5jZAjN7zMzuMrO3h8zYnyr5U/dmiNMPLABGufuewArgrAAZ6lbPQVxt7g3gdHcfBewDTIssf69TgT+EDtGgWcDt7r47sAewNHCeupnZtsApwF7u/kGSUe6ksKlqupbk/6/lzgQWuvuuwN20d+/0lT91b7a83N19obu/Wbr6IDC81RlSqucgrrbl7s+4+29Ll18hKZaojkEws+HAeODq0FnSKq2w9nP3awHc/Q13fylwrLQGAVuY2WBgc+CpwHn65e73A3+puHkC8IPS5R8Ah7c0VAp95W+kN0OfOOw44I7AGWqp5yCuKJjZvwJ7Ar8KmyS17wJfAWJ8g2gH4Hkzu7Y0VppjZpuFDlUvd38K+A7wJ5KDE9e5+8KwqRrynt49+Nz9GeA9gfMMRF292ZRyN7Ofl+ZzvV+/K33/dNk2XwPWu/sNzcggmzKztwE3AaeWVvBRMLNPAmtLrz6s9BWTwcBewOXuvhfwGsmIIApmtjXJqnd7YFvgbWZ2VNhUmYhxoZCqN5tyhKq7H9Tf/WZ2LMnL7E804/kztgYYUXZ9eOm2aJReTt8E/I+73xo6T0ofBw4zs/HAZsCWZnadux8TOFe9VgOr3P2h0vWbgJjelD8QeMLdXwAws/8F/g2IbVG21syGuvtaM3sv8GzoQGml7c0Qe8uMI3mJfZi7/63Vz9+AjQdxlfYSmERy0FZM/hv4g7vPCh0kLXc/291HuPuOJD/7uyMqdkqjgFVmNrJ001jiemP4T8DHzOyfzcxI8sfwhnDlq7z5wLGly/8BtPsiZ5P8jfRmy/dzN7MVwBDgz6WbHnT3k1saIqXSD3YWbx3EdVHgSHUzs48D9wK/I3kp6sDZ7n5n0GANMLMDgP9098NCZ0nDzPYgeTP4n4AngC+6+4thU9XPzM4n+cO6HvgNMKW0c0FbMrMbgE7gX4C1wPnALcCPge2AlcCR7r4uVMb+VMl/Nil7UwcxiYjkUOi9ZUREpAlU7iIiOaRyFxHJIZW7iEgOqdxFRHJI5S4ikkMqdxGRHFK5i4jk0P8DbKFfDbjhJEoAAAAASUVORK5CYII=\n", 799 | "text/plain": [ 800 | "" 801 | ] 802 | }, 803 | "metadata": {}, 804 | "output_type": "display_data" 805 | } 806 | ], 807 | "source": [ 808 | "pmu2 = pxt(7,mus)*pmu1\n", 809 | "pmu2 /= C*sum(pmu2)\n", 810 | "plot(xs,pmu2)" 811 | ] 812 | }, 813 | { 814 | "cell_type": "code", 815 | "execution_count": 13, 816 | "metadata": { 817 | "slideshow": { 818 | "slide_type": "slide" 819 | } 820 | }, 821 | "outputs": [ 822 | { 823 | "data": { 824 | "text/plain": [ 825 | "[]" 826 | ] 827 | }, 828 | "execution_count": 13, 829 | "metadata": {}, 830 | "output_type": "execute_result" 831 | }, 832 | { 833 | "data": { 834 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEACAYAAABfxaZOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAG5BJREFUeJzt3XuUXXV99/H3Z0jShKvtI6Q0IYBQAmEJVPqEKBenWiASAVm2SvAGFAjaIBWL3KokaFGxLBcsrCTlVig8sRV55FnlEqiOFG0kKlqhiQSQWwgUFNCIlJD5Pn/sM3g4zmWf2fvMb+9zPq+1smbOOXuf+eY3M5/57e++KSIwM7Pe0Je6ADMzmzgOfTOzHuLQNzPrIQ59M7Me4tA3M+shDn0zsx6SK/QlzZe0VtIDks4a5vXjJP2o8e9uSfs0vfZI4/l7Jd1TZvFmZtYejXWcvqQ+4AHg7cCTwGrg2IhY27TMPGBNRLwgaT6wJCLmNV57GNg/Ip7r0P/BzMxyyjPTnwusi4hHI2ITsAI4unmBiFgVES80Hq4CZjS9rJxfx8zMOixPGM8AHm96/ASvDfVWJwG3Nj0O4A5JqyWd3H6JZmZWlkllvpmkPwFOAA5qevrAiNggaXuy8F8TEXeX+XXNzCyfPKG/HpjV9Hhm47nXaOy8XQ7Mb+7fR8SGxsdnJN1E1i76rdCX5IsAmZm1KSLUzvJ52jurgd0l7SxpCnAscHPzApJmATcCH4iIh5qe31LS1o3PtwIOA+4bpfha/jt/m22Ixx5LXse46z///OQ1jPvf3/wN5/f3p6+jV8ff9Sf9Nx5jzvQjYrOkxcBKsj8SV0bEGkmLspdjOfBJ4PeAv5ckYFNEzAWmAzc1ZvGTgOsjYuW4KjUzs8Jy9fQj4jZgdstzy5o+Pxn4rZ20EfFTYL+CNZqZWUl8KGUJ+n/nd0BttdUqpb+/P3UJ4yfRv8suqasopNbjj+uvmzFPzpookqIqtbRt5kxYtSr7aBPrU5+CSZOyj2Y9RhLRgR25Npa6/rHqFh5/s9wc+mWpcXun1jzuZm1x6JuZ9RCHfhncXkjL42+Wm0O/LG4zpOFxN2uLQ9/MrIc49Mvg9kJaHn+z3Bz6ZXGbIQ2Pu1lbHPpmZj3EoV8GtxfS8vib5ebQL4vbDGl43M3a4tA3M+shDv0yuL2QlsffLDeHflncZkjD427WFoe+mVkPceiXwe2FtDz+Zrk59MviNkMaHneztjj0zcx6iEO/DG4vpOXxN8vNoV8WtxnS8LibtcWhb2bWQxz6ZXB7IS2Pv1luDv2yuM2QhsfdrC0OfTOzHuLQL4PbC2l5/M1yc+iXxW2GNDzuZm1x6JuZ9RCHfhncXkjL42+Wm0O/LG4zpOFxN2uLQ9/MrIc49Mvg9kJaHn+z3Bz6ZXGbIQ2Pu1lbHPpmZj3EoV8GtxfS8vib5ZYr9CXNl7RW0gOSzhrm9eMk/ajx725J++Rdt2u4zZCGx92sLWOGvqQ+4DLgcGBvYKGkPVsWexg4JCL2BT4DLG9jXTMzmyB5ZvpzgXUR8WhEbAJWAEc3LxARqyLihcbDVcCMvOt2BbcX0vL4m+WWJ/RnAI83PX6C34T6cE4Cbh3nuvXlNkMaHneztkwq880k/QlwAnDQeNZfsmTJq5/39/fT399fSl1mZt1gYGCAgYGBQu+RJ/TXA7OaHs9sPPcajZ23y4H5EfFcO+sOaQ79WnF7IS2Pv/WI1snw0qVL236PPO2d1cDuknaWNAU4Fri5eQFJs4AbgQ9ExEPtrNs13GZIw+Nu1pYxZ/oRsVnSYmAl2R+JKyNijaRF2cuxHPgk8HvA30sSsCki5o60bsf+N2ZmNqpcPf2IuA2Y3fLcsqbPTwZOzrtu13F7IS2Pv1luPiO3LG4zpOFxN2uLQ9/MrIc49Mvg9kJaHn+z3Bz6ZXGbIQ2Pu1lbHPpmZj3EoV8GtxfS8vib5ebQt3pze8esLQ79sjh8zKwGHPplcHshLY+/WW4Ofas3b2GZtcWhXxaHj5nVgEO/DG4vpOXxN8vNoW/15i0ss7Y49Mvi8DGzGnDol8HthbQ8/ma5OfSt3ryFZdYWh35ZHD5mVgMO/TK4vZCWx98sN4e+1Zu3sMza4tAvi8PHzGrAoW/15/aOWW4O/TI4dNLxFpZZWxz6ZXH4mFkNOPSt/rylZZabQ78MDp10vIVl1haHflkcPmZWAw59qz9vaZnl5tAvg0MnHW9hmbXFoV8Wh4+Z1YBD3+rPW1pmuTn0y+DQScdbWGZtceiXxeFjZjXg0Lf685aWWW4O/TI4dNLxFpZZWxz6ZXH4mFkN5Ap9SfMlrZX0gKSzhnl9tqTvSHpJ0hktrz0i6UeS7pV0T1mFm73KW1pmuU0aawFJfcBlwNuBJ4HVkr4eEWubFvsZcBrwrmHeYhDoj4jnSqi3mhw66XgLy6wteWb6c4F1EfFoRGwCVgBHNy8QEc9GxPeBV4ZZXzm/Tr05fMysBvKE8Qzg8abHTzSeyyuAOyStlnRyO8WZ5eItLbPcxmzvlODAiNggaXuy8F8TEXdPwNe1XuAtLLO25An99cCspsczG8/lEhEbGh+fkXQTWbto2NBfsmTJq5/39/fT39+f98uk5/Axsw4bGBhgYGCg0Hsoxtg0lrQF8BOyHbkbgHuAhRGxZphlzwc2RsTFjcdbAn0RsVHSVsBKYGlErBxm3RirlsqS3GJIZflyWL0a/uEfUldiNuEkERFtzTjHnOlHxGZJi8kCuw+4MiLWSFqUvRzLJU0HvgdsAwxKOh2YA2wP3CQpGl/r+uECv9Yc9ml5C8usLbl6+hFxGzC75bllTZ8/Dew0zKobgf2KFGhmZuXp/kMprft5a8sst4k4eqe7NQLnrkfvSlxIb/r9n69jm41Psc7jPywhDph5AFO2mJK6FKuIMXfkTpTa7sgdHGRw0hbMW/a/mTppaupqes6Cu55i74d+wUUn7JG6lEq677/v47pjrmPBHgtSl2Id0JEduZbPje+5kZ22G263hnXU5iug7z945wlXpq6kko76P0fxyuBwJ8pbr3JPv6g6bp10Ex+9Y9YWh34Jgmwzy6xqJBF4YmK/4dC3+vPWllluDv2iGoEjPNNPwltYoxKilgdIWMc49EsQcnvHqsk/l9bKoW/155nsqNzTt2YO/aLc3knLM9lRub1jrRz6JfCvlJnVhUO/JO6dJuSZ7Ih8yKa1cugX5cBJy39szdri0C9ByD19qyb39K2VQ78kbu8k5FAbkX8urZVDvygHTloOtTG5p2/NHPolCNzesWpye8daOfSt/hxqZrk59IsaOjnLbYY0PO6j8iGb1so3USlBCL79bdhucupKes/0NfC7T8HagdSVVNMzz8CgbypmTRz6ZQi44AKxzRapC+k9hz0Ff/xccOGS1JVU0+pdxIxfBsftk7oSqwqHflGN9s41V4s37p64ll50nWAlHHZd6kKq6dDLxZ1fgzjTnTDLuKdv1sVe/3qYPDkYGEhdiVWFQ78Evp5+Yj56Z0SSeNvbgy9/OXUlVhUO/aJePXoncR29ygM/poMOhDvugKeeSl2JVYFDvyQ+OcuqSBJTpwV//udw1VWpq7EqcOiXIPCEMym3d8a0aBEsXw6bN6euxFJz6BfVCJy+Pqd+Ev5rO6qhyzDsvz/ssAPcfnvqiiw1h35J3N6xKmo+wODUU+HyyxMWY5Xg0C9BdvRO6ip6mNs7oxq6DMN73wvf+Q488kjaeiwth35RQ+0dp34aHvdRNV9lc6ut4Pjj4bLL0tZkaTn0zXrI4sVwzTWwcWPqSiwVh34JAu/ITcrtnRG1XmVzl13gkEPg2mvT1WRpOfSLGgocZ34abu+07fTT4dJLYXAwdSWWgkO/JD56x6pouDtnHXIITJ0KK1cmKsqSyhX6kuZLWivpAUlnDfP6bEnfkfSSpDPaWbcbhLwjNym3d0Y03DWhpGy2f8klCQqy5MYMfUl9wGXA4cDewEJJe7Ys9jPgNOAL41i33nztnbQ88GMa7s5ZCxfCD34Aa9cmKMiSyjPTnwusi4hHI2ITsAI4unmBiHg2Ir4PvNLuut3CV9m0KhrpxuhTp2aXZrj00gRFWVJ5Qn8G8HjT4ycaz+VRZN3a8LV3EnN7Z1w+/GFYsQKefTZ1JTaRvCO3KJ+clZbHfVRi5Buj77gjvPvdPlmr1+S5XeJ6YFbT45mN5/Joa90lS5a8+nl/fz/9/f05v0x6zh6ro7/+azj4YDjzzOyMXau2gYEBBgreBi1P6K8Gdpe0M7ABOBZYOMryzfHX1rrNoV8nIR+ymZTbOyOShu/pD5k9Gw46KLvW/mmnTWBhNi6tk+GlS5e2/R5jtnciYjOwGFgJ3A+siIg1khZJOgVA0nRJjwMfA86T9JikrUdat+0qa8Bn5CbiTaxR5ZmMfOITcPHF8ErrYRjWlfLM9ImI24DZLc8ta/r8aWCnvOt2FR+yaRU3Uk9/yLx5sPPO8M//DMcdN0FFWTLekVuCwO2dpNzeGdFY7Z0hn/gEXHSRh7IXOPRL4pl+Ih74UhxxRHYrRV+aofs59Ivy7RKtwkY7ZPM1ywnOOgv+9m892+92Dv0ShPM+LadUKY49FjZsgG99K3Ul1kkO/ZL45KxEPO6jytvTB5g0Cc47D8ZxFKDViEO/KLd3rMLytneGvO998NhjcNddHSzKknLol8DNhcTc3inN5Mlw7rlwwQWpK7FOceiXxO2dRDzuo2qnvTPkgx+EBx+Eb3+7Q0VZUg79onxylnUZz/a7m0O/oIjIrr3j1E/H7Z0RtdvTH3L88dkNVlatKr8mS8uhXxJnfiIe+FGNdzIyZQqccw586lMlF2TJOfQLihgEnD1WXe329IeceCI89BB885slF2RJOfQLihjPxrOVyu2dEY23vQPZbP+CC7IZv4e4ezj0S+KZfiIe+I5auBBefBFuvjl1JVYWh35Bbu9YlY3nkM1mfX1w4YXZmbqbN5dYmCXj0C+Br72TmHsPHbVgAWy3HdxwQ+pKrAwO/YLCx+mn5YEfVZGe/qvvIfjsZ7MjeV5+uaTCLBmHflEOfauwss4fOeQQ2HNPWLZs7GWt2hz6BUXheZQV5vbOqIr09Jt97nPwmc/A88+X8naWiEO/ILd3EvPAj6qM9s6QffeFI4/MduxafTn0C4pBh771jk9/Gq66Ch5+OHUlNl4O/YKC8NE7qbm9M6Kih2y22nFH+NjH4OyzS3tLm2AO/YKcN4l5E2vCnXFGdiG2u+9OXYmNh0O/oNg8mLoEsxGV2dMfMm1adgjnGWfAoH/8a8ehXwJP9hPz5taIOnXJ74ULs48+Yat+HPoFDTpw0nJ7Z0xl9vSH9PXBJZdkvf1f/rL0t7cOcugXNDjo0Lfq6kR7Z8ib3wyHHgpLl3bk7a1DHPolcOwn5q2tZD7/ebj2Wrj//tSVWF4O/YLc3knM7Z1RlX3IZqsddsiuybN4sf/21oVDv6Bwe8d63KmnZpdmWLEidSWWh0O/qAjCs820PMUcUSd7+kMmTYIvfQnOPNM7devAoV+Q8yYx/8EdVacO2Wz1lrdkO3XPP39CvpwV4NAvaNBnp1jFdbKn3+yii+D66+F735uQL2fj5NAvaNA3Rk/Pm1sjmoj2zpDtt4e/+zs46STYtGlCvqSNg0Pf6s3tnUp5//th+nS4+OLUldhIHPoFub1jVdbpQzZ/++vB5ZdnM/516ybsy1obcoW+pPmS1kp6QNJZIyxzqaR1kn4o6Y+ann9E0o8k3SvpnrIKr4zwjdGTc3unUnbdFc47D045xd+aKhoz9CX1AZcBhwN7Awsl7dmyzDuA3SLiD4FFwJebXh4E+iPijyJibmmVV4Q7+om5vTOqiezpN/voR2HjxuyGK1YteWb6c4F1EfFoRGwCVgBHtyxzNHAtQER8F9hO0vTGa8r5dWrJJ2dZlU3UIZutttgCrrgCzjkHnngiSQk2gjxhPAN4vOnxE43nRltmfdMyAdwhabWkk8dbaFU58yvAPYRRTWRPv9m++2aXZ/iLv/C3qEomTcDXODAiNkjaniz810TEsPfcWbJkyauf9/f309/fPwHlFeRDNtNye2dUqdo7Q845Jztxa9my7HINVszAwAADAwOF3iNP6K8HZjU9ntl4rnWZnYZbJiI2ND4+I+kmsnbRmKFfFzEYOHbMhjd5cnYVzoMPzs7Y3W231BXVW+tkeOk4rmudp72zGthd0s6SpgDHAje3LHMz8EEASfOA5yPiaUlbStq68fxWwGHAfW1XWWHZprNjPyn3DkY00YdsDmevveDcc+H442Hz5qSlGDlCPyI2A4uBlcD9wIqIWCNpkaRTGsvcAvxU0oPAMuAjjdWnA3dLuhdYBfy/iFjZgf9HUo6chNzeqYW/+qvsbltf/GLqSixXTz8ibgNmtzy3rOXx4mHW+ymwX5ECq25wcJAtUhdhNoLUPf0hfX1wzTUwdy4cfji88Y2pK+pdXXso5URJ/+tkbu+MLNUhm8PZdVf4wheym6q/+GLqanqXQ7+giPAZuSlVKNSqKnVPv9mHPgT77AMf/3jqSnqXQ7+g8LV3rMKq0t4ZMnRtnpUr4WtfS11Nb3LoF1ShSVTv8jehVrbdFm64AT78YXjssdTV9B6HfkER7usn5fbOqKpwyOZwDjgAzjgD3vc+eOWV1NX0Fod+QVXadDZrpQqfQ3LmmTB1Kozj/CIrwKFfkC+4VgEVnMlWSVUnJn198E//BFdfDbfckrqa3uHQLygIwi2GdDz2o6pqe2fI9Onwla/ACSfAI4+krqY3OPSL8kzfrJADD8wuzPZnfwYvvZS6mu7n0C9osMKzqJ7h78GIqnbI5khOPx3e8Ibs5ivWWQ79ElT/V6qLub3TFSS48kq4666sx2+dMxHX0+9qvjG6VVnVe/rNttkmO2HrrW+FOXOywzqtfJ7pF1WP36fuVpNQS6HKh2wOZ86cbMb/7nfD+ta7dlgpPNMvKGrRMe1ibu+MqW4/oUcdBfffD+96V9bumTYtdUXdxTP9gnycvlVZndo7zc4+G/bYA0480RtyZXPoF+QfyArwN6HrSHDFFfDQQ/DZz6aupru4vVNQdmlltxiS8diPqi6HbA5n2jS46aZsh+5ee8Exx6SuqDs49IvyLNOsY2bMgK9/HebPhx13hHnzUldUf27vFOSTsyrA34MR1bWn32z//bNbLR5zDDz4YOpq6s+hX4J6/0rVnNs7o6rbIZsjWbAguxrnO94BzzyTupp6c+gXVPdZlHW/uvb0W51yCrz3vfDOd/oeu0U49AvyIZsV4D+8I+qG9k6zT38aZs/Obq7um6+Mj0O/BL4xekJu7/SUoUM5f/1rOPlk8FVQ2ufQL8g7cq3K6nzI5kimTMkO5XzggeyWi/4VbI9Dv6Bu2nSuLX8Pes5WW8G//isMDMAFF6Supl58nH5BEUF0yRESteT2zqi6raff7HWvg9tvh4MPzj4//fTUFdWDQ9+si3XLIZsjmT4d7rwzC/5tt81uu2ijc+gXFJu9Jym5Lp3JlqXbevqtZs2ClSvhbW+DyZPh/e9PXVG1OfQLGuzyX6jKc3tnVN3c3mk2ezbccQccemj22ME/Moe+mXWFOXMc/Hk49AvyyVkV0AMz2fHqxkM2R+PgH5tDv6AIn5yVlNs71qI5+CPgAx9IXVG1OPQLC+jyIySsvnqlp99qKPgPOwx+9Ss49dTUFVWHQ78ot3fS68FQy6vbD9kczZw58K1vZTP+F16As85KXVE1OPQLGuypjmkFub0zpl7+Cd1tN/j3f89m/M8/Dxde6B+ZXJdhkDRf0lpJD0ga9u+lpEslrZP0Q0n7tbNunfXiprPVR6+2d5rNmJHN+O+8Ez7yEV+kbczQl9QHXAYcDuwNLJS0Z8sy7wB2i4g/BBYBl+ddt/YC7tlU75+igYGB1CUUMvD886lLKKT241+D+l//evi3f4M1a7Ijel5++Tev1aH+MuWZ6c8F1kXEoxGxCVgBHN2yzNHAtQAR8V1gO0nTc65bazEYDv2UJAZ+8YvUVRTSyfGfiEM26/Lzs+22cOut2Y7dBQtg6MemLvWXJU/ozwAeb3r8ROO5PMvkWdfMbEJMmwY33gi77w6HHAJPPpm6oonXqR25ldlV8vJLLzMwZ6eOvf+WL/8P0et7hlLq68tumnrkkakrGb+f/AS+//2OvPWC5x9l9s/Xcc8nb+zI+wOs37CRe27+csfevxNOAN7za7h3Hjz0y42svLYz9W/aYhIL1q3vyHuPl8baySNpHrAkIuY3Hp8NRER8vmmZy4FvRsRXGo/XAm8Fdh1r3ab36O29TWZm4xDR3umheWb6q4HdJe0MbACOBRa2LHMz8JfAVxp/JJ6PiKclPZtj3XEVbmZm7Rsz9CNis6TFwEqyfQBXRsQaSYuyl2N5RNwi6QhJDwK/Itt6GnHdjv1vzMxsVGO2d8zMrHtU5h65ki6StKZxcteNkrZNXVMedT75TNJMSd+QdL+kH0v6aOqa2iWpT9IPJN2cupZ2SdpO0r80fu7vl3RA6praIeljku6T9J+Srpc0JXVNo5F0paSnJf1n03O/K2mlpJ9Iul3SdilrHM0I9bedm5UJfbIW0N4RsR+wDjgncT1j6oKTz14BzoiIvYE3A39Zs/oBTgf+K3UR43QJcEtE7AXsC9Sm9SnpD4DTgDdFxD5kreJj01Y1pqvJflebnQ3cGRGzgW9Q7dwZrv62c7MyoR8Rd0bE0FlOq4CZKevJqdYnn0XEUxHxw8bnG8lCpzbnUUiaCRwBXJG6lnY1ZmQHR8TVABHxSkTU7SyzLYCtJE0CtgQqfdR7RNwNPNfy9NHAPzY+/0fgXRNaVBuGq388uVmZ0G9xInBr6iJy6JqTzyTtAuwHfDdtJW35InAm1PKKYrsCz0q6utGeWi5pWuqi8oqIJ4GLgceA9WRH7N2Ztqpx2SEinoZsEgTskLieInLl5oSGvqQ7Gv2/oX8/bnw8smmZ84BNEXHDRNbWyyRtDXwVOL0x4688SQuApxtbKqJCJwTmNAl4E/CliHgT8CJZq6EWJL2ObJa8M/AHwNaSjktbVSnqOIFoKzcn9NLKEXHoaK9LOp5sc/1tE1JQceuBWU2PZzaeq43GpvlXgesi4uup62nDgcBRko4ApgHbSLo2Ij6YuK68ngAej4jvNR5/FajTgQB/CjwcET8HkPQ14C1A3SZrT0ua3jiv6PeB/05dULvazc3KtHckzSfbVD8qIv4ndT05vXriWuPIhWPJTlSrk6uA/4qIS1IX0o6IODciZkXEG8jG/Rs1CnwaLYXHJe3ReOrt1GuH9GPAPElTJYms/jrsiG7dKrwZOL7x+YeAqk98XlP/eHKzMsfpS1oHTAF+1nhqVUR8JGFJuTQG/RJ+c/LZ5xKXlJukA4G7gB+TbdYGcG5E3Ja0sDZJeivw8Yg4KnUt7ZC0L9lO6MnAw8AJEfFC2qryk3Q+2R/cTcC9wEmNAxoqSdINQD/wv4CngfOB/wv8C7AT8Cjwnoio5LW6R6j/XNrMzcqEvpmZdV5l2jtmZtZ5Dn0zsx7i0Dcz6yEOfTOzHuLQNzPrIQ59M7Me4tA3M+shDn0zsx7y/wH8enis/Iaw1wAAAABJRU5ErkJggg==\n", 835 | "text/plain": [ 836 | "" 837 | ] 838 | }, 839 | "metadata": {}, 840 | "output_type": "display_data" 841 | } 842 | ], 843 | "source": [ 844 | "result = zeros(xs.shape)\n", 845 | "for i,mu in enumerate(mus):\n", 846 | " weight = pmu2[i]\n", 847 | " result += weight * pxt(xs,mu)\n", 848 | "result /= C*sum(result)\n", 849 | "plot(xs,result)\n", 850 | "plot(xs,pxt(xs,7))\n", 851 | "plot(xs,pxt(xs,4))" 852 | ] 853 | }, 854 | { 855 | "cell_type": "markdown", 856 | "metadata": { 857 | "slideshow": { 858 | "slide_type": "slide" 859 | } 860 | }, 861 | "source": [ 862 | "This is even weirder. After seeing the first sample, the maximum likelihood\n", 863 | "estimator predicts only values between 0 and 4 occurring, but after seeing\n", 864 | "another training sample, it is changing its mind and now predicts that values\n", 865 | "between 0 and 7 can occur.\n", 866 | "\n", 867 | "The Bayesian estimator, in contrast, \"knows\" that the parameter must be greater than 7,\n", 868 | "so it predicts a uniform distribution for the interval [0...7] and then a tradeoff\n", 869 | "between the parameter distribution and the uniform distribution of the parameters." 870 | ] 871 | }, 872 | { 873 | "cell_type": "markdown", 874 | "metadata": { 875 | "slideshow": { 876 | "slide_type": "slide" 877 | } 878 | }, 879 | "source": [ 880 | "The last sample illustrates this further.\n", 881 | "\n", 882 | "A sample of $x_3=2$ doesn't cause any update to the maximum likelihood estimator,\n", 883 | "but it does cause an update to posterior distribution." 884 | ] 885 | }, 886 | { 887 | "cell_type": "code", 888 | "execution_count": 14, 889 | "metadata": { 890 | "slideshow": { 891 | "slide_type": "slide" 892 | } 893 | }, 894 | "outputs": [ 895 | { 896 | "data": { 897 | "text/plain": [ 898 | "[]" 899 | ] 900 | }, 901 | "execution_count": 14, 902 | "metadata": {}, 903 | "output_type": "execute_result" 904 | }, 905 | { 906 | "data": { 907 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEACAYAAABI5zaHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAHqxJREFUeJzt3XmUFPW5//H3MzMsIopbglcI4oo/+CloZDSurUZFwhZABZeruEAwEOKGijcySuJNcnBJrjGCC+KCRMGrKKBA4sREMVEUVxAUQUDFBRVF0YF57h/Vg80wvQ3dXVPdn9c5c+yurq5+Tod85jtP1fdb5u6IiEhxKQu7ABERyT2Fu4hIEVK4i4gUIYW7iEgRUriLiBQhhbuISBHKKNzNrIeZLTazJWZ2RZJ9Ymb2kpm9ZmZP5bZMERHJhqW7zt3MyoAlwAnAe8DzwCB3X5ywTxvgWeAkd19tZru5+8f5K1tERFLJZOReCSx19xXuXgNMBfrW2+cMYLq7rwZQsIuIhCuTcG8HrEx4viq+LdH+wC5m9pSZPW9mZ+eqQBERyV5FDo9zCHA8sD0w38zmu/tbOTq+iIhkIZNwXw10SHjePr4t0SrgY3ffAGwws6eBrsAW4W5mWshGRKQR3N2y2T+TtszzwL5mtqeZNQcGATPq7fMocJSZlZtZK+AwYFGSAiP7M3bs2NBrUP3h11GK9Ue59mKovzHSjtzdfZOZjQDmEPwyuNPdF5nZsOBln+jui83sSeAVYBMw0d3faFRFIiKyzTLqubv7E0Cnetsm1Hs+Hhifu9JERKSxNEM1C7FYLOwStonqD1eU649y7RD9+hsj7SSmnH6YmRfy80REioGZ4Xk4oSoiIhGjcBcRKUIKdxGRIqRwFxEpQgp3EZEipHAXESlCCncRkSKkcBcRKUIKd5GI+/ZbGDUKND9QEmmGqkjEffIJ7LYbLF4MnTql31+iRzNURUrYk0+GXYE0JQp3kYir+2NY4S6JFO4iRaBlS/jHP2DDhrArkaZC4S5SBFq3hi5d4J//DLsSaSoU7iIRV9eWOflktWbkOwp3kSLRo4fCXb6jcBcpAmbQvTusXh38iCjcRSKuri1TXg4nnABz5oRbjzQNCneRIqLWjNRRuItEnHvQlgE46SSYOxc2bQq3Jgmfwl2kiLRvD+3awb/+FXYlEjaFu0gRsIRVR3r1gsceC68WaRoU7iIRV38tvt694fHHw6lFmg6Fu0iRqayEDz+E5cvDrkTCpHAXKQKJbZnycujZU62ZUqdwF4m4hm6RoL67ZBTuZtbDzBab2RIzu6KB1481s8/M7MX4z3/lvlQRydRJJ8H8+fDFF2FXImFJG+5mVgbcApwMdAEGm9kBDez6tLsfEv/5dY7rFJEUrN49enbYAY44QrNVS1kmI/dKYKm7r3D3GmAq0LeB/bK6BZSI5EayO1eqNVPaMgn3dsDKhOer4tvq+5GZLTSzmWbWOSfViUij9e4Ns2ZptmqpytUJ1QVAB3fvRtDCeSRHxxWRDNRvywB07Aht22q2aqmqyGCf1UCHhOft49s2c/cvEx7PNrNbzWwXd19b/2BVVVWbH8diMWKxWJYli0iiZG0ZgJ/+FB5+OOi/S3RUV1dTXV29TccwT/UvAzCzcuBN4ATgfeDfwGB3X5SwT1t3XxN/XAk86O4dGziWp/s8EcnO6tXBxKWG1nFfuBD694e33254dC/RYGa4e1b/C6Ydubv7JjMbAcwhaOPc6e6LzGxY8LJPBAaa2XCgBvgaOD378kWksZIFd9euwWsLF8LBBxe2JglX2pF7Tj9MI3eRnFu1Cg4/PPhvQy6/HFq2hHHjCluX5E5jRu6aoSpS5AYMgOnTw65CCk3hLhJxiTfraEhlJaxbB4sWJd9Hio/CXaTIlZV9d9WMlA6Fu0gRSHclzIABCvdSo3AXibiaTRv5vNs4Ul2scNRRsHIlvPNOAQuTUCncRSLui28/Z90Pr2Hm0plJ96mogL59Ydq0AhYmoVK4ixSJ6/5+XcrR+6BBMHVqAQuSUCncRSKu1h37ZifW16xnztvJ1/iNxYJZrEuWFK42CY/CXaQImJdz9dFXM+7p5L338nI47TSN3kuFwl0k4pwgzE/vcjofrv+Q6uXVSfcdPBgeeCD1YmNSHBTuIhEXBLVRXlbOmKPHMO7p5OsMHH44fP01vPJKwcqTkCjcRYrImQeeyfLPlvPMu880+LpZcGL1gQcKXJgUnMJdJOLcHeJrSjUrb8aVR11J1d+rku4/eHDQd1drprgp3EUirn5In9vtXJZ9uixp7/2gg6BVK3juufzXJuFRuItEXN0J1TrNy5tzbexarv7b1Q1eOWMWjN6nTClUhRIGhbtI1MVPqCYa/P8H89mGz5i1dFaDbznrrKA18+23+S9PwqFwF4m4+iN3gPKycn593K/5r6f+i1qv3er1vfaCzp1hZvIVCyTiFO4iRcDYelnIfgf0o6KsgmlvNLygzLnnwt1357cuCY/CXSTikl31Ymb85vjfcM1T17CxduNWrw8cCH//O3z0UZ4LlFAo3EUiLvFSyPpO3PtEdm+9O5MXTt7qtR12gN69dWK1WCncRYqYmfH7E3/PNdXX8OW3X271ulozxUvhLhJxtWlmI1W2q+TYPY/lhmdv2Oq1446DTz6Bl1/OV3USFoW7SFFIfZ+960+4nj/++4+898V7W2wvK4Ozz4bJW3dtJOIU7iIR19ClkPV13KkjFxx8Ab/626+2eu2cc+D++6GmJh/VSVgU7iIR5w1MYmrImKPHMHPpTF7+YMsezP77Q6dOMGNGfuqTcCjcRUpEm5Zt+NUxv+KyuZdttSzBsGEwYUJIhUleKNxFIi/z5R2H/nAoKz9fudWyBAMGwEsvwbJlua5NwqJwF4k494ZnqDakWXkzbjr5Jn755C/5ZuM3m7e3bBmcWL399nxVKYWmcBeJuGT3TE3mlP1OofP3OnPD/C0vjRw6FCZN0mJixSKjcDezHma22MyWmNkVKfbrbmY1ZtY/dyWKSCruJJ2hmszNJ9/MjfNv5N3P39287YADgp9HH81xgRKKtOFuZmXALcDJQBdgsJkdkGS/3wJP5rpIEcmtvXbei5GVI7nkyUu22K4Tq8Ujk5F7JbDU3Ve4ew0wFejbwH4jgWnAhzmsT0TSCK5zz27kDjD6yNG89MFLzH177uZt/fsHN89+660cFiihyCTc2wErE56vim/bzMz2APq5+59pzL8yEWm0xt4Ldbtm23HzyTczcvZIvt0UNNpbtAjWm7ntttzVJ+GoyNFxbgYSe/FJA76qqmrz41gsRiwWy1EJIqUpkxmqyfTu1JsJCyYw/tnxjDl6DAAXXQSHHgpVVdC6dY6KlKxUV1dTXV29TcewdGfazexwoMrde8SfXwm4u/8uYZ+6q2MN2A1YDwx19xn1juXZntkXkdT+umAFPf5yDDW/X9Go9y//bDmHTjyU+efPZ79d9wOC9syJJ8Lw4bmsVBrLzHDP7qx5Jm2Z54F9zWxPM2sODAK2CG133zv+sxdB3/2i+sEuIvmxrQOmjjt1ZMzRYxj2+LDNxxo1Cv74R6jd+g59EhFpw93dNwEjgDnA68BUd19kZsPMbGhDb8lxjSKS1rad6vrFYb9g3TfrmLRwEgDHHAPNm8PcuWneKE1W2rZMTj9MbRmRnJv3wnJOeTBGze+Xb9NxFn6wkJPuPYlXh79K29ZtuesumDYNZs1K/17Jr3y1ZUSkCat1z8klat1278aQbkMY9cQoAM44AxYsgDffzMHBpeAU7iLFIMsZqsmMjY3lhfde4PElj9OyJVx4IdxyS04OLQWmcBeJuFy2Ols1a8Udfe7gZ4//jLVfr2X48OBGHmvX5uwjpEAU7iIRl+nNOjIV6xhjYOeBjJg1gnbtoG9fuPXWnB1eCkThLiJbuf6E61nw/gIeev0hRo8OWjNffx12VZINhbtIxAVtmdyu+tGqWSsm95vMyNkj2eUHazjssGA5YIkOhbuINOjw9odz3sHnMfTxoYwe7YwfDxs3hl2VZErhLhJx27K2TDpjjx3LO5++w9LtJ9OuXXDdu0SDwl0k4rK5zV62WlS04P7+93P53Ms5a9QSfvvbxq9CKYWlcBeJuHyO3AEObHsgVcdWcfvaM/h207c8qdvxRILCXaQY5GgSUzIXdb+Idju2Y+8Lr2bcOI3eo0DhLhJxebhYZitmxl197uLljVN5t/mTzJuX38+TbadwF4m4Qi3Gt2urXbm3/718ecIQxly/RqP3Jk7hLlIM8tyWqRPrGOOiH53Pok7/yZNzNxXkM6VxFO4iEVfoZbSvPW4sHfb+hqH3jdPovQlTuIsUhcLdl76irIK5w6by/n/cwX9Pm12wz5XsKNxFIi7fl0I2pF2b3Rmz/1SqXjqXdz5dXvDPl/QU7iIRl89JTKlcc+5R7Lr4Sk66YyAbNm4o+OdLagp3EWmU8nKYOOSXfPjm3oyYOTLscqQehbtIxNWGeFazVy+j81t3Mvv1Z7jthdtCq0O2pnAXKQqFb8sAmMEN1++AT5nB2KfG8tQ7T4VSh2xN4S4SeeFej3jEEdB9n33p7w8wePpg3l77dqj1SEDhLhJxYZ1QTXT99TB9/PGMrhxL7wd6s+6bdaHWIwp3EcmBLl2gd294/7HhHNfxOAZPH8ymWs1gDZPCXSTiwrjOvSG/+U1wK76f73szGzZu4NI5l4ZdUklTuItEnDsFW1smld13h8svh6tGN2P6adOZt2weN86/MeyySpbCXSTiCr22TCq//CW89ho8/4+dmHXmLG567ib+8tpfwi6rJCncRYpC+CN3gBYtYPx4uPhi2GP7Djw++HFGzh7J0yueDru0kpNRuJtZDzNbbGZLzOyKBl7vY2Yvm9lLZvZvMzsy96WKSEOa0sgdoF8/aNsWJkyArrt3ZcqAKZz60Km88dEbYZdWUtKGu5mVAbcAJwNdgMFmdkC93ea5e1d3Pxg4H7gj55WKSApNY+QOwcSmm2+Ga6+Fjz+GH+/9Y8afOJ5T7j+FlZ+vDLu8kpHJyL0SWOruK9y9BpgK9E3cwd2/SnjaGqjNXYkiEjUHHghnnAGjRwfPz+56Nr+o/AU/vvfHfLj+w3CLKxGZhHs7IPHX7ar4ti2YWT8zWwQ8BpyXm/JEJJ1a9yY0bv/OuHEwdy48HW+3X3rEpQzqMoiT7j2JT7/+NNziSkBFrg7k7o8Aj5jZUcCvgRMb2q+qqmrz41gsRiwWy1UJIiWs6cX7DjsE7Zmf/QwWLoTmzaEqVsW6b9bxkyk/Yc7Zc2jdvHXYZTZJ1dXVVFdXb9MxLN3JGDM7HKhy9x7x51cC7u6/S/Get4Hu7r623nZvaid/RKLuvnkvM3T22Xx1wythl7IVd+jVC446Cq66qm6bc+FjF7L8s+U8fsbjtKxoGW6REWBmuGc3mSGTtszzwL5mtqeZNQcGATPqffA+CY8PAZrXD3YRyaemN3KH4OTqLbfADTfAsmV124wJvSawa6tdGfjgQL7Z+E24RRaptOHu7puAEcAc4HVgqrsvMrNhZjY0vtsAM3vNzF4E/gc4LW8Vi0ik7LUXXHYZDB/O5htql5eVc99P76NlRUv6P9hfd3LKg7RtmZx+mNoyIjl379yFDHviHL664eWwS0mqpgYOOwxGjIDzEi63qNlUw5kPn8kX337B/57+v2rRJJGvtoyINHFhL/mbTrNmcPfdcMUVsGpVwvbyZkwZMIU2LdrQd2pfvq75OrQai43CXSTimsqqkOkcdBCMHAlDh37XngGoKKvgvv73set2uyrgc0jhLhJxQVA27ZF7nauugvffh8mTt9xeUVbBPT+9h+9v/31Ouf8U3ewjBxTuIpEXjZE7fNeeGT0aVq/e8rWKsgom95tM5+915rjJx/HR+o9CqbFYKNxFikETWM89U127BidWhwyB2noLlZSXlfOnnn+i5749OXrS0bz7+bvhFFkEFO4iERfFC9DGjIEvvwxmsNZnZow7fhzDfjiMoycdzeKPFxe+wCKQs+UHRCQcUTmhmqiiAu6/Hyor4fjjoVu3rfe5+EcXs8t2uxC7O8aMwTOobFdZ+EIjTCN3kSLQ1C+FbMhee8FNNwWrR371VcP7nNPtHCb2nshPpvyERxc/WtgCI07hLhJxtbXRG7nXOessOPjgYAZrMn069WHWGbO4aNZF/OG5PxSuuIhTuIsUheiN3OvceivMng0PP5x8n+7tuvPMec8w8cWJjJo9ik21mwpXYEQp3EUiLupLerRpAw8+GCwN/NZbyffruFNHnjnvGV776DUGPDiA9d+uL1yREaRwF5HQde8OVVUwcCB8nWKC6k4td2L2mbPZebuddalkGgp3kaIQ3bZMneHDoXPn4Br4VJqXN+euPndx1kFncdgdh/H0iqcLU2DEKNxFIq424m2ZOmYwcSLMnw933ZVuX+OSH13CPf3u4dSHTuVP//5T5NtTuaZwFykCUbwUsiGtW8O0acHqkS++mH7/E/c5kWfPe5bbFtzGhY9dqBt/JFC4i0RcFCcxpdK5M/z5z9CvH3zwQfr999llH+afP59PN3zKsXcfqz58nMJdJOoitCpkpgYODG7q0b8/fJPBYLx189ZMO3UaAzsPpPL2SmYumZn/Ips4hbuINEnXXAN77LH1+u/JmBmXHXEZ00+bzvCZw7ly3pVsrN2Y/0KbKIW7SMQVW1umTllZsO77K68EN9jO1JEdjmTB0AUs/GAhx00+jtXrVqd/UxFSuItEnHvxnFCtb/vt4dFH4cYbYcaMzN/3ve2/x6wzZ9Fjnx4cevuhvPh+Bmdni4zCXSTiinXkXqdDB3jkETj/fHjuuczfV2ZlXH3M1fTZvw9/XfbX/BXYRCncRYpCcY7c61RWwqRJwRU0S5dm994dW+xY9L8AG6JwF4m4Upm806sXXHcd9OgBa9Zk/j4zK5nvKJHCXaQIFGvPvb6hQ4P133v1Cu7klAnDNHIXkegptUHpddfBQQcFLZoNG9Lvr5G7iERSqY1K69ag2W03OPVUqKlJs79G7iISXaXRlqlTXg733hsE/VlnwaYU9+4wK63vpo7CXSTiSrHlANCsWXCTj7Vr4YILoLY2+b6l+B1lFO5m1sPMFpvZEjO7ooHXzzCzl+M//zSzA3NfqogkV5qj05Ytg2vgly6Fn/+84YBXWyYJMysDbgFOBroAg83sgHq7LQOOcfeuwK+B23NdqIhIQ7bfHmbNgldfDa6mqd+i0QnV5CqBpe6+wt1rgKlA38Qd3P05d/88/vQ5oF1uyxSRZIrlZh3bYscd4Ykn4O23YcgQ2JiwXphG7sm1A1YmPF9F6vC+AJi9LUWJSHZK5Tr3VFq3hpkzgzXgzzrru6toSnXkXpHLg5nZccAQ4Khk+1RVVW1+HIvFiMViuSxBpASVXnAl06pVsMDYgAFw+unwwAPRHLlXV1dTXV29TcfIJNxXAx0SnrePb9uCmR0ETAR6uPunyQ6WGO4isu28CG/WsS1atoSHHw5G7z17QvfR0LJF2FVlp/7A99prr836GJm0ZZ4H9jWzPc2sOTAI2GLxTTPrAEwHznb3t7OuQkQaLWqj0kJo0QKmToX99oN7Jhvr15fed5Q23N19EzACmAO8Dkx190VmNszMhsZ3+xWwC3Crmb1kZv/OW8UishVzjdzrKy8P7sW6887Gy6+UXrhn1HN39yeATvW2TUh4fCFwYW5LE5FMuKOuTBJmsFMbY1Nt6YW7ZqiKRF7pBVc2SvVqGYW7SMS5A2rLJBXFq2VyQeEuEnGOqy2TgpmV5B83CneRIqATqskZpXlFkcJdJOJKsZ+cDfXcRSS6NHBPSj13EYkutWWS0shdRCJJq0KmppG7iESWVoVMzsxK8hegwl0k4kpxVCrpKdxFok6rQqaknruISBEyU89dRCKoFIMrG4ZG7iISQe46oZqKRu4iElGlF1zZKNPIXUSiSyP3pEr0q1G4i0RcrWtVyFQ0iUlEpAjpUkgRiSR3LfmbSplOqIpINKktk4pG7iISXRq5J6Weu4hEUikuipUNMyjFy0UV7iJFQJOYkgtmqIZdReEp3EUirwSTKwuaoSoikVSKo9JsKNxFJLpMbZlkDCvJ34AKd5GIK8VRaTY0cheRaNIkppQ0iSkFM+thZovNbImZXdHA653M7Fkz22Bml+S+TBFJxjWJKb3Sy3Yq0u1gZmXALcAJwHvA82b2qLsvTtjtE2Ak0C8vVYqINJLaMslVAkvdfYW71wBTgb6JO7j7x+6+ANiYhxpFJAV3NEM1BbVlkmsHrEx4viq+TUSaAMd1sUwKus2eiESTg5ruyZVqWyZtzx1YDXRIeN4+vq1RqqqqNj+OxWLEYrHGHkpE0KWQ6ZgZUTujWl1dTXV19TYdI5Nwfx7Y18z2BN4HBgGDU+yfcgiRGO4ikhtaWya5INqjFe71B77XXntt1sdIG+7uvsnMRgBzCNo4d7r7IjMbFrzsE82sLfACsANQa2ajgM7u/mXWFYlIVkqxn5wNM4vawD0nMhm54+5PAJ3qbZuQ8HgN8IPcliYisu1KteeuE6oiRUFtmWQU7iISSbpZR2pluhRSRKJKJ1STi+LVMrmgcBeJvNILrmwE51NL7ztSuItEXOnFVnaCG2SXHoW7SFFQWyaZ4FLI0ot3hbtIxJXiycJs6GoZEYksnVBNTqtCikgklWJwZcNMl0KKSAS5VoVMKfhmFO4iEjmlF1zZCHrupUfhLlIENG5PTpOYRCSS1JZJTSdURSSSSjG4sqHb7IlIZOlSyOTUlhGRSCrFUWk21JYRkejSwD05LRwmIlHkOLjSPZkyK83vRuEuEnWlNyjNinruIhJZVqKj00wES/4q3EUkYkoxuLJRprVlRCSK3FHPPQW1ZUQkohx1ZZLTbfZEJJJKL7ayU6rnIxTuIkWhNAMsE5rEJCKRVIonC7NRpp67iESV1pZJTvdQFZFIKsXgyoZG7imYWQ8zW2xmS8zsiiT7/NHMlprZQjPrltsyRUQaSVfLNMzMyoBbgJOBLsBgMzug3j6nAPu4+37AMOC2PNQauurq6rBL2CaqP1z5qj/ouee3LRPl777MjJqVa8Muo+AyGblXAkvdfYW71wBTgb719ukL3APg7v8C2phZ25xW2gRE+R84qP6w5bP+fHfco/zdmxk1qz4Nu4yCyyTc2wErE56vim9Ltc/qBvYRkbzRCdVkSrXnXhF2Abny3w/O4eb5/5PXz/hy/pv8+fMFef2MfFL94cpX/V+wiv8oPzDnxy0WFeXlbKhYQ9uLe+ftM45v35MHLh2et+M3hqW7RtbMDgeq3L1H/PmVgLv77xL2uQ14yt3/En++GDjW3dfUO1bp/foUEckB9+wWEMpk5P48sK+Z7Qm8DwwCBtfbZwbwc+Av8V8Gn9UP9sYUJyIijZM23N19k5mNAOYQ9OjvdPdFZjYseNknuvssM+tpZm8B64Eh+S1bRERSSduWERGR6Cn4DFUz+72ZLYpPdppuZjsWuoZsZTKJq6kys/Zm9jcze93MXjWzX4RdU2OYWZmZvWhmM8KuJVtm1sbMHor/u3/dzA4Lu6ZsmNnFZvaamb1iZvebWfOwa0rFzO40szVm9krCtp3NbI6ZvWlmT5pZmzBrTCVJ/VnnZhjLD8wBurh7N2ApcFUINWQsk0lcTdxG4BJ37wL8CPh5xOqvMwp4I+wiGukPwCx3/39AV2BRyPVkzMz2AEYCh7j7QQSt3EHhVpXWJIL/vya6Epjn7p2Av9G0c6eh+rPOzYKHu7vPc/fa+NPngPaFriFLmUziarLc/QN3Xxh//CVBsERqDoKZtQd6AneEXUu24iOso919EoC7b3T3dSGXla1yYHszqwBaAe+FXE9K7v5PoP6spb7A5PjjyUC/ghaVhYbqb0xuhr1w2HnA7JBrSCeTSVyRYGYdgW7Av8KtJGs3AZcTzZkoewEfm9mkeFtpopltF3ZRmXL394AbgHcJJid+5u7zwq2qUb5fdwWfu38AfD/kerZFRrmZl3A3s7nx/lzdz6vx//ZO2OdqoMbdp+SjBtmSmbUGpgGj4iP4SDCznwBr4n99GNGbilkBHAL8yd0PAb4iaBFEgpntRDDq3RPYA2htZmeEW1VORHGgkFVu5mWGqrufmOp1MzuX4M/s4/Px+Tm2GuiQ8Lx9fFtkxP+cngbc6+6Phl1Plo4E+phZT2A7YAczu8fd/zPkujK1Cljp7i/En08DonRS/sfAMndfC2BmDwNHAFEblK0xs7buvsbMdgc+DLugbGWbm2FcLdOD4E/sPu7+TaE/vxE2T+KKXyUwiGDSVpTcBbzh7n8Iu5BsufsYd+/g7nsTfPd/i1CwE28FrDSz/eObTiBaJ4bfBQ43s5YW3Iz0BKJxQrj+X3kzgHPjj88BmvogZ4v6G5ObBb/O3cyWAs2BT+KbnnP3iwpaRJbiX+wf+G4S129DLiljZnYk8DTwKsGfog6McfcnQi2sEczsWOBSd+8Tdi3ZMLOuBCeDmwHLgCHu/nm4VWXOzMYS/GKtAV4CLohfXNAkmdkUIAbsCqwBxgKPAA8BPwBWAKe5+2dh1ZhKkvrHkGVuahKTiEgRCvtqGRERyQOFu4hIEVK4i4gUIYW7iEgRUriLiBQhhbuISBFSuIuIFCGFu4hIEfo/UX6V3U0jY6AAAAAASUVORK5CYII=\n", 908 | "text/plain": [ 909 | "" 910 | ] 911 | }, 912 | "metadata": {}, 913 | "output_type": "display_data" 914 | } 915 | ], 916 | "source": [ 917 | "pmu3 = pxt(2,mus)*pmu2\n", 918 | "pmu3 /= C*sum(pmu3)\n", 919 | "plot(xs,pmu3)\n", 920 | "plot(xs,pmu2)" 921 | ] 922 | }, 923 | { 924 | "cell_type": "code", 925 | "execution_count": 15, 926 | "metadata": { 927 | "slideshow": { 928 | "slide_type": "slide" 929 | } 930 | }, 931 | "outputs": [ 932 | { 933 | "data": { 934 | "text/plain": [ 935 | "[]" 936 | ] 937 | }, 938 | "execution_count": 15, 939 | "metadata": {}, 940 | "output_type": "execute_result" 941 | }, 942 | { 943 | "data": { 944 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEACAYAAABfxaZOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAH45JREFUeJzt3X2UXFWd7vHv0yTRIBCUkdxrQoIYDIiDWSghiEgNGYcQkcTXSXDxkisS1CBvw4XBWZLMGkVkBME4QyLKAhYShwiXjCJGBhtUkAlBXoQEmoEVkiDNBAQuqNy8/O4fVR2KstNd1XWqzz5Vz2etLKpO7ar6dTj9ZJ+99zlHEYGZmXWGrrwLMDOz4ePQNzPrIA59M7MO4tA3M+sgDn0zsw7i0Dcz6yB1hb6kGZLWSnpM0rn9vD5Z0l2S/iTprJrXxki6QdIaSQ9LOiSr4s3MrDEjBmsgqQtYDEwHngZWSbo5ItZWNXsOOA2Y3c9HXAbcEhGflDQC2Ln5ss3MbCjq6elPBXoiYl1EbAaWAbOqG0TEpohYDWyp3i5pN+DwiLiq0m5LRLyUTelmZtaoekJ/HLC+6vmGyrZ6vB3YJOkqSfdJWippdKNFmplZNlo9kTsCOAj4dkQcBPwBOK/F32lmZjsw6Jg+sBGYUPV8fGVbPTYA6yPi3srz5cCfTQQDSPJFgMzMGhQRaqR9PT39VcAkSRMljQLmACsGaL+9gIjoBdZLemdl03TgkR29MSIK+eeCCy7IvQbXn38drr+Yf4pc/1AM2tOPiK2SFgArKf8j8d2IWCNpfvnlWCppLHAvsCuwTdLpwLsi4mXgi8B1kkYCTwDzhlSpmZk1rZ7hHSLiVmByzbYlVY97gb128N4HgIObqNHMzDLiM3IzUCqV8i6hKa4/X64/X0Wvv1Ea6rhQ1iRFKrWYmRWBJKIFE7lmZtYmHPpmZh3EoW9m1kEc+mZmHcShb2bWQRz6ZmYdxKFvZtZBHPpmZh3EoW9m1kEc+mZmHcShb2bWQRz6ZmYdxKFvZtZBHPpmZh3EoW9m1kEc+mZmHcShb2bWQeq6R66kGcA3ee3G6BfVvD4ZuAo4CDg/Ii6peb2L8o3TN0TEsVkUnpL3Ln0vDzzzQN5lmP0ZSdx+wu0cPvHwvEuxRAwa+pXAXgxMB54GVkm6OSLWVjV7DjgNmL2DjzkdeATYrbly0/TsK8/y2GmPMWHMhLxLMXud2ctm8/s//T7vMiwh9fT0pwI9EbEOQNIyYBawPfQjYhOwSdIxtW+WNB6YCXwFOCuLolM0smskI7rqOnAyGzZd8giuvV49e8Q4YH3V8w2VbfW6FDgHaNu7nvuG7pYy759WraXdAEkfBnoj4n5AlT9tSWrbH80KzPul1apnPGIjUD1YPb6yrR6HAcdKmgmMBnaVdE1EnNBf44ULF25/XCqVKJVKdX5NvqJ9D2KsDXj/bB/d3d10d3c39Rka7NBP0k7Ao5Qncn8H/CcwNyLW9NP2AuDliPhGP68dAZy9o9U7kqKoh6HjLhnHPSffw/jdxudditnrzFo2i3lT5jF7vx2tsbAik0RENHQ4N2hPPyK2SloArOS1JZtrJM0vvxxLJY2lvCRzV2CbpNOBd0XEy43/GMWk9h25sgLzfmm16lpuEhG3ApNrti2petwL7DXIZ9wB3DGEGpNX1CMU6wzeP62a13OZmXUQh35GvErCUuT90mo59DPg1RGWMu+fVs2hnxFPmFmKvF9aLYe+mVkHcehnwKsjLGXeP62aQz8jnjCzFHm/tFoO/Qx4osxS5v3Tqjn0zcw6iEM/I14lYSnyfmm1HPoZ8ESZpcz7p1Vz6JuZdRCHfka8SsJS5P3Sajn0M+DVEZYy759WzaGfEU+YWYq8X1oth76ZWQdx6GfAqyMsZd4/rZpDPyOeMLMUeb+0Wg79DHiizFLm/dOq1XWPXEkzgG/y2o3RL6p5fTJwFXAQcH5EXFLZPh64BhgLbAO+ExGXZ1d+Gl56EcaPB/0x70rMXm/zLJg0C+a8O+9KLBWDhr6kLmAxMB14Glgl6eaIWFvV7DngNGB2zdu3AGdFxP2SdgFWS1pZ897C27YNfnmHOHBS3pWYvd77viYe+m3eVVhK6unpTwV6ImIdgKRlwCxge3BHxCZgk6Rjqt8YEc8Az1QevyxpDTCu+r3tIAhGvxFGj867ErPXe+tboedOD+/Ya+oZ0x8HrK96vqGyrSGS9gamAPc0+t4i8ISZpWiPPcSTT8DmzXlXYqmoa0y/WZWhneXA6RHx8o7aLVy4cPvjUqlEqVRqeW2ZceZbgkaNgr/4C3joITjooLyrsWZ1d3fT3d3d1GfUE/obgQlVz8dXttVF0gjKgX9tRNw8UNvq0C8U+fDZ0rXvO4O773bot4PazvCiRYsa/ox6hndWAZMkTZQ0CpgDrBigfW2f93vAIxFxWcPVFUiXh3csQULsuy/cfXfelVgqBg39iNgKLABWAg8DyyJijaT5kk4BkDRW0nrgTOBLkp6StIukw4BPA0dK+o2k+yrLP9uMe/qWrkmTwqFv29U1ph8RtwKTa7YtqXrcC+zVz1t/BezUTIFm1pz/+TZ4/nl49lnYc8+8q7G8+YzcjHh4x1IkiS7BIYd4iMfKHPqZ8PCOpSsIDj0Ufv3rvCuxFDj0M+KOvqVs2jT39K3MoZ8Rn5xlKeq7icohh8Dq1bBlS84FWe4c+llQ+NwsS1ZEsPvuMGECPPhg3tVY3hz6GXFP31JUvV8eeqiHeMyhb9YxPvAB+OUv867C8ubQz0R4IteS1XcTlcMPhzvvBN89sbM59DMij+pbgqr3y332KQf+k0/mWJDlzqGfgZB7+pauvhujS/DBD8IvfpFzQZYrh34WfLhsBdE3xGOdy6GfEXW5q2/pqV1Vdvjh7ul3Ood+Fnw9fUtYVB2Kvvvd8N//Dc88k2NBliuHvlkH6eqCww7z0s1O5tDPiK+yaSnqb1XZBz/ocf1O5tDPhId3LF1RszDf4/qdzaGfEff0LUX9XR7kve+Fnh548cUcCrLcOfSz4IlcS1jUHImOGgUHHwy/+lVOBVmuHPoZcUffisRDPJ2rrtCXNEPSWkmPSTq3n9cnS7pL0p8kndXIe9uFr7JpKdrR5UFKJejuHtZSLBGDhr6kLmAxcBRwADBX0n41zZ4DTgMuHsJ724CHdyxdtRO5UL7M8kMPwUsv5VCQ5aqenv5UoCci1kXEZmAZMKu6QURsiojVQO19eQZ9b7twR9+KZPRomDrVQzydqJ7QHwesr3q+obKtHs28t1B8lU1L0UD75ZFHwu23D2MxloQReRdQbeHChdsfl0olSqVSbrU0xlfZtHTVrt7pc+SRsGDBMBdjTenu7qa7ycmYekJ/IzCh6vn4yrZ6NPTe6tAvGvf0LUUDLTA4+GB4/HF47jnYY49hLMqGrLYzvGjRooY/o57hnVXAJEkTJY0C5gArBmhfvZc1+t7Cck/fimbkyPItFO+4I+9KbDgNGvoRsRVYAKwEHgaWRcQaSfMlnQIgaayk9cCZwJckPSVplx29t1U/TG58cpYlrL/VO308rt956hrTj4hbgck125ZUPe4F9qr3ve3I6/QtRYMNO/7VX8Hxxw9TMZYEn5GbCU/kWrp2NJELMGVK+dr6v/vdMBZkuXLom3WwnXaCI46An/8870psuDj0M+KrbFqK6hl2PPJIh34ncehnwRO5lrCBJnIBpk+Hn/0MBmlmbcKhnxF39K2o9t8ftmwpX2Pf2p9DPyNevWMpquekQQmOOgp++tNhKMhy59DPhI+LLV0Drd7p49DvHA79jHgi11JU7xHo9Onlm6W/+mqLC7LcOfTNjD32KI/t33VX3pVYqzn0syCfnGXpGmz1Th8P8XQGh35GPJFrKWrk6q8O/c7g0M+Ce/qWsHomcqF8J60nn4Te3hYXZLly6JsZUL7U8pFHlk/Usvbl0M+Ib6JiKWp02NFDPO3PoZ8RD+9YquqdyIVy6K9cCdu2tbAgy5VDPyOeyLUUNXoEuvfe5eWbq1e3ph7Ln0M/I858axfHHAM/+lHeVVirOPSb1Mihs1ke6l2908eh394c+hlxT99SNJRhx/e/v7x0c+PGFhRkuasr9CXNkLRW0mOSzt1Bm8sl9Ui6X9KUqu1nSvqtpAclXSdpVFbFp6DRXpTZcGv0aHTECJgxA265pUUFWa4GDX1JXcBi4CjgAGCupP1q2hwNvCMi9gXmA1dUtr8NOA04KCIOpHwj9jmZ/gRmljkP8bSvenr6U4GeiFgXEZuBZcCsmjazgGsAIuIeYIyksZXXdgLeJGkEsDPwdCaVpyTk4R1L0lDPH5kxo3wLxT/+MeOCLHf1hP44YH3V8w2VbQO12QiMi4ingW8AT1W2vRARtw293PR4ItdSN5QhyLe8BaZMge7u7OuxfI1o5YdL2p3yUcBE4EVguaTjIuL7/bVfuHDh9selUolSqdTK8jLlnr61m74hnqOPzrsS69Pd3U13k/8S1xP6G4EJVc/HV7bVttmrnzZ/DTwREc8DSLoReD8waOgXSjjxLU3NnDR4zDHlwF+82J2aVNR2hhctWtTwZ9QzvLMKmCRpYmXlzRxgRU2bFcAJAJKmUR7G6aU8rDNN0htV3vumA2sarjJhfYfO/qWwVA11CHL//csreR54IOOCLFeDhn5EbAUWACuBh4FlEbFG0nxJp1Ta3AI8KelxYAnw+cr2/wSWA78BHgAELG3FD5IvJ76lqZkLAUrw0Y/CTTdlWJDlrq4x/Yi4FZhcs21JzfMFO3jvIqDxYxAzy91HPwqnngpDGEWwRPmM3Cb1HTp7eMdS1cwJhIceCps2QU9PhgVZrhz6WfBEriWq2au/dnXBrFke4mknDv0meSLXUtfsuSQf+5hDv5049M1sQKUSPPqoL8DWLhz6mfBlGCxNWdzGc9Qo+PCH4eabMyjIcufQb9K2bb4Mg6UtiyvBfuxjcOONGRRjuXPom9mgjjoKVq2C55/PuxJrlkM/C169Y4nK6t7NO+8M06fDv/97Jh9nOXLoN2mbr7JpicvqSrAf/zjccEMmH2U5cuhnwj19S1MWE7l9jj0WfvELD/EUnUO/SZ7ItdRldUvPXXeFv/kbr9kvOoe+mdVtzhxYtizvKqwZDv0mReCJXEtWVhO5fWbOhHvvhd7eTD/WhpFDv0lZHTqbtUqWt/QcPbp8c5XlyzP7SBtmDn0za4iHeIrNod+kcifKwzuWpixX7/T50IfgkUdg/frMP9qGgUO/SVkeOpu1QtZDkKNGlW+u8m//lunH2jBx6DfJE7mWsqwncvt4iKe4HPpm1rBSCTZsKF9y2YqlrtCXNEPSWkmPSTp3B20ul9Qj6X5JU6q2j5F0g6Q1kh6WdEhWxafAl2Gw1LViCHLECDjuOLjmmsw/2lps0NCX1AUsBo4CDgDmStqvps3RwDsiYl9gPnBF1cuXAbdExP7Ae4A1GdWeEA/vWJpaMZHb58QT4dprYdu2ln2FtUA9Pf2pQE9ErIuIzcAyYFZNm1nANQARcQ8wRtJYSbsBh0fEVZXXtkTES9mVnz9P5FrqWnUuyYEHwlveAt3dLfl4a5F6Qn8cUL04a0Nl20BtNla2vR3YJOkqSfdJWippdDMFp8aZb53sxBM9xFM0I4bh8w8CvhAR90r6JnAecEF/jRcuXLj9calUolQqtbi8jHj1jiWqVat3+hx3HPzjP8Irr8Cb3tTSrzKgu7ub7iYPreoJ/Y3AhKrn4yvbatvstYM26yPi3srj5UC/E8Hw+tAvCk/kWupaOQQ5diwcdlj5VorHH9+yr7GK2s7wokWLGv6MeoZ3VgGTJE2UNAqYA6yoabMCOAFA0jTghYjojYheYL2kd1baTQceabhKM0vWCSd4iKdIBg39iNgKLABWAg8DyyJijaT5kk6ptLkFeFLS48AS4PNVH/FF4DpJ91NevfPVjH+GXPkyDJayVq7e6XPssXDffb4sQ1HUNaYfEbcCk2u2Lal5vmAH730AOHioBabOV9m01LV6H33jG+ETnygv3zz//JZ+lWXAZ+Rmwj19S1OrJ3L7nHwyXHml1+wXgUO/SZ7HNYP3vQ922w3+4z/yrsQG49Bvkk/OstQNxz4qwSmnwHe+0/KvsiY59Jvkq2xayoZjIrfPpz8NK1fCs88O21faEDj0m+SJXEvdcO2jY8aUr7N/9dXD8nU2RA79Jnl0x+w1fUM8/r1Il0M/Ex7esTQN1+qdPtOmle+sdccdw/q11gCHfpM8kWupG859tG9Cd+nSYftKa5BDv0mOfLPXO/54+MlPoLc370qsPw79LHj1jiVqOFfv9Hnzm8tn6Hr5Zpoc+k2Kbe7rW9ryWGG2YAFccQVs3jzsX22DcOhnII/elFk9hnsit8973gP77AM33ZTL19sAHPpN8vX0LXV5LTY47TRYvDiXr7YBOPTNrCVmz4YnnoAHHsi7Eqvm0M+Eh3csTXkOPY4cCaeeCt/6Vm4lWD8c+k3a5olcS1yelwo55RT44Q/huedyK8FqOPTNrGX23LN8Zy2frJUOh36TArxO35KV1+qdamefXR7iefXVvCsxcOg3zZdhsNTlvY8eeCD85V/CddflWoZV1BX6kmZIWivpMUnn7qDN5ZJ6JN0vaUrNa12S7pO0IouiU+Ibo1vKUjmH5Jxz4J//2bdTTMGgoS+pC1gMHAUcAMyVtF9Nm6OBd0TEvsB84IqajzkdeCSTis2scKZPhze8oXxNHstXPT39qUBPRKyLiM3AMmBWTZtZwDUAEXEPMEbSWABJ44GZwJWZVZ0Q30TFUpfCPirB3/1dubdv+aon9McB66ueb6hsG6jNxqo2lwLn0KYXpPTwjqUshYncPp/6FPzXf8G99+ZdSWcb0coPl/RhoDci7pdUYpB0XLhw4fbHpVKJUqnUyvIykfckmdlgUtlHR46EM86Aiy+GH/wg72qKqbu7m+7u7qY+o57Q3whMqHo+vrKtts1e/bT5BHCspJnAaGBXSddExAn9fVF16BdFGr9OZsXw2c/C174Gjz4KkyfnXU3x1HaGFy1a1PBn1DO8swqYJGmipFHAHKB2Fc4K4AQASdOAFyKiNyLOj4gJEbFP5X237yjwC83r9C1Rqaze6bPrruULsX31q3lX0rkG7elHxFZJC4CVlP+R+G5ErJE0v/xyLI2IWyTNlPQ48Aowr7Vlp8PX07fUpTCRW+2002DSpPLF2PbZJ+9qOk9dY/oRcSswuWbbkprnCwb5jDsA3y7ZrMPtvjt87nNw4YW+u1YefEZuBlI7hDbrk9LqnWpnnAE33ghPPZV3JZ3Hod8k30TFUpfK6p1qe+xRntS96KK8K+k8Dv1MpNmbMkv5KPSss+D66+Hpp/OupLM49JuUYCfKrBD23BNOOqm8hNOGj0O/SamtjDCrlfI+et555atvrluXdyWdw6HfpAi8Tt+SlepEbp8994TPfx6GcI6RDZFDv0kpTpKZVUt9Hz37bPjRj2DNmrwr6QwO/Syk3ZkyS9ruu5evwPnlL+ddSWdw6DfJwzuWspRX71RbsADuugtWr867kvbn0G9SypNkZlCMfXTnneEf/gG+9KW8K2l/Dv1m+Xr6lrDUJ3KrfeYz0NMDP/953pW0N4e+mSVh1Kjy9XjOOgu2bs27mvbl0G+SL8NgqUt99U61T34SRo+Ga6/Nu5L25dDPQFEmy6zzFG3flODSS8vj+6+8knc17cmh36Qi9aKsMxVhIrfaIYfAEUfA17+edyXtyaHfpGL9OpkVw4UXwuLFsGFD3pW0H4d+FrxO3xJVpNU71SZMgFNP9RLOVnDoN2mbb5doiSvqEOR558Ftt8Hdd+ddSXtx6JtZknbdFS6+uHxBti1b8q6mfdQV+pJmSFor6TFJ5+6gzeWSeiTdL2lKZdt4SbdLeljSQ5K+mGXxKQifnGUJK9rqnVpz55avzXPFFXlX0j4GDX1JXcBi4CjgAGCupP1q2hwNvCMi9gXmA33/i7YAZ0XEAcChwBdq31t8xTx0ts5RtNU71ST49rfLl17u7c27mvZQT09/KtATEesiYjOwDJhV02YWcA1ARNwDjJE0NiKeiYj7K9tfBtYA4zKrPhnF7k1Z+yrqRG61d70L5s2Dc87Ju5L2UE/ojwPWVz3fwJ8Hd22bjbVtJO0NTAHuabTIlBV0jsysUL785fI1ee68M+9Kim/EcHyJpF2A5cDplR5/vxYuXLj9calUolQqtby2ZhX50Nk6Q1FX71TbZZfymbqf+xzcdx+84Q15V5SP7u5uuru7m/qMekJ/IzCh6vn4yrbaNnv110bSCMqBf21E3DzQF1WHflFEFH+yzNpXO+2bH/94+Zo8F14IBYyKTNR2hhcN4T6T9QzvrAImSZooaRQwB1hR02YFcAKApGnACxHRN+3yPeCRiLis4eoKwD19S1277KMS/Mu/lCd2f/vbvKsprkFDPyK2AguAlcDDwLKIWCNpvqRTKm1uAZ6U9DiwBPgcgKTDgE8DR0r6jaT7JM1o0c+SizY4cjYrjHHj4CtfgZNP9uWXh6quMf2IuBWYXLNtSc3zBf2871fATs0UWAi+DIMlqh1W79Q6+WS4/nr41rfgjDPyrqZ4fEZuk6I8qG+WrHaYyK3W1QXf+Q780z/BE0/kXU3xOPSb1F6/TmbFMGlS+do88+Z5mKdRDv0seHjHEtVOq3dqnXlm+b+XXppvHUXj0G9S+Cqblrh2Wb1Ta6ed4Oqr4aKL4MEH866mOBz6mWjf3pQVWztO5Fbbe+/ylTiPPx5efTXvaorBod+kbRGOfEtau03k1jrxRNhnn/KlGmxwDn0zKzQJli4tn617xx15V5M+h34m3Ne3NLXzRG61t74VrryyPMyzaVPe1aTNod+kbW1+6GzF164TubVmzoS//Vs46STYti3vatLl0DeztvHVr8Jzz8Ell+RdSbqG5dLK7cy3S7SUtfvqnVojR8KyZTB1KnzgAzBtWt4Vpcc9/aZ1xqGzFVe7r96pNXEiLFkCc+bA73+fdzXpceg3KQKfkWvJ6pSJ3FqzZ5f/HH+8x/drOfTNrC1dfDG89FL5pur2God+k4Kgw4ZNrWA6ZfVOrZEj4YYb4Kqr4OYB79nXWRz6TfLwjqWs0yZya40dC8uXw2c/C2vX5l1NGhz6Teq0STIrnk7fR6dOLV+UbfZsePHFvKvJn0O/SZ3962RWDPPmwfTpMHcubNmSdzX5qiv0Jc2QtFbSY5LO3UGbyyX1SLpf0pRG3lt8nX0Ibenq1NU7/fnmN8uBf8YZnX1v60FDX1IXsBg4CjgAmCtpv5o2RwPviIh9gfnAFfW+t+hiW7Bt/St5l9GU7u7uvEtoiusfWKsncovy9983sdvdDZdf/tr2otSflXp6+lOBnohYFxGbgWXArJo2s4BrACLiHmCMpLF1vrfwtm34Q94lNKXoO73rz1eR6h8zBn78Y/j612HFivK2ItWfhXpCfxywvur5hsq2etrU897C8wG0parTV+/0Z+JEuOkm+MxnYPXqvKsZfq269k4ye9r/27yVvf737JZ9/qvxf0noxzV7nS518eOeH/OR6z/Ssu949KFHWX198dJz4t/Dod+CkWsf5V9fbE39IzSKjZf8sCWfPVQabDmXpGnAwoiYUXl+HhARcVFVmyuAn0fEDyrP1wJHAG8f7L1Vn9HBUytmZkMT0diJQvX09FcBkyRNBH4HzAHm1rRZAXwB+EHlH4kXIqJX0qY63jukws3MrHGDhn5EbJW0AFhJeQ7guxGxRtL88suxNCJukTRT0uPAK8C8gd7bsp/GzMwGNOjwjpmZtY9kzsiV9HVJayond/1Q0m5511SPIp98Jmm8pNslPSzpIUlfzLumRknqknSfpBV519IoSWMk3VDZ7x+WdEjeNTVC0pmSfivpQUnXSRqVd00DkfRdSb2SHqza9mZJKyU9KumnksbkWeNAdlB/w7mZTOhTHgI6ICKmAD3A3+dcz6Da4OSzLcBZEXEAcCjwhYLVD3A68EjeRQzRZcAtEbE/8B6gMEOfkt4GnAYcFBEHUh4qnpNvVYO6ivLvarXzgNsiYjJwO2nnTn/1N5ybyYR+RNwWEX23O/g1MD7PeupU6JPPIuKZiLi/8vhlyqFTmPMoJI0HZgJX5l1Loyo9ssMj4iqAiNgSES/lXFajdgLeJGkEsDPwdM71DCgifgnU3ktrFnB15fHVQOvWdzepv/qHkpvJhH6N/wX8JO8i6tA2J59J2huYAtyTbyUNuRQ4h2Je9+7twCZJV1WGp5ZKGp13UfWKiKeBbwBPARspr9i7Ld+qhmTPiOiFcicI2DPneppRV24Oa+hL+lll/K/vz0OV/36kqs2XgM0R8f3hrK2TSdoFWA6cXunxJ0/Sh4HeypGKKN4ZciOAg4BvR8RBwB8oDzUUgqTdKfeSJwJvA3aRdFy+VWWiiB2IhnKzVWfk9isiPjTQ65JOony4fuSwFNS8jcCEqufjK9sKo3Jovhy4NiKKdH+hw4BjJc0ERgO7SromIk7Iua56bQDWR8S9lefLgSItBPhr4ImIeB5A0o3A+4GiddZ6JY2tnFf0P4Bn8y6oUY3mZjLDO5JmUD5UPzYiXs27njptP3GtsnJhDuUT1Yrke8AjEXFZ3oU0IiLOj4gJEbEP5b/32wsU+FSGFNZLemdl03SKNSH9FDBN0htVvsDPdIoxEV17VLgCOKny+EQg9Y7P6+ofSm4ms05fUg8wCniusunXEfH5HEuqS+Uv/TJeO/nsazmXVDdJhwF3Ag9RPqwN4PyIuDXXwhok6Qjg7Ig4Nu9aGiHpPZQnoUcCTwDzIqIw93aSdAHlf3A3A78BTq4saEiSpO8DJWAPoBe4APg/wA3AXsA64FMR8UJeNQ5kB/WfT4O5mUzom5lZ6yUzvGNmZq3n0Dcz6yAOfTOzDuLQNzPrIA59M7MO4tA3M+sgDn0zsw7i0Dcz6yD/H8kKVNVXdNeXAAAAAElFTkSuQmCC\n", 945 | "text/plain": [ 946 | "" 947 | ] 948 | }, 949 | "metadata": {}, 950 | "output_type": "display_data" 951 | } 952 | ], 953 | "source": [ 954 | "result = zeros(xs.shape)\n", 955 | "total = 0\n", 956 | "for i,mu in enumerate(mus):\n", 957 | " weight = pmu3[i]\n", 958 | " result += weight * pxt(xs,mu)\n", 959 | " total += weight\n", 960 | "result /= total\n", 961 | "plot(xs,result)\n", 962 | "plot(xs,pxt(xs,7))" 963 | ] 964 | }, 965 | { 966 | "cell_type": "markdown", 967 | "metadata": { 968 | "slideshow": { 969 | "slide_type": "slide" 970 | } 971 | }, 972 | "source": [ 973 | "In fact, if we repeat the same process with a lot of samples (in this case\n", 974 | "the true parameter is 7), we see that the Bayesian parameter estimate\n", 975 | "becomes an increasingly peaked distribution close to the true value.\n", 976 | "\n", 977 | "I.e., if, out of 100 samples, we haven't seen a value greater than 7,\n", 978 | "then the probability that the mean is significantly greater than 7 must\n", 979 | "be very small." 980 | ] 981 | }, 982 | { 983 | "cell_type": "code", 984 | "execution_count": 16, 985 | "metadata": { 986 | "slideshow": { 987 | "slide_type": "slide" 988 | } 989 | }, 990 | "outputs": [ 991 | { 992 | "data": { 993 | "text/plain": [ 994 | "[]" 995 | ] 996 | }, 997 | "execution_count": 16, 998 | "metadata": {}, 999 | "output_type": "execute_result" 1000 | }, 1001 | { 1002 | "data": { 1003 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEACAYAAACj0I2EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAEndJREFUeJzt3XusnHWdx/H3tz2Ua1tUUkS6gizUNSTIEi+sLrsTK9JoBP/Y3SC7q2CWf1aBoAGh/tGDiUYlXshe/jBAwxqbTaxmxY2XgjhsWFddQShCKSQaiyD1slyD1kK/+8eZqQ/Dc3rmds4zv+n7lUw685zn8k3z9NPvfOd55kRmIkkq37KmC5AkjYeBLklTwkCXpClhoEvSlDDQJWlKGOiSNCUWDPSIuCEidkfE9pqffSgi9kXESxenPElSv/rp0DcD5/QujIi1wNnAz8ZdlCRpcAsGembeATxe86PPAleMvSJJ0lCGmqFHxLnAw5l575jrkSQNaWbQDSLicGAjc+OW/YvHVpEkaSgDBzrwx8CJwD0REcBa4M6IeENm/rJ35Yjwy2IkaQiZOVCz3O/IJToPMvPHmfnyzDwpM18F/Bz407owrxRV7GPTpk2N13Cw1l9y7dbf/KP0+ofRz2WLW4DvAusiYldEXNSb1zhykaTGLThyycwLFvj5SeMrR5I0LO8UXUCr1Wq6hJGUXH/JtYP1N630+ocRw85q+j5ARC72MSRp2kQEuUgfikqSJpyBLklTwkCXpClhoEvSlDDQJWlKGOiSNCUMdEmaEga6JE0JA10q0J498OijTVehSWOgSwXauBGOP77pKjRpDHSpQL+c98uqdTAz0KUCLV/edAWaRAa6VKBl/stVDU8LqUAGuup4WkgFMtBVx9NCKpCBrjqeFlKBDHTV8bSQCmSgq46nhVQgA111PC2kAhnoquNpIRXIQFedBU+LiLghInZHxPbKsk9FxI6IuDsivhwRqxa3TElVBrrq9HNabAbO6Vm2DTg1M08HHgKuHndhkubnrf+qs2CgZ+YdwOM9y27NzH2dl98D1i5CbZLmYYeuOuM4Ld4HfGMM+5HUJwNddWZG2TgiPgLszcwtB1pvdnZ2//NWq0Wr1RrlsNJBz0CfPu12m3a7PdI+IjMXXiniBOBrmXlaZdmFwMXAWzJzzwG2zX6OIal/mzbBRz8K/tOaXhFBZsYg2/TboUfn0T3QBuAK4C8OFOaSFocduur0c9niFuC7wLqI2BURFwH/BBwF3BIRd0XEvy5ynZIqDHTVWbBDz8wLahZvXoRaJPXJQFcdTwupQAa66nhaSAUy0FXH00IqkIGuOp4WUoG89V91DHSpQDHQ1ck6WBjoUoEcuaiOp4VUIEcuqmOgSwWyQ1cdTwupQAa66nhaSAUy0FXH00IqkIGuOp4WUoEMdNXxtJAKZKCrjqeFVKDujUX+ggtVGehSwZ5/vukKNEkMdKlA3c58375m69BkMdClgtmhq8pAlwpmh64qA10qUHfkYoeuKgNdKpgduqoMdKlgBrqqDHSpQI5cVMdAlwpmh66qBQM9Im6IiN0Rsb2y7CURsS0idkbEtyJi9eKWKamOHbqq+unQNwPn9Cy7Crg1M18N3AZcPe7CJM3PG4tUZ8FAz8w7gMd7Fp8H3NR5fhPwrjHXJakPduiqGnaGviYzdwNk5mPAmvGVJGkhduiqMzOm/RzwO99mZ2f3P2+1WrRarTEdVjq42aFPj3a7TbvdHmkfkX18/2ZEnAB8LTNP67zeAbQyc3dEvBz4Tma+Zp5ts59jSOrf9dfDxRfDzp2wbl3T1WgxRASZGYNs0+/IJTqPrpuBCzvP3wt8dZCDShqNIxfV6eeyxS3Ad4F1EbErIi4CPgGcHRE7gfWd15KWmCMXVS04Q8/MC+b50VvHXIukAdmhq8o7RaUCeeu/6hjoUsHs0FVloEsFs0NXlYEuFcirXFTHQJcKZoeuKgNdKpAduuoY6FLBDHRVGehSwRy5qMpAlwrkdeiqY6BLBXvuuaYr0CQx0KWC2aGrykCXCtQdudihq8pAlwpmh64qA10qmB26qgx0qUBe5aI6BrpUMDt0VRnoUsHs0FVloEsFcuSiOga6VDBHLqoy0KUC2aGrjoEuFcwOXVUGulQwO3RVGehSgbz1X3VGCvSIuDwifhwR2yPiixGxYlyFSVqYHbqqhg70iHgFcAlwRmaeBswA54+rMEkLs0NX1cyI2y8HjoyIfcARwKOjlyRpIV7lojpDd+iZ+SjwaWAX8AjwRGbeOq7CJC3MDl1VQ3foEXE0cB5wAvAksDUiLsjMLb3rzs7O7n/earVotVrDHlZShR369Gi327Tb7ZH2Edl97zbohhF/BZyTmRd3Xv898MbM/EDPejnsMSTV+9zn4PLL4cor4ZOfbLoaLYaIIDNjkG1GucplF3BmRBwWEQGsB3aMsD9JA3LkoqpRZug/ALYCPwLuAQL4/JjqktQHRy6qGukql8y8BrhmTLVI6lMmLF9uh64X8k5RqVAzM3boeiEDXSqQHbrqGOhSoezQ1ctAlwo1M2OHrhcy0KUCZdqh68UMdKlQBrp6GehSofxQVL0MdKlAjlxUx0CXCuWHouploEuFskNXLwNdKlB35GKHrioDXSqUHbp6GehSobzKRb0MdKlAXuWiOga6VChn6OploEsF6n7boh26qgx0qVCOXNTLQJcK5chFvQx0qUBeh646BrpUqBUrYO/epqvQJDHQpUIdcoiBrhcy0KUCZdqh68UMdKlQdujqNVKgR8TqiPhSROyIiPsi4o3jKkzSgfmhqHrNjLj9dcDXM/OvI2IGOGIMNUlagCMX1Rk60CNiFXBWZl4IkJnPAU+NqS5JC3Dkol6jjFxeBfw6IjZHxF0R8fmIOHxchUk6MANdvUYZucwAZwDvz8wfRsTngKuATb0rzs7O7n/earVotVojHFZS98aiffvmHsu8vKF47Xabdrs90j4iM4fbMOJY4H8y86TO6z8HPpyZ7+xZL4c9hqR6H/sYPPssXHstPP00HHpo0xVp3CKCzIxBthn6//XM3A08HBHrOovWA/cPuz9J/ev2SI5dVDXqVS6XAl+MiEOAnwAXjV6SpH4dcoiXLuoPRgr0zLwHeP2YapE0gIi5Obodurr8KEUqkCMX1THQpYIZ6Koy0KVCRRjoeiEDXSqQIxfVMdClQtmhq5eBLhXMQFeVgS4VqDty8St0VWWgS4Vy5KJeBrpUMANdVQa6VCCvclEdA10qlCMX9TLQpQLZoauOgS4VzEBXlYEuFar7bYtetqguA10qkCMX1THQpYIZ6Koy0KVCRcCKFfD73zddiSaFgS4VqDtyMdBVZaBLhYqAQw+FPXuarkSTwkCXCmagq8pAlwrUHbkY6Koy0KVCOXJRLwNdKpiBrqqRAz0ilkXEXRFx8zgKkrQwRy6qM44O/TLg/jHsR9IAvA5dvUYK9IhYC7wduH485Ujqhx266ozaoX8WuALIMdQiaUAGuqpmht0wIt4B7M7MuyOiBcR8687Ozu5/3mq1aLVawx5WUodXuUyXdrtNu90eaR+ROVxzHREfB/4OeA44HFgJfCUz39OzXg57DEn1rr4aVq6E170Orr0Wbrml6Yo0bhFBZs7bKNcZeuSSmRsz85WZeRJwPnBbb5hLWlx26KryOnSpUI5c1GvoGXpVZt4O3D6OfUlamN+2qDp26FKh7NDVy0CXCmagq8pAlwrkjUWqY6BLhXLkol4GulQwA11VBrpUIEcuqmOgS4Xqjlz27oV9+5quRpPAQJcK1O3QI+Cww+C3v222Hk0GA10q3BFHwLPPNl2FJoGBLhUqOl/bdMQRduiaY6BLBap+genhh9uha46BLhXOkYu6DHSpUI5c1MtAlwrkyEV1DHSpUNUO3UAXGOhS8Ry5qMtAlwrkyEV1DHSpUI5c1MtAlwrnyEVdBrpUIEcuqmOgS4Vy5KJeBrpUoGqHftRR8MwzzdWiyWGgS4VbuRKefrrpKjQJhg70iFgbEbdFxH0RcW9EXDrOwiQdWHfksmoVPPVUs7VoMsyMsO1zwAcz8+6IOAq4MyK2ZeYDY6pN0jyqIxc7dHUN3aFn5mOZeXfn+TPADuD4cRUmqT926Ooayww9Ik4ETge+P479SVpYd+SycqWBrjmjjFwA6IxbtgKXdTr1F5mdnd3/vNVq0Wq1Rj2sdFCrjlxWrXLkMg3a7TbtdnukfURWz4xBN46YAf4T+EZmXjfPOjnKMSS92KWXwsknz/359NNw3HFeujhtIoLMjEG2GXXkciNw/3xhLmnxHXnk3K3/zz/fdCVq2iiXLb4Z+FvgLRHxo4i4KyI2jK80SfOpvuldtmwu1O3QNfQMPTP/G1g+xlokDSAqb8ZXroQnn4TVq5urR83zTlFpCrz0pfD4401XoaYZ6FKBeq8zeNnL4De/aaYWTQ4DXSpUdeRioAsMdKlIduiqY6BLU8BAFxjoUrEcuaiXgS4VqHfkcswxBroMdGkqHHMM/OpXTVehphnoUqGqI5fjjoNf/KK5WjQZDHSpQL0jl+OPh0ceaaYWTQ4DXSpUtUNfswaeeAL27GmuHjXPQJemwLJlcOyx8NhjTVeiJhnoUoHqfsWAYxcZ6FKhoudXH5x4Ivz0p42UoglhoEtT4pRT4MEHm65CTTLQpQLVjVzWrYOHHlr6WjQ5DHSpUL0jl1NOgZ07m6lFk8FAlwpU16Gfeio88ADs3bv09WgyGOjSlFi5Ek44Ae67r+lK1BQDXSpU78gF4PWvh+9/f+lr0WQw0KUC1Y1cANavh23blrYWTQ4DXZoiGzbAt78Nv/td05WoCSMFekRsiIgHIuLBiPjwuIqStLC6kcuaNXDmmbB169LXo+YNHegRsQz4Z+Ac4FTg3RHxJ+MqbFK02+2mSxhJyfWXXDssbv3zjVwALrsMPv7x0a928e+/PKN06G8AHsrMn2XmXuDfgfPGU9bkKP2kKLn+kmuHxa+/rkOHubHLySfDJZfAvn3D79+///KMEujHAw9XXv+8s0xSgyLgC1+A7dvhbW+Db34Tnnmm6aq0FGaaLmAUn/kMfOc7i3uMnTvhzjsX9xiLqeT6S64dFrf+e+6BN71p/p+vXg233w433gjXXDO3/tFHz/0y6VWrYGYGli9/4aO343/wQfjhDxen/qVwoPrne3cziCuvhLPOGn0/4xR5oGHcgTaMOBOYzcwNnddXAZmZn+xZb7gDSNJBLjMH+q9nlEBfDuwE1gO/AH4AvDszdwy1Q0nSSIYeuWTm8xHxAWAbc7P4GwxzSWrO0B26JGmyLMmdohHxqYjYERF3R8SXI2LVUhx3FCXfNBURayPitoi4LyLujYhLm65pGBGxLCLuioibm65lUBGxOiK+1Dnv74uINzZd0yAi4vKI+HFEbI+IL0bEiqZrOpCIuCEidkfE9sqyl0TEtojYGRHfiojVTdZ4IPPUP3BuLtWt/9uAUzPzdOAh4OolOu5QpuCmqeeAD2bmqcCfAe8vrP6uy4D7my5iSNcBX8/M1wCvBYoZR0bEK4BLgDMy8zTmRrPnN1vVgjYz9++16irg1sx8NXAbk507dfUPnJtLEuiZeWtmdm9x+B6wdimOO4Kib5rKzMcy8+7O82eYC5Oi7hGIiLXA24Hrm65lUJ1O6qzM3AyQmc9l5lMNlzWo5cCRETEDHAE82nA9B5SZdwCP9yw+D7ip8/wm4F1LWtQA6uofJjeb+HKu9wHfaOC4g5iam6Yi4kTgdKC0L1X9LHAFUOKHPK8Cfh0Rmzsjo89HxOFNF9WvzHwU+DSwC3gEeCIzb222qqGsyczdMNfkAGsarmcUfeXm2AI9Im7pzNu6j3s7f76zss5HgL2ZuWVcx9X8IuIoYCtwWadTL0JEvAPY3XmXEZ1HSWaAM4B/ycwzgGeZe/tfhIg4mrnu9gTgFcBREXFBs1WNRYnNwUC5ObY7RTPz7AWKupC5t9BvGdcxF9EjwCsrr9d2lhWj81Z5K/CFzPxq0/UM6M3AuRHxduBwYGVE/Ftmvqfhuvr1c+DhzOzep7gVKOmD9bcCP8nM/wOIiK8AbwJKa8R2R8Sxmbk7Il4O/LLpggY1aG4u1VUuG5h7+3xuZu5ZimOO6H+BkyPihM6n++cDpV1pcSNwf2Ze13Qhg8rMjZn5ysw8ibm/+9sKCnM6b/Mfjoh1nUXrKevD3V3AmRFxWEQEc/WX8KFu77u5m4ELO8/fC0x6Y/OC+ofJzSW5Dj0iHgJWAL/pLPpeZv7joh94BJ2/zOv4w01Tn2i4pL5FxJuB/wLuZe5tZgIbM/ObjRY2hIj4S+BDmXlu07UMIiJey9wHuocAPwEuyswnm62qfxGxibn/TPcCPwL+oXOBwESKiC1AC3gZsBvYBPwH8CXgj4CfAX+TmU80VeOBzFP/RgbMTW8skqQp4a+gk6QpYaBL0pQw0CVpShjokjQlDHRJmhIGuiRNCQNdkqaEgS5JU+L/AZJJYMxLBHozAAAAAElFTkSuQmCC\n", 1004 | "text/plain": [ 1005 | "" 1006 | ] 1007 | }, 1008 | "metadata": {}, 1009 | "output_type": "display_data" 1010 | } 1011 | ], 1012 | "source": [ 1013 | "p = pmu.copy()\n", 1014 | "for i in range(100):\n", 1015 | " p = pxt(rand()*7,mus)*p\n", 1016 | " p /= C*sum(p)\n", 1017 | "plot(mus,p)" 1018 | ] 1019 | }, 1020 | { 1021 | "cell_type": "markdown", 1022 | "metadata": { 1023 | "slideshow": { 1024 | "slide_type": "slide" 1025 | } 1026 | }, 1027 | "source": [ 1028 | "Loss Functions for Parameter Estimation\n", 1029 | "========================================\n", 1030 | "\n", 1031 | "Consider $p(\\theta|x)$ from the previous example again." 1032 | ] 1033 | }, 1034 | { 1035 | "cell_type": "code", 1036 | "execution_count": 17, 1037 | "metadata": { 1038 | "slideshow": { 1039 | "slide_type": "slide" 1040 | } 1041 | }, 1042 | "outputs": [ 1043 | { 1044 | "data": { 1045 | "text/plain": [ 1046 | "[]" 1047 | ] 1048 | }, 1049 | "execution_count": 17, 1050 | "metadata": {}, 1051 | "output_type": "execute_result" 1052 | }, 1053 | { 1054 | "data": { 1055 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEACAYAAABI5zaHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAFnJJREFUeJzt3X20VHW9x/H398BlXTXNuhUpiNcn1Kg0S8qbD4fwAalEi7xg6TVFScGjeS3UfDhULrVVJlc0Jb2uvGlgelMsH4irZ6mZhVhYBkK6JEBFzfC5QvzeP/YcHKYzZ2bP2TO/+e39ea111pmH3cyH4+lzfvOdvfeYuyMiIvnSETqAiIhkT+UuIpJDKncRkRxSuYuI5JDKXUQkh1TuIiI5VFe5m9k4M1tmZsvNbEYf9x9gZuvM7OHS1znZRxURkXoNrrWBmXUAs4GxwFPAIjO71d2XVWx6r7sf1oSMIiKSUj0r99HACndf6e7rgbnAhD62s0yTiYhIw+op92HAqrLrq0u3VdrHzH5rZj8zs/dlkk5ERBpScyxTp8XACHd/zcwOBW4BRmb02CIiklI95b4GGFF2fXjpto3c/ZWyy3eY2RVm9k53f6F8OzPTiWxERBrg7qlG3/WMZRYBO5vZ9mY2BJgEzC/fwMyGll0eDVhlsZcFjPbr/PPPD55B+cPnKGL+mLPnIX8jaq7c3X2DmU0HFpD8MbjG3Zea2dTkbp8DTDSzk4D1wOvAvzeURkREMlHXzN3d7wR2rbjtqrLLlwOXZxtNREQapSNUU+js7AwdYUCUP6yY88ecHeLP3whrdJ7T0JOZeSufT0QkD8wMb8IbqiIiEhmVu4hIDqncRURySOUukgN6K0sqqdxFIvfKK7DjjvDqq6GTSDtRuYtE7m9/gyefhHnzQieRdqJyF8mJ730vdAJpJyp3kci5w9Zbw/PPw0MPhU4j7ULlLpIDgwbBiSfClVeGTiLtQuUukhPHHQc33wzr1oVOIu1A5S4SOXcwg6FD4eCD4Yc/DJ1I2oHKXSRHTjopeWNV+72Lyl0kRw44ADZsgPvvD51EQlO5i0SudywDyfcvfQmuuCJsJglP5S6SM8ceC3fdBWvW1NxUckzlLhK5yvn61lvDUUfpoKaiU7mL5IBVfIxDVxfMmQOvvx4mj4SnchfJoZEjYe+94YYbQieRUFTuIpGrttvjaafBrFnaLbKoVO4iOVA5lgE48MBkt8h77ml9HglP5S6SU2bJ7H3WrNBJJASVu0jk+hu7HH00PPAAPP546/JIe1C5i+RAX2MZgM03hylT4NJLW5tHwlO5i+RcVxdcfz0891zoJNJKKneRyNXaG2abbWDiRJg9uzV5pD2o3EVyoNpYptcZZyRHrOpDtItD5S5SACNHwn77wTXXhE4iraJyF4lcvQcpzZgB3/kOrF/f3DzSHlTuIjlQaywDMHo07LgjzJvX/DwSnspdpEC++lX41rd0SoIiULmLRC5NUY8bBx0d8NOfNi+PtAeVu0gO1DOW6d3u3HPh61/X6j3vVO4iBXPEEfDXv8Idd4ROIs2kcheJXPlnqNajoyNZvc+cqdV7ntVV7mY2zsyWmdlyM5vRz3Z7m9l6M/tMdhFFJGsTJ8IrrySftSr5VLPczawDmA0cAowCJpvZblW2uwjQr4tICzWy+tbqPf/qWbmPBla4+0p3Xw/MBSb0sd0pwE3AsxnmE5E6pBnL9Prc52DdOli4MPs8El495T4MWFV2fXXpto3MbFvgcHf/HtDAr5mItNqgQXDOOVq959XgjB7nUqB8Fl+14Lu7uzde7uzspLOzM6MIIsU0kGKeNAm+8Q34+c/h4IOzyyQD09PTQ09Pz4Aew7zGb4aZfQzodvdxpetnAu7uF5dt80TvReBdwKvAie4+v+KxvNbziUg6K1fC/vsn3xtx443JUauLFjU23pHmMzPcPdV/nXrGMouAnc1sezMbAkwCNiltd9+x9LUDydz95MpiF5H2NHEivPkm3Hxz6CSSpZrl7u4bgOnAAuBRYK67LzWzqWZ2Yl//k4wzikg/BvpiuKMDLrwwmb+/8UY2mSS8mmOZTJ9MYxmRzD35JHR2Jt8b5Q5jxiQfqH388RkFk8w0aywjIjlnlqzeZ85MTk0g8VO5i0QuqxfD++wDH/pQ8nF8Ej+Vu0gOZLWXywUXwEUXwUsvZfN4Eo7KXUQ2ev/74dBDkxGNxE3lLhK5rPdRuOACmDNnYG/QSngqd5EcyPLgo2HDoKsLzjwzu8eU1lO5i8g/OOMM+MUv4Je/DJ1EGqVyF4lcMw4d2WKLZDzz5S8nR69KfFTuIjnQjHPCfOELyRGr8+Zl/9jSfCp3EelTRwd897vJ7P3110OnkbRU7iKRS/sZqmnstx+MHg0XX1x7W2kvKncR6dcll8Ds2fDEE7W3lfahcheJXLPPxbfddsneM6ee2tznkWyp3EVyoNkfsnH66bBiBdx2W3OfR7KjcheRmoYMgcsuSw5u0purcVC5i0SuVR+RcNBBsPfeyYnFpP2p3EVyoFWffXrJJXD55cmIRtqbyl1E6jZ8OJx9Nkyd2rpXDNIYlbtI5Fpdsl1d8PLLcO21rX1eSUflLpIDrRrLAAweDFdfnRy5+swzrXteSUflLiKp7bEHTJkCp5wSOolUo3IXiVyo2fe558KSJXDLLWGeX/qnchfJgVaOZXptthl8//swfTq8+GLrn1/6p3IXkYYdcACMHw9f+UroJFJJ5S4SudC7JH7727BgAdx+e9gcsimVu0gOhBjL9Npqq2S3yBNOgBdeCJdDNqVyF5EBGzMGJk6EadNCJ5FeKneRyIUey/S68EJ4+GG48cbQSQRU7iK5EHIs02vzzeG665J9359+OnQaUbmLSGY++tFk9j5lSvu8oigqlbtI5NqtRM87D559NvloPglncOgAIjJw7TCW6TVkCPzoR7DPPskHbO+5Z+hExaSVu4hkbued4dJLYdIkePXV0GmKSeUuEjn39lq59/r855MZfFdX6CTFpHIXkaaZPRvuuw/mzg2dpHjqKnczG2dmy8xsuZnN6OP+w8xsiZn9xsx+bWYfzz6qiPSl3d5QLbfllsn8vasLHn88dJpiqVnuZtYBzAYOAUYBk81st4rNFrr7Hu7+IeB44OrMk4pIVe04lun14Q8ne9B89rPw2muh0xRHPSv30cAKd1/p7uuBucCE8g3cvfw/2duAN7OLKCKxmzYNRo2Ck09u71caeVJPuQ8DVpVdX126bRNmdriZLQVuA47LJp6I1BJDWZrBnDmweHHyXZovszdU3f0Wd98dOBz4ZlaPKyK1tfNYptcWW8DNNyef4LRoUeg0+VfPQUxrgBFl14eXbuuTu99vZjua2Tvd/R9OANrd3b3xcmdnJ52dnXWHFZG4jRwJV12VnEFy8WJ417tCJ2pPPT099PT0DOgxzGu8pjOzQcBjwFjgaeDXwGR3X1q2zU7u/njp8l7Are6+XR+P5bWeT0TSWbIEjj4aHnkkdJL6zZiRlPudd8JgHSdfk5nh7qlen9Ucy7j7BmA6sAB4FJjr7kvNbKqZnVja7LNm9nszexi4DDgyZXYRGYAYxjLlLrggKfXTTw+dJL9qrtwzfTKt3EUyt2QJHHNM8j0m69Yl55857TSYOjV0mvbWyMpdL4hEIhfremnrrWH+fNh332QWP2ZM6ET5otMPiORAbGOZXrvsAjfcAJMnwx//GDpNvqjcRSSosWOTI1gPOwxefDF0mvxQuYtELtaxTLmTT4ZPfCLZRfLvfw+dJh9U7iI5EOtYptysWcnnsOoj+rKhcheRtjBoUHIGyeXLk6NYZWBU7iKRy9Mqd/PN4bbbYN685EhWaZx2hRTJgTyMZXq9+93Jkav77gvDhsGnPhU6UZy0cheRtrPTTnDrrXDccXD//aHTxEnlLhK5PI1lyo0eDddfD5/5DDz8cOg08VG5i+RAnsYy5Q46KJm9f/KTsHRp7e3lLZq5i0Quryv3XkccAS+/DAcfDPfeCzvsEDpRHFTuIjmQ15V7r2OOSQr+wAPhvvtg221DJ2p/KncRicK0afDSS0nB33MPDB0aOlF7U7mLRC7vY5lyZ52VnJ5gzBi4+25473tDJ2pfKneRHMj7WKbc+ecn/97egt9mm9CJ2pPKXUSic955bxX8Pfeo4PuicheJXJHGMuXOPRc6OlTw1ajcRXKgSGOZcl/7WlLw++8PCxfC9tuHTtQ+VO4iErWzzoIttoD99oO77oLddw+dqD2o3EUiV9SxTLmuLnjHO5IP/LjtNvjIR0InCk/lLpIDRR3LlDv6aNhqKxg/Hm68ETo7QycKS+eWEZHcmDAhORf8kUfC/Pmh04SllbtI5DSW2dSYMfCznyUfuP3cc3D88aEThaFyF8kBjWU2tffeyUnGxo2DVaveOvCpSDSWEZFc2mUXeOAB+MlP4LrrQqdpPZW7SOQ0lqlu6NDkVMFr14ZO0noqd5EcKNrIQWpTuYtIrpkV89WNyl0kckUsrjRU7iISLY1lpJLKXSRyRVyVpqGVu4hIDqncRSRaGstUV9SfjcpdJHJFXJWmVcSfkcpdJAeKujqth8Yy/TCzcWa2zMyWm9mMPu4/ysyWlL7uN7MPZB9VRCQ9lXsVZtYBzAYOAUYBk81st4rNngD2d/c9gG8C3886qIj0rYjFJbXVs3IfDaxw95Xuvh6YC0wo38DdH3T3F0tXHwSGZRtTRPqjsUx1WrlXNwxYVXZ9Nf2X9xTgjoGEEhHJSlHLPdPzuZvZGOCLwL5ZPq6IVFfE4pLa6in3NcCIsuvDS7dtwsw+CMwBxrn7X6o9WHd398bLnZ2ddBb9gw5FMqCxTHUxrtx7enro6ekZ0GOY1/hXm9kg4DFgLPA08GtgsrsvLdtmBPB/wNHu/mA/j+W1nk9E0rn3XjjnnOS7/KPu7qTcZ84MnaRxZoa7p/oTXnPl7u4bzGw6sIBkRn+Nuy81s6nJ3T4HOBd4J3CFmRmw3t1Hp/8niEhaWi/1L8aVexbqmrm7+53ArhW3XVV2+QTghGyjiUi9NJbpXxHLXUeoikiuFXXlrnIXiVwRiysNlbuIREtjGamkcheRXNPKXUSiVMTiSkPlLiLR0lhGKqncRSTXtHIXkSgVsbjSULmLSLQ0lqlO5S4iUSpicUltKncRyTWt3EUkWhrLVKdyF5EoFbG4pDaVu0gOaOVenVbuIiI5pHIXkSgVsbjSKOqrGpW7SA4UtcDqVcQ/gCp3Eck1jWVEJEpFLK40VO4iEi2NZaSSyl1Eck0rdxGJUhGLKw2Vu4hES2MZqaRyF5Fc08pdRKJUxOJKQ+UuItHSWKa6ov5sVO4ikntauYtIdIpYXGloLCMi0Srq6KEeKncRiVIRi0tqU7mLSK5p5S4i0dJYpjqVu4hEqYjFJbWp3EUk17RyF5FoaSxTncpdRKJUxOJKQ+XeDzMbZ2bLzGy5mc3o4/5dzewBM/urmZ2efUwR6Y9W7lJpcK0NzKwDmA2MBZ4CFpnZre6+rGyzPwOnAIc3JaWISIO0cq9uNLDC3Ve6+3pgLjChfAN3f97dFwNvNCGjiPSjiMWVhsq9umHAqrLrq0u3iUib0FhGKtUcy2Stu7t74+XOzk46OztbHUFECiTGlXtPTw89PT0Deox6yn0NMKLs+vDSbQ0pL3cRGbjYiqvVYiz3yoXvzJkzUz9GPWOZRcDOZra9mQ0BJgHz+9leLxBFWkxjGalUc+Xu7hvMbDqwgOSPwTXuvtTMpiZ3+xwzGwo8BGwJvGlmpwLvc/dXmhleRKSWGFfuWahr5u7udwK7Vtx2VdnltcB22UYTkXoUsbjSKGq56whVkRzQWKY6lbuIiOSGyl0kckVclaahlbuIREtjmepU7iISpSIWl9SmcheRXNPKXUSipbFMdSp3EYlSEYtLalO5i0iuaeUuItHSWKY6lbuIRKmIxZWGyl1ERHJD5S6SAxrLVKeVu4hEqYjFlYbKXUSipZW7VFK5i0iuaeUuIlEqYnGloXIXkWhpLNM/lbuISM4U9Q+fyl0kckVclaahsYyIRKuoq9N6qNxFRCQ3VO4ikSviqjQNrdxFJFoay1SncheRKBWxuKQ2lbuI5JpW7iISLY1lqlO5i0iUilhcaajcRUQkN1TuIjmgsUx1WrmLSJSKWFxpqNxFRCQ3VO4iOaCxTHVauYtIlIpYXGmo3EVEJDfqKnczG2dmy8xsuZnNqLLNf5nZCjP7rZntmW1MEemPxjLVaeVehZl1ALOBQ4BRwGQz261im0OBndx9F2AqcGUTsgbX09MTOsKAKH9YzcrfiuKK+WdvBi+80BM6RsvVs3IfDaxw95Xuvh6YC0yo2GYCcB2Au/8KeLuZDc00aRuI+RcclD+0ZuZv9so95p+9yr26YcCqsuurS7f1t82aPrYREZEWGRw6QFYWLIDLLmvuczz2GCxe3NznaCblD6tZ+Vevhg98IPvHzYtBg2DtWvj0p5v3HOPHw0knNe/xG2FeY2BnZh8Dut19XOn6mYC7+8Vl21wJ3OPu80rXlwEHuPvaiscq4NsaIiID5+6phm/1rNwXATub2fbA08AkYHLFNvOBacC80h+DdZXF3kg4ERFpTM1yd/cNZjYdWEAyo7/G3Zea2dTkbp/j7reb2Xgz+yPwKvDF5sYWEZH+1BzLiIhIfFp+hKqZfcvMlpYOdrrZzLZqdYa06jmIq12Z2XAzu9vMHjWz35lZV+hMjTCzDjN72Mzmh86Slpm93cx+XPq9f9TMPho6Uxpm9mUz+72ZPWJm15vZkNCZ+mNm15jZWjN7pOy2d5jZAjN7zMzuMrO3h8zYnyr5U/dmiNMPLABGufuewArgrAAZ6lbPQVxt7g3gdHcfBewDTIssf69TgT+EDtGgWcDt7r47sAewNHCeupnZtsApwF7u/kGSUe6ksKlqupbk/6/lzgQWuvuuwN20d+/0lT91b7a83N19obu/Wbr6IDC81RlSqucgrrbl7s+4+29Ll18hKZaojkEws+HAeODq0FnSKq2w9nP3awHc/Q13fylwrLQGAVuY2WBgc+CpwHn65e73A3+puHkC8IPS5R8Ah7c0VAp95W+kN0OfOOw44I7AGWqp5yCuKJjZvwJ7Ar8KmyS17wJfAWJ8g2gH4Hkzu7Y0VppjZpuFDlUvd38K+A7wJ5KDE9e5+8KwqRrynt49+Nz9GeA9gfMMRF292ZRyN7Ofl+ZzvV+/K33/dNk2XwPWu/sNzcggmzKztwE3AaeWVvBRMLNPAmtLrz6s9BWTwcBewOXuvhfwGsmIIApmtjXJqnd7YFvgbWZ2VNhUmYhxoZCqN5tyhKq7H9Tf/WZ2LMnL7E804/kztgYYUXZ9eOm2aJReTt8E/I+73xo6T0ofBw4zs/HAZsCWZnadux8TOFe9VgOr3P2h0vWbgJjelD8QeMLdXwAws/8F/g2IbVG21syGuvtaM3sv8GzoQGml7c0Qe8uMI3mJfZi7/63Vz9+AjQdxlfYSmERy0FZM/hv4g7vPCh0kLXc/291HuPuOJD/7uyMqdkqjgFVmNrJ001jiemP4T8DHzOyfzcxI8sfwhnDlq7z5wLGly/8BtPsiZ5P8jfRmy/dzN7MVwBDgz6WbHnT3k1saIqXSD3YWbx3EdVHgSHUzs48D9wK/I3kp6sDZ7n5n0GANMLMDgP9098NCZ0nDzPYgeTP4n4AngC+6+4thU9XPzM4n+cO6HvgNMKW0c0FbMrMbgE7gX4C1wPnALcCPge2AlcCR7r4uVMb+VMl/Nil7UwcxiYjkUOi9ZUREpAlU7iIiOaRyFxHJIZW7iEgOqdxFRHJI5S4ikkMqdxGRHFK5i4jk0P8DbKFfDbjhJEoAAAAASUVORK5CYII=\n", 1056 | "text/plain": [ 1057 | "" 1058 | ] 1059 | }, 1060 | "metadata": {}, 1061 | "output_type": "display_data" 1062 | } 1063 | ], 1064 | "source": [ 1065 | "plot(mus,pmu2)" 1066 | ] 1067 | }, 1068 | { 1069 | "cell_type": "markdown", 1070 | "metadata": { 1071 | "slideshow": { 1072 | "slide_type": "slide" 1073 | } 1074 | }, 1075 | "source": [ 1076 | "Assume now that we are supposed to return a \"best estimate\" of the parameter.\n", 1077 | "\n", 1078 | "By itself, that isn't sufficient.\n", 1079 | "\n", 1080 | "But now assume that we are given a loss function: if our estimate is with in $\\pm 0.5$ of the true value,\n", 1081 | "we don't need to pay a penalty, otherwise, we need to pay a penalty of 1. What value should we return?\n", 1082 | "\n", 1083 | "The most likely value is 7, but no values less than 7 can occur." 1084 | ] 1085 | }, 1086 | { 1087 | "cell_type": "markdown", 1088 | "metadata": { 1089 | "slideshow": { 1090 | "slide_type": "slide" 1091 | } 1092 | }, 1093 | "source": [ 1094 | "Therefore, it is better to return 7.5. That way, not only do we have the most likely value, but we also\n", 1095 | "get all the probability mass between 7 and 8 as well and our expected loss is about half of what it would be\n", 1096 | "if we had returned 7." 1097 | ] 1098 | }, 1099 | { 1100 | "cell_type": "markdown", 1101 | "metadata": { 1102 | "slideshow": { 1103 | "slide_type": "slide" 1104 | } 1105 | }, 1106 | "source": [ 1107 | "Now assume we are penalized if we are outside the range $\\pm 1$ from the true value.\n", 1108 | "By the same reasoning, our parameter estimate should now be 8." 1109 | ] 1110 | }, 1111 | { 1112 | "cell_type": "markdown", 1113 | "metadata": { 1114 | "slideshow": { 1115 | "slide_type": "slide" 1116 | } 1117 | }, 1118 | "source": [ 1119 | "As you can see from this simple example, there is not \"best\" answer to the parameter estimation\n", 1120 | "problem; our answer depends on the loss function.\n", 1121 | "\n", 1122 | "But we can see that for any symmetric loss function, the value 7 (the maximum likelihood estimate)\n", 1123 | "is never the optimal answer." 1124 | ] 1125 | }, 1126 | { 1127 | "cell_type": "code", 1128 | "execution_count": null, 1129 | "metadata": {}, 1130 | "outputs": [], 1131 | "source": [] 1132 | } 1133 | ], 1134 | "metadata": { 1135 | "celltoolbar": "Slideshow", 1136 | "kernelspec": { 1137 | "display_name": "Python 2", 1138 | "language": "python", 1139 | "name": "python2" 1140 | }, 1141 | "language_info": { 1142 | "codemirror_mode": { 1143 | "name": "ipython", 1144 | "version": 2 1145 | }, 1146 | "file_extension": ".py", 1147 | "mimetype": "text/x-python", 1148 | "name": "python", 1149 | "nbconvert_exporter": "python", 1150 | "pygments_lexer": "ipython2", 1151 | "version": "2.7.15" 1152 | } 1153 | }, 1154 | "nbformat": 4, 1155 | "nbformat_minor": 1 1156 | } 1157 | -------------------------------------------------------------------------------- /PDF/010-understanding-DL-models.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmbdev-tutorials/dl-2018/7b42601e90a9cad1d0fb8d399c55c178561313ab/PDF/010-understanding-DL-models.pdf -------------------------------------------------------------------------------- /PDF/020-dl-pca-ica.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmbdev-tutorials/dl-2018/7b42601e90a9cad1d0fb8d399c55c178561313ab/PDF/020-dl-pca-ica.pdf -------------------------------------------------------------------------------- /PDF/030-2d-layers.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmbdev-tutorials/dl-2018/7b42601e90a9cad1d0fb8d399c55c178561313ab/PDF/030-2d-layers.pdf -------------------------------------------------------------------------------- /PDF/040-bayesian-decision-theory.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmbdev-tutorials/dl-2018/7b42601e90a9cad1d0fb8d399c55c178561313ab/PDF/040-bayesian-decision-theory.pdf -------------------------------------------------------------------------------- /PDF/050-parameter-estimation.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmbdev-tutorials/dl-2018/7b42601e90a9cad1d0fb8d399c55c178561313ab/PDF/050-parameter-estimation.pdf -------------------------------------------------------------------------------- /PDF/100-Large Scale DL.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmbdev-tutorials/dl-2018/7b42601e90a9cad1d0fb8d399c55c178561313ab/PDF/100-Large Scale DL.pdf -------------------------------------------------------------------------------- /chapter.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmbdev-tutorials/dl-2018/7b42601e90a9cad1d0fb8d399c55c178561313ab/chapter.png -------------------------------------------------------------------------------- /flex.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2017 NVIDIA CORPORATION. All rights reserved. 2 | # See the LICENSE file for licensing terms (BSD-style). 3 | 4 | import numpy as np 5 | import torch 6 | from torch import nn 7 | from torch import autograd 8 | from torch.autograd import Variable 9 | from torch.legacy import nn as legnn 10 | import numpy as np 11 | import torch 12 | from torch import nn 13 | from torch import autograd 14 | from torch.autograd import Variable 15 | from torch.legacy import nn as legnn 16 | import layers 17 | 18 | class Flex(nn.Module): 19 | def __init__(self, creator): 20 | super(Flex, self).__init__() 21 | self.creator = creator 22 | self.layer = None 23 | self.dummy = nn.Parameter(torch.zeros(1)) 24 | def forward(self, *args): 25 | if self.layer is None: 26 | self.layer = self.creator(*args) 27 | self.layer.to(self.dummy.device) 28 | return self.layer.forward(*args) 29 | def __repr__(self): 30 | return "Flex:"+repr(self.layer) 31 | def __str__(self): 32 | return "Flex:"+str(self.layer) 33 | 34 | def Linear(*args, **kw): 35 | def creator(x): 36 | assert x.ndimension()==2 37 | return nn.Linear(x.size(1), *args, **kw) 38 | return Flex(creator) 39 | 40 | 41 | def Conv1d(*args, **kw): 42 | def creator(x): 43 | assert x.ndimension()==3 44 | d = x.size(1) 45 | return nn.Conv1d(x.size(1), *args, **kw) 46 | return Flex(creator) 47 | 48 | 49 | def Conv2d(*args, **kw): 50 | def creator(x): 51 | assert x.ndimension()==4 52 | return nn.Conv2d(x.size(1), *args, **kw) 53 | return Flex(creator) 54 | 55 | 56 | def Conv3d(*args, **kw): 57 | def creator(x): 58 | assert x.ndimension()==5 59 | return nn.Conv3d(x.size(1), *args, **kw) 60 | return Flex(creator) 61 | 62 | 63 | def Lstm1(*args, **kw): 64 | def creator(x): 65 | assert x.ndimension()==3 66 | return layers.LSTM1(x.size(1), *args, **kw) 67 | return Flex(creator) 68 | 69 | 70 | def LSTM1to0(*args, **kw): 71 | def creator(x): 72 | assert x.ndimension()==3 73 | return layers.Lstm1to0(x.size(1), *args, **kw) 74 | return Flex(creator) 75 | 76 | 77 | def Lstm2(*args, **kw): 78 | def creator(x): 79 | assert x.ndimension()==4 80 | return layers.LSTM2(x.size(1), *args, **kw) 81 | return Flex(creator) 82 | 83 | 84 | def Lstm2to1(*args, **kw): 85 | def creator(x): 86 | assert x.ndimension()==4 87 | return layers.LSTM2to1(x.size(1), *args, **kw) 88 | return Flex(creator) 89 | 90 | def BatchNorm1d(*args, **kw): 91 | def creator(x): 92 | assert x.ndimension()==3 93 | return nn.BatchNorm1d(x.size(1), *args, **kw) 94 | return Flex(creator) 95 | 96 | def BatchNorm2d(*args, **kw): 97 | def creator(x): 98 | assert x.ndimension()==4 99 | return nn.BatchNorm2d(x.size(1), *args, **kw) 100 | return Flex(creator) 101 | 102 | def BatchNorm3d(*args, **kw): 103 | def creator(x): 104 | assert x.ndimension()==5 105 | return nn.BatchNorm3d(x.size(1), *args, **kw) 106 | return Flex(creator) 107 | 108 | def replace_modules(model, f): 109 | for key in model._modules.keys(): 110 | sub = model._modules[key] 111 | replacement = f(sub) 112 | if replacement is not None: 113 | model._modules[key] = replacement 114 | else: 115 | replace_modules(sub, f) 116 | 117 | def flex_replacer(module): 118 | if isinstance(module, Flex): 119 | return module.layer 120 | else: 121 | return None 122 | 123 | def flex_freeze(model): 124 | replace_modules(model, flex_replacer) 125 | 126 | def delete_modules(model, f): 127 | for key in model._modules.keys(): 128 | if f(model._modules[key]): 129 | del model._modules[key] 130 | 131 | -------------------------------------------------------------------------------- /helpers.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2017 NVIDIA CORPORATION. All rights reserved. 2 | # See the LICENSE file for licensing terms (BSD-style). 3 | 4 | """A set of helper functions for dealing uniformly with tensors and 5 | ndarrays.""" 6 | 7 | import numpy as np 8 | import torch 9 | from torch import autograd, nn, optim 10 | from torch.autograd import Variable 11 | import torch.nn.functional as F 12 | from scipy import ndimage 13 | 14 | torch_tensor_types = tuple([ 15 | torch.Tensor, 16 | torch.FloatTensor, torch.IntTensor, torch.LongTensor, 17 | torch.cuda.FloatTensor, torch.cuda.IntTensor, torch.cuda.LongTensor 18 | ]) 19 | 20 | def asnd(x): 21 | """Convert torch/numpy to numpy.""" 22 | if isinstance(x, np.ndarray): 23 | return x 24 | if isinstance(x, Variable): 25 | x = x.data 26 | if isinstance(x, (torch.cuda.FloatTensor, torch.cuda.DoubleTensor, torch.cuda.IntTensor)): 27 | x = x.cpu() 28 | return x.numpy() 29 | 30 | def as_nda(x, transpose_on_convert=None): 31 | """Turns any tensor into an ndarray.""" 32 | if isinstance(x, np.ndarray): 33 | return x 34 | if isinstance(x, list): 35 | return np.array(x) 36 | if isinstance(x, autograd.Variable): 37 | x = x.data 38 | if isinstance(x, torch_tensor_types): 39 | x = x.cpu().numpy() 40 | return np.ascontiguousarray(maybe_transpose(x, transpose_on_convert)) 41 | raise ValueError("{}: can't convert to np.array".format(type(x))) 42 | 43 | def astorch(x, single=True): 44 | """Convert torch/numpy to torch.""" 45 | if isinstance(x, np.ndarray): 46 | if x.dtype == np.dtype("f"): 47 | return torch.FloatTensor(x) 48 | elif x.dtype == np.dtype("d"): 49 | if single: 50 | return torch.FloatTensor(x) 51 | else: 52 | return torch.DoubleTensor(x) 53 | elif x.dtype == np.dtype("i"): 54 | return torch.IntTensor(x) 55 | else: 56 | error("unknown np.dtype") 57 | return x 58 | 59 | def as_torch(x, transpose_on_convert=None, single=True): 60 | """Converts any kind of tensor/array into a torch tensor.""" 61 | if isinstance(x, Variable): 62 | return x.data 63 | if isinstance(x, torch_tensor_types): 64 | return x 65 | if isinstance(x, list): 66 | x = np.array(x) 67 | if isinstance(x, np.ndarray): 68 | x = maybe_transpose(x, transpose_on_convert) 69 | if x.dtype == np.dtype("f"): 70 | return torch.FloatTensor(x) 71 | elif x.dtype == np.dtype("d"): 72 | if single: 73 | return torch.FloatTensor(x) 74 | else: 75 | return torch.DoubleTensor(x) 76 | elif x.dtype in [np.dtype("i"), np.dtype("int64")]: 77 | return torch.LongTensor(x) 78 | else: 79 | raise ValueError("{} {}: unknown dtype".format(x, x.dtype)) 80 | raise ValueError("{} {}: unknown type".format(x, type(x))) 81 | 82 | def is_tensor(x): 83 | if isinstance(x, Variable): 84 | x = x.data 85 | return isinstance(x, torch_tensor_types) 86 | 87 | def rank(x): 88 | """Return the rank of the ndarray or tensor.""" 89 | if isinstance(x, np.ndarray): 90 | return x.ndim 91 | else: 92 | return x.dim() 93 | 94 | def size(x, i): 95 | """Return the size of dimension i.""" 96 | if isinstance(x, np.ndarray): 97 | return x.shape[i] 98 | else: 99 | return x.size(i) 100 | 101 | def shp(x): 102 | """Returns the shape of a tensor or ndarray as a tuple.""" 103 | if isinstance(x, Variable): 104 | return tuple(x.data.size()) 105 | elif isinstance(x, np.ndarray): 106 | return tuple(x.shape) 107 | elif isinstance(x, torch_tensor_types): 108 | return tuple(x.size()) 109 | else: 110 | raise ValueError("{}: unknown type".format(type(x))) 111 | 112 | def novar(x): 113 | """Turns a variable into a tensor; does nothing for a tensor.""" 114 | if isinstance(x, Variable): 115 | return x.data 116 | return x 117 | 118 | def maybe_transpose(x, axes): 119 | if axes is None: return x 120 | return x.transpose(axes) 121 | 122 | def typeas(x, y): 123 | """Make x the same type as y, for numpy, torch, torch.cuda.""" 124 | assert not isinstance(x, Variable) 125 | if isinstance(y, Variable): 126 | y = y.data 127 | if isinstance(y, np.ndarray): 128 | return asnd(x) 129 | if isinstance(x, np.ndarray): 130 | if isinstance(y, (torch.FloatTensor, torch.cuda.FloatTensor)): 131 | x = torch.FloatTensor(x) 132 | else: 133 | x = torch.DoubleTensor(x) 134 | return x.type_as(y) 135 | 136 | def sequence_is_normalized(x, d, eps=1e-3): 137 | """Check whether a batch of sequences BDL is normalized in d.""" 138 | if isinstance(x, Variable): 139 | x = x.data 140 | assert x.dim() == 3 141 | marginal = x.sum(d) 142 | return (marginal - 1.0).abs().lt(eps).all() 143 | 144 | def bhwd2bdhw(images, depth1=False): 145 | images = as_torch(images) 146 | if depth1: 147 | assert len(shp(images)) == 3, shp(images) 148 | images = images.unsqueeze(3) 149 | assert len(shp(images)) == 4, shp(images) 150 | return images.permute(0, 3, 1, 2) 151 | 152 | def bdhw2bhwd(images, depth1=False): 153 | images = as_torch(images) 154 | assert len(shp(images)) == 4, shp(images) 155 | images = images.permute(0, 2, 3, 1) 156 | if depth1: 157 | assert images.size(3) == 1 158 | images = images.index_select(3, 0) 159 | return images 160 | 161 | def reorder(batch, inp, out): 162 | """Reorder the dimensions of the batch from inp to out order. 163 | 164 | E.g. BHWD -> BDHW. 165 | """ 166 | if inp is None: return batch 167 | if out is None: return batch 168 | assert isinstance(inp, str) 169 | assert isinstance(out, str) 170 | assert len(inp) == len(out), (inp, out) 171 | assert rank(batch) == len(inp), (rank(batch), inp) 172 | result = [inp.find(c) for c in out] 173 | # print ">>>>>>>>>>>>>>>> reorder", result 174 | for x in result: assert x >= 0, result 175 | if is_tensor(batch): 176 | return batch.permute(*result) 177 | elif isinstance(batch, np.ndarray): 178 | return batch.transpose(*result) 179 | 180 | def assign(dest, src, transpose_on_convert=None): 181 | """Resizes the destination and copies the source.""" 182 | src = as_torch(src, transpose_on_convert) 183 | if isinstance(dest, Variable): 184 | dest.data.resize_(*shp(src)).copy_(src) 185 | elif isinstance(dest, torch.Tensor): 186 | dest.resize_(*shp(src)).copy_(src) 187 | else: 188 | raise ValueError("{}: unknown type".format(type(dest))) 189 | 190 | def one_sequence_softmax(x): 191 | """Compute softmax over a sequence; shape is (l, d)""" 192 | y = asnd(x) 193 | assert y.ndim==2, "%s: input should be (length, depth)" % y.shape 194 | l, d = y.shape 195 | y = np.amax(y, axis=1)[:, np.newaxis] -y 196 | y = np.clip(y, -80, 80) 197 | y = np.exp(y) 198 | y = y / np.sum(y, axis=1)[:, np.newaxis] 199 | return typeas(y, x) 200 | 201 | def sequence_softmax(x): 202 | """Compute sotmax over a batch of sequences; shape is (b, l, d).""" 203 | y = asnd(x) 204 | assert y.ndim==3, "%s: input should be (batch, length, depth)" % y.shape 205 | for i in range(len(y)): 206 | y[i] = one_sequence_softmax(y[i]) 207 | return typeas(y, x) 208 | 209 | def ctc_align(prob, target): 210 | """Perform CTC alignment on torch sequence batches (using ocrolstm)""" 211 | import cctc 212 | prob_ = prob.cpu() 213 | target = target.cpu() 214 | b, l, d = prob.size() 215 | bt, lt, dt = target.size() 216 | assert bt==b, (bt, b) 217 | assert dt==d, (dt, d) 218 | assert sequence_is_normalized(prob, 2), prob 219 | assert sequence_is_normalized(target, 2), target 220 | result = torch.rand(1) 221 | cctc.ctc_align_targets_batch(result, prob_, target) 222 | return typeas(result, prob) 223 | 224 | def ctc_loss(logits, target): 225 | """A CTC loss function for BLD sequence training.""" 226 | assert logits.is_contiguous() 227 | assert target.is_contiguous() 228 | probs = sequence_softmax(logits) 229 | aligned = ctc_align(probs, target) 230 | assert aligned.size()==probs.size(), (aligned.size(), probs.size()) 231 | deltas = aligned - probs 232 | logits.backward(deltas.contiguous()) 233 | return deltas, aligned 234 | 235 | class LearningRateSchedule(object): 236 | def __init__(self, schedule): 237 | if ":" in schedule: 238 | self.learning_rates = [[float(y) for y in x.split(",")] for x in schedule.split(":")] 239 | assert self.learning_rates[0][0] == 0 240 | else: 241 | lr0 = float(schedule) 242 | self.learning_rates = [[0, lr0]] 243 | def __call__(self, count): 244 | _, lr = self.learning_rates[0] 245 | for n, l in self.learning_rates: 246 | if count < n: break 247 | lr = l 248 | return lr 249 | 250 | -------------------------------------------------------------------------------- /imgclass.py: -------------------------------------------------------------------------------- 1 | from numpy import * 2 | import numpy as np 3 | from random import randint, uniform 4 | from pylab import randn 5 | import h5py 6 | import torch 7 | from torch import nn 8 | from torch import optim 9 | import numpy as np 10 | import torchvision 11 | from scipy import ndimage as ndi 12 | import flex 13 | import layers 14 | import time 15 | import copy 16 | import pylab 17 | from IPython import display 18 | import matplotlib.pyplot as plt 19 | 20 | training_figsize = (4, 4) 21 | 22 | def one_hot(classes, nclasses=None, value=1.0): 23 | if nclasses is None: nclasses = 1+np.amax(classes) 24 | targets = torch.FloatTensor(len(classes), nclasses) 25 | targets[:, :] = 0 26 | return targets.scatter(1, classes.reshape(-1, 1), value) 27 | 28 | def C(images): 29 | if isinstance(images, np.ndarray): 30 | raise Error("accepts only Torch tensors") 31 | if images.dtype == torch.uint8: 32 | return images.type(torch.float)/255.0 33 | elif images.dtype == torch.float: 34 | return images 35 | else: 36 | raise Error("unknown dtype", images.dtype) 37 | 38 | def evaluate(model, images, classes, bs=200, return_results=False): 39 | results = [] 40 | errs = 0 41 | total = 0 42 | with torch.no_grad(): 43 | for i in range(0, len(images), bs): 44 | outputs = model.forward(C(images[i:i+bs])) 45 | _, indexes = outputs.max(1) 46 | results.append(indexes) 47 | errs += int((indexes!=classes[i:i+bs]).sum()) 48 | total += outputs.size(0) 49 | erate = float(errs) / total 50 | if return_results: return erate, total, results 51 | return erate 52 | 53 | def train(model, images, classes, ntrain=100000, bs=20, lr=0.001, momentum=0.9, decay=0, mode="ce"): 54 | 55 | with torch.no_grad(): 56 | model.forward(C(images[:bs])) 57 | if mode.lower() in ["ce", "crossentropy"]: 58 | criterion = nn.CrossEntropyLoss() 59 | expand = lambda target, output: target 60 | elif model.lower() in ["mse", "meansquared", "meansquarederror"]: 61 | criterion = nn.MSELoss() 62 | expand = lambda target, output: one_hot(target, output.size(1)) 63 | optimizer = optim.SGD(model.parameters(), lr=lr, momentum=momentum, weight_decay=decay) 64 | losses = [] 65 | for i in range(ntrain//bs): 66 | with torch.set_grad_enabled(True): 67 | optimizer.zero_grad() 68 | start = randint(0, len(images)-bs) 69 | inputs = C(images[start:start+bs]) 70 | outputs = model(inputs) 71 | target = expand(classes[start:start+bs], outputs) 72 | loss = criterion(outputs, target) 73 | losses.append(float(loss)) 74 | loss.backward() 75 | optimizer.step() 76 | return mean(losses[-100:]) 77 | 78 | class AutoMLP(object): 79 | def __init__(self, make_model, images, classes, test_images, test_classes, 80 | initial_bs=50, 81 | initial_lrs=10**linspace(-6, 2, 20), 82 | initial_ntrain=100000, 83 | momentum=0.9, 84 | decay=0, 85 | ntrain=50000, 86 | maxtrain=3e6, 87 | selection_noise=0.05, 88 | mintrain=1e6, 89 | stop_no_improvement=2.0 90 | ): 91 | self.make_model = make_model 92 | self.images = images 93 | self.classes = classes 94 | self.test_images = test_images 95 | self.test_classes = test_classes 96 | self.verbose = False 97 | self.initial_bs = initial_bs 98 | self.initial_lrs = initial_lrs 99 | self.initial_ntrain = int(initial_ntrain) 100 | self.ntrain = int(ntrain) 101 | self.momentum = momentum 102 | self.decay = decay 103 | self.maxrounds = int(maxtrain) // int(ntrain) 104 | self.selection_noise = selection_noise 105 | self.mintrain = int(mintrain) 106 | self.stop_no_improvement = stop_no_improvement 107 | self.best_model = None 108 | 109 | def initial_population(self, make_model): 110 | population = [] 111 | for lr in self.initial_lrs: 112 | model = make_model().cuda() 113 | model.PARAMS = dict(bs=self.initial_bs, lr=lr, id=randint(0, 1000000000)) 114 | model.LOG = [] 115 | population.append(model) 116 | return population 117 | 118 | def selection(self, population, size=4, key="training_loss"): 119 | if len(population) == 0: return [] 120 | for model in population: 121 | model.KEY = (1.0+randn()*self.selection_noise) * model.LOG[-1][key] 122 | population = sorted(population, key=lambda m: m.KEY) 123 | while len(population) > size: del population[-1] 124 | return population 125 | 126 | def mutation(self, old_population, variants=1, variation=0.8): 127 | population = [] 128 | for model in old_population: 129 | population += [model] 130 | for _ in range(variants): 131 | cloned = copy.deepcopy(model) 132 | cloned.PARAMS = dict( 133 | lr = clip(cloned.PARAMS["lr"] * uniform(variation, 1.0/variation), 1e-7, 1e2), 134 | bs = clip(int(cloned.PARAMS["bs"] * uniform(variation, 1.0/variation)), 1, 1000), 135 | id = randint(0, 1000000000) 136 | ) 137 | population += [cloned] 138 | return population 139 | 140 | def is_better(self, model, other): 141 | if model is None: return False 142 | if other is None: return True 143 | return model.LOG[-1]["test_loss"] < other.LOG[-1]["test_loss"] 144 | 145 | def train_population(self, population, ntrain=50000, momentum=0.9, verbose=False): 146 | infos = [] 147 | for model in population: 148 | lr, bs = [model.PARAMS[name] for name in "lr bs".split()] 149 | ntrained = 0 if len(model.LOG)==0 else model.LOG[-1]["ntrain"] 150 | training_loss = train(model, self.images, self.classes, lr=lr, bs=bs, 151 | momentum=self.momentum, ntrain=ntrain, decay=self.decay) 152 | test_loss = evaluate(model, self.test_images, self.test_classes) 153 | info = dict( 154 | training_loss=training_loss, 155 | test_loss=test_loss, 156 | lr=lr, 157 | ntrain=ntrain+ntrained, 158 | momentum=momentum, 159 | bs=bs) 160 | if self.verbose: print info 161 | model.LOG += [info] 162 | infos += [info] 163 | if self.is_better(model, self.best_model): 164 | self.best_model = model 165 | return infos 166 | 167 | def to(self, device): 168 | for model in self.population: 169 | model.to(device) 170 | 171 | def cpu(self): 172 | for model in self.population: 173 | model.cpu() 174 | 175 | def train(self): 176 | self.fig = plt.figure(figsize=training_figsize) 177 | self.fig.add_subplot(1,1,1) 178 | self.ax = self.fig.get_axes()[0] 179 | self.infos = [] 180 | self.best_model = None 181 | self.population = self.initial_population(self.make_model) 182 | initial_infos = self.train_population(self.population, ntrain=self.initial_ntrain) 183 | # infos += initial_infos 184 | self.population = self.selection(self.population) 185 | for r in xrange(self.maxrounds): 186 | old_population = [copy.deepcopy(model) for model in self.population] 187 | self.population = self.mutation(self.population) 188 | self.infos += self.train_population(self.population, ntrain=self.ntrain) 189 | self.population = self.selection(self.population + old_population) 190 | # information display 191 | if len(self.infos)>0: self.display() 192 | maxtrained = amax([m.LOG[-1]["ntrain"] for m in self.population]) 193 | l = self.best_model.LOG[-1] 194 | print "#best", l["test_loss"], "@", l["ntrain"], "of", maxtrained 195 | last_best = l["ntrain"] 196 | if maxtrained > self.mintrain and maxtrained > self.stop_no_improvement * maxtrained: 197 | print "# stopping b/c no improvement" 198 | break 199 | self.cpu() 200 | return self.population[0].cpu() 201 | 202 | def display(self, key="training_loss", yscale="log", ylim=None): 203 | self.ax.cla() 204 | self.ax.set_yscale(yscale) 205 | if ylim is not None: self.ax.set_ylim(ylim) 206 | self.ax.scatter(*zip(*[(l["ntrain"], l["training_loss"]) for l in self.infos])) 207 | display.clear_output(wait=True) 208 | display.display(self.fig) 209 | -------------------------------------------------------------------------------- /imgimg.py: -------------------------------------------------------------------------------- 1 | from numpy import * 2 | import numpy as np 3 | from random import randint, uniform 4 | from pylab import randn 5 | import h5py 6 | import torch 7 | from torch import nn 8 | from torch import optim 9 | import torchvision 10 | from scipy import ndimage as ndi 11 | import flex 12 | import layers 13 | import time 14 | import copy 15 | import pylab 16 | import matplotlib.pyplot as plt 17 | from IPython import display 18 | 19 | training_figsize = (4,4) 20 | 21 | 22 | def evaluate(model, images, targets, bs=200): 23 | assert images.dtype == torch.float 24 | assert targets.dtype == torch.float 25 | losses = [] 26 | with torch.no_grad(): 27 | for i in range(0, len(images), bs): 28 | outputs = model.forward(images[i:i+bs].type(torch.float)) 29 | results.append(outputs) 30 | loss = criterion(outputs, targets[i:i+bs]) 31 | losses.append(float(loss)) 32 | return mean(losses) 33 | 34 | def train(model, images, targets, ntrain=100000, bs=20, lr=0.001, momentum=0.9, decay=0.0): 35 | assert images.dtype == torch.float 36 | assert targets.dtype == torch.float 37 | with torch.no_grad(): 38 | model.forward(images[:bs].type(torch.float)) 39 | criterion = nn.MSELoss() 40 | optimizer = optim.SGD(model.parameters(), lr=lr, momentum=momentum, weight_decay=decay) 41 | losses = [] 42 | for i in range(ntrain//bs): 43 | with torch.set_grad_enabled(True): 44 | optimizer.zero_grad() 45 | start = randint(0, len(images)-bs) 46 | inputs = images[start:start+bs].type(torch.float) 47 | outputs = model(inputs) 48 | if isinstance(outputs, tuple): 49 | loss = criterion(outputs[0], targets[start:start+bs].type(torch.float)) + \ 50 | outputs[1] 51 | else: 52 | loss = criterion(outputs, targets[start:start+bs].type(torch.float)) 53 | losses.append(float(loss)) 54 | loss.backward() 55 | optimizer.step() 56 | return losses 57 | 58 | 59 | class AutoMLP(object): 60 | def __init__(self, make_model, images, targets, 61 | initial_bs=50, 62 | initial_lrs=10**linspace(-6, 2, 20), 63 | initial_ntrain=10000, 64 | momentum=0.9, 65 | ntrain=10000, 66 | maxtrain=1e6, 67 | selection_noise=0.05, 68 | mintrain=1e6, 69 | stop_no_improvement=2.0, 70 | decay=1e-6 71 | ): 72 | self.make_model = make_model 73 | self.images = images 74 | self.targets = targets 75 | self.verbose = False 76 | self.initial_bs = initial_bs 77 | self.initial_lrs = initial_lrs 78 | self.initial_ntrain = int(initial_ntrain) 79 | self.ntrain = int(ntrain) 80 | self.momentum = momentum 81 | self.decay = decay 82 | self.maxrounds = int(maxtrain) // int(ntrain) 83 | self.selection_noise = selection_noise 84 | self.mintrain = int(mintrain) 85 | self.stop_no_improvement = stop_no_improvement 86 | self.best_model = None 87 | 88 | def initial_population(self, make_model): 89 | population = [] 90 | for lr in self.initial_lrs: 91 | model = make_model().cuda() 92 | model.PARAMS = dict(bs=self.initial_bs, lr=lr, id=randint(0, 1000000000)) 93 | model.LOG = [] 94 | population.append(model) 95 | return population 96 | 97 | def selection(self, population, size=4, key="training_loss"): 98 | if len(population) == 0: return [] 99 | for model in population: 100 | model.KEY = (1.0+randn()*self.selection_noise) * model.LOG[-1][key] 101 | population = sorted(population, key=lambda m: m.KEY) 102 | while len(population) > size: del population[-1] 103 | return population 104 | 105 | def mutation(self, old_population, variants=1, variation=0.8): 106 | population = [] 107 | for model in old_population: 108 | population += [model] 109 | for _ in range(variants): 110 | cloned = copy.deepcopy(model) 111 | cloned.PARAMS = dict( 112 | lr = clip(cloned.PARAMS["lr"] * uniform(variation, 1.0/variation), 1e-7, 1e2), 113 | bs = clip(int(cloned.PARAMS["bs"] * uniform(variation, 1.0/variation)), 1, 1000), 114 | id = randint(0, 1000000000) 115 | ) 116 | population += [cloned] 117 | return population 118 | 119 | def train_population(self, population, ntrain=50000, momentum=0.9, verbose=False): 120 | infos = [] 121 | for model in population: 122 | lr, bs = [model.PARAMS[name] for name in "lr bs".split()] 123 | ntrained = 0 if len(model.LOG)==0 else model.LOG[-1]["ntrain"] 124 | training_loss = train(model, self.images, self.targets, lr=lr, bs=bs, 125 | momentum=self.momentum, 126 | decay=self.decay, 127 | ntrain=ntrain) 128 | if isinstance(training_loss, list): 129 | training_loss = mean(training_loss[-100:]) 130 | info = dict( 131 | training_loss=training_loss, 132 | lr=lr, 133 | ntrain=ntrain+ntrained, 134 | momentum=momentum, 135 | bs=bs) 136 | if self.verbose: print info 137 | model.LOG += [info] 138 | infos += [info] 139 | return infos 140 | 141 | def to(self, device): 142 | for model in self.population: 143 | model.to(device) 144 | 145 | def cpu(self): 146 | for model in self.population: 147 | model.cpu() 148 | 149 | def train(self): 150 | self.fig = plt.figure(figsize=training_figsize) 151 | self.fig.add_subplot(1,1,1) 152 | self.ax = self.fig.get_axes()[0] 153 | self.infos = [] 154 | self.population = self.initial_population(self.make_model) 155 | initial_infos = self.train_population(self.population, ntrain=self.initial_ntrain) 156 | # infos += initial_infos 157 | self.population = self.selection(self.population) 158 | for r in xrange(self.maxrounds): 159 | old_population = [copy.deepcopy(model) for model in self.population] 160 | self.population = self.mutation(self.population) 161 | self.infos += self.train_population(self.population, ntrain=self.ntrain) 162 | self.population = self.selection(self.population + old_population) 163 | # information display 164 | if len(self.infos)>0: self.display() 165 | self.cpu() 166 | self.fig.clf() 167 | return self.best() 168 | 169 | def display(self, key="training_loss", yscale="log", ylim=None): 170 | self.ax.cla() 171 | self.ax.set_yscale(yscale) 172 | if ylim is not None: self.ax.set_ylim(ylim) 173 | self.ax.scatter(*zip(*[(l["ntrain"], l["training_loss"]) for l in self.infos])) 174 | display.clear_output(wait=True) 175 | display.display(self.fig) 176 | 177 | def best(self): 178 | return self.population[0].cuda() 179 | 180 | def info(self, model=None): 181 | for i, key in enumerate("training_loss lr bs".split()): 182 | pylab.subplot(2, 2, i+1) 183 | pylab.plot(*zip(*[(l["ntrain"], l[key]) for l in model.LOG])) 184 | -------------------------------------------------------------------------------- /init.py: -------------------------------------------------------------------------------- 1 | import scipy 2 | import scipy.ndimage as ndi 3 | 4 | rc("image", cmap="gray", interpolation="bicubic") 5 | figsize(10, 10) 6 | 7 | import dlinputs as dli 8 | import dltrainers as dlt 9 | import dlinputs.filters as dlf 10 | -------------------------------------------------------------------------------- /layers.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2017 NVIDIA CORPORATION. All rights reserved. 2 | # See the LICENSE file for licensing terms (BSD-style). 3 | 4 | import numpy as np 5 | import torch 6 | from torch import nn 7 | from torch import autograd 8 | from torch.legacy import nn as legnn 9 | import numpy as np 10 | import torch 11 | from torch import nn 12 | from torch import autograd 13 | from torch.legacy import nn as legnn 14 | 15 | BD = "BD" 16 | LBD = "LBD" 17 | LDB = "LDB" 18 | BDL = "BDL" 19 | BLD = "BLD" 20 | BWHD = "BWHD" 21 | BDWH = "BDWH" 22 | BWH = "BWH" 23 | 24 | def deprecated(f): 25 | def g(*args, **kw): 26 | raise Exception("deprecated") 27 | return g 28 | 29 | def lbd2bdl(x): 30 | assert len(x.size()) == 3 31 | return x.permute(1, 2, 0).contiguous() 32 | 33 | 34 | def bdl2lbd(x): 35 | assert len(x.size()) == 3 36 | return x.permute(2, 0, 1).contiguous() 37 | 38 | def data(x): 39 | return x 40 | 41 | @deprecated 42 | def typeas(x, y): 43 | """Make x the same type as y, for numpy, torch, torch.cuda.""" 44 | if isinstance(y, np.ndarray): 45 | return asnd(x) 46 | if isinstance(x, np.ndarray): 47 | if isinstance(y, (torch.FloatTensor, torch.cuda.FloatTensor)): 48 | x = torch.FloatTensor(x) 49 | else: 50 | x = torch.DoubleTensor(x) 51 | return x.type_as(y) 52 | 53 | class Fun(nn.Module): 54 | def __init__(self, f, info=None): 55 | nn.Module.__init__(self) 56 | assert isinstance(f, str) 57 | self.f = eval(f) 58 | self.f_str = f 59 | self.info = info 60 | def __getnewargs__(self): 61 | return (self.f_str, self.info) 62 | def forward(self, x): 63 | return self.f(x) 64 | def __repr__(self): 65 | return "Fun {} {}".format(self.info, self.f) 66 | 67 | class PixelsToBatch(nn.Module): 68 | def forward(self, x): 69 | b, d, h, w = x.size() 70 | return x.permute(0, 2, 3, 1).contiguous().view(b*h*w, d) 71 | 72 | class WeightedGrad(autograd.Function): 73 | def forward(self, input, weights): 74 | self.weights = weights 75 | return input 76 | def backward(self, grad_output): 77 | return grad_output * self.weights, None 78 | 79 | def weighted_grad(x, y): 80 | return WeightedGrad()(x, y) 81 | 82 | class Info(nn.Module): 83 | def __init__(self, info="", count=1, mod=1): 84 | nn.Module.__init__(self) 85 | self.mod = mod 86 | self.count = count 87 | self.steps = 0 88 | self.outputs = 0 89 | self.info = info 90 | def forward(self, x): 91 | if self.outputs < self.count: 92 | if self.steps % self.mod == 0: 93 | print "Info", self.info, x.size(), float(x.min()), float(x.max()) 94 | self.outputs += 1 95 | self.steps += 1 96 | return x 97 | def __repr__(self): 98 | return "Info {}".format(self.info) 99 | 100 | class CheckSizes(nn.Module): 101 | def __init__(self, *args, **kw): 102 | nn.Module.__init__(self) 103 | self.order = kw.get("order") 104 | self.name = kw.get("name") 105 | self.limits = [(x, x) if isinstance(x, int) else x for x in args] 106 | def forward(self, x): 107 | for (i, actual), (lo, hi) in zip(enumerate(tuple(x.size())), self.limits): 108 | if actual < lo: 109 | raise Exception("{} ({}): index {} too low ({} not >= {})" 110 | .format(self.name, self.order, 111 | i, actual, lo)) 112 | if actual > hi: 113 | raise Exception("{} ({}): index {} too high ({} not <= {})" 114 | .format(self.name, self.order, 115 | i, actual, hi)) 116 | return x 117 | 118 | def __repr__(self): 119 | return "CheckSizes {}".format(self.limits) 120 | 121 | 122 | class AutoDevice(nn.Module): 123 | def __init__(self): 124 | nn.Module.__init__(self) 125 | self.dummy = nn.Parameter(torch.zeros(1)) 126 | def forward(self, x): 127 | return x.type(type(self.dummy.data)) 128 | def __repr__(self): 129 | return "AutoDevice:{}:{}".format(type(self.dummy.data), self.dummy.device) 130 | 131 | class Cpu(nn.Module): 132 | def __init__(self): 133 | nn.Module.__init__(self) 134 | def forward(self, x): 135 | return x.cpu() 136 | def __repr__(self): 137 | return "Cpu" 138 | 139 | class Check(nn.Module): 140 | def __init__(self, *shape, **kw): 141 | nn.Module.__init__(self) 142 | self.expected = tuple(shape) 143 | self.valid = kw.get("valid", (-1e-5, 1+1e-5)) 144 | def forward(self, x): 145 | expected_shape = self.expected 146 | actual_shape = tuple(x.size()) 147 | assert len(actual_shape)==len(expected_shape) 148 | for i in range(len(actual_shape)): 149 | assert expected_shape[i]<0 or expected_shape[i]==actual_shape[i], \ 150 | (expected_shape, actual_shape, i) 151 | assert data(x).min() >= self.valid[0], (data(x).min(), self.valid) 152 | assert data(x).max() <= self.valid[1], (data(x).max(), self.valid) 153 | return x 154 | 155 | class Reorder(nn.Module): 156 | def __init__(self, old, new): 157 | self.old = old 158 | self.new = new 159 | nn.Module.__init__(self) 160 | self.permutation = tuple([old.find(c) for c in new]) 161 | def forward(self, x): 162 | return x.permute(*self.permutation).contiguous() 163 | def __repr__(self): 164 | return "Reorder {}->{}".format(self.old, self.new) 165 | 166 | class Permute(nn.Module): 167 | def __init__(self, *args): 168 | nn.Module.__init__(self) 169 | self.permutation = args 170 | def forward(self, x): 171 | return x.permute(*self.permutation).contiguous() 172 | def __repr__(self): 173 | return "Permute({})".format(self.permutation) 174 | 175 | class Reshape(nn.Module): 176 | def __init__(self, *args): 177 | nn.Module.__init__(self) 178 | self.shape = args 179 | def forward(self, x): 180 | newshape = [] 181 | for s in self.shape: 182 | if isinstance(s, int): 183 | newshape.append(int(x.size(s))) 184 | elif isinstance(s, (tuple, list)): 185 | total = 1 186 | for j in s: 187 | total *= int(x.size(j)) 188 | newshape.append(total) 189 | else: 190 | raise ValueError("shape spec must be either int or tuple, got {}".format(s)) 191 | return x.view(*newshape) 192 | def __repr__(self): 193 | return "Reshape({})".format(self.shape) 194 | 195 | class Viewer(nn.Module): 196 | def __init__(self, *args): 197 | nn.Module.__init__(self) 198 | self.shape = args 199 | 200 | def forward(self, x): 201 | return x.view(*self.shape) 202 | 203 | def __repr__(self): 204 | return "Viewer %s" % (self.shape,) 205 | 206 | class Norm(nn.Module): 207 | def __init__(self, r=2): 208 | nn.Module.__init__(self) 209 | self.r = r 210 | 211 | def forward(self, x): 212 | assert x.ndimension() == 2 213 | r = self.r 214 | return x / (((x.abs()**r).sum(1))**(1.0/r)).unsqueeze(1) 215 | 216 | def __repr__(self): 217 | return "Norm-{}".format(self.r) 218 | 219 | class Flat(nn.Module): 220 | def __init__(self): 221 | nn.Module.__init__(self) 222 | 223 | def forward(self, x): 224 | rank = len(x.size()) 225 | assert rank > 2 226 | new_depth = np.prod(tuple(x.size())[1:]) 227 | return x.view(-1, new_depth) 228 | 229 | def __repr__(self): 230 | return "Flat" 231 | 232 | class Img2FlatSum(nn.Module): 233 | input_order = BDWH 234 | output_order = BD 235 | 236 | def __init__(self): 237 | nn.Module.__init__(self) 238 | 239 | def forward(self, img): 240 | # BDWH -> BD 241 | return img.sum(3).sum(2) 242 | 243 | def __repr__(self): 244 | return "Img2FlatSum" 245 | 246 | class Img2FlatMax(nn.Module): 247 | input_order = BDWH 248 | output_order = BD 249 | 250 | def __init__(self): 251 | nn.Module.__init__(self) 252 | 253 | def forward(self, img): 254 | # BDWH -> BD 255 | return img.max(3)[0].max(2)[0] 256 | 257 | def __repr__(self): 258 | return "Img2FlatMax" 259 | 260 | class Textline2Img(nn.Module): 261 | input_order = BWH 262 | output_order = BDWH 263 | 264 | def __init__(self): 265 | nn.Module.__init__(self) 266 | 267 | def forward(self, seq): 268 | b, l, d = seq.size() 269 | return seq.view(b, 1, l, d) 270 | 271 | def __repr__(self): 272 | return "Textline2Img" 273 | 274 | 275 | class Img2Seq(nn.Module): 276 | input_order = BDWH 277 | output_order = BDL 278 | 279 | def __init__(self): 280 | nn.Module.__init__(self) 281 | 282 | def forward(self, img): 283 | b, d, w, h = img.size() 284 | perm = img.permute(0, 1, 3, 2).contiguous() 285 | return perm.view(b, d * h, w) 286 | 287 | def __repr__(self): 288 | return "Img2Seq" 289 | 290 | class ImgMaxSeq(nn.Module): 291 | input_order = BDWH 292 | output_order = BDL 293 | 294 | def __init__(self): 295 | nn.Module.__init__(self) 296 | 297 | def forward(self, img): 298 | # BDWH -> BDW -> BWD 299 | return img.max(3)[0].squeeze(3) 300 | 301 | def __repr__(self): 302 | return "ImgMaxSeq" 303 | 304 | class ImgSumSeq(nn.Module): 305 | input_order = BDWH 306 | output_order = BDL 307 | 308 | def __init__(self): 309 | nn.Module.__init__(self) 310 | 311 | def forward(self, img): 312 | # BDWH -> BDW -> BWD 313 | return img.sum(3)[0].squeeze(3).permute(0, 2, 1).contiguous() 314 | 315 | def __repr__(self): 316 | return "ImgSumSeq" 317 | 318 | 319 | class LSTM1(nn.Module): 320 | """A simple bidirectional LSTM. 321 | 322 | All the sequence processing layers use BDL order by default to 323 | be consistent with 1D convolutions. 324 | """ 325 | input_order = BDL 326 | output_order = BDL 327 | 328 | def __init__(self, ninput=None, noutput=None, ndir=2): 329 | nn.Module.__init__(self) 330 | assert ninput is not None 331 | assert noutput is not None 332 | self.ndir = ndir 333 | self.ninput = ninput 334 | self.noutput = noutput 335 | self.lstm = nn.LSTM(ninput, noutput, 1, bidirectional=self.ndir - 1) 336 | self.lstm.flatten_parameters() 337 | 338 | def forward(self, seq): 339 | seq = bdl2lbd(seq) 340 | l, bs, d = seq.size() 341 | assert d == self.ninput, seq.size() 342 | h0 = torch.zeros(self.ndir, bs, self.noutput, dtype=seq.dtype) 343 | c0 = torch.zeros(self.ndir, bs, self.noutput, dtype=seq.dtype) 344 | post_lstm, _ = self.lstm(seq, (h0, c0)) 345 | return lbd2bdl(post_lstm) 346 | 347 | def __repr__(self): 348 | return "LSTM1:"+self.lstm.__repr__() 349 | 350 | 351 | 352 | 353 | class LSTM2to1(nn.Module): 354 | """An LSTM that summarizes one dimension.""" 355 | input_order = BDWH 356 | output_order = BDL 357 | 358 | def __init__(self, ninput=None, noutput=None): 359 | nn.Module.__init__(self) 360 | self.ninput = ninput 361 | self.noutput = noutput 362 | self.lstm = nn.LSTM(ninput, noutput, 1, bidirectional=False) 363 | self.lstm.flatten_parameters() 364 | 365 | def forward(self, img): 366 | # BDWH -> HBWD -> HBsD 367 | b, d, w, h = img.size() 368 | seq = img.permute(3, 0, 2, 1).contiguous().view(h, b * w, d) 369 | bs = b * w 370 | h0 = torch.zeros(1, bs, self.noutput, dtype=img.dtype) 371 | c0 = torch.zeros(1, bs, self.noutput, dtype=img.dtype) 372 | # HBsD -> HBsD 373 | assert seq.size() == (h, b * w, d), (seq.size(), (h, b * w, d)) 374 | post_lstm, _ = self.lstm(seq, (h0, c0)) 375 | assert post_lstm.size() == (h, b * w, self.noutput), (post_lstm.size(), 376 | (h, b * w, self.noutput)) 377 | # HBsD -> BsD -> BWD 378 | final = post_lstm.select(0, h - 1).view(b, w, self.noutput) 379 | assert final.size() == (b, w, self.noutput), (final.size(), (b, w, self.noutput)) 380 | # BWD -> BDW 381 | final = final.permute(0, 2, 1).contiguous() 382 | assert final.size() == (b, self.noutput, w), (final.size(), 383 | (b, self.noutput, self.noutput)) 384 | return final 385 | 386 | 387 | class LSTM1to0(nn.Module): 388 | """An LSTM that summarizes one dimension.""" 389 | input_order = BDL 390 | output_order = BD 391 | 392 | def __init__(self, ninput=None, noutput=None): 393 | nn.Module.__init__(self) 394 | self.ninput = ninput 395 | self.noutput = noutput 396 | self.lstm = nn.LSTM(ninput, noutput, 1, bidirectional=False) 397 | self.lstm.flatten_parameters() 398 | 399 | def forward(self, seq): 400 | seq = bdl2lbd(seq) 401 | l, b, d = seq.size() 402 | assert d == self.ninput, (d, self.ninput) 403 | h0 = torch.zeros(1, b, self.noutput, dtype=seq.dtype) 404 | c0 = torch.zeros(1, b, self.noutput, dtype=seq.dtype) 405 | assert seq.size() == (l, b, d) 406 | post_lstm, _ = self.lstm(seq, (h0, c0)) 407 | assert post_lstm.size() == (l, b, self.noutput) 408 | final = post_lstm.select(0, l - 1).view(b, self.noutput) 409 | return final 410 | 411 | 412 | class RowwiseLSTM(nn.Module): 413 | def __init__(self, ninput=None, noutput=None, ndir=2): 414 | nn.Module.__init__(self) 415 | self.ndir = ndir 416 | self.ninput = ninput 417 | self.noutput = noutput 418 | self.lstm = nn.LSTM(ninput, noutput, 1, bidirectional=self.ndir - 1) 419 | self.lstm.flatten_parameters() 420 | 421 | def forward(self, img): 422 | b, d, h, w = img.size() 423 | # BDHW -> WHBD -> WB'D 424 | seq = img.permute(3, 2, 0, 1).contiguous().view(w, h * b, d) 425 | # WB'D 426 | h0 = torch.zeros(self.ndir, h * b, self.noutput, dtype=img.dtype) 427 | c0 = torch.zeros(self.ndir, h * b, self.noutput, dtype=img.dtype) 428 | seqresult, _ = self.lstm(seq, (h0, c0)) 429 | # WB'D' -> BD'HW 430 | result = seqresult.view( 431 | w, h, b, self.noutput * self.ndir).permute(2, 3, 1, 0) 432 | return result 433 | 434 | 435 | class LSTM2(nn.Module): 436 | """A 2D LSTM module.""" 437 | 438 | def __init__(self, ninput=None, noutput=None, nhidden=None, ndir=2): 439 | nn.Module.__init__(self) 440 | assert ndir in [1, 2] 441 | nhidden = nhidden or noutput 442 | self.hlstm = RowwiseLSTM(ninput, nhidden, ndir=ndir) 443 | self.vlstm = RowwiseLSTM(nhidden * ndir, noutput, ndir=ndir) 444 | 445 | def forward(self, img): 446 | horiz = self.hlstm(img) 447 | horizT = horiz.permute(0, 1, 3, 2).contiguous() 448 | vert = self.vlstm(horizT) 449 | vertT = vert.permute(0, 1, 3, 2).contiguous() 450 | return vertT 451 | 452 | -------------------------------------------------------------------------------- /nb.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmbdev-tutorials/dl-2018/7b42601e90a9cad1d0fb8d399c55c178561313ab/nb.png -------------------------------------------------------------------------------- /note.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmbdev-tutorials/dl-2018/7b42601e90a9cad1d0fb8d399c55c178561313ab/note.png -------------------------------------------------------------------------------- /roadmodel.py: -------------------------------------------------------------------------------- 1 | from numpy import * 2 | from pylab import rand, randn, randint 3 | import scipy 4 | import scipy.ndimage as ndi 5 | 6 | 7 | def spatial_sampler(image): 8 | h, w = image.shape 9 | probs = image.reshape(h*w) 10 | assert amin(probs) >= 0 11 | probs = probs * 1.0 / sum(probs) 12 | probs = add.accumulate(probs) 13 | def f(): 14 | x = rand() 15 | i = searchsorted(probs, x) 16 | return (i//w, i%w) 17 | return f 18 | 19 | class RoadModel(object): 20 | def __init__(self, n=256, background=0.01, sigma=10.0): 21 | self.n = n 22 | self.background = background 23 | self.minsize = 3 24 | self.scale = 0.05 25 | 26 | obstacle_prior = zeros([n, n]) 27 | xs, ys = meshgrid(linspace(-1, 1, n), linspace(-1, 1, n)) 28 | prior = 1.0 * maximum((xs>0.5+0.5*ys), (xs<-0.5-0.5*ys)) 29 | self.road_map = prior 30 | 31 | prior = ndi.gaussian_filter(prior, sigma) 32 | prior -= amin(prior); prior /= amax(prior) 33 | prior = maximum(prior, self.background) 34 | self.prior = prior 35 | 36 | self.xs, self.ys = meshgrid(range(n), range(n)) 37 | self.sampler = spatial_sampler(self.prior) 38 | 39 | def sampled_image(self, k=100000): 40 | n = self.n 41 | result = zeros([n, n]) 42 | for _ in xrange(k): 43 | i, j = self.sampler() 44 | result[i, j] += 1.0 45 | return result 46 | 47 | def sample(self, k): 48 | if isinstance(k, tuple): k = randint(*k) 49 | return [self.sampler() for _ in range(k)] 50 | 51 | def render(self, samples): 52 | n = self.n 53 | target = zeros([n, n], 'f') 54 | for i, j in samples: 55 | target[i, j] = 1.0 56 | d = ndi.distance_transform_cdt(target==0) 57 | target = 1.0*(d < maximum(self.minsize, self.ys*self.scale)) 58 | return target 59 | 60 | def road_truth(self, k=(5, 20)): 61 | return self.render(self.sample(k)) 62 | 63 | def sense(self, target, fg=0.2, bg=0.02): 64 | n = self.n 65 | return 1.0*maximum(rand(n, n) < fg * target, rand(n, n) < bg) 66 | -------------------------------------------------------------------------------- /sicon.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmbdev-tutorials/dl-2018/7b42601e90a9cad1d0fb8d399c55c178561313ab/sicon.png -------------------------------------------------------------------------------- /summary.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmbdev-tutorials/dl-2018/7b42601e90a9cad1d0fb8d399c55c178561313ab/summary.png -------------------------------------------------------------------------------- /texture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmbdev-tutorials/dl-2018/7b42601e90a9cad1d0fb8d399c55c178561313ab/texture.png -------------------------------------------------------------------------------- /texture3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmbdev-tutorials/dl-2018/7b42601e90a9cad1d0fb8d399c55c178561313ab/texture3.png -------------------------------------------------------------------------------- /texture4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmbdev-tutorials/dl-2018/7b42601e90a9cad1d0fb8d399c55c178561313ab/texture4.png -------------------------------------------------------------------------------- /trainers.py: -------------------------------------------------------------------------------- 1 | # copyright (c) 2017 NVIDIA CORPORATION. All rights reserved. 2 | # See the LICENSE file for licensing terms (BSD-style). 3 | 4 | """A set of "trainers", classes that wrap around Torch models 5 | and provide methods for training and evaluation.""" 6 | 7 | import time 8 | import types 9 | import platform 10 | import numpy as np 11 | import torch 12 | from torch import autograd, nn, optim 13 | from torch.autograd import Variable, Function 14 | import torch.nn.functional as F 15 | from scipy import ndimage 16 | import helpers as dlh 17 | 18 | def add_log(log, logname, **kw): 19 | entry = dict(kw, __log__=logname, __at__=time.time(), __node__=platform.node()) 20 | log.append(entry) 21 | 22 | def get_log(log, logname, **kw): 23 | records = [x for x in log if x.get("__log__")==logname] 24 | return records 25 | 26 | def update_display(): 27 | from matplotlib import pyplot 28 | from IPython import display 29 | display.clear_output(wait=True) 30 | display.display(pyplot.gcf()) 31 | 32 | class Weighted(Function): 33 | def forward(self, x, weights): 34 | self.saved_for_backward = [weights] 35 | return x 36 | def backward(self, grad_output): 37 | weights, = self.saved_for_backward 38 | grad_input = weights * grad_output 39 | return grad_input 40 | 41 | 42 | class BasicTrainer(object): 43 | """Trainers take care of bookkeeping for training models. 44 | 45 | The basic method is `train_batch(inputs, targets)`. It catches errors 46 | during forward propagation and reports the model and input shapes 47 | (shape mismatches are the most common source of errors. 48 | 49 | Trainers are just a temporary tool that's wrapped around a model 50 | for training purposes, so you can create, use, and discard them 51 | as convenient. 52 | """ 53 | 54 | def __init__(self, model, use_cuda=True, 55 | fields = ("input", "output"), 56 | input_axes = None, 57 | output_axes = None): 58 | self.use_cuda = use_cuda 59 | self.model = self._cuda(model) 60 | self.init_loss() 61 | self.input_name, self.output_name = fields 62 | self.no_display = False 63 | self.current_lr = None 64 | self.optimizer = None 65 | self.weighted = Weighted() 66 | self.ntrain = 0 67 | self.log = [] 68 | 69 | def _cuda(self, x): 70 | """Convert object to CUDA if use_cuda==True.""" 71 | if self.use_cuda: 72 | return x.cuda() 73 | else: 74 | return x.cpu() 75 | 76 | def set_training(self, mode=True): 77 | """Set training or prediction mode.""" 78 | if mode: 79 | if not self.model.training: 80 | self.model.train() 81 | self.cuinput = autograd.Variable( 82 | torch.randn(1, 1, 100, 100).cuda()) 83 | self.cutarget = autograd.Variable(torch.randn(1, 11).cuda()) 84 | else: 85 | if self.model.training: 86 | self.model.eval() 87 | self.cuinput = autograd.Variable(torch.randn(1, 1, 100, 100).cuda(), 88 | volatile=True) 89 | self.cutarget = autograd.Variable(torch.randn(1, 11).cuda(), 90 | volatile=True) 91 | 92 | def set_lr(self, lr, momentum=0.9, weight_decay=0.0): 93 | """Set the optimizer to SGD with the given parameters.""" 94 | self.current_lr = lr 95 | self.optimizer = optim.SGD(self.model.parameters(), 96 | lr=lr, 97 | momentum=momentum, 98 | weight_decay=weight_decay) 99 | 100 | def get_outputs(self): 101 | """Performs any necessary transformations on the output tensor. 102 | """ 103 | return dlh.novar(self.cuoutput).cpu() 104 | 105 | def set_inputs(self, batch): 106 | """Sets the cuinput variable from the input data. 107 | """ 108 | assert isinstance(batch, torch.Tensor) 109 | dlh.assign(self.cuinput, batch) 110 | 111 | def set_targets(self, targets, weights=None): 112 | """Sets the cutarget variable from the given tensor. 113 | """ 114 | dlh.assign(self.cutarget, targets, False) 115 | assert self.cuoutput.size() == self.cutargets.size() 116 | if weights is not None: 117 | dlh.assign(self.cuweights, weights, False) 118 | assert self.cuoutput.size() == self.cuweights.size() 119 | else: 120 | self.cuweights = None 121 | 122 | def init_loss(self, loss=nn.MSELoss()): 123 | self.criterion = self._cuda(loss) 124 | 125 | def compute_loss(self, targets, weights=None): 126 | self.set_targets(targets, weights=weights) 127 | return self.criterion(self.cuoutput, self.cutarget) 128 | 129 | def forward(self): 130 | try: 131 | self.cuoutput = self.model(self.cuinput) 132 | except RuntimeError, err: 133 | print "runtime error in forward step:" 134 | print "input", self.cuinput.size() 135 | raise err 136 | 137 | def train_batch(self, inputs, targets, weights=None, update=True, logname="train"): 138 | if update: 139 | self.set_training(True) 140 | self.optimizer.zero_grad() 141 | else: 142 | self.set_training(False) 143 | self.set_inputs(inputs) 144 | self.forward() 145 | if weights is not None: 146 | self.cuweights = autograd.Variable(torch.randn(1, 1).cuda()) 147 | dlh.assign(self.cuweights, weights, False) 148 | self.cuoutput = self.weighted(self.cuoutput, self.cuweights) 149 | culoss = self.compute_loss(targets, weights=weights) 150 | if update: 151 | culoss.backward() 152 | self.optimizer.step() 153 | ploss = dlh.novar(culoss)[0] 154 | self.ntrain += dlh.size(inputs, 0) 155 | add_log(self.log, logname, loss=ploss, ntrain=self.ntrain, lr=self.current_lr) 156 | return self.get_outputs(), ploss 157 | 158 | def eval_batch(self, inputs, targets): 159 | return self.train_batch(inputs, targets, update=False, logname="eval") 160 | 161 | def predict_batch(self, inputs): 162 | self.set_training(False) 163 | self.set_inputs(inputs) 164 | self.forward() 165 | return self.get_outputs() 166 | 167 | def loss_curve(self, logname): 168 | records = get_log(self.log, logname) 169 | records = [(x["ntrain"], x["loss"]) for x in records] 170 | records = sorted(records) 171 | if len(records)==0: 172 | return [], [] 173 | else: 174 | return zip(*records) 175 | 176 | def plot_loss(self, every=100, smooth=1e-2, yscale=None): 177 | if self.no_display: return 178 | # we import these locally to avoid dependence on display 179 | # functions for training 180 | import matplotlib as mpl 181 | from matplotlib import pyplot 182 | from scipy.ndimage import filters 183 | x, y = self.loss_curve("train") 184 | pyplot.plot(x, y) 185 | x, y = self.loss_curve("test") 186 | pyplot.plot(x, y) 187 | 188 | def display_loss(self, *args, **kw): 189 | pyplot.clf() 190 | self.plot_loss(*args, **kw) 191 | update_display() 192 | 193 | def set_sample_fields(self, input_name, output_name): 194 | self.input_name = input_name 195 | self.output_name = output_name 196 | 197 | def train_for(self, training, training_size=1e99): 198 | if isinstance(training, types.FunctionType): 199 | training = training() 200 | count = 0 201 | losses = [] 202 | for batch in training: 203 | if count >= training_size: break 204 | input_tensor = batch[self.input_name] 205 | output_tensor = batch[self.output_name] 206 | _, loss = self.train_batch(input_tensor, output_tensor) 207 | count += len(input_tensor) 208 | losses.append(loss) 209 | loss = np.mean(losses) 210 | return loss, count 211 | 212 | def eval_for(self, testset, testset_size=1e99): 213 | if isinstance(testset, types.FunctionType): 214 | testset = testset() 215 | count = 0 216 | losses = [] 217 | for batch in testset: 218 | if count >= testset_size: break 219 | input_tensor = batch[self.input_name] 220 | output_tensor = batch[self.output_name] 221 | _, loss = self.eval_batch(input_tensor, output_tensor) 222 | count += len(input_tensor) 223 | losses.append(loss) 224 | loss = np.mean(losses) 225 | return loss, count 226 | 227 | class ImageClassifierTrainer(BasicTrainer): 228 | def __init__(self, *args, **kw): 229 | BasicTrainer.__init__(self, *args, **kw) 230 | 231 | def set_inputs(self, images, depth1=False): 232 | dlh.assign(self.cuinput, images, transpose_on_convert=(0, 3, 1, 2)) 233 | 234 | def set_targets(self, targets, weights=None): 235 | assert weights is None, "weights not implemented" 236 | if isinstance(targets, list): 237 | targets = np.array(targets) 238 | if dlh.rank(targets) == 1: 239 | targets = dlh.as_torch(targets) 240 | targets = targets.unsqueeze(1) 241 | b, c = dlh.shp(self.cuoutput) 242 | onehot = torch.zeros(b, c) 243 | onehot.scatter_(1, targets, 1) 244 | dlh.assign(self.cutarget, onehot) 245 | else: 246 | assert dlh.shp(targets) == dlh.shp(self.cuoutput) 247 | dlh.assign(self.cutarget, targets) 248 | 249 | 250 | def zoom_like(batch, target_shape, order=0): 251 | assert isinstance(batch, np.ndarray) 252 | scales = [r * 1.0 / b for r, b in zip(target_shape, batch.shape)] 253 | result = np.zeros(target_shape) 254 | ndimage.zoom(batch, scales, order=order, output=result) 255 | return result 256 | 257 | def pixels_to_batch(x): 258 | b, d, h, w = x.size() 259 | return x.permute(0, 2, 3, 1).contiguous().view(b*h*w, d) 260 | 261 | class Image2ImageTrainer(BasicTrainer): 262 | """Train image to image models.""" 263 | def __init__(self, *args, **kw): 264 | BasicTrainer.__init__(self, *args, **kw) 265 | 266 | def compute_loss(self, targets, weights=None): 267 | self.set_targets(targets, weights=weights) 268 | return self.criterion(pixels_to_batch(self.cuoutput), 269 | pixels_to_batch(self.cutarget)) 270 | 271 | def set_inputs(self, images): 272 | dlh.assign(self.cuinput, images, (0, 3, 1, 2)) 273 | 274 | def get_outputs(self): 275 | return dlh.as_nda(self.cuoutput, (0, 2, 3, 1)) 276 | 277 | def set_targets(self, targets, weights=None): 278 | b, d, h, w = tuple(self.cuoutput.size()) 279 | targets = dlh.as_nda(targets, (0, 2, 3, 1)) 280 | targets = zoom_like(targets, (b, h, w, d)) 281 | dlh.assign(self.cutarget, targets, (0, 3, 1, 2)) 282 | assert self.cutarget.size() == self.cuoutput.size() 283 | if weights is not None: 284 | weights = dlh.as_nda(weights, (0, 2, 3, 1)) 285 | weights = zoom_like(weights, (b, h, w, d)) 286 | dlh.assign(self.cuweights, weights, (0, 3, 1, 2)) 287 | 288 | def ctc_align(prob, target): 289 | """Perform CTC alignment on torch sequence batches (using ocrolstm). 290 | 291 | Inputs are in BDL format. 292 | """ 293 | import cctc 294 | assert dlh.sequence_is_normalized(prob), prob 295 | assert dlh.sequence_is_normalized(target), target 296 | # inputs are BDL 297 | prob_ = dlh.novar(prob).permute(0, 2, 1).cpu().contiguous() 298 | target_ = dlh.novar(target).permute(0, 2, 1).cpu().contiguous() 299 | # prob_ and target_ are both BLD now 300 | assert prob_.size(0) == target_.size(0), (prob_.size(), target_.size()) 301 | assert prob_.size(2) == target_.size(2), (prob_.size(), target_.size()) 302 | assert prob_.size(1) >= target_.size(1), (prob_.size(), target_.size()) 303 | result = torch.rand(1) 304 | cctc.ctc_align_targets_batch(result, prob_, target_) 305 | return dlh.typeas(result.permute(0, 2, 1).contiguous(), prob) 306 | 307 | def sequence_softmax(seq): 308 | """Given a BDL sequence, computes the softmax for each time step.""" 309 | b, d, l = seq.size() 310 | batch = seq.permute(0, 2, 1).contiguous().view(b*l, d) 311 | smbatch = F.softmax(batch) 312 | result = smbatch.view(b, l, d).permute(0, 2, 1).contiguous() 313 | return result 314 | 315 | class Image2SeqTrainer(BasicTrainer): 316 | """Train image to sequence models using CTC. 317 | 318 | This takes images in BHWD order, plus output sequences 319 | consisting of lists of integers. 320 | """ 321 | def __init__(self, *args, **kw): 322 | BasicTrainer.__init__(self, *args, **kw) 323 | 324 | def init_loss(self, loss=None): 325 | assert loss is None, "Image2SeqTrainer must be trained with BCELoss (default)" 326 | self.criterion = nn.BCELoss(size_average=False) 327 | 328 | def compute_loss(self, targets, weights=None): 329 | self.cutargets = None # not used 330 | assert weights is None 331 | logits = self.cuoutput 332 | b, d, l = logits.size() 333 | probs = sequence_softmax(logits) 334 | assert dlh.sequence_is_normalized(probs), probs 335 | ttargets = torch.FloatTensor(targets) 336 | target_b, target_d, target_l = ttargets.size() 337 | assert b == target_b, (b, target_b) 338 | assert dlh.sequence_is_normalized(ttargets), ttargets 339 | aligned = ctc_align(probs.cpu(), ttargets.cpu()) 340 | assert dlh.sequence_is_normalized(aligned) 341 | return self.criterion(probs, Variable(self._cuda(aligned))) 342 | 343 | def set_inputs(self, images): 344 | dlh.assign(self.cuinput, images, (0, 3, 1, 2)) 345 | 346 | def set_targets(self, targets, outputs, weights=None): 347 | raise Exception("overridden by compute_loss") 348 | --------------------------------------------------------------------------------