├── README.md
├── ch.3
    ├── .ipynb_checkpoints
    │   ├── Bayesian Spam Detection-checkpoint.ipynb
    │   ├── Multiple Hypothesis Testing via QPSK-checkpoint.ipynb
    │   └── Neyman-Pearson Gaussian detectors-checkpoint.ipynb
    ├── Bayesian Spam Detection.ipynb
    ├── Multiple Hypothesis Testing via QPSK.ipynb
    └── Neyman-Pearson Gaussian detectors.ipynb
├── ch.4
    ├── .ipynb_checkpoints
    │   ├── Bayesian Inference with Exponentials-checkpoint.ipynb
    │   ├── Bias, Variance, and the Bias-variance Trade-off-checkpoint.ipynb
    │   ├── Machine Learning Sampler-checkpoint.ipynb
    │   ├── Noisy Deconvolution-checkpoint.ipynb
    │   └── The Cramer-Rao Bound and Asymptotic Efficiency-checkpoint.ipynb
    ├── Bias, Variance, and the Bias-variance Trade-off.ipynb
    ├── Machine Learning Sampler.ipynb
    ├── Noisy Deconvolution.ipynb
    └── The Cramer-Rao Bound and Asymptotic Efficiency.ipynb
├── ch.5
    ├── .ipynb_checkpoints
    │   ├── Bayesian Deconvolution-checkpoint.ipynb
    │   ├── Bayesian Inference with Scalars-checkpoint.ipynb
    │   ├── Conjugate Priors-checkpoint.ipynb
    │   ├── Ridge Regression and LASSO-checkpoint.ipynb
    │   └── Sparse Coding-checkpoint.ipynb
    ├── Bayesian Deconvolution.ipynb
    ├── Bayesian Inference with Scalars.ipynb
    ├── Conjugate Priors.ipynb
    ├── Ridge Regression and LASSO.ipynb
    └── Sparse Coding.ipynb
├── ch.6
    ├── Kalman Filtering - 2D Kinematics.ipynb
    ├── Kalman Filtering-Brownian Motion.ipynb
    └── empty
├── ch1.intro
    ├── .ipynb_checkpoints
    │   ├── Course Overview-checkpoint.ipynb
    │   ├── Denoising via Gradient Descent-checkpoint.ipynb
    │   ├── Expectation, the Law of Large Numbers, and the Central Limit Theorem-checkpoint.ipynb
    │   └── Multivariate Gaussians-checkpoint.ipynb
    ├── Course Overview.ipynb
    ├── Denoising via Gradient Descent.ipynb
    ├── Expectation, the Law of Large Numbers, and the Central Limit Theorem.ipynb
    └── Multivariate Gaussians.ipynb
└── course-notes.pdf


/README.md:
--------------------------------------------------------------------------------
 1 | # detection-estimation-learninng
 2 | 
 3 | Course material for my graduate class on Detection, Estimation, and Learning. The course covers the fundamentals of detection and estimation theory, including hypothesis testing, maximum-likelihood and Bayes estimation, and tracking of linear systems via the Kalman filter. It also covers applications of these ideas to machine learning: regression, classification, feature extraction, and sparse coding and signal recovery.
 4 | 
 5 | The file course-notes.pdf is a ~120-page monograph of my lecture materials for the class.
 6 | 
 7 | The folders contain Jupyter notebooks with Python code for in-class demonstrations of the course material. They follow the sequence of chapters in the lecture notes, which go as follows:
 8 | 
 9 | Chapter 1: Course introduction: review of important concepts from probability theory, linear algebra, and optimization theory.
10 | 
11 | Chapter 3: Detection theory: Binary and multiple hypothesis testing, ROC curves, minimization of Bayes Risk.
12 | 
13 | Chapter 4: Parameter estimation: Maximum-likelihood estimation, the MVUE and sufficient statistics. Machine learning applications of estmation theory, including linear regression, logistic regression, PCA, and k-means.
14 | 
15 | Chapter 5: Bayesian estimation: MMSE/MAP estimators, conjugate priors, and Gaussian signal processing. Machine learning applications including sparse coding/signal processing, ridge regression, and the LASSO.
16 | 
17 | Chapter 6: Kalman filter: Linear state space models, Kalman filter, extended Kalman filter.
18 | 


--------------------------------------------------------------------------------
/ch.3/.ipynb_checkpoints/Bayesian Spam Detection-checkpoint.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# Bayesian Spam Detection\n",
  8 |     "\n",
  9 |     "Here, we'll take a look at spam detection using the bag-of-words model and the application of Bayesian detection."
 10 |    ]
 11 |   },
 12 |   {
 13 |    "cell_type": "markdown",
 14 |    "metadata": {},
 15 |    "source": [
 16 |     "The two hypotheses are H_0 (normal email) and H_1 (spam/phishing email).\n",
 17 |     "\n",
 18 |     "The first step is to choose a dictionary. I've chosen this fairly arbitrarily from examples I found online. In a real-world system, you would need to estimate the probabilities for each word given the two hypotheses, but here I have made them up entirely."
 19 |    ]
 20 |   },
 21 |   {
 22 |    "cell_type": "code",
 23 |    "execution_count": 1,
 24 |    "metadata": {
 25 |     "collapsed": true
 26 |    },
 27 |    "outputs": [],
 28 |    "source": [
 29 |     "import numpy as np\n",
 30 |     "import matplotlib.pyplot as plt\n",
 31 |     "\n",
 32 |     "dictionary = ['order','report','mail','colleague','language','one','report','send','money','free']\n",
 33 |     "p_0 = [10,5,15,10,5,20,20,10,5,1]\n",
 34 |     "p_0 = p_0/np.sum(p_0)\n",
 35 |     "\n",
 36 |     "p_1 = [7,5,20,15,10,15,13,20,15,10]\n",
 37 |     "p_1 = p_1/np.sum(p_1)\n",
 38 |     "\n",
 39 |     "#Note that p_0 and p_1 are NOT the likelihood functions!"
 40 |    ]
 41 |   },
 42 |   {
 43 |    "cell_type": "markdown",
 44 |    "metadata": {},
 45 |    "source": [
 46 |     "Let's generate a \"message\" according to each hypothesis. Obviously this is not going to generate a realistic-looking message owing to the limited dictionary and the bag-of-words model."
 47 |    ]
 48 |   },
 49 |   {
 50 |    "cell_type": "code",
 51 |    "execution_count": 2,
 52 |    "metadata": {},
 53 |    "outputs": [
 54 |     {
 55 |      "name": "stdout",
 56 |      "output_type": "stream",
 57 |      "text": [
 58 |       "['mail' 'language' 'report' 'money' 'report' 'report' 'report' 'money'\n",
 59 |       " 'report' 'colleague' 'mail' 'report' 'money' 'report' 'report' 'one'\n",
 60 |       " 'send' 'mail' 'report' 'one' 'one' 'mail' 'order' 'colleague' 'language'\n",
 61 |       " 'report' 'colleague' 'mail' 'colleague' 'free' 'colleague' 'mail' 'one'\n",
 62 |       " 'order' 'report' 'one' 'report' 'order' 'colleague' 'order' 'mail' 'one'\n",
 63 |       " 'money' 'mail' 'send' 'report' 'report' 'mail' 'colleague' 'report']\n",
 64 |       "['send' 'send' 'mail' 'money' 'order' 'mail' 'send' 'mail' 'send' 'one'\n",
 65 |       " 'send' 'colleague' 'colleague' 'report' 'one' 'mail' 'send' 'mail' 'money'\n",
 66 |       " 'one' 'colleague' 'free' 'money' 'money' 'send' 'one' 'report' 'order'\n",
 67 |       " 'one' 'mail' 'language' 'order' 'report' 'language' 'free' 'mail' 'money'\n",
 68 |       " 'one' 'send' 'one' 'colleague' 'one' 'money' 'order' 'one' 'send' 'report'\n",
 69 |       " 'language' 'order' 'language']\n"
 70 |      ]
 71 |     }
 72 |    ],
 73 |    "source": [
 74 |     "N = 50\n",
 75 |     "\n",
 76 |     "normal_message = np.random.choice(dictionary,size=N,p=p_0)\n",
 77 |     "spam_message = np.random.choice(dictionary,size=N,p=p_1)\n",
 78 |     "\n",
 79 |     "print(normal_message)\n",
 80 |     "print(spam_message)"
 81 |    ]
 82 |   },
 83 |   {
 84 |    "cell_type": "markdown",
 85 |    "metadata": {},
 86 |    "source": [
 87 |     "Now, let's build a Bayes classifier. To do so, we need to choose priors and cost functions, then we compute the LLR."
 88 |    ]
 89 |   },
 90 |   {
 91 |    "cell_type": "code",
 92 |    "execution_count": 11,
 93 |    "metadata": {
 94 |     "collapsed": true
 95 |    },
 96 |    "outputs": [],
 97 |    "source": [
 98 |     "pi_0 = 0.9\n",
 99 |     "pi_1 = 1-pi_0\n",
100 |     "\n",
101 |     "C_00 = 0\n",
102 |     "C_11 = 0\n",
103 |     "C_10 = 1\n",
104 |     "C_01 = 1"
105 |    ]
106 |   },
107 |   {
108 |    "cell_type": "markdown",
109 |    "metadata": {},
110 |    "source": [
111 |     "Let's sample directly from the spam bag of words model using np.multinomial."
112 |    ]
113 |   },
114 |   {
115 |    "cell_type": "code",
116 |    "execution_count": 12,
117 |    "metadata": {},
118 |    "outputs": [
119 |     {
120 |      "name": "stdout",
121 |      "output_type": "stream",
122 |      "text": [
123 |       "[0 0 1 2 2 1 1 1 1 1]\n"
124 |      ]
125 |     }
126 |    ],
127 |    "source": [
128 |     "y = np.random.multinomial(N,p_1)\n",
129 |     "print(y)"
130 |    ]
131 |   },
132 |   {
133 |    "cell_type": "markdown",
134 |    "metadata": {},
135 |    "source": [
136 |     "Now, we compute the LLR and classify."
137 |    ]
138 |   },
139 |   {
140 |    "cell_type": "code",
141 |    "execution_count": 13,
142 |    "metadata": {},
143 |    "outputs": [
144 |     {
145 |      "name": "stdout",
146 |      "output_type": "stream",
147 |      "text": [
148 |       "3.33664688732\n",
149 |       "2.8903717579\n"
150 |      ]
151 |     }
152 |    ],
153 |    "source": [
154 |     "L = 0\n",
155 |     "for i in range(0,len(dictionary)):\n",
156 |     "    L += y[i]*np.log(p_1[i]/p_0[i])\n",
157 |     "\n",
158 |     "threshold = np.log(((C_10-C_00)*pi_0)/((C_01-C_11)*pi_1))\n",
159 |     "print(L)\n",
160 |     "print(threshold)"
161 |    ]
162 |   },
163 |   {
164 |    "cell_type": "markdown",
165 |    "metadata": {},
166 |    "source": [
167 |     "Did we correctly detect the spam? Let's get a better sense of the detector performance by looking at the Bayes risk, averaged over many samples and for many priors."
168 |    ]
169 |   },
170 |   {
171 |    "cell_type": "code",
172 |    "execution_count": 17,
173 |    "metadata": {},
174 |    "outputs": [
175 |     {
176 |      "name": "stderr",
177 |      "output_type": "stream",
178 |      "text": [
179 |       "C:\\Users\\fy4311\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\ipykernel_launcher.py:10: RuntimeWarning: divide by zero encountered in log\n",
180 |       "  # Remove the CWD from sys.path while we load stuff.\n",
181 |       "C:\\Users\\fy4311\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\ipykernel_launcher.py:10: RuntimeWarning: divide by zero encountered in double_scalars\n",
182 |       "  # Remove the CWD from sys.path while we load stuff.\n"
183 |      ]
184 |     },
185 |     {
186 |      "data": {
187 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xd8leX9//HXJ5sRVsLekAAiQzAy\nlKVYwFalrtaqlbZaW5RKrdra77fra23rz7qrdVtHtRZHldaBm4QpQZkiWYwERE4CYYWQdf3+OAeb\nUiAHyDn3Ge/n45EHZ9z3yZv7Afnkuu/rvj7mnENERCTSJHgdQERE5HBUoEREJCKpQImISERSgRIR\nkYikAiUiIhFJBUpERCKSCpSIiEQkFSgREYlIKlAiIhKRkrwO0FwyMzNdnz59vI4hIiJNWL58eblz\nrmNT28VMgerTpw/5+flexxARkSaY2aZgttMpPhERiUgqUCIiEpFUoEREJCKpQImISERSgRIRkYik\nAiUiIhFJBUpERCKSCpRIM6qqqWNhUTkby/d5HUUk6sXMjboiXqiurefjTTtZXFLB4uIKVpRWUtfg\nSDC4cGQPZk/OpmeHll7HFIlKKlAix+BAXT0rS3exqLicxcUVfFJaSU1dAwkGQ3u04+rx/RjdtwML\ni8p5ZskmXluxhW+N6sWsM7Po1CbN6/giUcWcc15naBY5OTlOSx1Jc6utb2BV2S6WBEZI+Zt2UF3b\ngBmc3K0NY/tlMLZ/Bqf16UB6WvJ/7Pv5rv386f0i5iwrJSnRmHF6H344oT/tW6V49LcRiQxmttw5\nl9PkdipQIv9W3+BYs2XXl6fs8jfuYF9NPQCDuqQzJlCQxvTNoG3L5CY+zW9TxT7ue7eQf6zYQuuU\nJK4e34/vjevzXwVNJF6oQIkEoaHBsW7bbhYXV7CkpIKlG3awp7oOgKxOrb8cIY3u24GM1qkn9L0K\nvtjD3W8X8NbabbRvmczMSf25cmwf0pITm+OvIhI1VKBEDsM5R8EXe1lcXM7iQEGqrKoFoE9GS//o\nqF8GY/tlhOya0aqySu58u4DcAh+d0lP50eRsvpnTk5QkTaqV+KACJYK/IJWU72NxcQWLSypYUlxB\nxb4aALq3a8Hp/TO+LErd2rUIa7alJRXc+fZ6lm3cSc8OLZg9eQAXjOhOYoKFNYdIuKlASVxyzrF5\nR9WXBWlxcQXb9xwAoEubNMb2z/jytF0kTP92zjG/wMedb69nzZbdZHVqzU++MoBpJ3chQYVKYlSw\nBUrTzCXqbanc7y9IgetIWyr3A5DZOoUx/TI4vX8mY/tn0CejJWaR9UPfzJg0sBMTB3TkrTXbuOud\nAq597mOGdG/DjVMGMmlAx4jLLBIuGkFJ1CrbWcUvXl3Dh+t9ALRvmfzlLLux/TLI6tQ66n641zc4\nXluxhXveLaB0x35yerfn5qkDGd0vw+toIs1Gp/gkZtU3OJ5etJE7314PwLWT+jP5pM4M7JweM6fF\nauoamJNfyp/eL+SL3QcYn53JTVMGMrxnO6+jiZwwFSiJSeu37eFnL69iRWklkwZ25LavD6FHe++v\nJYVKdW09f12yiQc/KGJnVS1TT+7MjVMGMqBzutfRRI6bCpTElAN19Tz4fhF//rCYNi2S+fV5gzl/\neLeoO4V3vPYeqOPJBRt4LLeEvTV1TB/ejR+fPYA+ma28jiZyzFSgJGbkb9zBz15eRbFvHxeM6M4v\nzx1MhzhdLqiyqoaH55fw1KIN1NU7LsnpyfWTs+jaNrxT5EVOhAqURL091bXc8dZ6nl2yie7tWvC7\nC4YwaWAnr2NFhO27q3nwgyKe/2gzZsa3x/Rm5qT+ZJ7gahci4aACJVHtvXVf8ItX17BtdzXfPb0v\nN04ZQKtU3RVxqLKdVdz/XiEvLS8jLTmR753Rl+9P6EfbFlrnTyKXCpREJd+eA/zfP9fyr1WfM7Bz\nOrdfNJQRvdp7HSviFfv2cs87Bfxr1ee0SUviBxP7890z+tAyRUVdIo8KlEQV5xwvLS/jttfXsb+m\nnllnZfHDif21Pt0xWrt1F3e/XcB7n20ns3UK152ZxWWje5GapAVpJXKoQEnU2FxRxf/8YzULisrJ\n6d2e2y8aSlYnTaM+Ecs37eSut9ezqLiCbm3TuH5yNhef2oOkRBV88Z4KlES8uvoGnlq0kbveLiAx\nwfjZOYO4fFSvmLnZNhIsLCrnj/PWs6K0klN6tuOubwynf8fWXseSOKcCJRHt0627ueWVVawq28Xk\nQZ347deHhH018XjhnOOfqz7nV6+tYX9NPT+dNojvnt5HvwiIZ7RYrESk6tp6/vR+IY/ML6Fdy2Qe\nuGwEXxvaNW5uuPWCmXH+8G6M6deBn7+8mt/+61Pmrd3GnRcPp1dG7K7CIdFPIygJmyUlFfzPK6sp\nKd/Hxaf24BdfO4l2LePzhluvHJyMcus/P6XeOf73aydx2ahe+gVBwkojKIkYu6tr+cMbn/G3jzbT\ns0ML/nrVaMZlZ3odKy6ZGZfk9OT0rEx+9tIq/vcfa3hrzTbuuHiYVqOQiKMRlITUvLXb+NVra/Dt\nOcBV4/pyw1cG6N6cCOGc469LN/P719eRlGj85ryTuXBkd42mJOQ0ghJPbd9TzW/mruWN1dsY1CWd\nx67MYVgPtYqIJAeXSJqQncnNL67ixhdX8tbabfz+gqF0TNeSSeI9jaCkWTnnmJNfyu9eX0d1XQOz\nJ2dzzYR+JOv+m4hW3+D4y8IN3DFvPa1SErnt60P52rCuXseSGKURlITdxvJ9/PyV1SwuqWBU3w7c\nfuFQ+umem6iQmGBcPb4fkwZ25MY5K7nu+Y95c01Xfjt9CO3jdOV48Z4KlJywuvoGHl+wgXveKSAl\nMYHfXzCUS0/rqftsolBWp3Rennk6D88v5r73Clm6YQd/uGAoZw/u7HU0iUMhPe9iZtPMbL2ZFZnZ\nLYd5/ydm9qmZrTKz98ysd6P3ZphZYeBrRihzyvFbs2UX0x9cyO1vfsbEAR1598aJXDZaq0FEs6TE\nBGadlc1r140jo1UKVz+Tz00vrmR3da3X0STOhOwalJklAgXAV4AyYBnwLefcp422ORNY6pyrMrOZ\nwCTn3DfNrAOQD+QADlgOnOqc23mk76drUOFVXVvPPe8W8HjeBjq0SuG3009m2hBds4g1NXUN3P9e\nIQ/NL6Zzeip3XDxctwjICQv2GlQoR1CjgCLnXIlzrgZ4AZjeeAPn3AfOuarA0yVAj8DjqcA7zrkd\ngaL0DjAthFnlGCwqLmfqvbk8Mr+ES07twbs3TFRxilEpSQncNHUgL888nRYpiVzxxFJ+8epq9h2o\n8zqaxIFQFqjuQGmj52WB147kKuDNY9nXzK4xs3wzy/f5fCcYV5pSVVPHz15axWWPLQXg+atHc/tF\nw2jbUs3xYt0pPdvx+vXjuXpcX55buplz7svjow07vI4lMS6UBepwFyEOez7RzK7Afzrvj8eyr3Pu\nUedcjnMup2PHjscdVJrW0OD48QsreHF5KT+Y2I95P57A6Vk61RNP0pIT+cW5g/n7NWMB+Oaji7nt\nX59SXVvvcTKJVaEsUGVAz0bPewBbD93IzM4G/hc43zl34Fj2lfC5Y9563v70C3557mB+fs5JpCWr\nAV68GtW3A2/OHs/lo3vx+IINfO3+PFaUVnodS2JQKAvUMiDbzPqaWQpwKTC38QZmNgJ4BH9x2t7o\nrXnAFDNrb2btgSmB18QDL+aX8vD8Yi4f3YvvnN7H6zgSAVqlJnHb14fy7FWjqKqp56KHFnHnvPXU\n1DV4HU1iSMgKlHOuDpiFv7CsA+Y459aa2a1mdn5gsz8CrYEXzWyFmc0N7LsD+C3+IrcMuDXwmoTZ\n0pIK/ucfqxmXlclvzj9Z67TJfxif3ZF5N0zgwhHdeeCDIqY/uJBPt+72OpbECC11JEe0qWIf0x9c\nSEarFF659gzattBkCDmydz/9glteWc2u/TXMnpzNDyf2V4t5OaxImGYuUWzX/lq+99QyAJ6YcZqK\nkzTp7MGdeeeGCUwb0pU73y7goocWUbR9j9exJIqpQMl/qa1vYNbzH7N5RxUPX3EqfTJbeR1JokT7\nVin86VsjeOCyEWzeUcVX71/A43kl1DfExpkaCS8VKPkPzjn+759rySss53cXDGVMvwyvI0kUOndY\nN+bdMIEJ2R257fV1XProYjZV7PM6lkQZFSj5D08v2shfl2zmBxP78Y2cnk3vIHIEndLTeOzKU7nr\nkuF8tm0P0+7N49klm4iV694SeipQ8qUP1m/n1n99ypTBnfnZ1EFex5EYYGZcdGoP3r5hAjl92vPL\nV9dw5ZMfsbVyv9fRJAqoQAkA67ft4UfPf8JJXdtw76WnaDVyaVZd27bgme+N4ncXDGH5pp1MvSeX\nOfmlGk3JUalACeV7D/C9p5bRMiWRx2fk0DJFbcKk+ZkZl4/uzVuzJ3BStzb8NLCuY7Fvr9fRJEKp\nQMW56tp6fvDscir2HeDxGTl0bdvC60gS43pltOSF74/hdxcMYe3WXZxzbx53v1OgNf3kv6hAxTHn\nHLe8vIrlm3Zy9zdOYViPdl5HkjiRkOAfTb134yS+OrQL979XyNR7c8ktUFcC+TcVqDj2wPtFvLpi\nKzdNGcBXh6qfk4Rfx/RU7r10BM9dPZoEM6588iN+9LdP2L672utoEgFUoOLU66s+5653CrhgRHeu\nOzPL6zgS587IyuTN2eO54ewBzFu7jcl3zeeZxRt1g2+cU4GKQytLK/nJnBXk9G7P7RcN1QKwEhHS\nkhOZfXY28348geE92/Gr19Zy4Z8XsmbLLq+jiUdUoOLM1sr9XP1MPh3TU3nk26eSmqS+ThJZ+ma2\n4tmrRnHfpaewpbKa8x9YwP/9cy17qmu9jiZhpgIVR/YdqOPqp/PZX1PPk985jYzWqV5HEjksM2P6\nKd1578aJXD66N08t2sjZd8/njdWf696pOKICFSfqGxyzX1jBZ9t288BlIxjQOd3rSCJNatsimd9+\nfQj/uPYMMlqlcu1zH/Pdp5axuaLK62gSBipQceKOtz7j3XVf8KtzBzNpYCev44gck1N6tmPurDP4\n5bmDWbZhB1+5Zz4PflCkDr4xTgUqDsxZVsojuSV8e0xvZqhlu0SppMQErhrXl3dvnMhZgzrxx3nr\n+er9eSwpqfA6moSIClSMW1zsb9k+PjuTX583WDP2JOp1bduCh644lSe/k0N1bT2XPrqEG+espGLv\nAa+jSTNTgYphG8v3MfO55fTOaMkDl41U+22JKWcN6sw7N0zk2kn9eW3FFibfPZ+/L9tMg+6dihn6\niRWjdlXV8r2nl2HAk99Ry3aJTS1SEvnptEG8MXs8Azql87OXV/ONRxazfptazccCFagYVFvfwLXP\nL6d0RxWPfDuH3hlq2S6xbUDndP7+gzHccfEwin17+dr9efzhzXVU1dR5HU1OgApUjHHO8eu5a1lY\nVMEfLhzGqL4dvI4kEhZmxjdyevLejZO4cGR3HplfwlfuzuXdT7/wOpocJxWoGPOXhRt5fulmZk7q\nz8Wn9vA6jkjYdWiVwh0XD2fOD8bSMiWRq5/J5wfP5quLbxRSgYoh73/2Bbe9/ilTT+7MzVMGeh1H\nxFOj+nbg9evH87Npg5hf4OPsu+fzeF4JdfW6dypaqEDFiM+27eZHz3/C4G5tuOebatkuApCSlMDM\nSf1554aJjOmXwW2vr+O8Bxby8eadXkeTIKhAxQDfngNc9VQ+rdOSePzK09SyXeQQPTu05IkZOTx8\nxUh27qvhoocW8T//WM2uKi1AG8lUoKJcdW091zyb72/ZfuVpdGmb5nUkkYhkZkwb0pV3b5zI987o\nywsfbWby3R/yj0/KtABthFKBimLOOX760io+2VzJvd88haE92nodSSTitU5N4pfnDmburHF0b9+S\nG/6+kssfX0qxb6/X0eQQKlBR7E/vFzF35VZunjqQaUPUsl3kWAzp3pZXZp7ObV8fwuotuzjn3jwe\nzyvxOpY0ogIVpf65cit3v1PAhSO7c+2k/l7HEYlKiQnGFWN6896NExmXncnv31hH4RdahSJSqEBF\noU827+SmF1dyWp/2/OFCtWwXOVGd0tP448XDaJmSxF1vF3gdRwJUoKLMlsr9fP+Z5XRqk8rDV6hl\nu0hzyWidytXj+/LW2m2sKK30Oo6gAhVV9h6o46qnlnGgtp4nZ6hlu0hzu3p8Pzq0SuGP8z7zOoqg\nAhU16hscP37hEwq+2MMDl48kWy3bRZpd69Qkrjszi4VFFSwoLPc6TtxTgYoSt7+5jnfXbec355/M\nxAEdvY4jErMuH92L7u1a8Md5n+n+KI+pQEWBFz7azGN5G5gxtjdXju3jdRyRmJaWnMjss7NZWbaL\neWu3eR0nroW0QJnZNDNbb2ZFZnbLYd6fYGYfm1mdmV18yHv1ZrYi8DU3lDkj2aLicn7x6homDOjI\nL88d7HUckbhw4YjuZHVqzR/nrdfish465gJlZgPN7LEgtksEHgTOAQYD3zKzQ3/Cbga+Azx/mI/Y\n75w7JfB1/rHmjAUlvr3M/OvH9M1sxQOXjVDLdpEwSUpM4KYpAyj27eOVT7Z4HSduHfEnnpkNM7O3\nzWyNmd1mZp3N7GXgPeDTID57FFDknCtxztUALwDTG2/gnNvonFsF6FeUQ9TVN3D1M/kkJhhPzDiN\nNmlq2S4STlNP7sLwHm25950CqmvrvY4Tl472K/lj+Ec2FwE+4GOgBMhyzt0TxGd3B0obPS8LvBas\nNDPLN7MlZvb1w21gZtcEtsn3+XzH8NGRb2VZJSW+ffzq3MH0ymjpdRyRuGNm/HTaILbuqua5pZu9\njhOXjlagUp1zTznn1jvn7sM/yrnFOVcd5GcfbnmDY5kS08s5lwNcBtxrZv+1no9z7lHnXI5zLqdj\nx9ia2Ta/oJwEgzMHdvI6ikjcOiMrk3FZmTz4QRF7D9R5HSfuHK1ApZnZCDMbaWYjgb3AsEbPm1IG\n9Gz0vAewNdhgzrmtgT9LgA+BEcHuGwvyCn0M79mOti11ak/ESzdPHciOfTVaSNYDR+ts9zlwd6Pn\n2xo9d8BZTXz2MiDbzPoCW4BL8Y+GmmRm7YEq59wBM8sEzgDuCGbfWLCrqpaVpZXMOivb6ygicW94\nz3acM6QLj+WW8O0xvbWCSxgdsUA55848kQ92ztWZ2SxgHpAIPOmcW2tmtwL5zrm5ZnYa8A+gPXCe\nmf2fc+5k4CTgETNrwD/Ku905F8zEjJiwsLicBgcTsjO9jiIiwI1TBjBv7Tb+/GGxbvcIo5D2BnfO\nvQG8cchrv2r0eBn+U3+H7rcIGBrKbJEst8BHemoSp/Rs53UUEQGyOqVz8ak9eHbxJr43ri/d27Xw\nOlJc0I01EcY5R15hOadnZei+J5EIMvvsAQDc967acYSLfgJGmGLfPrZU7meC1tsTiSjd27Xg22N7\n89LyMoq2q6lhODRZoALXjBo/TzSz50IXKb7lFfrv55qQrQIlEmmundSfFsmJamoYJsGMoHqZ2c8B\nzCwV/6SGwpCmimO5BT76ZraiZwfdnCsSaTJap/L9Cf14c802VqqpYcgFU6C+CwwNFKl/Ah84534T\n0lRx6kBdPUtKdjBes/dEIta/mxqu9zpKzDvaWnwHb8gdAdwHfBP/yGl+kDfqyjFavmkn+2vrdXpP\nJIIdbGq4oKichUVqahhKR5tmftchz3fiX5X8LoK7UVeOUW5BOUkJxpj+GV5HEZGjuHx0L57IK+GO\ntz7j1evOwOxwK7vJiQrZjbpy7PIKfZzauz2tU0N6e5qInKC05ER+/JUB/PSlVcxbu41pQ7p6HSkm\nBTOLb7aZtTG/xwMNBqeEI1w88e05wNqtuzW9XCRKXDiiO/07tuLOtwvU1DBEgpkk8T3n3G5gCtAJ\n/6SJ20OaKg4dPJetCRIi0SEpMYGbpw6kaPteNTUMkWAK1MGTq18F/uKcW8nhW2nICcgt8NG+ZTJD\nurX1OoqIBOlgU8P73i3kQJ2aGja3YArUcjN7G3+Bmmdm6agDbrNyzpFbWM647I4kJKj2i0SLg00N\nt1Tu57klamrY3IIpUFcBtwCnOeeqgBT8p/mkmaz7fA/lew9o9XKRKHRGViZnZGXwgJoaNruj3Qc1\nKPDwlMCf/QL3P/UmxKugx5uDyxuN1/1PIlHpp1MHsWNfDU/kbfA6Skw5WqH5CXAN/30/FOg+qGaV\nV1jOwM7pdGmb5nUUETkOw3u2Y9rJXXgsr4Rvj+1Nh1YpXkeKCUccQTnnrgn8eeZhvlScmsn+mno+\n2qjljUSi3U1TB1BVU8efPyjyOkrMOOZ2G2b2FTN7JxRh4tHSDRXU1DXo/ieRKJfVKZ2LRvbgmSWb\n2Fq53+s4MeFo16DOMrMCM9trZn81s8Fmlo//HqiHwhcxtuUWlJOalMCovh28jiIiJ+jHXxkADu57\nVw0fmsPRRlB34b8GlQG8BCwBnnXOneqceyUc4eJBXqGPUX07kJac6HUUETlB3du14IoxvXlxeSlF\n2/d6HSfqHa1AOefch865A865VwGfc+6+cAWLB1sr91O4fa9WLxeJIded6W9qePc7asdxoo42i6+d\nmV3Y6Lk1fq5R1IlbUBhY3miAJkiIxIqM1qlcPb4f971XyKqySob1aOd1pKh1tBHUfOC8Rl+Nn58b\n+mixb36hj07pqQzsnO51FBFpRleP76umhs3gaO02tFpECNU3OBYWlTN5UGf1khGJMelpyVw7qT+3\nvb6ORUXlnJ6lsyTH45inmUvzWL1lF5VVtUzQ6T2RmHTFmN50a5vG/5u3Huec13GikgqUR/IKfJjB\nOP1mJRKT0pIT+fHZA1hZWsm8tV94HScqqUB5JK+wnCHd2pLROtXrKCISIheOPNjUcD31DRpFHatg\nOupeEmixgZn9wsxeCSwaK8dpT3UtH2/eqeWNRGJcUmICN00JNDX8uMzrOFEnmBHUL51ze8xsHDAV\neBqtJHFCFhdXUNfgtLyRSByYNqQLw3q05V41NTxmwRSog0f0a8BDzrnX8PeEkuOUW+ijZUoiI3u1\n9zqKiISYmfHTqWpqeDyCKVBbzOwR4BvAG2aWGuR+cgR5heWM7ZdBSpIOo0g8GJftb2r4oJoaHpNg\nfkJ+A5gHTHPOVQIdgJtDmiqGbarYx6aKKp3eE4kzN08dRIWaGh6TJgtUoM37dmBc4KU6QEv1Hqfc\ng8sbaYKESFw5pVFTwx37aryOExWCmcX3a+BnwM8DLyUDfw1lqFiWW+CjR/sW9M1s5XUUEQmzg00N\nH/pQTQ2DEcwpvguA84F9AM65rYAWjzsOtfUNLC6uYHx2Ry1vJBKHDjY1fHqxmhoGI5gCVeP863Q4\nADPTr/7HaUVpJXsP1DFRyxuJxK2DTQ3vf09XSpoSTIGaE5jF187Mvg+8CzwWzIeb2TQzW29mRWZ2\ny2Hen2BmH5tZnZldfMh7M8ysMPA1I5jvF+lyC3wkJhhj+6tAicSrg00N5+SXUuxTU8OjCWaSxJ34\nO+q+DAwEfuWc+1NT+5lZIvAgcA4wGPiWmQ0+ZLPNwHeA5w/ZtwPwa2A0MAr4tZlF/U1DuYXlnNKz\nHW1bJHsdRUQ89GVTw7cLvI4S0YKZJDELyHfO3eycu8k5906Qnz0KKHLOlTjnaoAXgOmNN3DObXTO\nrQIaDtl3KvCOc26Hc24n8A4wLcjvG5F27qthVVmlZu+JyJdNDV9f/Tmry3Z5HSdiBXOKrwuwzMzm\nBE7ZBXt1vztQ2uh5WeC1ZtvXzK4xs3wzy/f5fEF+tDcWFpfjHLr/SUQAf1PD9i2TuWPeZ15HiVjB\nnOL7BZANPIH/dFyhmf3ezPo3sevhClmwy/kGta9z7lHnXI5zLqdjx8j+wZ9b4KNNWhLDurf1OoqI\nRID0tGSuOzOLvMJyFhWXex0nIgW11k5gFt+2wFcd0B54yczuOMpuZUDPRs97AFuDzHUi+0Yc5xx5\nheWckZVJUqKWNxIRv4NNDe94S00NDyeYa1DXm9ly4A5gITDUOTcTOBW46Ci7LgOyzayvmaUAlwJz\ng8w1D5hiZu0DkyOmBF6LSkXb9/L5rmqd3hOR/5CWnMjss7NZUVrJ25+qqeGhgvl1PhO40Dk31Tn3\nonOuFsA51wCce6SdnHN1wCz8hWUdMMc5t9bMbjWz8wHM7DQzKwMuAR4xs7WBfXcAv8Vf5JYBtwZe\ni0pa3khEjuSikT3o17EVd85TU8NDWbDDSjPrBKQdfO6ci6h143Nyclx+fr7XMQ5rxpMfUbqzivdv\nnOR1FBGJQG+s/pxrn/uYOy8ZzsWn9vA6TsiZ2XLnXE5T2wVziu88MysENgDzgY3AmyecME5U19az\ndEMFE7J1ek9EDu+cIV0Y2r0t97xToKaGjQRziu82YAxQ4JzrC0zGfy1KgrB8006qaxuYoOWNROQI\nzIyfThvIlsr9PL80ok5OeSqYAlXrnKsAEswswTn3AXBKiHPFjNwCH8mJxui+GV5HEZEINi4rk9P7\nZ/DA+2pqeFAwBarSzFoDucBzZnYf/qnmEoTcwnJyenegVWqS11FEJIKZGTdPHUjFvhqeXKCmhhBc\ngZoOVAE3AG8BxcB5oQwVK7bvqWbd57sZr9N7IhKEEb3aM/XkzjyWq6aGENxKEvuccw2BaeOvA38K\nnPKTJiwITC/XBAkRCdZNUwayT00NgaMUKDMbY2YfmtkrZjbCzNYAa4AvzCyqF24Nl9wCHxmtUhjc\ntY3XUUQkSmR3TufCQFPDz3fFd1PDo42gHgB+D/wNeB+42jnXBZgA/CEM2aJaQ4NjQVE547IzSUhQ\n91wRCd6Pz84GB/e9G99NDY9WoJKcc287514EtjnnlgA457T0bhA+/Xw35XtrdHpPRI5Zj/YtuXxM\nL15cXhbXTQ2PVqAa92g6dJyp9TiakKfljUTkBFx3ZhZpSQncG8ejqKMVqOFmttvM9gDDAo8PPh8a\npnxRK6/Qx6Au6XRqk9b0xiIih8hsncqM0/vwr1VbKdq+x+s4njhigXLOJTrn2jjn0p1zSYHHB5+r\nZ/lRVNXUkb9xp1YvF5ETcvX4frRITuT+9+JzRp+aE4XA0pId1NQ36PqTiJyQDq1SuHJsH/65aitF\n2+PvWpQKVAjML/CRlpxATp/Q4fDEAAAQmElEQVT2XkcRkSj3/fF9aZGcyAPvx9+1KBWoEMgr9DG6\nbwZpyYleRxGRKJfROpVvj+3N3JVb425GnwpUM9tSuZ9i3z7N3hORZnPN+H6kJiXywPvxdS1KBaqZ\n5RX4ADRBQkSaTUbrVK4c25vXVmyhJI5GUSpQzSy30EeXNmlkd2rtdRQRiSHfnxB/oygVqGZU3+BY\nUFjO+OxMzLS8kYg0n8zWqVwxphevrtjChvJ9XscJCxWoZrSyrJLd1XU6vSciIXHNhP6kJCXEzShK\nBaoZ5RWUY+bvjCki0tw6pqdyxejevLpiCxvjYBSlAtWM8gp9DOvelvatUryOIiIx6pqJ/UhKMB74\nIPZHUSpQzWR3dS2flFYyXqtHiEgIdUpP44oxvfnHJ1vYVBHboygVqGayqKiC+gan608iEnI/ODiK\nivFrUSpQzSS30Efr1CRG9GrndRQRiXGd0tO4bHQvXvlkC5srqryOEzIqUM3AOUdugY+x/TNITtQh\nFZHQmzmxP0kJxoMxfC1KP02bwcaKKsp27meCljcSkTDp1CaNb43qxcsfl1G6IzZHUSpQzSCv0L+8\nkSZIiEg4zZzUn4QYHkWpQDWD3AIfvTq0pE9mK6+jiEgc6dwmjctG9eKl5bE5ilKBOkE1dQ0sLq7Q\n6uUi4okfTuxPghl//jD2RlEqUCfok8072VdTr+nlIuKJLm3TuHRUT17ML6NsZ2yNolSgTlBuoY/E\nBGNs/wyvo4hInJo56eAoqtjrKM1KBeoE5RWWM7JXO9qkJXsdRUTiVNe2LfjmaT15Mb+ULZX7vY7T\nbFSgTsCOfTWs3rJLs/dExHMzJ/UH4M8xNKNPBeoELCgqxzl1zxUR73Vr5x9FzckvZWuMjKJUoE5A\nboGPdi2TGdq9rddRRESYOSkLIGZm9IW0QJnZNDNbb2ZFZnbLYd5PNbO/B95famZ9Aq/3MbP9ZrYi\n8PVwKHMeD+cceYU+zsjKJDFB3XNFxHvd27XgkpyezFlWxue7on8UFbICZWaJwIPAOcBg4FtmNviQ\nza4CdjrnsoB7gP/X6L1i59wpga8fhirn8Sr4Yi9f7D6g5Y1EJKJcO6k/DsdDMTCjL5QjqFFAkXOu\nxDlXA7wATD9km+nA04HHLwGTzSwqhiNa3khEIlGP9i25+NSevPBRadSPokJZoLoDpY2elwVeO+w2\nzrk6YBdw8Iaivmb2iZnNN7Pxh/sGZnaNmeWbWb7P52ve9E3ILSwnq1NrurVrEdbvKyLSlGsn9afB\nOR6O8lFUKAvU4UZCLshtPgd6OedGAD8BnjezNv+1oXOPOudynHM5HTuGbyRTXVvP0hItbyQikaln\nh5ZcktODvy0rZduuaq/jHLdQFqgyoGej5z2ArUfaxsySgLbADufcAedcBYBzbjlQDAwIYdZjsmzj\nDg7UNWh6uYhErGsnZdHQ4Hh4fvSOokJZoJYB2WbW18xSgEuBuYdsMxeYEXh8MfC+c86ZWcfAJAvM\nrB+QDZSEMOsxyS3wkZKYwOi+HbyOIiJyWD07tOSikT14/qPNfLE7OkdRIStQgWtKs4B5wDpgjnNu\nrZndambnBzZ7AsgwsyL8p/IOTkWfAKwys5X4J0/80Dm3I1RZj1VeYTmn9W1Py5Qkr6OIiBzRdWdG\n9ygqpD9hnXNvAG8c8tqvGj2uBi45zH4vAy+HMtvx+mJ3NZ9t28Mt5wzyOoqIyFH1ymjJhSO78/zS\nzcyc2J9ObdK8jnRMtJLEMcorLAdggqaXi0gUmHVmNnUNjofnR8xVkqCpQB2j3AIfma1TGdQl3eso\nIiJN6pXRkgtHdOe5pZvYvie6rkWpQB2DhgbHgqJyxmdnkqDljUQkSsw6K4u6BscjUTaKUoE6Bmu3\n7mbHvhomDND9TyISPXpntOLrp/hHUb49B7yOEzQVqGOQG1jeaFyWrj+JSHT50VlZ1NY7Hs2Nnhl9\nKlDHIK/Qx+CubeiYnup1FBGRY9InsxXTT+nGs0uiZxSlAhWkfQfqWL5pJ+N1ek9EotSPzsqmpq6B\nx/Ki41qUClSQlpRUUFvvmKjp5SISpfpm+q9FPbt4E+V7I38UpQIVpNwCHy2SEzm1T3uvo4iIHLdZ\nZ2VxoK6ex3IjfxSlAhWkvMJyxvTrQGpSotdRRESOW7+OrTl/eDeeWbyJiggfRalABaF0RxUl5fvU\nnFBEYsKss7L9o6i8DV5HOSoVqCB8ubyR2muISAzI6tSa84Z345nFG9mxr8brOEekAhWE3AIf3dqm\n0b9jK6+jiIg0ix+dlcX+2vqIntGnAtWEuvoGFhaXMz67I2Za3khEYkNWp3TOG9aNZxZF7ihKBaoJ\nK8t2sae6Tqf3RCTmXD85i6raeh6P0FGUClQTcgt8JBickZXhdRQRkWaV1Smdrw3tytOLNrIzAkdR\nKlBNyCv0MaxHO9q1TPE6iohIs7t+cjZVtfU8sSDyZvSpQB3FrqpaVpRWMiFbyxuJSGwa0Dmdrw7t\nylOLNlJZFVmjKBWoo1hUXE6D0/RyEYlt15+Vzd4DdRE3ilKBOorcQh/pqUkM79nO6ygiIiEzsIv/\nWtRTCyNrFKUCdQTOOXILyjk9K4PkRB0mEYltP5qcxZ4DdTwZQaMo/eQ9gpLyfWyp3K/ljUQkLgzq\n0oZzhnThLws3squq1us4gArUEeUV+LvnTtT1JxGJE9dPzvaPohZGxihKBeoIcgvL6ZPRkp4dWnod\nRUQkLE7q2oZpJ3fhyYUb2LXf+1GUCtRhHKirZ3FxhU7viUjcuX5yNnuq6/hLBIyiVKAO4+NNleyv\nrdf0chGJO4O7tWHK4M48ucD7UZQK1GHkFvpISjDG9OvgdRQRkbC7fnI2u6vreGrhRk9zqEAdRl6h\nj5G925Oelux1FBGRsBvSvS1fGdyZJxaUsLvau1GUCtQhyvceYM2W3VreSETi2uzAKOppD0dRKlCH\nWFik7rkiIkO6t+Xskzrz+IIN7PFoFKUCdYj5BT7at0zm5G5tvY4iIuKp2ZOz2bW/lqcXbfTk+6tA\nNeKcI6+wnHHZHUlMUPdcEYlvQ3u0ZfKgTp6NolSgGvls2x58ew4wXtefREQAmH12NpVVtTyzeFPY\nv7cKVCN5hf7ljVSgRET8hvVox1mDOvFYXgl7D9SF9XurQDWSV1jOgM6t6dq2hddRREQixuzJB0dR\nG8P6fUNaoMxsmpmtN7MiM7vlMO+nmtnfA+8vNbM+jd77eeD19WY2NZQ5AfbX1LN0ww4tbyQicojh\nPdtx5sCOPJZbwr4wjqJCVqDMLBF4EDgHGAx8y8wGH7LZVcBO51wWcA/w/wL7DgYuBU4GpgF/Dnxe\nyHy0cQc1dQ2aXi4ichizzx7AzjBfiwrlCGoUUOScK3HO1QAvANMP2WY68HTg8UvAZDOzwOsvOOcO\nOOc2AEWBzwuZ3AIfKUkJjOqj5Y1ERA51Ss92TBzQkcfywjeKCmWB6g6UNnpeFnjtsNs45+qAXUBG\nkPtiZteYWb6Z5ft8vhMKu7ConNF9O9AiJaQDNRGRqDX77Gx27KvhrTXbwvL9kkL42Ye7kcgFuU0w\n++KcexR4FCAnJ+e/3j8Wc344lh17a07kI0REYtrIXu3556xxDOneJizfL5QFqgzo2eh5D2DrEbYp\nM7MkoC2wI8h9m1WbtGTaaHFYEZGjGtojfKvshPIU3zIg28z6mlkK/kkPcw/ZZi4wI/D4YuB955wL\nvH5pYJZfXyAb+CiEWUVEJMKEbATlnKszs1nAPCAReNI5t9bMbgXynXNzgSeAZ82sCP/I6dLAvmvN\nbA7wKVAHXOecqw9VVhERiTzmH7BEv5ycHJefn+91DBERaYKZLXfO5TS1nVaSEBGRiKQCJSIiEUkF\nSkREIpIKlIiIRCQVKBERiUgxM4vPzHzAia5imAmUN0OcaKfj4Kfj4Kfj4Kfj8G8neix6O+eaXJk7\nZgpUczCz/GCmPsY6HQc/HQc/HQc/HYd/C9ex0Ck+ERGJSCpQIiISkVSg/tOjXgeIEDoOfjoOfjoO\nfjoO/xaWY6FrUCIiEpE0ghIRkYikAiUiIhEpLguUmU0zs/VmVmRmtxzm/VQz+3vg/aVm1if8KUMv\niOPwEzP71MxWmdl7Ztbbi5yh1tRxaLTdxWbmzCwmpxoHcxzM7BuBfxNrzez5cGcMhyD+X/Qysw/M\n7JPA/42vepEz1MzsSTPbbmZrjvC+mdn9geO0ysxGNnsI51xcfeHvTVUM9ANSgJXA4EO2uRZ4OPD4\nUuDvXuf26DicCbQMPJ4Zr8chsF06kAssAXK8zu3Rv4ds4BOgfeB5J69ze3QcHgVmBh4PBjZ6nTtE\nx2ICMBJYc4T3vwq8CRgwBlja3BnicQQ1CihyzpU452qAF4Dph2wzHXg68PglYLKZWRgzhkOTx8E5\n94FzrirwdAnQI8wZwyGYfw8AvwXuAKrDGS6MgjkO3wcedM7tBHDObQ9zxnAI5jg4oE3gcVtgaxjz\nhY1zLhd/I9kjmQ484/yWAO3MrGtzZojHAtUdKG30vCzw2mG3cc7VAbuAjLCkC59gjkNjV+H/bSnW\nNHkczGwE0NM5969wBguzYP49DAAGmNlCM1tiZtPCli58gjkOvwGuMLMy4A3gR+GJFnGO9WfIMQtZ\ny/cIdriR0KFz7YPZJtoF/Xc0syuAHGBiSBN546jHwcwSgHuA74QrkEeC+feQhP803yT8o+k8Mxvi\nnKsMcbZwCuY4fAt4yjl3l5mNBZ4NHIeG0MeLKCH/ORmPI6gyoGej5z347yH6l9uYWRL+YfzRhrrR\nKJjjgJmdDfwvcL5z7kCYsoVTU8chHRgCfGhmG/Gfa58bgxMlgv1/8ZpzrtY5twFYj79gxZJgjsNV\nwBwA59xiIA3/4qnxJqifISciHgvUMiDbzPqaWQr+SRBzD9lmLjAj8Phi4H0XuCoYQ5o8DoFTW4/g\nL06xeL0BmjgOzrldzrlM51wf51wf/NfiznfO5XsTN2SC+X/xKv6JM5hZJv5TfiVhTRl6wRyHzcBk\nADM7CX+B8oU1ZWSYC1wZmM03BtjlnPu8Ob9B3J3ic87VmdksYB7+GTtPOufWmtmtQL5zbi7wBP5h\nexH+kdOl3iUOjSCPwx+B1sCLgTkim51z53sWOgSCPA4xL8jjMA+YYmafAvXAzc65Cu9SN78gj8ON\nwGNmdgP+U1rficFfYDGzv+E/nZsZuN72ayAZwDn3MP7rb18FioAq4LvNniEGj6uIiMSAeDzFJyIi\nUUAFSkREIpIKlIiIRCQVKBERiUgqUCIiEpFUoEREJCKpQIlEGDN73MwGH+X9U81sdaDNwf0xuJCx\nCKD7oESijpl9BMzGv6rFG8D9zrlYXMhX4pxGUCIeMbM+ZvaZmT0daPj2kpm1NLMPj7TWX6CdQRvn\n3OLA6gXPAF8Pa3CRMFGBEvHWQOBR59wwYDf+ZplH0x3/Ip0HNXuLA5FIoQIl4q1S59zCwOO/AuOa\n2D4eWsGIACpQIl47tLg0VWzK+M/Oxs3e4kAkUqhAiXirV6DpHfgb4S042saBdgZ7zGxMYPbelcBr\nIc4o4gkVKBFvrQNmmNkqoAPwUBD7zAQex9/moBjQDD6JSZpmLuIRM+sD/Ms5N8TjKCIRSSMoERGJ\nSBpBiUQoM1sKpB7y8redc6u9yCMSbipQIiISkXSKT0REIpIKlIiIRCQVKBERiUgqUCIiEpH+P0gS\n1IszcFufAAAAAElFTkSuQmCC\n",
188 |       "text/plain": [
189 |        "<matplotlib.figure.Figure at 0x1d0b776d748>"
190 |       ]
191 |      },
192 |      "metadata": {},
193 |      "output_type": "display_data"
194 |     }
195 |    ],
196 |    "source": [
197 |     "NUM_SAMPLES = 2000\n",
198 |     "N = 10\n",
199 |     "pi_0s = np.linspace(0,1,10)\n",
200 |     "\n",
201 |     "R = np.zeros(pi_0s.size)\n",
202 |     "\n",
203 |     "for i in range(0,pi_0s.size):\n",
204 |     "    this_pi0 = pi_0s[i]\n",
205 |     "    this_pi1 = 1- this_pi0\n",
206 |     "    threshold = np.log(((C_10-C_00)*this_pi0)/((C_01-C_11)*this_pi1))\n",
207 |     "\n",
208 |     "    for k in range(0,NUM_SAMPLES):\n",
209 |     "        #draw a sample from H, and then from p(y|H)\n",
210 |     "        H = np.random.choice(2,p=[this_pi0,this_pi1])\n",
211 |     "        if(H==0):\n",
212 |     "            y = np.random.multinomial(N,p_0)\n",
213 |     "            L = 0\n",
214 |     "            for j in range(0,len(dictionary)):\n",
215 |     "                L += y[j]*np.log(p_1[j]/p_0[j])\n",
216 |     "            if(L<threshold):\n",
217 |     "                R[i] += C_00/NUM_SAMPLES\n",
218 |     "            else:\n",
219 |     "                R[i] += C_10/NUM_SAMPLES\n",
220 |     "        else:\n",
221 |     "            y = np.random.multinomial(N,p_1)\n",
222 |     "            L = 0\n",
223 |     "            for j in range(0,len(dictionary)):\n",
224 |     "                L += y[j]*np.log(p_1[j]/p_0[j])\n",
225 |     "            if(L<threshold):\n",
226 |     "                R[i] += C_01/NUM_SAMPLES\n",
227 |     "            else:\n",
228 |     "                R[i] += C_11/NUM_SAMPLES  \n",
229 |     "\n",
230 |     "plt.plot(pi_0s,R)\n",
231 |     "\n",
232 |     "plt.xlabel(\"pi_0\")\n",
233 |     "plt.ylabel(\"Bayes Risk R\")\n",
234 |     "\n",
235 |     "plt.tight_layout()\n",
236 |     "plt.show()"
237 |    ]
238 |   },
239 |   {
240 |    "cell_type": "markdown",
241 |    "metadata": {},
242 |    "source": [
243 |     "It's worth fiddling both with the costs and the number of words N per sample. How do the affect the Bayes risk? Which prior is the worst-case prior, corresponding with the minimax detector? How does changing the costs impact this prior?"
244 |    ]
245 |   }
246 |  ],
247 |  "metadata": {
248 |   "kernelspec": {
249 |    "display_name": "Python 3",
250 |    "language": "python",
251 |    "name": "python3"
252 |   },
253 |   "language_info": {
254 |    "codemirror_mode": {
255 |     "name": "ipython",
256 |     "version": 3
257 |    },
258 |    "file_extension": ".py",
259 |    "mimetype": "text/x-python",
260 |    "name": "python",
261 |    "nbconvert_exporter": "python",
262 |    "pygments_lexer": "ipython3",
263 |    "version": "3.6.3"
264 |   }
265 |  },
266 |  "nbformat": 4,
267 |  "nbformat_minor": 2
268 | }
269 | 


--------------------------------------------------------------------------------
/ch.3/.ipynb_checkpoints/Multiple Hypothesis Testing via QPSK-checkpoint.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# Multiple hypothesis testing: QPSK\n",
  8 |     "\n",
  9 |     "We'll look at multiple hypothesis testing through digital communications. We'll simulate a simple quaternary phase shift keying (QPSK) system and build its matched filter receiver."
 10 |    ]
 11 |   },
 12 |   {
 13 |    "cell_type": "markdown",
 14 |    "metadata": {},
 15 |    "source": [
 16 |     "First, we'll define the (discrete-time) carrier frequency and the duration of each transmit symbol and plot all four symbols."
 17 |    ]
 18 |   },
 19 |   {
 20 |    "cell_type": "code",
 21 |    "execution_count": 16,
 22 |    "metadata": {},
 23 |    "outputs": [
 24 |     {
 25 |      "data": {
 26 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAH5tJREFUeJztnXGMXdWd3z8/xjY7kHWnzkwTPDAM\nqSIaBFUcuaTIbRShpGbTVXFQorC7ilarRkZtiVpVgdraqqFSK7v1tttVFLXQhM1uu3XSUjBoi+Ru\n6o2oUATYHQIG6i4bwPGYLCbWxA1MsT3z6x/zZjwZv3nvvnvOu+fee74faWS/e+45v+/3nOvfXN9z\n3znm7gghhGgPV6QWIIQQIi5K7EII0TKU2IUQomUosQshRMtQYhdCiJahxC6EEC1DiV0IIVqGErsQ\nQrSMKIndzB42s7fM7HiM9oQQQpTHYnzz1Mw+AfwM+H13v7nf+ePj4z49PT1QjLl3L/Djc/+PCwuL\nbBy5gg9u/gXGrtpYTnDDyNW7fOflG/L2XoRjx4697e4T/c7bECOYuz9lZtNFz5+enubo0aOF2z80\nM8veR19k/MLCyrGNG0f4x3fdwq5tk4NIbRy5epfvvHxD3t6LYmZvFDmvEc/YDxw+wfyqwQaYv7DA\ngcMnEimqjly9y/clcvANeXuPTWWJ3cx2m9lRMzt65syZgeqenpsf6HibyNW7fBc73iZy9h6byhK7\nuz/k7tvdffvERN9HRD/H1rHRgY63iVy9y3ex420iZ++xacSjmPt23sjoxpGfOza6cYT7dt6YSFF1\n5Opdvi+Rg2/I23tsokyemtlB4JPAuJmdAr7q7t+M0TawMnFy/yMvcH5hkcmxUe7beePPTagcmpnl\nwOETnJ6bZ2uX8jrTS3s/77n67le/7qynPVffkO+YD4MorzsOyvbt232Qt2KW+cKD3wfgO/fc9nPH\nl2fTV0+8jG4cYV8DZtOLau/mPVffg9SvI0W05+ob8vbeDzM75u7b+53XiEcx/WjybHqI9lx9x6if\nEo35JTTmw9HeisTe5Nn0EO25+o5RPyUa82LHY9dPSdXaW5HYmzybHqI9V98x6qdEY17seOz6Kala\neysSe5Nn00O05+o7Rv2UaMwvoTEfjvYob8Wkpshsel0J0Z6r7xj1U6Ix15gPW3srEjssddzBZ08C\nl8+m150Q7bn6jlE/JRpzjfkwacWjGCGEEJdQYhdCiJahxC6EEC1DiV0IIVqGErsQQrQMJXYhhGgZ\nsTazvsPMTpjZq2a2J0abQgghyhH8HruZjQBfBz4NnAKeM7Mn3P3l0LZjknK5T8VOs8Rqrt5zjZ06\nfmrvq4nxBaVbgVfd/YcAZvZt4E6gNol97ZKZs3Pz7H30RYChd7xiVx87dXzF1phX7X0tweuxm9nn\ngDvc/Uudz18EPu7u965Xp+x67L/7K1/mg2d+xE3XbO5a/vKb5wAuK585Ocd7FxcuO//KDSNsmxpb\n+fz6T94BYPr9V192bq+yXuVFY6+nvUj5MH2XLa/Cd6/yIvGH4bto7F7a+5X1Kq9izFNe673KmzLm\nP564jt84+LWu7fej6HrsMe7Yrcuxy35bmNluYDfA1NRUqUBbrr6Sq346sm75VZu6l3Xr8G7H3z3f\n/bx+Zb3Ki8ZeT3uR8mH6Lltehe9e5UXiD8N30dhQbkz7lVcx5imv9V7lTRnzLVdf2TNGDGLcsd8G\nPODuOzuf9wK4+7716pS9Yy/Ljv1HmO2y7vHk2ChP77l95fN6O7f0K+tVXjT2MIjhu2x5St9F4w/D\nd9HYw6KKMa+j76Lxmz7mVe6g9BzwYTO7wcw2AXcDT0RoNxopl/tU7Opjp46v2NXHTh0/tfe1BD+K\ncfeLZnYvcBgYAR5295eClUVkefIixYy1Yqd5SyBX77nGTh0/tfe1RFm2192fBJ6M0dawWL0DvGK3\nP3bq+Iqdhpy9r0bfPBVCiJahxC6EEC1DiV0IIVqGErsQQrQMJXYhhGgZSuxCCNEylNiFEKJlKLF3\nODQzy8zJOZ557Sw79h/h0MxsJXVTE6q9qd5z9Q261nMYcyV2Li25eX5hEbi05GaRgQupm5pQ7U31\nnqtv0LWey5grsbP0NeDldZSXmb+wwIHDJ4ZaNzWh2pvqPVffoGt9NW0ecyV24HSXVdl6HY9VNzWh\n2pvqPVffoGu96PHY9atGiR3YOjY60PFYdVMTqr2p3nP1DbrWix6PXb9qlNgJW3Kzbst1DkKo9qZ6\nz9U36FpfTZvHPCixm9nnzewlM1s0s76Lv9eVXdsm2XfXLUyOjWIsLY6/765bVlZq6zUb3q9unSmi\nvY3ec/UNYd5z9V20fp0I2kHJzD4CLAIPAl9x90LbIlW9g1IIazephaXf1HUe1Fjk6j1X35Cv96b4\nrmQHJXd/xd3rOS0ciabNhsckV++5+oZ8vbfNd2XP2M1st5kdNbOjZ86cqSpsME2bDY9Jrt5z9Q35\nem+b776J3cy+a2bHu/zcOUggd3/I3be7+/aJiYnyiiumabPhMcnVe66+IV/vbfPdN7G7+6fc/eYu\nP49XITA1TZsNj0mu3nP1Dfl6b5vvoMnTlUbMvscAk6dmdgZ4o2S4ceDtknVLccXo5i0j79syaSMb\nNvnCxfMLPzs7uzh/7mwdtA1AKW0DeK9c2zBZ8X3Fhk2+ODTfoQyl3yKNee3GdBVdtVV0rZfStorr\n3b3vI4/Qt2I+C3wNmADmgOfdfWfpBovFPFpkVjgF0lYOaSuHtJUjB20bQiq7+2PAY6EihBBCxEPf\nPBVCiJbRxMT+UGoBPZC2ckhbOaStHK3XFmXyVAghRH1o4h27EEPBzB4zs39mZv/TzH5sZp9KrUmI\nMiixC3GJm4E5d//rwN8Ffi2xHiFKESWxm9nDZvaWmR2P0Z4QVWNmVwF/DvjtzqENLL3CK0TjiPUF\npU8APwN+391v7nf++Pi4T09PB8ctyty7F5idm2dxldcrzJgcG2Xsqo2V6aiaXH1Dvt5z9Q15eD92\n7NjbRb6gFPQe+zLu/pSZTRc9f3p6miqX7d2x/wgXuyzm84GxUZ7ec3tlOqomV9+Qr/dcfUMe3s2s\n0Df2s1jdsW0rtxUlV9+Qr/dcfUPe3tdSWWJPubpj21ZuK0quviFf77n6hry9ryWLt2LatnJbUXL1\nDfl6z9U35O19LVGesded5a2t7n/kBc4vLDI5Nsp9O2+s1ZZXwyBX35Cv91x9Q97e1xIlsZvZQeCT\nwLiZnQK+6u7fjNF2UQ7NzHLg8AlOz82ztcuA7to2ycFnTwLwnXtuG7h+XcnVN4R5z9V3kfp1Jtcx\nH5RYb8X8Sox2yrJ2I9rZuXn2PvoiQKGBC62filx9Q5j2XH3HqJ+SXMe8DK14xh66EW1TN7LN1TeE\nac/Vd4z6Kcl1zMvQisQe+ppTU1+TytU3hGnP1XeM+inJdczL0IrEHvqaU1Nfk8rVN4Rpz9V3jPop\nyXXMy9CKxB76mlNTX5PK1TeEac/Vd4z6Kcl1zMvQitcdQ19zauprUrn6hjDtufqOUT8luY55GVqR\n2KH/K17Drp+KXH1DmPZcfceon5Jcx3xQWvEoRgghxCWU2IUQomUosQshRMtQYhdCiJahxC6EEC1D\niV0IIVpGrM2s7zCzE2b2qpntidGmEEKIcgS/x25mI8DXgU8Dp4DnzOwJd385tO3VpFxyM/Vyn7l6\nV2xdb7nEjk2MLyjdCrzq7j8EMLNvA3cC0RJ7yiU3Uy/3mat3xdb1lkvsYWDuHtaA2eeAO9z9S53P\nXwQ+7u73rldn+/btfvTo0cIxduw/wuzcPPe88Dgf+unsyvErN4ywbWps5fPLb54D4KZrNndtp0z5\nzMk53ru4cNm5a2O//pN3AJh+/9Vd2y5bXiT+MHwXjd1Le7+yXuVFY4d4H6bvsuVV+O5VXiR+ymu9\nl/Yi5XUY87mtN3DnN36ra3k/zOyYu2/vd16MO3brcuyy3xZmthvYDTA1NTVQgPWW1lw7EFdtGul6\nXkh5t8Hudvzd893PCy0vEn8YvovGht7ehukbwrwP03fZ8ip89yovEj/ltQ7NH/Oz77zX85wYxLhj\nvw14wN13dj7vBXD3fevVKXvHvpbJsVGe3nP7wJoHoWjsLzz4fWD9NSjKljfde66+y5an9F00vq71\n+N6LUvSOPcZbMc8BHzazG8xsE3A38ESEdldIueRm6uU+c/Wu2NXHTh0/19jDIPhRjLtfNLN7gcPA\nCPCwu78UrGwVy5MXKWasU8ZOHV+x84qdOn6usYdBrM2snwSejNHWeuzaNpmsk1PGTh1fsfOKnTp+\nrrFjo2+eCiFEy1BiF0KIlqHELoQQLUOJXQghWoYSuxBCtAwldiGEaBlK7EII0TKU2CNwaGaWmZNz\nPPPaWXbsP8Khmdn+lSLWT0mI9lx9x6ifilx9Q7O8K7EHsrzc5/mFReDScp9FBy20fkpCtOfqO0b9\nVOTqG5rnXYk9kAOHT6ys4bzM/IUFDhw+UUn9lIRoz9V3jPqpyNU3NM+7Ensg6y0pvN7x2PVTEqI9\nV98x6qciV9/QPO9K7IFsHRsd6Hjs+ikJ0Z6r7xj1U5Grb2iedyX2QEKX+2zycqEh2nP1HaN+KnL1\nDc3zHpTYzezzZvaSmS2aWd/F39vIrm2T7LvrFibHRjGWFubfd9ctP7dKXK/Z8CL160o/7bn6hnZ6\nz9U3NM970A5KZvYRYBF4EPiKuxfaFmnQHZSazNpNcmHpN3VTLuiy5Oob8vWeq2+oznslOyi5+yvu\nXv8p7YQ0+U2AEHL1Dfl6z9U31M97Zc/YzWy3mR01s6NnzpypKmxymvwmQAi5+oZ8vefqG+rnvW9i\nN7PvmtnxLj93DhLI3R9y9+3uvn1iYqK84obR5DcBQsjVN+TrPVffUD/vQc/YVxox+x4DPGM3szPA\nGyXDjQNvl6w7bC7TdsXo5i0bNk9cj9mlX6LuixfPnXljcf7c2ZTahsmAvhs1pv2ocMxr1W81utb7\nEb3fInrvp+16d+97Zxxlz9NBKSJsPczsaJHJgxRIWzmkrRzSVo4ctIW+7vhZMzsF3Ab8NzM7HCpI\nCCFEGEF37O7+GPBYJC1CCCEi0MRvnj6UWkAPpK0c0lYOaStH67VFmTwVQghRH5p4xy6EEKIHSuxC\nCNEyoiR2M3vYzN4ys+Mx2hNCCFGeWHfs3wLuiNSWEEKIAKJ8QcndnzKz6aLnj4+P+/R04dOTMvfu\nBWbn5llcNcl8hRmTY6OMXbUxobLhk6v3XH1Dvt6b4vvYsWNv1/abp9PT0zRl2d4d+49wsctCPh8Y\nG+XpPbcnUFQduXrP1Tfk670pvs2s0FIsWt2xD3Vbta1KcvWeq2/I13vbfFeW2Ju6umPdVm2rkly9\n5+ob8vXeNt963bEPTd6nMZRcvefqG/L13jbfUZ6xm9lB4JPAeGdRsK+6+zdjtF0Vh2ZmOXD4BKfn\n5tk6Nsp9O29k17bJlW2t7n/kBc4vLDK5qqxf3SbQS3ubvefqG8K85+q7X/26kWRJgbrteVpkv8Iv\nPPh9AL5zz20D160rRbW3zXuuviHMe66+B6k/bCrZ87QthOxXWLe9DgchVHtTvefqG3Str6bNY67E\nTtiMeJNn00O1N9V7rr5B13rR47HrV40SO2Ez4k2eTQ/V3lTvufoGXetFj8euXzVK7ITNiDd5Nj1U\ne1O95+obdK2vps1jnuSbp3WjyIz4MOqmJlR7U73n6ht0recy5krsHXZtm+TgsyeBy2fEh1k3NaHa\nm+o9V9+gax3aP+Z6FCOEEC1DiV0IIVqGErsQQrQMJXYhhGgZSuxCCNEylNiFEKJlKLELIUTLiLVs\n7x3A7wAjwDfcfX+MdmOScslNxU6zzGmu3nONnTp+au+rCU7sZjYCfB34NHAKeM7MnnD3l0PbjsXa\nJTdn5+bZ++iLAEPveMWuPnbq+IqtMa/a+1qC12M3s9uAB9x9Z+fzXgB337denbLrsT/+pa8wdvo1\npt9/ddfy13/yDsBl5TMn53jv4sJl51+5YYRtU2Mrn19+8xwAN12z+bJze5X1Ki8aez3tRcqH6bts\neRW+e5UXiT8M30Vj99Ler6xXeRVjnvJa71XelDGf23oDd37jt7q234+i67HHeBQzCfxo1edTwMe7\nCNoN7AaYmpoqFejsO++x6fzlnbfMu+uUdevwbsev2jTS9bx+Zb3Ki8ZeT3uR8mH6Lltehe9e5UXi\nD8N30dhQbkz7lVcx5imv9V7lTRnzs++81zNGDGLcsX8e2OnuX+p8/iJwq7t/eb06Ze/Y19vdpF/5\njv1HmO2ybvLk2ChP77l9YB2DUDR2WW+9ytruu1d5073n6jukvA3e+1HlDkqngOtWfb4WOB2h3Wik\nXHJTsauPnTq+YlcfO3X81N7XEuNRzHPAh83sBmAWuBv41QjtRmN58iLFjLVip3lLIFfvucZOHT+1\n97UEJ3Z3v2hm9wKHWXrd8WF3fylYWWR2bZtM1smKnYZcvecaO3X81N5XE+U9dnd/EngyRltCCCHC\n0DdPhRCiZSixCyFEy1BiF0KIlqHELoQQLUOJXQghWoYSuxBCtAwldiGEaBmtSeyHZmaZOTnHM6+d\nZcf+IxyamU0tqTAh2nP1HaN+SjTmGvNham9FYl9eC/n8wiJwaS3kJgx6iPZcfceonxKNucZ82Npb\nkdgPHD6xssD9MvMXFjhw+EQiRcUJ0Z6r7xj1U6Ixv4TGfDjaW5HYT3dZLrPX8ToRoj1X3zHqp0Rj\nXux47PopqVp7KxL71rHRgY7XiRDtufqOUT8lGvNix2PXT0nV2oMSu5l93sxeMrNFM+u7+PuwqNta\nyIMQoj1X3zHqp0RjfgmN+XC0h96xHwfuAp6KoKUnvWaUd22bZN9dtzA5NoqxtGvJvrtuqc0Smr0o\non0977n6Llq/rvTTnqtvyNt7TIK3xgMws+8BX3H3QvvdDbo13todwGHpt11TBjWEXL3Ld16+IW/v\nRalya7yh0+TZ8FBy9S7fl8jBN+TtPTZ9N9ows+8CH+xS9Jvu/njRQGa2G9gNMDU1VVggNHs2PJRc\nvct3seNtImfvsel7x+7un3L3m7v8FE7qnXYecvft7r59YmJiIJFNng0PJVfv8l3seJvI2Xtskjxj\nN7MzwBtF279idPOWDZsnrsfs0i8i98WL5868sTh/7uygeofIOPB2zAYjeo+uLSKXaavRmFfabwP6\nbtSY9qPCMW9yv13v7n3vjIMSu5l9FvgaMAHMAc+7+87SDRaLebTI5EEKpK0c0lYOaStHDtqCNrN2\n98eAx0JFCCGEiEcj3ooRQghRnCYm9odSC+iBtJVD2sohbeVovbYok6dCCCHqQxPv2IUQQvRAiV0I\nIVpGlMRuZg+b2VtmdjxGe0IIIcoT6479W8AdkdoSQggRQNB77Mu4+1NmNl30/PHxcZ+eLnx6o5l7\n9wKzc/MsrpqkvsKMybFRxq7amFDZcMnVN+TrPVffUJ33Y8eOvV3km6dREvugTE9PM8iyvU1mx/4j\nXOyyiNEHxkZ5es/tCRRVQ66+IV/vufqG6rybWaGlWCqbPDWz3WZ21MyOnjlzpqqwycl1xbpcfUO+\n3nP1DfXzXlliD1ndscnkumJdrr4hX++5+ob6edfrjkOmyfs0hpCrb8jXe66+oX7eozxjN7ODwCeB\ncTM7BXzV3b8Zo+0mcGhmlgOHT3B6bp6tY6Pct/PGla28lv+8/5EXOL+wyOSa8n7168562nP1De0e\n81x9Q7O8J1lSYNA9T+tM0X0av/Dg9wH4zj23lapfR4poz9U3tM97rr6hPt5btedpnQndp7HJ+zyG\naM/Vd4z6qcjVNzTPuxJ7IKGz4XWbTR+EEO25+o5RPxW5+obmeVdiDyR0Nrxus+mDEKI9V98x6qci\nV9/QPO9K7IGEzobXbTZ9EEK05+o7Rv1U5Oobmuc9yTdP20SR2fBh1k9JiPZcfceon4pcfUPzvCux\nR2DXtkkOPnsSuHw2vIr6KQnRnqvvGPVTkatvaJZ3PYoRQoiWocQuhBAtQ4ldCCFahhK7EEK0DCV2\nIYRoGUrsQgjRMpTYhRCiZcRatvcO4HeAEeAb7r4/RrurSbncZ+qlRnP1rti63nKJHZvgxG5mI8DX\ngU8Dp4DnzOwJd385tO1l1i55OTs3z95HXwQYesenjJ06vmLnFTt1/FxjD4Pg9djN7DbgAXff2fm8\nF8Dd961XZ9D12HfsP8Ls3Dz3vPA4H/rp7MrxKzeMsG1qbOXz6z95B4Dp91/dtZ0y5TMn53jv4sJl\n566N/fKb5wC46ZrNXdsuW14k/jB8F43dS3u/sl7lRWOHeB+m77LlVfjuVV4kfsprvZf2IuV1GPMf\nT1zHbxz8WtfyfhRdjz3Go5hJ4EerPp8CPt5F0G5gN8DU1NRAAdZb2nLtQLx7/vKBCS3vNtjdjl+1\naaTreaHlReIPw3fR2NDb2zB9Q5j3YfouW16F717lReKnvNah+WO+5eore54Tgxh37J8Hdrr7lzqf\nvwjc6u5fXq9O2Tv2tUyOjfL0nttXPq+3e0lIedHYw6JI/GH4Lhp7WFQx5rn67lXedu919D0IVe6g\ndAq4btXna4HTEdpdIeVyn6mXGs3Vu2JXHzt1/FxjD4MYj2KeAz5sZjcAs8DdwK9GaHeF5cmLFDPW\nKWOnjq/YecVOHT/X2MMgOLG7+0Uzuxc4zNLrjg+7+0vBytawa9tksk5OGTt1fMXOK3bq+LnGjk2U\n99jd/UngyRhtCSGECEPfPBVCiJahxC6EEC1DiV0IIVqGErsQQrQMJXYhhGgZSuxCCNEyWpPYD83M\nMnNyjmdeO8uO/Uc4NDPbv1LE+qnI1TeEac/Vd4z6Kcl1zAelFYl9ecnN8wuLwKUlN4sOXGj9VOTq\nG8K05+o7Rv2U5DrmZWhFYj9w+MTKOsrLzF9Y4MDhE5XUT0WuviFMe66+Y9RPSa5jXoZWJPb1lvVd\n73js+qnI1TeEac/Vd4z6Kcl1zMvQisS+dWx0oOOx66ciV98Qpj1X3zHqpyTXMS9DKxJ76JKbTV2y\nM1ffEKY9V98x6qck1zEvQ9AiYJ1NNh4APsLS5hrFd8+ISOiSm01dsjNX3xCmPVffMeqnJNcxL0PQ\nDkpm9hFgEXgQ+ErRxD7oDkoxODQzy/2PvMD5hUUmWz6oq8nVN+TrPVff0H7vlex56u6vdIKFNDN0\n1nvVCZq5A3lRcvUN+XrP1Tfk7X0trXjG3o/cXnVaJlffkK/3XH1D3t7X0veO3cy+C3ywS9Fvuvvj\nRQOZ2W5gN8DU1FRhgTHI7VWnZXL1Dfl6z9U35O19LX3v2N39U+5+c5efwkm9085D7r7d3bdPTEyU\nV1yC3F51WiZX35Cv91x9Q97e1xI0ebrSiNn3GGDy1MzOAG+UDDcOvD1IhStGN2/ZsHnieswu/SJz\nX7x47swbi/PnzpbUEUXbMKnQdyjR+y2i91qN6Rou01ajMa+83wbw3qgxXcP17t73zjj0rZjPAl8D\nJoA54Hl331m6wWIxjxaZFU6BtJVD2sohbeXIQVvoWzGPAY+FihBCCBGPLN6KEUKInGhiYn8otYAe\nSFs5pK0c0laO1muLMnkqhBCiPjTxjl0IIUQPGpXYzewOMzthZq+a2Z7UelZjZq+b2Ytm9ryZJVkM\nbZWWh83sLTM7vurYFjP7IzP7k86ff75G2h4ws9lO3z1vZp9JpO06M/tjM3vFzF4ys7/fOZ6873po\nS953ZvYLZvasmf2go+2fdo7fYGbPdPrtO2a2qUbavmVmr63qt49WrW2VxhEzmzGzP+x8Du83d2/E\nDzAC/CnwIWAT8APgptS6Vul7HRhPraOj5RPAx4Djq479S2BP5+97gH9RI20PsPQ9iNT9dg3wsc7f\nfxH4P8BNdei7HtqS9x1gwPs6f98IPAP8VeA/A3d3jv874O/USNu3gM+lvuY6uv4h8J+AP+x8Du63\nJt2x3wq86u4/dPfzwLeBOxNrqiXu/hSw9ssodwK/1/n77wG7KhXVYR1ttcDd33T3/9X5+/8FXgEm\nqUHf9dCWHF/iZ52PGzs/DtwOPNI5nqrf1tNWC8zsWuBvAt/ofDYi9FuTEvsk8KNVn09Rkwu7gwP/\n3cyOddbFqRsfcPc3YSlJAH8hsZ613GtmL3Qe1SR5TLQaM5sGtrF0h1ervlujDWrQd53HCc8DbwF/\nxNL/rufc/WLnlGT/Xtdqc/flfvvnnX77bTO7MoU24N8A97O0/DnA+4nQb01K7N3WBq7Nb15gh7t/\nDPgl4O+Z2SdSC2oQ/xb4i8BHgTeBf5VSjJm9D/ivwD9w93Mptayli7Za9J27L7j7R4FrWfrf9Ue6\nnVatqk7QNdrM7GZgL/CXgL8CbAH+UdW6zOyXgbfc/djqw11OHbjfmpTYTwHXrfp8LXA6kZbLcPfT\nnT/fYunbuLemVXQZf2Zm1wB0/nwrsZ4V3P3POv/4FoF/T8K+M7ONLCXOP3D3RzuHa9F33bTVqe86\neuaA77H0HHvMzJa/3Z783+sqbXd0Hm25u78H/C5p+m0H8LfM7HWWHi3fztIdfHC/NSmxPwd8uDNj\nvAm4G3gisSYAzOxqM/vF5b8DfwM43rtW5TwB/Hrn778ODLQ65zBZTpodPkuivus83/wm8Iq7/+tV\nRcn7bj1tdeg7M5sws7HO30eBT7E0B/DHwOc6p6Xqt27a/veqX9TG0jPsyvvN3fe6+7XuPs1SPjvi\n7r9GjH5LPSM84OzxZ1h6G+BPWVoPPrmmjq4PsfSWzg+Al1JrAw6y9N/yCyz9T+dvs/Ts7n8Af9L5\nc0uNtP0H4EXgBZaS6DWJtP01lv7b+wLwfOfnM3Xoux7akvcd8JeBmY6G48A/6Rz/EPAs8CrwX4Ar\na6TtSKffjgP/kc6bM6l+gE9y6a2Y4H7TN0+FEKJlNOlRjBBCiAIosQshRMtQYhdCiJahxC6EEC1D\niV0IIVqGErsQQrQMJXYhhGgZSuxCCNEy/j94q2ZtNaCSigAAAABJRU5ErkJggg==\n",
 27 |       "text/plain": [
 28 |        "<matplotlib.figure.Figure at 0x1a66d438a90>"
 29 |       ]
 30 |      },
 31 |      "metadata": {},
 32 |      "output_type": "display_data"
 33 |     }
 34 |    ],
 35 |    "source": [
 36 |     "import numpy as np\n",
 37 |     "import matplotlib.pyplot as plt\n",
 38 |     "\n",
 39 |     "omega = np.pi/4\n",
 40 |     "T = np.round(2*np.pi/omega*5).astype(int) #This is the number of samples of the cosine that each symbol will consist of\n",
 41 |     "\n",
 42 |     "n = np.arange(0,T)\n",
 43 |     "s = np.zeros((4,T))\n",
 44 |     "s[0,:] = np.cos(omega*n) + 0*np.sin(omega*n)\n",
 45 |     "s[1,:] = 0*np.cos(omega*n) + np.sin(omega*n)\n",
 46 |     "s[2,:] = -1*np.cos(omega*n) + 0*np.sin(omega*n)\n",
 47 |     "s[3,:] = 0*np.cos(omega*n) + -1*np.sin(omega*n)\n",
 48 |     "\n",
 49 |     "fig, axs = plt.subplots(nrows=4, sharex=True)\n",
 50 |     "\n",
 51 |     "axs[0].stem(n,s[0,:])\n",
 52 |     "axs[1].stem(n,s[1,:])\n",
 53 |     "axs[2].stem(n,s[2,:])\n",
 54 |     "axs[3].stem(n,s[3,:])\n",
 55 |     "axs[0].set_xlabel('$n$')\n",
 56 |     "\n",
 57 |     "plt.show()"
 58 |    ]
 59 |   },
 60 |   {
 61 |    "cell_type": "markdown",
 62 |    "metadata": {},
 63 |    "source": [
 64 |     "As you can see, the four \"symbols\" are just cosines with different phases---hence the name!. We'll assume a very simple channel in which the symbols are corrupted by i.i.d. Gaussian noise."
 65 |    ]
 66 |   },
 67 |   {
 68 |    "cell_type": "code",
 69 |    "execution_count": 43,
 70 |    "metadata": {},
 71 |    "outputs": [
 72 |     {
 73 |      "data": {
 74 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEKCAYAAAAGvn7fAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAGHZJREFUeJzt3X+QXWddx/HPN9tNWWgnCyQo2W0I\nYCdYwg5hFpRpRabpmLaADQ7GMowyVScodlSUQGLHiFWZakbLII4ShRYVSyOmaeWHoTQ4SIdfCQmb\n1BJafrTNptIgbCmy7W52H/845ya7N/fec/aen89z3q+ZTPaeu+ee73PO3e997vPrmHNOAIBwLKs6\nAABAvkjsABAYEjsABIbEDgCBIbEDQGBI7AAQGBI7AASGxA4AgSGxA0BgzqvioCtXrnRr166t4tAA\n4K1Dhw591zm3Kun3Kknsa9eu1cGDB6s4NAB4y8weSvN7NMUAQGBI7AAQGBI7AASGxA4AgSGxA0Bg\nSOwA0pvYI928XnrXcPT/xJ6qI0IHlQx3BOChiT3Sv/+2NDsdPX78keixJI1tqS4unIMaO4B07rnx\nbFJvmZ2OtqNWSOwA0nn8xNK2ozIkdgDprBhd2nZUhsQOIJ2NO6XBocXbBoei7agVEjuAdMa2SK97\nrzRwfvR4xUXRYzpOa4dRMQDSG9siHfpQ9PN1H682FnRFjR0AAkNiDx0TSoDGoSkmZEwoARqJGnvI\nmFCCTvgWFzxq7CFjQgna8S2uEaixh4wJJWjHt7hGILGHjAklaMe3uEYgsYeMCSVox7e4RiCxh25s\nizT6cul5l0lvO0ZSbzq+xTUCiR1oEr7FNQKjYoCmYVmA4FFjB4DAkNhRHCbCAJWgKQbFYCJMcSb2\nROPOHz8RjWbZuJNzikWosaMYTIQpRusD8/FHJLmzH5h8G8ICJHYUg4kwxeADEymQ2FEMJsIUgw9M\npEBiRzGYCFMMPjCRAokdxWAiTDH4wEQKjIpBcZgIk7/WB+Od10tzT0UfmIyKQRsSO+AbPjCRgKYY\nAAhMLjV2M/ugpNdKesw5tz6P12y37/Ckdu0/rpNT01o9PKRtm9Zp84aRIg5VO00tO+VuVrmlZpc9\nT3k1xdwq6X2S/jGn11tk3+FJ7dh7VNOzc5Kkyalp7dh7VJKCv+hNLTvlbla5pWaXPW+5NMU45z4r\n6Xt5vFYnu/YfP3OxW6Zn57Rr//GiDlkbQZe9x1oyQZe7h6aWW2p22fNWWuepmW2VtFWS1qxZs6R9\nT05NL2l7SIIte8JaMsGWO0FTyy01u+x5K63z1Dm32zk37pwbX7Vq1ZL2XT08tKTtIQm27AlT44Mt\nd4Kmlltqdtnz5sWomG2b1mlocGDRtqHBAW3btK6iiMoTbNkTpsYHW+4ETS231ICyl7iMtRfj2Fsd\nJ+/46IRm5uY10qDe8mDLvmI0XqGww3YFXO4ETS23FHjZS17GOq/hjrdJerWklWZ2QtIfOec+kMdr\nt2zeMKLbvvSwJOn2t7wyz5euvSDLvnHn4je6dM7U+CDLnUJTyy0FXPZeTY91TezOuTfm8TpoEKbG\no0lKXpXTi6YYBIqp8WiKhKbHvHnReQoAXit5VU4SO+prYo/e951f0W2PXsXNsOG3kpexpikG9RSP\nIlg1z82wEYgSmx6psaOeuLcn0Ldgauw+rwqXJXbfy73m4e9rZm5ev3/TgcWxpxhF4HvZs1zzruet\n5rJeM9+veVnXLYgae2tVuMmpaTmdXRVu3+HJqkNLlCX2EMo9MzcvqUPsCff2DKHsWa551/NWY1mv\nWe2veY+ZpWVftyASu8+rwmWJPehyJ4wiCLrsBe1btayx17rsrZmljz8iyZ3tE4qTe9mxB5HYa78q\nXI9P8iyx177cPSTGHo8imNGgnHTOKIKgy17QvlXLGnuq/Utcj2WRhD6hsq9bEIm91qvCJXySZ4m9\n1uVOkCr2sS16YPmL9N/LXyK97dii0TDBl72AfauWNfbE/RP+1gqV0CdU9nULIrHXelW4hE/yLLHX\nutwJssbe1LI3tdyp9q9yJFVCn1DZ1y2IUTG1XhUu4ZM8S+y1LneCVox2h+SkJcceQtmzXPN+z1uV\nsl6zxP1LXo9lkYRF7cq+bkEkdqnGq8KlWCMiS+y1LXcKmzeM6L5PRm/Be7df3tf+Ppc9yzXPct6q\nlPWa9dy/5PVYFkmxqF2Z1y2Ipphay2ONCKbWA8lKXo/lHGNbpNGXS8+77Jw+obIFU2OvrazL0zK1\nHkiHpaDPILGXIcsaESUv0A94jaWgJdEUU39VdggB8BKJve4ShlEBQDsSe91V3SEElImBArmgjb3u\n4nb0mb1v1aBmZQ3uEELgGCiQG2rsPugxtR4IBmvw54bEDqAeGCiQGxI7gHpgoEBuSOwA6oGBArkh\nsQOoh4Q1+JEeo2IA5GdiT9TZ+fiJqAllqSO4xrbogY+9V5L04rd9rqAgw0diB5CP1o0uZhmuWDWa\nYgDkg+GKtUFiB5APhivWBokdQD4YrlgbJHYA+ShjuCJryaRC5ymAfBR9owvWkkmNxA4gP0Xe6IKb\nzqRGUwwAP9A5mxqJHYAf6JxNzZ/ETqcJ0GysJZOaH23sdJoA4KYzqflRY2dGGwCJm86klEtiN7Mr\nzey4mT1oZtvzeM1F6DQBgNQyJ3YzG5D0N5KuknSJpDea2SVZX3eRHDpN9h2e1KU3HdDzt39cl950\nQPsOT+YUXLpjf+Xh7+sL3/pfjl2iqq95U49d9TVv6vt9oTxq7K+Q9KBz7pvOuRlJH5F0TQ6ve1bG\nTpN9hye1Y+9RTU5Ny0manJrWjr1HSznxrWPPzM1LHLu0N3sdrnlTj131NW/i+72dOeeyvYDZGyRd\n6Zz79fjxL0v6Kefc9d32GR8fdwcPHlzagSb26NEb3q6nps6T3DLNnn665ueetvh3lj0R/T9/4aLN\nP3zytOad0wVPfzJ6/KNov2VmuuBpZ/uPLd7fte2f9Fyv51vHHr7w/yRJU088o+Oxu8XeMjA4JUma\nmx0+98mEcicdu9+y9Xo+7bF7livF81mueRHlTnvsXrEnPqfu5yXtsc+L9z/d5bz2Knu3fUt5r/d4\nPs3xi77mad7vNmx64SeOd3z9JGZ2yDk3nvR7eYyKsQ7bzvm0MLOtkrZK0po1a5Z+lLEt+sHsu2Uz\nT3U96ecNzEmSTs8v3j4ff3i1nm/f3rIsfn6ubf+k53o93zrG6bmBnsfuFvuZ32/bP82+aY/db9l6\nPZ/22L3Kleb5LNe8iHKnPXav2JOek7qfl7THbr8u7XqVvdu+aa75soEnNTj4VPTAzXasoPX7nkh3\n/GKveZr3u5s/v/OL5yiPxH5C0kULHo9KOtn+S8653ZJ2S1GNvZ8DrfuPI71/4ZbXRP+3TWW+9KYD\nmpya1keW/4kk6dqZP5QkjQwP6d7tl5/5vfvefZkk6cV/cO6dW3o91+v51rHbtR+7W+ypJJQ76dj9\nlq3X86nLnVWGa15EudMeu1fsic/1kPbYv/T+z0uSbn/LKzu+Tq+yd9s38Zq334hDippUc7r9XZr3\nXNHXvNexy5RHG/uXJV1sZs83s+WSrpV0Vw6vm5ttm9ZpaHDxJ+nQ4IC2bVrHsQM8dtXH59hdjl3w\nsOVal71kmWvszrnTZna9pP2SBiR90Dl3X+bIcrR5w4gkafldyzQzN6+R4SFt27TuzPYyjr1r/3Gd\nnJrWao5d+LEXHr/Ka97UY3e95gUPW048/sQeXTzzNQ1qNpq9nuPkpqrf7+1ymXnqnPuEpE/k8VpF\n2bxhRDryTEnSvdeV+9Vo84aRyi5wU4/dOn6V17znsSf2SCe+HC1vW0SSqeN7fcVoNGu80/aijx83\nAy3XbPS4gNnrVb/fF/Jj5ilQN3Ht75KZo0tfu6jV1jwXdyK2kkzo6x9VudZLw2avk9iBpVpQ+zNp\n6Ym5YUnmjLEtUUfpioskWfR/Th2niRo2e92PRcCAOsl6w4eGJZlFxrZUs75LCc1AdUKNHViqrImZ\ndcXL17Alf0nswFJlTcwNSzK1EDcDnVr2HM13awbK0m9SMzTFAEu1cWfniTZpE3MrmdxzY1TLXzHK\nuuJlGNui6z8fzaU8Z2JWCaNmykRiB5Yq/kM/te8GPXv+lJb1k5iramtGZ4HdKJvEDvSjV+0P/gms\nQ5s2dgAIrEObxF4HrVmID33O+04bwEuBdWiT2KvW1FmIQJ2kGTXjkXDa2Atce6NQgXXaAN4KqN8k\njBq7z7XewDptAFQvjMTu89obgXXaAKheGInd51pvYJ02AKoXRmL3udZb5Yp3oWO0ERoqjM7TrFO8\nq8YsxPx163eRONcIXhg19ibXeqmVduZzvwuQURg1dqmZtV5qpd353O8CZBRGjb2pqJV253O/C5AR\nid1n1Eq7Y7QRGozE7jNqpd353O9CvwkyCqeNvYl8Hw2UVdIyEj72u9BvghxQY/eZz7XSrHxeRqIX\n+k2QA2rsvvOxVpqHUBdPo98EOaDGHtt3eFI/fPK0nnjytC696YD2HZ4sZd+qZY29srJnTIB5lPvw\nw1P64re+l2+5S+g3yRJ7YeUuQdbYffo7J7ErumA79h6Vix9PTk1rx96jqS5cln2rljX2SsueIQHm\nVe6Zufm+9u+p4NE8WWIvtNwFyxq7b3/nJHZJu/Yf1/Ts3KJt07Nz2rX/eKH7Vi1r7JWWPUMCrHW5\nC+434b1+Vm2ueQGa08beYwTFyanpjrt0257md9LsW7WssVda9laiu+fGqPllxWjqm6vUvtwF9puk\nin1ijy6e+ZoGNbvob4X3ev/7l60ZiT1hCNnq4SFNdrhAq4eHztnW6Xf63bdqWWOvvOx9JkDvy51B\nYuzx38pyzUaPF/ytrB5eGW65C96/bM1oikkYQrZt0zoNDQ4senpocEDbNq1LfOks+1Yta+y+lr2p\n5ZZSxN7jbyXoche8f9maUWNPGEGxecOIJGn5Xcs0MzevkeEhbdu07sz2XrLsW7Wssfta9rzKvWv/\ncZ2cmtZqT8otpYi9x99K0OVOub8v7/VmJPYVo9FXyk7bY5s3jEhHnilJuve6y5f08ln2rVrW2H0t\nex7lrusfdZKesSf8rQRb7pT7+/Jeb0ZTTJoRFKzPAbB4WiCakdiThpCFOj0dWKr4b+XUsudovmnL\nVASkGU0xUu8RFKFOTwf6MbZF13/+IknS7W95ZcXBoB/NqLEnYX0OAAEhsUusaw4gKCR2iQ4jAEHJ\nlNjN7BfN7D4zmzez8byCKl1C56pPq7rlrallb2q5Jb9XcMwipHJn7Tw9JukXJL0/h1iq1aVztbWq\n2y0WPW6t6ibJ2/G8aTW17E0tt9R9FUQp7LKHVu5MNXbn3P3OuXoub5YT31Z1y1NTy55LuT2dF8E1\nP8vncpfWxm5mW83soJkdPHXqVFmHzcy3Vd3y1NSyZy63x/MiuObpttddYmI3s0+b2bEO/65ZyoGc\nc7udc+POufFVq1b1H3HJuq3eVtdV3fLU1LJnLrfH9y3lmqfbXneJid05d4Vzbn2Hf3eWEWDVfFvV\nLU9NLXvmcns8L4JrfpbP5W7OzNM++baqW56aWvbM5U6x6Fxd+byCYxahlTtTYjez10v6a0mrJH3c\nzI445zblElmN+LSqW96aWvZM5d64M2pTX9gc49G8CJ9XcMwipHJnSuzOuTsk3ZFTLEAYMty2D8gD\nTTFAEQq8bymQhCUFACAwJHYACAyJHQACQ2LPg6fTxwGEicSelcfTxwGEicSelcfTxwGEicSelcfT\nxwGEicSeFbfVA1AzJPasuK0egJohsWeVcFs9ACgbSwrkgenjAGqEGjsABIbEDgCBIbEDQGBI7AAQ\nGBI7AASGxA4AgSGxA0AaHq3iSmIHgCSereJKYgeAJJ6t4kpiB4Aknq3iSmJHbx61KwKF8WwVVxI7\nuvOsXREojGeruJLY0Z1n7YpAYTxbxZXVHdGdZ+2KQKE8WsWVGju686xdEUCExI7uPGtXBBAhsaM7\nz9oVAURoY0dvHrUrAohQYweAwJDYgdAwqazxSOxASJhUBpHYgbAwqQwisQNhYVIZRGIH7bFhYVIZ\nRGJvtjzaY/lgqBcmlUEk9mbL2h5LR139MKkMyjhBycx2SXqdpBlJ35B0nXNuKo/AUIKs7bG9Phjq\nnkha3zTmnoq+aWzcWf+Y02JSWeNlrbHfLWm9c25M0tcl7cgeEkqTtT3W1446vmkgcJkSu3PuU865\n0/HDL0iih8YnWdtjfe2oY0ggApdnG/uvSvpkjq+HomVtj/W1o87XbxpASolt7Gb2aUk/3uGpG5xz\nd8a/c4Ok05I+3ON1tkraKklr1qzpK1gUIEt7bGu/e26MkuKKUT/aqleMRs0vnbYDAUhM7M65K3o9\nb2ZvlvRaSRudc67H6+yWtFuSxsfHu/4ePONjR93GnVGb+sLmGB++aQApZR0Vc6Wkd0r6Wefcj/IJ\nCSiYr980gJSyrsf+PknnS7rbzCTpC86538gcFVC0pG8aIQ+HRPAyJXbn3E/kFQhQG92GQ0okd3iB\nmadAO4ZDwnMkdqAdwyHhORI70M7XiVdAjMQOtPN14hUQI7ED7VghEZ7LOtwRCJOPE6+AGDV2AAgM\niR0AAkNiB4DAkNgBIDAkdgAIDIkdAAJDYgeAwJDYASAwJHYACAyJHQACQ2IHgMCQ2NNo3Sbtoc9F\nt0mb2FN1RADQFYk9SbfbpJHcAdQUiT0Jt0kD4BkSexJukwbAMyT2JNwmDYBnSOxJuE0aAM+Q2JNw\nmzQAnuHWeGlwmzQAHqHGDgCBIbEDQGBI7AAQGBI7AASGxA4AgTHnXPkHNTsl6aE+d18p6bs5hpMn\nYusPsfWH2Prjc2zPc86tSnqRShJ7FmZ20Dk3XnUcnRBbf4itP8TWnybERlMMAASGxA4AgfExse+u\nOoAeiK0/xNYfYutP8LF518YOAOjNxxo7AKAHrxK7mV1pZsfN7EEz2151PAuZ2bfN7KiZHTGzgxXH\n8kEze8zMji3Y9iwzu9vMHoj/f2aNYnuXmU3G5+6ImV1dUWwXmdlnzOx+M7vPzH4n3l75uesRW+Xn\nzsyeZmZfMrOvxrH9cbz9+Wb2xfi83W5my2sU261m9q0F5+2lZce2IMYBMztsZh+LH2c/b845L/5J\nGpD0DUkvkLRc0lclXVJ1XAvi+7aklVXHEcfyKkkvk3Rswba/kLQ9/nm7pD+vUWzvkvT2Gpy350p6\nWfzzhZK+LumSOpy7HrFVfu4kmaQL4p8HJX1R0k9L2iPp2nj730n6zRrFdqukN1T9novj+j1J/yLp\nY/HjzOfNpxr7KyQ96Jz7pnNuRtJHJF1TcUy15Jz7rKTvtW2+RtKH4p8/JGlzqUHFusRWC865R51z\nX4l/fkLS/ZJGVINz1yO2yrnID+OHg/E/J+lySR+Nt1d13rrFVgtmNirpNZL+IX5syuG8+ZTYRyQ9\nsuDxCdXkjR1zkj5lZofMbGvVwXTwY865R6UoSUh6TsXxtLvezCbipppKmokWMrO1kjYoquHV6ty1\nxSbV4NzFzQlHJD0m6W5F366nnHOn41+p7O+1PTbnXOu8/Vl83m42s/OriE3SeyS9Q9J8/PjZyuG8\n+ZTYrcO22nzySrrUOfcySVdJ+i0ze1XVAXnkbyW9UNJLJT0q6S+rDMbMLpD0b5J+1zn3gypjadch\ntlqcO+fcnHPupZJGFX27/slOv1ZuVPFB22Izs/WSdkh6kaSXS3qWpHeWHZeZvVbSY865Qws3d/jV\nJZ83nxL7CUkXLXg8KulkRbGcwzl3Mv7/MUl3KHpz18l3zOy5khT//1jF8ZzhnPtO/Mc3L+nvVeG5\nM7NBRYnzw865vfHmWpy7TrHV6dzF8UxJ+k9F7djDZta6S1vlf68LYrsybtpyzrmnJN2ias7bpZJ+\n3sy+rahp+XJFNfjM582nxP5lSRfHPcbLJV0r6a6KY5IkmdkzzOzC1s+Sfk7Ssd57le4uSW+Of36z\npDsrjGWRVtKMvV4Vnbu4ffMDku53zv3VgqcqP3fdYqvDuTOzVWY2HP88JOkKRX0An5H0hvjXqjpv\nnWL72oIPalPUhl36eXPO7XDOjTrn1irKZwecc29SHuet6h7hJfYeX61oNMA3JN1QdTwL4nqBolE6\nX5V0X9WxSbpN0dfyWUXfdH5NUdvdPZIeiP9/Vo1i+ydJRyVNKEqiz60otssUfe2dkHQk/nd1Hc5d\nj9gqP3eSxiQdjmM4JmlnvP0Fkr4k6UFJ/yrp/BrFdiA+b8ck/bPikTNV/ZP0ap0dFZP5vDHzFAAC\n41NTDAAgBRI7AASGxA4AgSGxA0BgSOwAEBgSOwAEhsQOAIEhsQMxM7vDzP7UzP7LzP7HzK6oOiag\nHyR24Kz1ilbW+xlJb5X0porjAfpCYgckmdnTJa2QdHO86TxJU9VFBPSPxA5EXizpkHNuLn48pvot\n5AakQmIHIusVLazVMqZo4SjAOyR2IPISLU7s60WNHZ5idUcACAw1dgAIDIkdAAJDYgeAwJDYASAw\nJHYACAyJHQACQ2IHgMCQ2AEgMP8PZOO3STJYtNEAAAAASUVORK5CYII=\n",
 75 |       "text/plain": [
 76 |        "<matplotlib.figure.Figure at 0x1a66d40be80>"
 77 |       ]
 78 |      },
 79 |      "metadata": {},
 80 |      "output_type": "display_data"
 81 |     }
 82 |    ],
 83 |    "source": [
 84 |     "sigma_2 = 1\n",
 85 |     "\n",
 86 |     "y = s[0,:] + np.random.normal(0,np.sqrt(sigma_2),T)\n",
 87 |     "\n",
 88 |     "plt.stem(n,s[0,:])\n",
 89 |     "plt.stem(n,y,linefmt='C1-',markerfmt='C1o')\n",
 90 |     "plt.gca().set_xlabel(\"$n$\")\n",
 91 |     "plt.show()"
 92 |    ]
 93 |   },
 94 |   {
 95 |    "cell_type": "markdown",
 96 |    "metadata": {},
 97 |    "source": [
 98 |     "As we saw in class, the ideal detector is a matched filter receiver, where we multiply the received signal by a sine and cosine at frequency omega and integrate the results. This gives us 2D points corresponding to the sine and cosine component of the received signal. Let's plot a bunch of realizations of the output of the matched filter."
 99 |    ]
100 |   },
101 |   {
102 |    "cell_type": "code",
103 |    "execution_count": 57,
104 |    "metadata": {},
105 |    "outputs": [
106 |     {
107 |      "data": {
108 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJztnX2QXNV55p93WleoB6fUIigxNAxS\nWCIlrIwmzCK5VLVlkRgRtMBYGGRCskltsqrsxpWFJVMZYpUlXGSZrMqB7Nq1DnFcTgoKCwxui4is\nHEdyeZdEBCkzQpaRshgsoRZVViwNjpkG9cy8+8f0bd3uPud+9L39eZ9flUrT3bfvPf113nPej+cV\nVQUhhJD0MdDpARBCCOkMNACEEJJSaAAIISSl0AAQQkhKoQEghJCUQgNACCEphQaAEEJSCg0AIYSk\nFBoAQghJKYs6PQA/Lr/8cl2xYkWnh0EIIT3D4cOH/1lVl4c5tqsNwIoVK3Do0KFOD4MQQnoGETkZ\n9li6gAghJKXQABBCSEqhASCEkJRCA0AIISmFBoAQQlIKDQAhhKSUrk4DJaTVFCaL2LXvBM5Ml3Bl\nLouxTaswOpzv9LAIaQs0ACS1FCaLeOj5oyiV5wAAxekSHnr+KADQCJBUQBcQSS279p2oTv4upfIc\ndu070aEREdJeEjEAIvIlEfmBiHzH8vhHROQdEZmq/Pt0EtclJA5npkuR7iek30hqB/BlALcGHPN/\nVHVt5d9nErouIU1zZS4b6X5C+o1EDICqfhvAuSTORUi7GNu0ClknU3Nf1slgbNOqDo2IkPbSzhjA\nh0XkiIj8tYhc38brEmJkdDiPR7esQT6XhQDI57J4dMsaBoBJamhXFtA/ArhGVX8sIrcBKAC4znSg\niGwDsA0AhoaG2jQ8klZGh/Oc8ElqacsOQFV/pKo/rvz9IgBHRC63HPuEqo6o6sjy5aEkrQkhhDRB\nWwyAiHxQRKTy902V6/6wHdcmhBBiJhEXkIg8DeAjAC4XkdMAdgBwAEBVvwDg4wD+k4jMAigB+ISq\nahLXJqRVsEqY9DuJGABVvTfg8c8B+FwS1yKkHbBKmKQBVgITYoBVwiQN0AAQYoBVwiQN0AAQYoBV\nwiQN0AAQYiDJKuHCZBEbJvZj5fhebJjYj8JkMalhEhILykETYsAN9DaTBeTNHsoNOvjxe7Mozy8k\nvTGYTLoJ6eZszJGRET106FCnh0FIaOqzh2zkc1m8NH5zm0ZF0oSIHFbVkTDH0gVESIKYsodMFKdL\ndAmRjkMXECF1xCkAi5IlpKBLiHQW7gAI8eC6cIrTpZoJOuwqvZksIdYXkE5BA0CIh7gFYGObVsHJ\nSOTrujuH7YWjuPahF7FifC+ufehFbC8cjXwuQsJCFxAhHhIpAGsir+LKXBbbC0fx5MFT1fvmVKu3\nHxldE/2khATAHQAhHuIWgO3ad6Ka8hkWt77g6ZffMj5uu5+QuNAAEOLBVADmDAhmLsw2ZO2YCrzC\n7BQ2XHsZ8hWDkhGpupjmLCnZtvsJiQtdQCSV+GX6LHEGqnGArDOA2XnF+ZkygItB4UMnz+G5w8Ua\ntdAHdk+F8v7cPbLQ6a5ebdSPDRP7KUdNEoeFYKTrSVqX31SslXUyuOvGfM2kDgACs0s/IxJrZS4C\nRH161smwZzEJhIVgpG+Im5Zpwpbp8/TLbzXcb5uj47pl/J6eEXMWEdNFSdLQAJCuphW6/DY/fZRJ\n3TZJx2XZoIPvPXobbGenHDVJEhoA0tXEScu0qXDaMnrCTupORnDvuqsbgsVJcH6mjBXje607D8pR\nkyShASBdTbNpmX6uI5vUc9hJ/dLFi/DI6Bo8umVNNZunHTQrR02IDRoA0tU0q8vv5zoaHc7j0S1r\nkMs61ceWOAMYueayUJP6O6WFjKDR4fxC5e9Aa9xBXvK5bE0AmD0GSBIwDZR0Nc3q8odxHb0/O1/9\n+/xMGQ89fxR33RicYePdfTRT+BUVAarS0YXJIh5+4Vg1LRWgoBxpHhoA0vWMDucjN2IZsKRpupO3\nbYfw1MFTvrn8AtTsPqIEZZ0BwTyAuYgGwx2zX68B7+6GkLDQAJC+oH5yNE3+XteRrfAqaGpWoJqB\nNDqcx5W5rG8Rl5vvn89lMXNhtmbl7rI4I7gwZ7/yxtXLsWFif2CxGDOESFQSiQGIyJdE5Aci8h3L\n4yIi/0NEXheRV0XkF5K4LiEutkYsIgurdq8PvTBZtKZZ2vAe77pctheOYubCrO/zVBeyhsY2rcK0\nYfIH4Dv5A8Bzh4uBkz/ADCESnaR2AF8G8DkAf2l5/JcBXFf5tw7A/6r8T0gi2Fa/qsDjW9fWuEZ2\n7TsRSbDTVPVbKs/VKHf6UZ5T7Np3InC3YLt2mA5jzBAizZDIDkBVvw3gnM8hdwL4S13gIICciFyR\nxLUJAfxXv/VFY36TsCnjKAkxtjPTpaYm6DDXzmUdSkSQpmhXGmgegFfT9nTlPkISwW9y9e4O/Nw/\nrpson8vWuI2SyPUfEMGhk35rJPuY/B57fOtaTO24BQCYFkoi064gsOk3Z1zaiMg2ANsAYGhoqJVj\nIj2MSSBu2aBjDLLWp22avnhudo+bceSe/4HdU1iadeBkBOUAX70fc6p4KqTLyMvG1csbMpPqReHq\nA+BMCyVhaZcBOA3gas/tqwCcMR2oqk8AeAJYUANt/dBIr2Ga8Fwp5nr1znrfuDVWANRM+O9emK1O\n+NOlMpwBwbJBB9MzZVyZy+IHPyqhPG88lVVBNOqX+dLFGTx3uFjzPAFw1435hpiGX9EbITba5QLa\nA+DfV7KB1gN4R1XfbtO1SZ9hmvDU87+73ayvngXssQIBqrIR06Vyw2q/PK8YXLwIb05sxkvjN+MD\nSxzjeZYNOnhzYnPkLKN6nIzAyQwYX+eB42dr7kukjSVJJUmlgT4N4O8BrBKR0yLymyLy2yLy25VD\nXgTwBoDXAfwZgP+cxHVJOgma2BQLk/9L4zc3rIBN0hK2FbvfdW0pne79cVMyd338hqrkhN84/K7F\ntFASRFJZQPeq6hWq6qjqVar656r6BVX9QuVxVdXfUdVrVXWNqrLLC2mapVnz6tuLzUi4OkDeQG9Y\n14x3Qs0Nmsfg3r9x9fKQZ20kn8tWi8yCxgE0r5dECCuBSc8RRrXZb/VbLy0RpsrWO6EWJouYtqzO\n3azNejdNFFzjMbZplbFzWf3E3qxeEiE0AKTnsLlfXKKufk0TrTMg+MCSRdWgrzuhugFoW3q+67aJ\n4393jUeUiT2sXhIhXmgASM/hV1Gbj7j6ddM9S+W5asWv3zlskhMuCmDF+N5YPYO9xoMTO2klNACk\n57C5RqJWwxYmixh79khVztmdsIvTpZrqYe8KPKyUQ5zqYZv7ylT7QONA4kADQHqOpHzeO/ccs2r5\nF6dLGPvqEUBRPaY4XQqdMeTi7gQGBHAv5Z4jV1dvANjdV2GLvWgkSBREE9A5aRUjIyN66BAThkhr\nWDG+t+XXEACPbV1rjTGcnynXuJ42rl6OA8fPNkzgtkC1m+4KmPsF2GIZpH8RkcOqOhLmWO4ACGkh\nV+ayxrhBeV6rshVzqsg6GWxcvRzPHS4aV/lhir2CrkOJCFIPdwCkJ0nC1TH8mW8YtYOSwo1LuDIV\nQdgCx64gnGkHkMs6uPSSRThTqWIOg3fXQPqPKDsANoUnPYfr6nClG9yVbVQFzB23Xw8nE120YZml\nCMyLV4YibEWuLXDsSknXF3s5A4J3L8xW34ewUCKCuNAAkJ7DT/wsCqPDeez6+A3VquBlgw5yWSdQ\nx2fzh67wPWbZoINz776P+3dPYcX4XkzPXIAzEGxobIe41cVLnIs/11zWwQeWLGpKoZQSEcSFMQDS\ncyQpfmbLs/erDj5w/CzuWz9kbSBf71Z690K4jl6AomSQGH2vPNcQ3H1/dt63HkGABlVT9zqUiCAu\n3AGQnqMd4mdBDWYeGV2Dx7auTaRZDAA8umUN3rPoS5fKjZO93+Sfz2Xx5sRmTO24pWaHY1JHJemG\nOwDSc4TVyAmLLaD88AvHfBvMuLuHleN7I2v9e3HF33btOxG5Z3A99e8DK4mJHzQApOdIohDMnfTr\ni7u8qZI7br8+lKFpptm76Xw2w7bEGfDNVsqIYF6Vef4kMjQApCeJs7KtL5iqX727AWU3VTLI0Jgm\n7jDUaw7ZDBsA3/PPq+LNic2Rrk0IQANAUkiQoBtwMaAcxtDUT9y5QQfvleeMAV3AX7fI73oPPnPE\nmCrKrB7SLDQAJHWEyRaKOqn6TdxJFK25xycZ+4gKdYb6DxoAkjqCfPZJT6pJBWI72fglrBgd6S0o\nBUFSR2GyaJVnyIjgs/fckMhqvp9WzGHE6Eh3QCkIQnwYHc5b0zbnVX0n9LASFEnJVXQLSRbfke6B\nBoCkElsBl5/v3yZBcf/uKWyY2F8zuSclV9EttKP4jrQfGgCSSkziakG+f7/Vbv0Kv99WzM28X6T7\noQEgqWR0OI9Ht6yJJJMQtNr1rvD7bcXczPtFup9EsoBE5FYAfwIgA+CLqjpR9/hvANgFwN0jf05V\nv5jEtQlplqjZOWEKvtwVftJyFd0AZSX6j9gGQEQyAD4P4KMATgN4RUT2qOp36w7draqfjHs9QjqF\nNw3Tlkbq1Qlyj+2HLCDSnySxA7gJwOuq+gYAiMhXANwJoN4AENLzuKtgU/9dCrGRXiMJA5AH8Jbn\n9mkA6wzH3SUi/xbAPwF4QFXfMhwDEdkGYBsADA0NJTA8QpKHK/zo9FNdRL+QhAEw9TGqT7N+AcDT\nqvq+iPw2gL8AYKweUdUnADwBLBSCJTA+0iTd9IPtprG4cIUfHlYSdydJGIDTAK723L4KwBnvAar6\nQ8/NPwPwRwlcl7SAMDLJ7f7BcvLofWx1EQ8+cwQAP8dOkUQa6CsArhORlSKyGMAnAOzxHiAiV3hu\n3gHgtQSuSxLGW70K2GWSk77mhon9WDm+t6GYyqUXiqrCvI40Y6t/mFPt6QrpXie2AVDVWQCfBLAP\nCxP7M6p6TEQ+IyJ3VA77XRE5JiJHAPwugN+Ie12SPFFkkpMgrFxCtxdV9ZvsQyvwq3/oNmOeJhIp\nBFPVF1X1Z1X1WlX9w8p9n1bVPZW/H1LV61X1BlXdqKrHk7guSZZWyCTbKEwW8eAzR0Kt7Lu9qKoX\ndiidxlRJ7KVbjHnaYCUwqRI0oSZVyOSumE3NTYDGyaDbZQhsNQFx+/v2E24lcUZMOSPdY8zTBvsB\nkCqm6lU3EFzfvjAOQa6m+smg21MuMyJGY2ab7HqZwmQRO/ccw3RpoUfxskEHO26/PtRn0Q1NbUgt\nNACkSrsmWr/tvm0y6OaUS9tOxnZ/r1KYLGLs2SMoz198Xednyrh/9xQeev5VLHEymJ4pG7833jTe\n3KCDSxYN4J2S+VjSPmgASA31E62b3ZKkQbB15MqI9KTAWN7yemyS073Krn0naiZ/L6XyfLUHcn2a\nbn0a7/mZMrJOBo9tXdtzn3W/wY5gBIC50Aowb9fjTtI2GYX683Zj8ZeJsK+n11k5vtfaSMdERgTz\nqhiwuMjYTaw1ROkIxh0AsRZaLXEGrNktcSa2MK6mXir+6vYYRVLkBh2cnymHPt6d9G2usOJ0CYXJ\nYt+9T70EDQCxpjHaArVhU/b8VvBBPn2/1MpunDC6OUaRBIXJIn783mzi531g9xQOnTyHR0bXJH5u\nEgwNAImcg+3N0rFN8s2s4L3nsrkamC/ePHFcan7+/zgogKcOnsLINZdVvzf9vpPqJmgAiDUom8s6\neH923pqyZ5vkD508h6dffqth6++3gjf50W1jJdGJ61JrpeFVADv3HAOAnnH79QssBCPWQqudd1zv\n2wbQ5qZ58uCp0EVeLmFkKJgv3jxB1cpBWkZRDW/UCojpUhk79xxjRXWb4Q6ANAQxl2YdiCz4Z/22\n4c2sCm0Tid+5pPI8ugOax09PKczuYGzTKjyweyp0FlAzziK3uMw0RtIauAMgABZ+6C+N34zHtq7F\n+7PzOD9TDhQ2a8YdU5wuRVph5nNZvDmxGS+N38zJPwZ+ekphtIxGh/NNTepJQLdf66ABSAFRpIqj\nCJttXL088lYfaDQqhcki3n2/McOELp/k8NNTsq2wXWPtfm+WDTrtGGoN/A60FrqA+hzT9n7sq0ew\nc88xYyl+WOnlwmQRzx0uNr0q9BoVU/A3isYMCcavVsGvyb17f6eE7e66sb/TazsNDUCfY1rRl+e0\n6m+t9/cuzTpGX+zSbO3qL0zQNojidMl6nsHFi3x/+H7pp0wjNGOrVRjbtKpB46db2Pvq2zhw/Cw/\nzxZBA9DnhAmgua357t89ZT2mXtgyiRVhRqSpZi9+6afPHS4yjTAio8N5/MHzr3alATg/U65WH/Pz\nTB7GALqQJNsLhg2gBSlXTtdJACQhdTynanUh1e84vNjiFE+//BbTCJtge+EoZipCbu0gzjeHn2ey\ncAfQZSStgWPS+G+GekPSaqnjf3l/Fmsf/kakOIVf7QFdQ2YKk0U8dfBU266XdTL4haGlOPjG+aa/\nQ+7nz880PjQAXUbSGjj1wb/BxRm8eyGaMRCgJhOjMFm0NkFJirl5e5zCT07aNKbcoFPj4y5OlzD2\n7JHq+fodv4ly174TbUvvzIjgrhvzeO5wMdZ3Z0AE2wtH6e5LAMpBdxk2yV0B8ObE5tjn3zCxvyn/\n/eNb11azRdwuYe3GlQ+2yS+7k0v9/YBWteq95LIOpnbc0o6hdwzTe+Xt8ub3XRABok4PTkZw04pl\neOl75xoeG3QGsHhRxlrwZRwDzN81v+9gkt3repEoctCMAXQZSTRA94shNFNVuWzQwUPPH61OFp1a\nMrh56Q/snsISZwC5rFMjUfHI6BqjdIVp8gfslaf9hGlH6X5+rjE3IQDuWzfk28jdiALHzvyL8aGZ\n8nzk99z2XfP7DhanS3hg9xS2F45GulYaoQuoyzD57KMUwwTFEGzuExtORqCK2DGEJBBczD6ydZUy\npTr6ZTf1O0EGX9G4mhYA960fwiOjazByzWW+dQL1lD2uu05SrzJKzCSyAxCRW0XkhIi8LiLjhscv\nEZHdlcdfFpEVSVy3HxkdzvsKsAURVMlrqgj1Y9GAdMUPGmhc9YXNCLFVsHaisrXdhNk5uu4gwcJ7\nsjTr4KmDp7BhYj8A4KXxm/H41rXRdwMRiZodFHS8AswYCiD2DkBEMgA+D+CjAE4DeEVE9qjqdz2H\n/SaA86r6r0TkEwD+CMDWuNfuV+I0FwnKq68PCucGHaja3SE290m3ECbDZ8ft12Psq0dQnrtoQpyM\nYMft13diyG0lTBaYLbbiulK8DVu87/O5d99P7Pvx+Na11fOH2W24u5QDx8/6Hk8hOX+ScAHdBOB1\nVX0DAETkKwDuBOA1AHcC2Fn5+6sAPiciot0cge5RbC4e70rQZGCaDQ53mlwlPuGXDZKWlo0mvK/d\nFMD3uhdt8YInD57C3lffxo7br6/p4Tv8mW8kZgAeev4oHt2yBmObVgW67LwuKmDB7WlTKqWQnD9J\nGIA8gLc8t08DWGc7RlVnReQdAD8J4J8TuD7x0EwMwSbG1u1knYwxPmFKm+33lo1+eF+7327Jb7V8\nfqbcYFjriwPjENadZ8rwGR3O49DJc3jq4CmrcSNmkjAAJldcvTEOc8zCgSLbAGwDgKGhoXgjSyFR\nV7u2TlyXLs5gpjwXOQ2wleSyDi69ZFHN63rAslrk1t+MnyEMShDwTtKtqB8I+swerwv4e/EGrNO2\ny4tDEgbgNICrPbevAnDGcsxpEVkEYCmAxkRhAKr6BIAngIU6gATGlzqirHZtYmwzF+balu6ZswjQ\n1bPzjkZ1UJvPmFv/6IRp+uK62FqRFeZ+ZjYj5Bof23c7zbu8ZkkiC+gVANeJyEoRWQzgEwD21B2z\nB8CvV/7+OID99P93B7ZVV7s+nHwui6kdtyAfMGEPiPmH76dzT6IxOpzHfeuHArNrwk7+UbJ63M/M\nL0vNrzkRaY7YBkBVZwF8EsA+AK8BeEZVj4nIZ0Tkjsphfw7gJ0XkdQD/FUBDqijpDH6ia63GO1GP\nbVrlO2HMK4zFbXHTZtNCWIHBR0bX4LFKRk5cHtu6NtCwA7WfmffzNOEq19IIJEMihWCq+iKAF+vu\n+7Tn7/cA3J3EtUiyJCDqGYmMCOZVG3y0biDvSR9hMm9zEm9A0rb1p1jYAlEFBoOaxETBz6/v9em7\nBqo+vmPaic6pUvcnIagFlHJs2kOtRIBq/UG92uf2wtGGbA4bbv66CZteUBp3B7YU36jvX1TyAT59\nYGFBsP5nluEfT73T8FldsmjANzbkN/40Qy0gEhpbsLSVVbKKhbTC6VJj43nXBRHGdeC3uozS27jf\nidp0x905xQ30npkuBVaez6nipe+dM35WIvB9LjO94kMDkHJsQdTNH7oi9rmjeJe8k/PocD7Uyk6x\nsLrdXjja4N9uptNYP1KYLGLA4uczGX935Z+E+yc36DTEaKIwPVPGo1vWWJsPMdMrPjQAKac+6JYR\nqXbXikNGpLqSD/vD907OhcliqOcVp0t48uApFKdLNbuJnGUHk6ZJw53MTdr7tkypJFb+LqqNcZgo\nXJnLYnQ4j8/ecwMzvVoEDUCKsGWCjA7nqzsBd7KI2+xliTNQLdIK69LxZiTFKTQqVQrY0j5p2Cbz\njIg1FhJWhycM06VydTfhGueweJsQMdOrdVAOOiUEZYIkufIDUO065l7H1Kylnh+9V0ZhsojR4Xxs\nF8Q7pTIeqzSxSWsWkM3dNa9qfR8GZCHl1obbeOevjrwdWLzn7iabQVGb4cMir9ZAA5ASglpNttI3\nXirP4cDxs3h0yxrf9MJ5BR585ohV3iEKrvsgzZNGGGFAL4XJou/k79XhGbnmspo2m/U4GalRX41K\nmB0jiQ9dQCkhKCjajG/cGQgf1jszXaoGd/2eNacaOy01ba4eG1GrpP0ypNyUS6/WlG3yBxB68s9l\nndS76joJDUBKCGo1GbZRTEak6ofddfcNoVdq3qBs0oHYQWeA/mEDUX3nfm63jauX19xOasf4TqlM\n/34HoQsoJQTJRNeriNrWb3OqDZK8YQqGvDHlsU2rGhq0hMEZEEBqV5dZJ4P/xgnDSv3nahNUC5JW\nOHD8bM3tqK1FbdBV11loAFJCGJlo7w/Rr0GMqVPUzj3HfIOC73gec6/x8AvHcL6iKS8Co/R0vXRE\n0GsgtYSRgXCP8cNd8btpnabmMlHxZvp4x8vPt31QCoIYCSMFIEBNU/bCZBEPPnPEmELqbTto+oGH\nkW7g5BCdMDIQYbvB5bIO3r0wW7MDi2MENlx7GZ76jx+u3qZ8RzJEkYLgDoAYqW8laMJtul3fetHm\nagqzGrVN8FEFzcgCYSqiw/rzTTu8OMvH7/+wtvDPtHgwdXcjyUEDkGKCVtSuSyjIHVSv4uime9af\nd8PEft9UVD9fcFAaKzETJhU0KX9+PQMC/Mq6IavCq9etZKtY9h5HkodZQClle+EoHtg91SChYAoG\n+mn1S+W53nMAwEvjN+PNic01qYNx9Hmo7dMcYVJBbcfE5VfWLTRuv3Sx+VxuZlhQEWIne1b0OzQA\nKaQwWTRKLtvUMm2dokz+Xz/FzaBUVD/iPDfNhEkFtR0TtxjLzRxyMuZpxl3wBxnxdy/MsgFMi6AL\nKIX46ezYfoympts2t4HtHEGpqCb8sk5YMJQcNvdbnJ4A7vfgHUt2mHt/kAuqPKd09bUI7gBSiN+K\ny29F7YrGXZnL4sx0KbJMb9TCpHppYsVFITIWDIXD+x4Gufrqqf+8IhR+A7j4PUiiCJGuvtbAHUAK\nsa246vOy64PEG1cvrxF0iyIz7GJaadqC0SbfsIKdoKIQN3jufl6FySLGnj2CeUug1m93FqUI0bYT\nqDciTAlOBhqAFGL6QQqA+9YP1aRdesW+XN19E7Y+v+556n+owMV0z6V1ueXeQHK/BH47OVkl9R7a\ntH9yWQdTO27xfY1RihBttQD1CxOmBCcDDUAKCfOD3LnnmK/Yl5d5Vbw5sbnhftMPdezZIzVyDqbc\ncneFGlXNshvp9GSV1HtoMxjTpTI2TOzH2KZVvruy+p2fqQm89xi/7yZTgpODMYCU4ipzPrZ1LQDg\ngd1TNU1igrTevdhaCz74zJGGH2p5XkNpANn6yfZa4LfTvYmTeg/9DEaUuAIQHJdwv5v1acQu/bIz\n7AZoAFJMnAChi2kyCSrsCYMrEtbrSpFJTla2jm5+JPUeBgVqoxi1uEaRKcHJEcsFJCKXAdgNYAWA\n7wO4R1XPG46bA+CqTZ1S1TviXJckg98PcdmgUxVqs7Fs0MGO269vmEzidherDxD20oRfT1IumDiu\npCTewzCB2rBGLa5RbCadmJiJuwMYB/C3qnodgL+t3DZRUtW1lX+c/LsEvx/ijtuvh5Pxz/sbXLzI\nOLH4/ZCdAWk4rzMgWDbo9Owq34+kXDCddiUBF10ztgKxsEYt7gq+H3aG3ULcIPCdAD5S+fsvAHwL\nwO/HPCdpE36r0zgrPtt5MyLYdfcN1fOmIYUvTFAzDN3k9467Ag96fpisqV7fGXYLcQ3AT6vq2wCg\nqm+LyE9ZjlsiIocAzAKYUNVCzOuSBAiTn+0nBmdbsdnO612lpeHHWz+ReaWzo9JNGVFxjZrf8zud\nNZU2AvsBiMg3AXzQ8NCnAPyFquY8x55X1WWGc1ypqmdE5GcA7Afwi6r6Pcv1tgHYBgBDQ0M3njx5\nMvSLIdEJs9pqRqc97YU6Sfc3SItWfpj+BcSfKP0AYjWEEZETAD5SWf1fAeBbquq7DxSRLwP4K1X9\natD52RCme+j3CT3p1xc0kdGomlk5vteoUyWAsdaENNLOhjB7APw6gInK/183DGYZgBlVfV9ELgew\nAcB/j3ld0mb62efaCrdDkM++mWKmfv4MXLrJ1ZUG4mYBTQD4qIj8PwAfrdyGiIyIyBcrx/wcgEMi\ncgTAASzEAL4b87qEJEYrMmyCMl26KajbTfRD8V8vEWsHoKo/BPCLhvsPAfityt9/B2BNnOsQ0kpa\nMRnb9JY2rl4OIN5K16+vcq+7iJLKmiLhoBYQST2tcDuMDudx6OS5msY7CuC5w0WMXHNZ06mUNnfV\noZPnapRaezl7Jg2urm6BUhDhXMCdAAAOHUlEQVQk9bTK7XDg+Flrx7QwxUwm6Qebu+rpl9/qeKEY\n6T24AyB9T5BrpL7oLSNSM3m2KhBsW+kWJovYuedYjSCfu6K3SWywoTppBu4ASF8TVvDO7XaWdTLV\nybQZcTwvNheSX5Nzd7w2mWxbF7ao3dkIAbgDIH1OlHTLZlIzvbuLpVkHIsD0TLnaQW33P7zV0FfB\nbXJuOmeQkN6cKrJOpiF2cNeN+ZoYgHs/s2eIH9wBkL4mSoZP1Gyg+t3FdKmM8zPl6k7jucNFLF7U\n+BNzm5xHuZaLGyuojx08MrqGAmkkMtwBkL4mSoZPmGNN/nkbfiv54nQJK8f3NsQkbGMAACcjDZ2z\nvDB7hkSFOwDS10TJ8BnbtArOQKNUtVelcuzZI5G6pflhikmMbVoFmwj3pRb5bUKahTsA0tdELiyq\nn30FOHTynK8sdly8cYbR4Tzu3z1lPC7I8PRDIRhpLzQApO8J6xrZte9EQ7/i8pzWFHO1Cq/vP29x\nAwlgDR5TRpk0A11AhFSwBWBbPfkDtXEGVy7CNA5b8LgbOoaR3oM7AJJa6l0mS7NOYv59G8sGHbxX\nnvdN1/yrI29bnx81U4mFYMQPGgCSSrYXjta4dorTJTgZgTMgDXn7fgwIEOFwnJ8p1zwvb/DV+xkh\nP5VRyiiTqNAFRFJHYbJo9OuX5xQfWLLIWlXr5VfXD+H7E5vxx/esbcgycjKCXKXa13auea1N6wyL\nrbCLMsqkGWgASOp4+IVjVr/+9EwZ8yG65B04fhYAjKJuuz5+A6Z23ILHt67FB5cusZ6jPKfYuedY\nw/2DjvlneenijG+zGBaCkajQBURSRWGyWHXDmHBdJkEpn17fuinLyNTy0cR0qVyT2VOYLBpdUAMC\n/OHH/NtqsBCMRIU7AJIq/LJiBAuuFJM7pZ4g33qQpo+Xh1+4uAswpaICCwJynNxJ0nAHQFKF38r+\nvvVD1Un20MlzePrlt4wyy1I5z4aJ/di4ejkOHD/bUHwVJfvm/MzFXYDtedM+uxZCmoU7AJIqbEFZ\nEeCR0QUXS2GyiOcOF60a+97MoScPnjJKTUfNvnF3JrbnDYg0LUtNiA0aAJIqrJO65+4o7pt63OKr\nMG4kL8XpEgqTRWxcvdyoBTSnGqs3ASEm6AIifYefJo5NZiHvWXnHLZ4qTpcauowJgiuKx549Aoj9\nuKDeBIREhTsA0lcEdQALky+fRPGU69N/afxm5HPZUHIS5Xk1BoC9sLKXJAkNAOkrgjRxwuTLR3Xf\n2MbhkuSkzcpekiSxXEAicjeAnQB+DsBNqnrIctytAP4EQAbAF1V1Is51CbERRhMnKF/efSxs4xcT\nxekStheO4sDxs4mJybGylyRN3BjAdwBsAfCntgNEJAPg8wA+CuA0gFdEZI+qfjfmtQlpIClNHNdI\nuPGEZnoBPHnwVKTjnQHBPIA5QyHYskEHO26/3ioFHbYPAHsGEC+xXECq+pqqBunN3gTgdVV9Q1Uv\nAPgKgDvjXJcQG81q4hQmi9gwsR8rx/diw8T+aszA9eO3CjctNZ/LYtfdN+AnLjGvyQYt3cCCYh7N\nHkvSgWgI3ZPAk4h8C8DvmVxAIvJxALeq6m9Vbv8agHWq+knLubYB2AYAQ0NDN548eTL2+Ei6iLrK\nNck2ZJ0M7roxXy3yGhCxppDGxc0QyvhcQwC8ObG54f4NE/uNu5OMCOZVa16/7VjArEpKehMROayq\nI2GODXQBicg3AXzQ8NCnVPXrYcZjuM/6S1LVJwA8AQAjIyPt6MVB+owwmjheI2Ga3EvluRrFUNPE\n7GQWcjajyEeb8LuGi82FZYt5uOfydgbzC0azg1g6CTQAqvpLMa9xGsDVnttXATgT85yENE39ij+o\n4teEANj6bxa+1q1uGennwrLFPLy4WVBBx7LOIH20Iw30FQDXichKEVkM4BMA9rThuoQYiVPp66JY\nCPI+/fJbLZ38vWmqpjhF2JTVM9OlUMeyziBdxE0D/RiA/wlgOYC9IjKlqptE5EospHvepqqzIvJJ\nAPuwkAb6JVVtFEEnpE0kOclFjQs4GQks9nLJ57LVALSp6fv9u6eqx4osyFnY4ghX5rIN1ckmWGeQ\nLuJmAX1NVa9S1UtU9adVdVPl/jOqepvnuBdV9WdV9VpV/cO4gyYkDp2c5MJO/vVun6Bdizvnr/+Z\nZb5ZUG5W0+NbGzuZsc4gfbASmKSOIFdImJaQrcCbElpfnRx21/J33zuHu27MB3YGYwcxAlAMjqQQ\nd5J7+IVjDd3B3PTP5w4XI8UJ/FI4g/C6emyECfYCC7GJA8fPhqpdYAcxwh0ASSWjw3lMfnqhb2/9\nKviR0TV4dMuaamP3ILJOBveuu7qpnUNYt0sUfSIGcklYEikEaxUjIyN66JBRXoiQtrC9cNQ3zTPr\nDOCuG68KvWNYNuhgcPGipqQYwspSuEVdlHxIJ4kWghHSD4StDi5MFmtcQ7msg/vWD1l1fUrleWvr\nSBPvleesmj71bC8crZ47I4J7111dkxX0B8+/ipnyfM1zsk4GG1cvb8gYYpEXMcEdAOl7bFIP3qBn\nYbIYS/0zCgMC/PE9a30F20yTOwD86vqhautK99h6w2bbJYSJNZDeJ8oOgAaA9D02DRx3QjQZiFYz\n6Axg2aWXNOxIgsaSEcH3Hr3N+JjLyvG9RpeVTU+I9Bd0ARHiIahHQBKVwVGZKc9jpnL94nRpoR1k\niLGEcTUlJYlN+h9mAZG+xzbxLc06vgqZ7aQ8r9i551hgBk+YTKNmJbFJ+qABIH2PaUJ0BgTvXpjt\nisnfZbpUDlyl37vuat/HARZ5kfDQBUT6Hq8Gjutzn7kw21AE1g1sXL3cmFIqAty3rjYA7AeLvEgY\nGAQmqSJKi8dlgw5U4ZsZNACgMVeneeob0TCHn0SFQWBCDETJ9sk6mWq+vl/Xrc/ec0ONKmdcSuW5\n0FIOhMSFMQCSGqJk+7jNUQB7UPWz99yA0eE8lg2Gk4wIC6UcSLugASCpIerE6h4fFFTd/KErQp3v\nV9cPhdLzYbomaRd0AZHUYMuP92ui4uIXVD1w/GzgtfO5LB4ZXYORay7Dg88csebzM12TtBPuAEhq\nsLly7l13day8+TA7C29DlnmfxAuma5J2wh0ASQ2mdFA3w2bkmsuaVs8M0uoXAA/snsKufScwtmmV\n9fi8p21jPWHF7AiJAtNACYlJ1OwiU8OZenG6oPP7HU/STZQ0ULqACAlJYbKIDRP7sXJ8LzZM7Edh\nsgigNkgMXJRrMMk2uGmeUSp1TdlL3iwlQpqFLiBCQlC/Cq/X2DcFiVeO7zWe68x0KVKlbpCYHSHN\nwh0AISFoZhVuS+eMmuaZ1HkIqSeWARCRu0XkmIjMi4jV5yQi3xeRoyIyJSJ06pOeo5lVeFKqnFT3\nJK0irgvoOwC2APjTEMduVNV/jnk9QjpCMxr7fllHUUjqPITUk0gWkIh8C8DvqapxdS8i3wcwEtUA\nMAuIdAumTBwnI7h08SK8U5Fx5qRMuoFuFINTAN8QEQXwp6r6RJuuS4iVKLn19avw3KCDH783W1UK\nZeN10osEGgAR+SaADxoe+pSqfj3kdTao6hkR+SkAfyMix1X125brbQOwDQCGhoZCnp6QaARl9Zjw\nZu5smNjf0E/ADQrTAJBeIdAAqOovxb2Iqp6p/P8DEfkagJsAGA1AZXfwBLDgAop7bUJM+GX1hJnA\nmZpJ+oGWp4GKyKUi8hPu3wBuwULwmJCOEXcCZ2om6QfipoF+TEROA/gwgL0isq9y/5Ui8mLlsJ8G\n8H9F5AiAfwCwV1X/d5zrEhKXuBM4UzNJPxArCKyqXwPwNcP9ZwDcVvn7DQA3xLkOIUkztmmVUV8n\n7ATO1EzSD1AKgqSSJCZwNl4nvQ4NAEktnMBJ2qEWECGEpBQaAEIISSk0AIQQklJoAAghJKXQABBC\nSEqhASCEkJTS1U3hReQsgJOdHkdMLgeQ1j4IaX7tQLpfP19757hGVZeHObCrDUA/ICKHwmpz9xtp\nfu1Aul8/X3tvvHa6gAghJKXQABBCSEqhAWg9ae5+lubXDqT79fO19wCMARBCSErhDoAQQlIKDUAb\nEJFdInJcRF4Vka+JSK7TY2oXInK3iBwTkXkR6YnMiLiIyK0ickJEXheR8U6Pp52IyJdE5Acikrqu\nfyJytYgcEJHXKt/5/9LpMQVBA9Ae/gbAv1bVDwH4JwAPdXg87eQ7ALbA0gO63xCRDIDPA/hlAD8P\n4F4R+fnOjqqtfBnArZ0eRIeYBfCgqv4cgPUAfqfbP3sagDagqt9Q1dnKzYMArurkeNqJqr6mqic6\nPY42chOA11X1DVW9AOArAO7s8Jjahqp+G8C5To+jE6jq26r6j5W//wXAawC6uuEEDUD7+Q8A/rrT\ngyAtIw/gLc/t0+jySYAkj4isADAM4OXOjsQfdgRLCBH5JoAPGh76lKp+vXLMp7CwTXyqnWNrNWFe\ne4oQw31MtUsRIvIBAM8BuF9Vf9Tp8fhBA5AQqvpLfo+LyK8D+HcAflH7LPc26LWnjNMArvbcvgrA\nmQ6NhbQZEXGwMPk/parPd3o8QdAF1AZE5FYAvw/gDlWd6fR4SEt5BcB1IrJSRBYD+ASAPR0eE2kD\nIiIA/hzAa6r6x50eTxhoANrD5wD8BIC/EZEpEflCpwfULkTkYyJyGsCHAewVkX2dHlMrqQT7Pwlg\nHxaCgM+o6rHOjqp9iMjTAP4ewCoROS0iv9npMbWRDQB+DcDNld/5lIjc1ulB+cFKYEIISSncARBC\nSEqhASCEkJRCA0AIISmFBoAQQlIKDQAhhKQUGgBCCEkpNACEEJJSaAAIISSl/H/6HC3+NvpJ4gAA\nAABJRU5ErkJggg==\n",
109 |       "text/plain": [
110 |        "<matplotlib.figure.Figure at 0x1a66ce30e10>"
111 |       ]
112 |      },
113 |      "metadata": {},
114 |      "output_type": "display_data"
115 |     }
116 |    ],
117 |    "source": [
118 |     "NUM_TRIALS = 500\n",
119 |     "filter_output = np.zeros((2,NUM_TRIALS))\n",
120 |     "\n",
121 |     "for i in range(0,NUM_TRIALS):\n",
122 |     "    h = np.random.choice(4) #which symbol do we send?\n",
123 |     "    this_s = s[h,:]\n",
124 |     "    this_y = this_s + np.random.normal(0,np.sqrt(sigma_2),T)\n",
125 |     "    \n",
126 |     "    filter_output[0,i] = 2/T*this_y.T@np.cos(omega*n)\n",
127 |     "    filter_output[1,i] = 2/T*this_y.T@np.sin(omega*n)\n",
128 |     "\n",
129 |     "plt.scatter(filter_output[0],filter_output[1])\n",
130 |     "plt.axis('equal')\n",
131 |     "plt.show()"
132 |    ]
133 |   },
134 |   {
135 |    "cell_type": "markdown",
136 |    "metadata": {},
137 |    "source": [
138 |     "We see a point cloud for each symbol. How do the clouds change if we alter the noise variance or the symbol time? How do they change if we alter the frequency? The maximum-likelihood detector after the matched filter is simple: find the symbol closest in the Euclidean sense to the matched filter output. "
139 |    ]
140 |   }
141 |  ],
142 |  "metadata": {
143 |   "kernelspec": {
144 |    "display_name": "Python 3",
145 |    "language": "python",
146 |    "name": "python3"
147 |   },
148 |   "language_info": {
149 |    "codemirror_mode": {
150 |     "name": "ipython",
151 |     "version": 3
152 |    },
153 |    "file_extension": ".py",
154 |    "mimetype": "text/x-python",
155 |    "name": "python",
156 |    "nbconvert_exporter": "python",
157 |    "pygments_lexer": "ipython3",
158 |    "version": "3.6.3"
159 |   }
160 |  },
161 |  "nbformat": 4,
162 |  "nbformat_minor": 2
163 | }
164 | 


--------------------------------------------------------------------------------
/ch.3/Bayesian Spam Detection.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# Bayesian Spam Detection\n",
  8 |     "\n",
  9 |     "Here, we'll take a look at spam detection using the bag-of-words model and the application of Bayesian detection."
 10 |    ]
 11 |   },
 12 |   {
 13 |    "cell_type": "markdown",
 14 |    "metadata": {},
 15 |    "source": [
 16 |     "The two hypotheses are H_0 (normal email) and H_1 (spam/phishing email).\n",
 17 |     "\n",
 18 |     "The first step is to choose a dictionary. I've chosen this fairly arbitrarily from examples I found online. In a real-world system, you would need to estimate the probabilities for each word given the two hypotheses, but here I have made them up entirely."
 19 |    ]
 20 |   },
 21 |   {
 22 |    "cell_type": "code",
 23 |    "execution_count": 4,
 24 |    "metadata": {
 25 |     "collapsed": true
 26 |    },
 27 |    "outputs": [],
 28 |    "source": [
 29 |     "import numpy as np\n",
 30 |     "import matplotlib.pyplot as plt\n",
 31 |     "\n",
 32 |     "dictionary = ['order','report','mail','colleague','language','one','report','send','money','free']\n",
 33 |     "p_0 = [10,5,15,10,5,20,20,10,5,1]\n",
 34 |     "p_0 = p_0/np.sum(p_0)\n",
 35 |     "\n",
 36 |     "p_1 = [7,5,20,15,10,15,13,20,15,10]\n",
 37 |     "p_1 = p_1/np.sum(p_1)\n",
 38 |     "\n",
 39 |     "#Note that p_0 and p_1 are NOT the likelihood functions!"
 40 |    ]
 41 |   },
 42 |   {
 43 |    "cell_type": "markdown",
 44 |    "metadata": {},
 45 |    "source": [
 46 |     "Let's generate a \"message\" according to each hypothesis. Obviously this is not going to generate a realistic-looking message owing to the limited dictionary and the bag-of-words model."
 47 |    ]
 48 |   },
 49 |   {
 50 |    "cell_type": "code",
 51 |    "execution_count": 5,
 52 |    "metadata": {
 53 |     "collapsed": false
 54 |    },
 55 |    "outputs": [
 56 |     {
 57 |      "name": "stdout",
 58 |      "output_type": "stream",
 59 |      "text": [
 60 |       "['one' 'report' 'one' 'one' 'order' 'money' 'one' 'colleague' 'report'\n",
 61 |       " 'send' 'mail' 'report' 'report' 'mail' 'order' 'report' 'order' 'one'\n",
 62 |       " 'report' 'report' 'order' 'money' 'report' 'mail' 'send' 'send' 'order'\n",
 63 |       " 'one' 'money' 'one' 'report' 'send' 'one' 'colleague' 'report' 'one'\n",
 64 |       " 'money' 'colleague' 'one' 'order' 'one' 'mail' 'one' 'report' 'colleague'\n",
 65 |       " 'mail' 'one' 'report' 'send' 'report']\n",
 66 |       "['one' 'one' 'send' 'colleague' 'send' 'colleague' 'language' 'order'\n",
 67 |       " 'order' 'report' 'colleague' 'report' 'one' 'report' 'money' 'one' 'mail'\n",
 68 |       " 'one' 'one' 'report' 'language' 'send' 'send' 'send' 'report' 'send'\n",
 69 |       " 'language' 'one' 'colleague' 'money' 'colleague' 'colleague' 'colleague'\n",
 70 |       " 'one' 'send' 'colleague' 'report' 'mail' 'one' 'report' 'language'\n",
 71 |       " 'colleague' 'report' 'report' 'order' 'send' 'free' 'send' 'one'\n",
 72 |       " 'language']\n"
 73 |      ]
 74 |     }
 75 |    ],
 76 |    "source": [
 77 |     "N = 50\n",
 78 |     "\n",
 79 |     "normal_message = np.random.choice(dictionary,size=N,p=p_0)\n",
 80 |     "spam_message = np.random.choice(dictionary,size=N,p=p_1)\n",
 81 |     "\n",
 82 |     "print(normal_message)\n",
 83 |     "print(spam_message)"
 84 |    ]
 85 |   },
 86 |   {
 87 |    "cell_type": "markdown",
 88 |    "metadata": {},
 89 |    "source": [
 90 |     "Now, let's build a Bayes classifier. To do so, we need to choose priors and cost functions, then we compute the LLR."
 91 |    ]
 92 |   },
 93 |   {
 94 |    "cell_type": "code",
 95 |    "execution_count": 36,
 96 |    "metadata": {
 97 |     "collapsed": true
 98 |    },
 99 |    "outputs": [],
100 |    "source": [
101 |     "pi_0 = 0.9\n",
102 |     "pi_1 = 1-pi_0\n",
103 |     "\n",
104 |     "C_00 = 1\n",
105 |     "C_11 = 1\n",
106 |     "C_10 = 2\n",
107 |     "C_01 = 6"
108 |    ]
109 |   },
110 |   {
111 |    "cell_type": "markdown",
112 |    "metadata": {},
113 |    "source": [
114 |     "Let's sample directly from the spam bag of words model using np.multinomial."
115 |    ]
116 |   },
117 |   {
118 |    "cell_type": "code",
119 |    "execution_count": 31,
120 |    "metadata": {
121 |     "collapsed": false
122 |    },
123 |    "outputs": [
124 |     {
125 |      "name": "stdout",
126 |      "output_type": "stream",
127 |      "text": [
128 |       "[0 0 3 1 1 0 1 3 1 0]\n"
129 |      ]
130 |     }
131 |    ],
132 |    "source": [
133 |     "y = np.random.multinomial(N,p_1)\n",
134 |     "print(y)"
135 |    ]
136 |   },
137 |   {
138 |    "cell_type": "markdown",
139 |    "metadata": {},
140 |    "source": [
141 |     "Now, we compute the LLR and classify."
142 |    ]
143 |   },
144 |   {
145 |    "cell_type": "code",
146 |    "execution_count": 32,
147 |    "metadata": {
148 |     "collapsed": false
149 |    },
150 |    "outputs": [
151 |     {
152 |      "name": "stdout",
153 |      "output_type": "stream",
154 |      "text": [
155 |       "2.18479008414\n",
156 |       "0.587786664902\n"
157 |      ]
158 |     }
159 |    ],
160 |    "source": [
161 |     "L = 0\n",
162 |     "for i in range(0,len(dictionary)):\n",
163 |     "    L += y[i]*np.log(p_1[i]/p_0[i])\n",
164 |     "\n",
165 |     "threshold = np.log(((C_10-C_00)*pi_0)/((C_01-C_11)*pi_1))\n",
166 |     "print(L)\n",
167 |     "print(threshold)"
168 |    ]
169 |   },
170 |   {
171 |    "cell_type": "markdown",
172 |    "metadata": {},
173 |    "source": [
174 |     "Did we correctly detect the spam? Let's get a better sense of the detector performance by looking at the Bayes risk, averaged over many samples and for many priors."
175 |    ]
176 |   },
177 |   {
178 |    "cell_type": "code",
179 |    "execution_count": 37,
180 |    "metadata": {
181 |     "collapsed": false
182 |    },
183 |    "outputs": [
184 |     {
185 |      "name": "stderr",
186 |      "output_type": "stream",
187 |      "text": [
188 |       "C:\\Users\\fy4311\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\ipykernel_launcher.py:10: RuntimeWarning: divide by zero encountered in log\n",
189 |       "  # Remove the CWD from sys.path while we load stuff.\n",
190 |       "C:\\Users\\fy4311\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\ipykernel_launcher.py:10: RuntimeWarning: divide by zero encountered in double_scalars\n",
191 |       "  # Remove the CWD from sys.path while we load stuff.\n"
192 |      ]
193 |     },
194 |     {
195 |      "data": {
196 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xd8VFX6x/HPkw4hgSRDh1ASeocQ\nIFFXd1fFil2xK4i6um511V1/uqvu6uo2Xdcui6JgRUXFunZ66L2EEiIthBASQuo8vz9mcLNIkgEy\nc6c879crL2buvZP55pLkyTn33HNEVTHGGGOCTZTTAYwxxpgjsQJljDEmKFmBMsYYE5SsQBljjAlK\nVqCMMcYEJStQxhhjgpIVKGOMMUHJCpQxxpigZAXKGGNMUIpxOkBzcrlc2r17d6djGGOMacSiRYv2\nqGrbpo4LqwLVvXt38vLynI5hjDGmESKy1ZfjrIvPGGNMULICZYwxJihZgTLGGBOUrEAZY4wJSlag\njDHGBCUrUMYYY4KSFShjjDFByQqUMSbgSg5Us3DLXtxudTqKCWJhdaOuMSZ4qSqLtpbw8vwC3l+x\ng+paN6f1b8/fLh1Kq3j7VWS+z74rjDF+tb+yhreXfMvL8wpYt6uMVvExXJrVlXZJ8fzjPxu44InZ\nPHt1Ft3SEp2OaoKMFShjjF+sKCzl5flbeWfpdg7W1DGwczIPXTCIc4Z0ItHbYhreLYVbpi3m3Mdn\n8/jlwzixV5PTs5kIIqr+6QMWkcnA2cBuVR14hP3jgPsBN1AL/FxVv/HuqwNWeA8tUNVzfXnPrKws\ntbn4jHFORXUtM5duZ9qCApYXltIiNppzh3TiitHpDO7S5oivKSiu4IYX89iwu4zfntmPCSf0QEQC\nnNwEkogsUtWsJo/zY4E6CSgHXmygQLUCDqiqishg4DVV7evdV66qrY72Pa1AGeOMtTv3M21+AW8t\n/payqlp6t2/FlaO7cd6wziQnxDb5+gNVtfzqtWV8uGonFwzrzJ8uGERCbHQAkhsn+Fqg/NbFp6pf\niUj3RvaX13uaCNhwHmNCSGVNHR+s3MHL8wrI21pCXEwUZw3qyBWj0hnRLeWoWkGJ8TE8ccVwHv98\nI3/7ZD35ReU8fVUWHVon+PErMMHO0WtQInI+8CDQDjir3q4EEcnD0/X3kKq+7UQ+Y8z3bSoqZ/qC\nAl5fVMi+ihp6uBL53Zn9uHBEF1IT447580ZFCbf9qBd9OiTxy1eXcs7j3/DUlSMY0S2lGdObUOK3\nLj4AbwvqvSN18R123EnAPar6Y+/zTqq6XUR6Ap8BP1LV/AZeOwmYBJCenj5i61aflhkxxhyF6lo3\nn6zexbQFW5m9sZiYKOH0AR24fFQ6Y3qmERXVvNeM1u8q44YX89ixr5L7zxvApSPTm/XzG2c5fg3K\nG6I7PhQo77GbgZGquuew7VO8n+ONpj6HXYMypnlt21vBKwsLeHVhIXvKq+jcpgWXj0rn4qwutEvy\nb/fbvopqfjp9CV9v2MM1Y7px99n9iY22uQXCgePXoJoiIplAvneQxHAgDigWkRSgQlWrRMQF5AIP\nO5XTmEhT51Y+X7ubl+dv5Yv1RQjww77tuGJUN07q3ZboZm4tNaRNyzj+fe1I/vzhWp79ejPrdpXx\nxBUjjqsb0YQWvxUoEZkOnAy4RKQQuBeIBVDVp4ALgatFpAY4CFzqLVb9gKdFxI1nKqaHVHW1v3Ia\nYzx27a/k1YXbeGVBAdtLK2mXFM9PT8nk0ux0Ordp4UimmOgofndWf/p1TObOGSs455/f8OzVWfTv\nlOxIHhNYfu3iCzTr4jPm6Ljdyjcb9zBtfgGfrNlFnVs5sZeLK0al86N+7YOqS23Ztn3cOHURpQdr\n+MvFQzhrcEenI5ljFPRdfMYY5xSXV/H6okKmzS+gYG8FqYlxTDyxB+NHptPdFZxTDg3p2oaZP83l\npqmLuGXaYtbsyOSXp/Zu9gEaJnhYgTImQqgqCzbv5eX5BXy4cifVdW6ye6Tyq9N6M3ZgB+Jjgv/G\n2HZJCUyfNJp73l7F459vZO3O/fz90qEk+XAzsAk9VqCMCXOlFTXMWFLIy/ML2Li7nOSEGK4Ync7l\n2en0ap/kdLyjFh8TzUMXDmJA52T+8O5qzn9iDs9cNYKebY968hkT5OwalDFh6NDSFtO8S1tU1boZ\n2rUNV4xK5+zBnWgRF/ytJV/MzS/mlmmLqalz88/xwzi5TzunIxkfBMV9UIFmBcpEun0V1cxY/C3T\nFxSwYXc5SfExnDesM5dld2VAp9ZOx/OLbXsrmDR1Eet27ueOsX2ZdFJPm2w2yNkgCWMihKqSt7WE\n6Ye1lh6+cDBnD+lIy7jw/jHvmtqSN28ew+2vL+fBD9ayesd+/nzhYJtsNgyE93euMWFsX0U1b3pb\nSxu9raVLsroyPjs94u4TahkXw+OXD6P/F8n85eN1300269T9W6Z5WBefMSFEVVm4pYTpC/67bPqw\n9DaMz07n7MHh31ryxX/W7OJnrywlITaKJ68cwcjuqU5HMoexa1DGhJGSA9W8ubiQ6QsKyC86QFJ8\nDOcP78xlIyOvteSLjbvLuOHFRRSWVPCHcwdy+SibbDaY2DUoY0LcofuWpi8oYNbKnd+1lh65aDBn\nWWupUZntknj7llxum76E3761glXbS7n3nAHExQTPzBimafYdbkyQOdRamraggE1FB0hKiGH8yK5c\nlp1Ov47WWvJV6xaxTL52JA9/tJanv9zEhl3lPHHlcFyt4p2OZnxkXXzGBAFVZb63tfTBCs8sD8PT\n23D5qG6cNahj2Ny35JR3ln7Lb95YTlpiHM9cncXAzuE55D5UWBefMSFg74FqZhzWWrp8VDqXZXel\nbwdrLTWXcUM709PVihun5nHRU3N4+KIhnDukk9OxTBOsQBkTYKrKvE2e1tKhOfFGdEvhLxdnWmvJ\njwZ1ac3Mn57AzS8t4rbpS1i9fT+3n94nYOtbmaNnBcqYANl7oJo3F3lG4m3ac4Bkb2tpfHY6fTqE\n3px4ocjVKp6XJ47mD++u4qkv81m7cz+PXjaM1i1sstlgZAXKGD9SVeZuKmb6gm185G0tZXVL4ZZT\nMjnTWkuOiIuJ4o/nD6Jfx2R+P3MV5/9rNs9cnUVmO5tsNthYgTLGD4rLq7z3LW1jc73W0uWj0ukd\ngjOIh6MrR3ejd/skbn5pEef/azaPjh/KD/u2dzqWqcdG8RnTjCqqa/nj+2t4Pa+Q6jo3I7unMD47\nnTMHdbS54YLUt/sOcuPUPFZt38+dY/ty4w8ynI4U9mwUnzEBtm5nGbdMW0x+UTlXjErn6jHdrbUU\nAjq3acHrN+Zw+xvLePCDtUSJcMNJPZ2OZbACZcxxU1VeXbiNe2euIikhlpcmjCI30+V0LHMUWsRF\n8+hlw1CFP85aQ1JCDJdl2/RITrMCZcxxKKus4bdvreTdZds5IdPF3y8dStskm6kgFEVHCX+/dCjl\nVbXc9dYKWiXEcPZgu1fKSX6dmEpEJovIbhFZ2cD+cSKyXESWikieiJxQb981IrLB+3GNP3MacyxW\nflvK2f/8hveXb+f20/vw4vXZVpxCXFxMFE9dOYKR3VL5+StL+XztbqcjRTR/z5w4BRjbyP7/AENU\ndShwPfAcgIikAvcCo4Bs4F4RSfFvVGN8o6pMmb2ZC56YQ1WNm1cmjeGWUzKJshs+w0KLuGieuzaL\nvh2TuOmlRczfVOx0pIjl1wKlql8BexvZX67/HUaYCBx6fDrwiaruVdUS4BMaL3TGBERpRQ03Tl3E\n799dzQm9XMz62Ylk97D1hsJNckIsL1yXTZeUFkx4IY8VhaVOR4pIjs89LyLni8ha4H08rSiAzsC2\neocVercd6fWTvN2DeUVFRf4NayLa4oISznzsaz5bu5u7z+rH89dkkZoY53Qs4ydpreJ5aeIoWreI\n5erJ89mwq8zpSBHH8QKlqm+pal/gPOB+7+Yj9ZUc8YYtVX1GVbNUNatt27b+imkimNutPP1lPpc8\nNZeoKHjj5hwmntgTEevSC3cdW7fg5YmjiImO4srn57Ntb4XTkSKK4wXqEG93YIaIuPC0mLrW290F\n2O5IMBPRisuruP6FhTz4wVpOG9Ce9356IkO7tnE6lgmg7q5Epk7IprLGzZXPz2f3/kqnI0UMRwuU\niGSK989QERkOxAHFwEfAaSKS4h0ccZp3mzEBM29TMWc+9jVz8ou5/7yB/Ovy4TapaITq2yGZKdeN\npKisiiufn0/JgWqnI0UEfw8znw7MBfqISKGITBCRm0TkJu8hFwIrRWQp8C/gUvXYi6e7b6H34z7v\nNmP8rs6tPPafDVz+7DwS42J46yc5XDW6m3XpRbhh6Sk8d00WW4oruHbKQsqrap2OFPZsLj5j6tm9\nv5Kfv7qUOfnFnDe0Ew+cP4hW8XY/u/mvT1bv4qaXFpHdPZV/XzfS5lg8Br7OxRc016CMcdpX64s4\n87GvWVxQwsMXDebvlw614mS+59T+7fnbJUOYt7mYW6ctpqbO7XSksGUFykS82jo3D3+4lqsnLyA1\nMY53bz2BS7K6WpeeadC4oZ25b9xAPl2zm1+/vgy3O3x6ooKJ/XloItr2fQe5bfoS8raWcNnIrtx7\nzgBbRND45KrR3SirrOHhD9fRKj6GB84baH/UNDMrUCZifbp6F79+Yxk1tW4evWwo44Ye8V5wYxr0\nk5MzKaus5ckv8kluEcsdY/s6HSmsWIEyEae61s2fP1zL899sZkCnZB6/fDg9XIlOxzIh6jen92H/\nwRqe/CKfpIQYfnJyptORwoYVKBNRCooruHX6YpYXlnJtTnfuOrMv8THWpWeOnYhw/7iBlFfV8vCH\n60hKiOWq0d2cjhUWrECZiPH+8h3c+eZyROCpK4czdmBHpyOZMBEVJfzl4iEcqKrlnndWkhQfw3nD\nrMv4eNkoPhP2KmvquPvtFdwybTEZ7Vrx/m0nWnEyzS42OorHLx/O6B5p/Or1ZXyyepfTkUKeFSgT\n1vKLyjnvX7N5aV4BN57Uk9dvGkPX1JZOxzJhKiE2mmevyWJg59bcMm0xczbucTpSSLMCZcLWjMWF\nnPPPb9hdVsW/rxvJXWf2IzbavuWNf7WKj+GF60bSIy2RiS/msaSgxOlIIct+Wk3Yqaiu5devL+OX\nry1jYOfWzLrtRE7p087pWCaCtGkZx9QJ2bhaxXPtvxeydud+pyOFJCtQJqys3bmfc/75DW8uLuS2\nH/Vi2sRRdGid4HQsE4HaJSfw8sRRtIiN5qrnF7BlzwGnI4UcK1AmLKgq0xcUMO7x2eyvrOWlCaP4\n5am9ibEuPeOgrqkteWliNrV1bq54bj47Sg86HSmk2E+vCXlllTXc9spS7pqxguweqcy67URyM11O\nxzIGgMx2Sbx4/Sj2H6zhyufmU1xe5XSkkGEFyoS0dTvLOPuf3zBrxQ5uP70PL1yXTdukeKdjGfM/\nBnVpzfPXjqSw5CDX/HsB+ytrnI4UEqxAmZC1Zc8BrnhuPger63hl0mhuOSWTqCibrNMEp+weqTx1\n1QjW7ihj4pQ8DlbXOR0p6FmBMiFpZ2klVz4/nzq3m2k3jGJk91SnIxnTpFP6tOMflw0lb+tebnpp\nEdW1tpZUY6xAmZCzr6KaqyfPp+RANS9cn01muySnIxnjs7MHd+LBCwbx5foifvHqUupsLakG2Vx8\nJqQcqKrl2n8vZEtxBVOuG8ngLm2cjmTMUbt0ZDpllbU88P4aWsXH8NCFg2wtqSOwAmVCRlVtHTdO\nXcTywn08eeUIcjJspJ4JXRNP7Mn+gzU89tlGWiXEcPdZ/axIHcZvBUpEJgNnA7tVdeAR9l8B3OF9\nWg7crKrLvPu2AGVAHVCrqln+ymlCQ51b+fkrS/lm4x4euWgwpw/o4HQkY47bL07tzf7KWp7/ZjOt\nW8Ry2496OR0pqPizBTUFeBx4sYH9m4EfqGqJiJwBPAOMqrf/FFW1mRYNqspvZ6zgg5U7ufusflyc\n1dXpSMY0CxHhnrP7U1ZZy98+WU+r+BiuP6GH07GCht8KlKp+JSLdG9k/p97TeUAXf2Uxoe2hD9fy\nat42fvrDTCae2NPpOMY0q6go4c8XDuJAVS33vbeapIQY+yPMK1hG8U0APqj3XIGPRWSRiExq7IUi\nMklE8kQkr6ioyK8hTeA9+UU+T3+5iatGd+OXp/Z2Oo4xfhETHcWj44dyYi8Xd7y5nA9X7nA6UlBw\nvECJyCl4CtQd9Tbnqupw4AzgFhE5qaHXq+ozqpqlqllt27b1c1oTSNMXFPDnD9dy7pBO/OHcAXYB\n2YS1+Jhonr5qBMPSU/jp9CV8td7+4Ha0QInIYOA5YJyqFh/arqrbvf/uBt4Csp1JaJzy/vId/Pat\nFZzSpy1/vWSIzRBhIkLLuBgmXzuSzHZJTJqaR96WvU5HcpRjBUpE0oEZwFWqur7e9kQRSTr0GDgN\nWOlMSuOEr9YX8fNXl5DVLYUnrhhhiwyaiNK6RSxTJ2TTqXULrpuykDU7InctKb/95IvIdGAu0EdE\nCkVkgojcJCI3eQ+5B0gDnhCRpSKS593eHvhGRJYBC4D3VfVDf+U0wWXR1hJunLqIzHZJPHfNSFrE\nRTsdyZiAc7WKZ+rEUcRFR/GPT9c3/YIw5c9RfOOb2D8RmHiE7ZuAIf7KZYLX2p37uX7KQtonx/Pi\n9dm0bhHrdCRjHNO5TQt+3K89H6zcQZ1biY7Abm7rOzFBoaC4gqueX0BCbBRTJ4yyJTOMAXIy09hf\nWcvq7ZHZzWcFyjhu937PzOQ1dW5emjCKrqktnY5kTFAYk5EGwOz8yJyzwAqUcVRpRQ1XPb+APeVV\nTLkum17tbWZyYw5pl5RAr3atmL3RCpQxAVVRXct1Uxawec8Bnr06i6FdbWZyYw6Xm+li4Za9Ebl2\n1FEXKBHpIyLP+iOMiRzVtW5unLqIpdv28dj4oeRm2szkxhxJTkYalTVulhSUOB0l4BosUCIyWEQ+\nFpGVIvKAiLQXkTeB/wCrAxfRhJs6t/KL15by9YY9PHTBYMYO7Oh0JGOC1qieaUQJzMkvbvrgMNNY\nC+pZYBpwIVAELAY2AZmq+vcAZDNhSFW5++2VvL98B787sx+XjLRJMY1pTOsWsQzq3Jo5EThQorEC\nFa+qU1R1nao+CriBO1W1MkDZTBh65KN1TF9QwE9OzuCGk2xmcmN8kZPpYknBPg5U1TodJaAaK1AJ\nIjJMRIaLyHA8iwoOrvfcmKPyzFf5PPFFPpePSuf20/s4HceYkJGTkUatW1kYYXPzNTaTxA7gb/We\n76z3XIEf+iuUCT+vLizgT7PWcvbgjtw/bqDNTG7MUcjqlkpcdBRz8os5uU87p+METIMFSlVPCWQQ\nE74+XLmDu2as4Ae92/K3S4ZG5JQtxhyPFnHRDEtvE3H3Q9l9UMavvtmwh9umL2VYegpPXjmcuBj7\nljPmWORmuli9Yz8lB6qdjhIw9tvC+M2SghImTc2jZ9tEJl8zkpZxfpub2Jiwl5uZhirM2xQ5w82t\nQBm/WL+rjOumLKRtkndm8pY2M7kxx2NwlzYkxkVH1Lx8TRYoEbnvsOfRIvKy/yKZULdtbwVXPT+f\nuOgoXpowinbJCU5HMibkxUZHkd0jNaJu2PWlBZUuIncBiEg8niXYN/g1lQlZu8s8M5NX1riZajOT\nG9OscjNdbCo6wM7SyLgd1ZcCdR0wyFuk3gU+V9Xf+zWVCUmlB2u4+vkFFJVV8e/rRtKng81Mbkxz\nOrT8RqTMKtHYXHyHbsgdBjwKXIqn5fSl3ahrDnewuo4JUxaSX1TO01eNYHh6itORjAk7/Tokk9Iy\nltkbI6Obr7FhVX897HkJ0N+73W7UNd+prnVz88uLWFxQwuOXD+fEXm2djmRMWIqKEsZkpDEnfw+q\nGvY3vNuNuua41LmVX72+jC/WFfHQBYM4c5DNTG6MP+VkuJi1Yidbiivo4Up0Oo5f+TKK72cikiwe\nz4nIYhE5zYfXTRaR3SKysoH9V4jIcu/HHBEZUm/fWBFZJyIbReTOo/uSTKCoKvfOXMm7y7Zz5xl9\nuSw73elIxoS9Q2unRcKsEr4MkrheVfcDpwHt8AyaeMiH100BxjayfzPwA1UdDNwPPAOeYezAv4Az\n8HQpjheR/j68nwmwv368npfmFXDTDzK46QcZTscxJiJ0T2tJx9YJETFQwpcCdaiT80zg36q6rN62\nBqnqV0CDU++q6hxVPbRE5Dygi/dxNrBRVTepajXwCjDOh5wmgJ77ehOPf76R8dlduWOszUxuTKCI\nCDkZLubmF+N2q9Nx/MqXArVIRD7GU6A+EpEkPGtDNacJwAfex52BbfX2FXq3HZGITBKRPBHJKyoq\nauZY5khey9vGA++v4axBHXngvEFhf6HWmGCTm5lGSUUNa3budzqKX/lSoCYAdwIjVbUCiMPTzdcs\nROQU73vccWjTEQ5r8M8EVX1GVbNUNattWxs95m8frdrJnW8u58ReLv526RCbmdwYB+RkeK5DzQ3z\nWSUauw+qr/fhUO+/Pb33P3Wj8eHpPhORwcBzwDhVPXSmC4H664B3AbY3x/uZ47Py21J+Om0JQ7q2\n4emrRhAfE+10JGMiUofWCfRsmxj2AyUaKzS/BCbx/fuhoBnugxKRdGAGcJWqrq+3ayHQS0R6AN8C\nlwGXH897meNXXevm168vo03LWJuZ3JggkJvhYsbiQmrq3MRGh+e8343dBzXJ++8x3Q8lItOBkwGX\niBQC9wKx3s/5FHAPkAY84b2GUevtqqsVkVuBj4BoYLKqrjqWDKb5PPHFRtbuLOO5q7NISYxzOo4x\nES8nI42p87ayvHAfI7qlOh3HL476z2ARORX4jaqe2thxqjq+if0TgYkN7JsFzDrabMY/Vm/fz+Of\nbeT8YZ35cf/2TscxxgCje6YhArM3FodtgWrsGtQPRWS9iJSLyEsi0l9E8vDcA/Vk4CIaJ9XUubn9\njWW0aRnHvefY7WjGBIuUxDj6d0wO6+tQjXVc/hXPNag04A089ypNVdURqjojEOGM857+Mp9V2/fz\nwHkDadPSuvaMCSa5mS6WFOzjYHWd01H8orECpar6hapWqerbQJGqPhqoYMZ563aW8eh/NnDOkE6M\nHdjB6TjGmMPkZKRRXecmb2uDcyKEtMauQbURkQvqPZf6z60VFd5qvV17yQmx/OHcAU7HMcYcwcju\nqcRECbM3FoflKgKNFagvgXMaeK54hoibMPXs15tZXljKvy4fTqqN2jMmKCXGxzAsvQ1zw3RevsaG\nmTfbbBEmtGzcXcbfP13PGQM7cNZgWz7DmGCWk+Hin59toPRgDa1bxDodp1mF591d5pjVuZXb31hO\nYlw0940b6HQcY0wTcjLScCvM3xR+0x5ZgTL/Y/I3m1lSsI/fnzuAtknxTscxxjRhWHoKCbFRzAnD\nefmsQJnvbCoq5y8fr+PU/u05d0gnp+MYY3wQFxPFyO6pYXk/lC8r6l7sXWIDEblbRGZ4J401YaTO\nrfzmjeUkxEbzx/MG2hIaxoSQ3EwXG3aXs7us0ukozcqXFtT/qWqZiJwAnA68gM0kEXZemLOFvK0l\n3HtOf9olJzgdxxhzFHLDdPkNXwrUoVuUzwKeVNV38KwJZcLElj0HePijtfywbzvOH9bg2pDGmCDV\nv1MyyQkxYdfN50uB+lZEngYuAWaJSLyPrzMhwO1WfvPmcmKjo/jT+bY6rjGhKDpKGJORFnYDJXwp\nNJfgWfpirKruA1KB2/2aygTMS/O3smDzXv7v7P50aG1de8aEqtxMF4UlBykornA6SrNpskB5l3nf\nDZzg3VQLbPBnKBMY2/ZW8NAHazmpd1suHtHF6TjGmOOQk5EGwOwwmlXCl1F89wJ3AHd5N8UCL/kz\nlPE/VeWON5cTJcJDF1jXnjGhLqNtK9olxYdVN58vXXznA+cCBwBUdTuQ5M9Qxv+mLShgTn4xvzur\nH53atHA6jjHmOIkIORlpzM3fg6o6HadZ+FKgqtXz1SqAiCT6N5Lxt2/3HeTBWWs5IdPFZSO7Oh3H\nGNNMcjJd7CmvZv2ucqejNAtfCtRr3lF8bUTkBuBT4Fn/xjL+oqrc+eZy3Ko8aF17xoSV765Dhclw\nc18GSfwFz4q6bwJ9gHtU9Z/+Dmb847W8bXy9YQ93ndGXrqktnY5jjGlGXVJa0i2tJXPCZKCEL4Mk\nbgXyVPV2Vf21qn7iyycWkckisltEVjawv6+IzBWRKhH59WH7tojIChFZKiJ5Pn0lpkk7Sg/ywHtr\nGN0zlStGdXM6jjHGD3IyXMzftJfaOrfTUY6bL118HYCFIvKaiIwV3/uEpgBjG9m/F7gN+EsD+09R\n1aGqmuXj+5lGqCq/nbGCWrfy8IVDiIqyrj1jwlFuZhplVbWs+LbU6SjHzZcuvruBXsDzwLXABhH5\nk4hkNPG6r/AUoYb271bVhUDNUSU2x+TNxd/y+boifjO2D+lp1rVnTLga09NzHSochpv7NGWRdxTf\nTu9HLZACvCEiD/splwIfi8giEZnU2IEiMklE8kQkr6ioyE9xQtuu/ZXc9+4qsruncs2Y7k7HMcb4\nUVqrePp2SAqL61C+XIO6TUQWAQ8Ds4FBqnozMAK40E+5clV1OHAGcIuInNTQgar6jKpmqWpW27Zt\n/RQndKkqv3trBVW1bv580WDr2jMmAuRkuMjbUkJlTV3TBwcxX1pQLuACVT1dVV9X1RoAVXUDZ/sj\nlPdmYFR1N/AWkO2P94kE7yzdzqdrdnP76X3o4bJb2IyJBLmZaVTVullcUOJ0lOPiyzWoe1R1q4i0\nE5H0Qx/efWuaO5CIJNZbIDEROA044khA07jdZZX8/t1VDE9vw3W5PZyOY4wJkOweqURHCXM2hvZ1\nqJimDhCRc4C/AZ3wTBrbDVgDDGjiddOBkwGXiBQC9+KZxw9VfUpEOgB5QDLgFpGfA/3xtNje8g4W\njAGmqeqHx/LFRTJV5f/eXklFdR0PXzSEaOvaMyZiJCXEMrhLa2bn7+HX9HE6zjFrskABDwCjgU9V\ndZiInAKMb+pFqtroMaq6EzjSFNr7gSE+5DKNeG/5Dj5atYs7z+hLZrtWTscxxgRYboaLJ7/Mp6yy\nhqSEWKfjHBNfrkHVqGoxECWkKBFvAAAUN0lEQVQiUar6OTDUz7nMcSgur+LemasY0qU1E0+wrj1j\nIlFOZhp1bmXB5gbv9gl6vhSofSLSCvgKeFlEHsUz1NwEqXtmrqK8spZHLh5CTLQtfmxMJBqenkJ8\nTBSzQ/g6lC+/vcYBFcAvgA+BfOAcf4Yyx+6DFTt4f/kOfvbjXvRub6uiGBOpEmKjyeqeEtL3Q/ky\niu+AqrpVtRZ4H/int8vPBJm9B6r5v3dWMrBzMpNO6ul0HGOMw3IyXKzdWcae8iqnoxyTBguUiIwW\nkS9EZIaIDPNO+roS2CUijc2xZxzyh3dXUXqwhkcuGkKsde0ZE/EOLb8xN0SnPWrst9jjwJ+A6cBn\nwERV7QCcBDwYgGzmKHy8aifvLN3Oraf0ol/HZKfjGGOCwKDOrUmKjwnZefkaK1Axqvqxqr4O7FTV\neQCqujYw0Yyv9lVU87u3V9KvYzI/OaXROXyNMREkJjqKUT1TQ/Y6VGMFqv5iIgcP2xceC96Hifve\nW03JgWoeuWiwde0ZY/5HToaLrcUVFJZUOB3lqDX222yIiOwXkTJgsPfxoeeDApTPNOGztbuYsfhb\nfnJyBgM7t3Y6jjEmyORmuoDQXH6jwQKlqtGqmqyqSaoa43186Hlo3pYcZkoP1nDXjBX0aZ/ErT/s\n5XQcY0wQ6t2+Fa5WcczZGHrdfNYfFML++P5q9pRX88jFg4mLsf9KY8z3iQhjMlzMyS/Gs7Rf6LDf\naiHqy/VFvJZXyKSTejK4Sxun4xhjglhORhq7y6rILyp3OspRsQIVgsoqa7jzzeVktmvFz35kXXvG\nmMblZniuQ4XatEdWoELQn2atZdf+Sh65aDAJsdFOxzHGBLn0tJZ0SWkRcsPNrUCFmG827GH6ggIm\nntiTYekpTscxxoSInIw05uYXU+cOnetQVqBCSHlVLXe8uZyerkR+eWpvp+MYY0JIbqaL/ZW1rN6+\n3+koPrMCFUL+/MFatpce5GHr2jPGHKUx3nn5ZodQN58VqBAxN7+YqfO2cl1OD7K6pzodxxgTYtol\nJdCrXStmh9D9UFagQkBFtadrr1taS24/vY/TcYwxISo308XCLXuprnU3fXAQsAIVAh7+cB0Feyt4\n+MLBtIizrj1jzLEZk5FGZY2bJQUlTkfxid8KlIhMFpHd3nWkjrS/r4jMFZEqEfn1YfvGisg6Edko\nInf6K2MoWLhlLy/M3cI1Y7oxqmea03GMMSFsdM80ogRmh8i8fP5sQU0BGlvYcC9wG/CX+htFJBr4\nF3AG0B8YLyL9/ZQxqB2sruM3byynS0oLfjO2r9NxjDEhrnWLWAZ1bs3cEBko4bcCpapf4SlCDe3f\nraoLgZrDdmUDG1V1k6pWA68A4/yVM5j99eN1bN5zgD9fMJjE+Bin4xhjwsCYDBdLCvZxoKrW6ShN\nCsZrUJ2BbfWeF3q3HZGITBKRPBHJKyoq8nu4QFm2bR+TZ2/m8lHp5HinyzfGmOOVm5lGrVtZsKXB\n9kPQCMYCJUfY1uCtz6r6jKpmqWpW27Zt/RgrcGrq3Nw5YwWuVvHceYZ17Rljmk9Wt1TioqOYGwLX\noYKx36gQ6FrveRdgu0NZHDH5m82s2bGfp64cTnKCLb1ljGk+LeKiGZbeJiTuhwrGFtRCoJeI9BCR\nOOAyYKbDmQKmoLiCv3+6nlP7t+f0AR2cjmOMCUO5mS5W79hPyYFqp6M0yp/DzKcDc4E+IlIoIhNE\n5CYRucm7v4OIFAK/BO72HpOsqrXArcBHwBrgNVVd5a+cwURV+d3bK4gW4b5xAxA5Um+nMcYcn5yM\nNFRh3qbg7ubzWxefqo5vYv9OPN13R9o3C5jlj1zBbOay7Xy9YQ9/OHcAHVu3cDqOMSZMDenahsS4\naGbn7+GMQR2djtOgYOzii0glB6q5793VDO3ahitHd3M6jjEmjMVGR5HdI5U5QT5QwgpUkPjTrDWU\nHqzhwQsGER1lXXvGGP/KyXCxqegAO0srnY7SICtQQWDOxj28vqiQG07qSb+OyU7HMcZEgJxM7/Ib\nQTyazwqUwypr6vjtWyvoltaSn/2ol9NxjDERol+HZFJaxgZ1N58VKIc9/tlGthRX8MfzBtkihMaY\ngImKEsZkpDEnfw+qwbkMvBUoB63bWcZTX+ZzwfDOnNDLpjMyxgRWToaLHaWVbCmucDrKEVmBcojb\nrdw1YzlJCTHcfVZETtZujHFYTkZwX4eyAuWQlxcUsLhgH3ef1Z/UxDin4xhjIlAPVyIdWycwJ0iX\n37AC5YCdpZU8/MFacjPTuGB4gxO1G2OMX4kIORku5uYX43YH33UoK1AO+P3MVVTXufnjeYNsOiNj\njKNyMtIoqahhzc79Tkf5HitQAfbxqp18uGonP/txL7q7Ep2OY4yJcLne9ebmbAy+4eZWoAKorLKG\ne95ZRd8OSdxwYk+n4xhjDB1aJ9CzbWJQXoeyAhVAf/14PbvKKnnwgkHERtupN8YEh5yMNBZs3ktN\nndvpKP/DfksGyJKCEl6Yu4WrR3djWHqK03GMMeY7uRkuDlTXsWzbPqej/A8rUAFQU+fmrhkraJ+U\nwK9P7+N0HGOM+R+je6YhQtBNe2QFKgCe/XoTa3eWcd+4ASTZEu7GmCCTkhhH/47JQXfDrhUoP9uy\n5wCPfrqBsQM6cJot4W6MCVK5mS6WFOzjYHWd01G+YwXKjw4t4R4XHcXvzx3gdBxjjGnQmIw0quvc\n5G3d63SU71iB8qO3lnzL7I3F/OaMvnRoneB0HGOMaVB291RiooTZQXQ/lBUoP9l7oJr731vN8PQ2\nXJGd7nQcY4xpVGJ8DMPS2zA3iO6H8luBEpHJIrJbRFY2sF9E5DER2Sgiy0VkeL19dSKy1Psx018Z\n/emB91dTVlnLgxcMJsqWcDfGhIAxGS5WfFtK6cEap6MA/m1BTQHGNrL/DKCX92MS8GS9fQdVdaj3\n41z/RfSPbzbsYcbib7npBxn06ZDkdBxjjPFJbkYaboV5m4Kjm89vBUpVvwIau9o2DnhRPeYBbUSk\no7/yBMrBas8S7j1cidz6w0yn4xhjjM+GpaeQEBvF3CC5H8rJa1CdgW31nhd6twEkiEieiMwTkfMa\n+yQiMsl7bF5RUZG/svrssc82ULC3gj+eP9CWcDfGhJS4mChGdk8NmvuhnCxQR7owc2hBknRVzQIu\nB/4hIhkNfRJVfUZVs1Q1q23btv7I6bM1O/bz7FebuHhEF3IybAl3Y0zoyc10sWF3ObvLKp2O4miB\nKgS61nveBdgOoKqH/t0EfAEMC3S4o1XnVu6asYLkFrH89sx+TscxxphjcmgZ+GDo5nOyQM0ErvaO\n5hsNlKrqDhFJEZF4ABFxAbnAagdz+uSleVtZum0f95zdnxRbwt0YE6IGdGpNckJMUHTzxfjrE4vI\ndOBkwCUihcC9QCyAqj4FzALOBDYCFcB13pf2A54WETeeAvqQqgZ1gdq+7yAPf7iWE3u5GDe0k9Nx\njDHmmEVHCWMy0oJi4li/FShVHd/EfgVuOcL2OcAgf+VqbqrKPe+sok7VlnA3xoSFnAwXH63aRUFx\nBelpLR3LYTNJHKePVu3k0zW7+MWPezv6H2mMMc0lN9NzHWq2w7NKWIE6Dvsra7h35ir6d0xmwgk9\nnI5jjDHNIqNtK9olxTvezWcF6jg88uE6isqqePCCQcTYEu7GmDAhIuRkpDE3fw+eqzHOsN+qx2jR\n1r28NH8r1+R0Z0jXNk7HMcaYZpWT6WJPeTXrdpU5lsEK1DGorvUs4d4xOYFfnWZLuBtjws+h+6Hm\nOLj8hhWoY/DMV/ms31XOfeMG0irebwMhjTHGMV1SWtItrSVzHBwoYQXqKG0qKuexzzZy1qCO/Lh/\ne6fjGGOM3+RkuJi/aS+1dW5H3t8K1FFQVX731kriY6K495z+Tscxxhi/yslIo6yqlhXfljry/lag\njsIbiwqZu6mYO8/oS7tkW8LdGBPevrsO5dBwcytQPtpTXsUfZ60hq1sK40faEu7GmPCX1iqevh2S\nHLsOZQXKRw+8t5oDVbU8eMEgW8LdGBMxcjJc5G0pobKmLuDvbQXKB1+uL+Ltpdu5+eRMerW3JdyN\nMZEjNzONqlo3i7eWBPy9rUA14WB1HXe/vYKebRP5yckNrptojDFhKbtHKtFR4sh1KCtQTfjHf9az\nbe9B/nT+IFvC3RgTcZISYhncpbUjE8dagWrEqu2lPPf1Zi7N6sronmlOxzHGGEfkZrhYXlhKWWVN\nQN/XClQDDi3hntIylrvO7Ot0HGOMcUxORhp1bmXB5r0BfV8rUA14Yc4WlheWcs85A2jT0pZwN8ZE\nruHdUoiPiWJ2gOflswJ1BN/uO8hfPl7HyX3acs7gjk7HMcYYRyXERpPVPSXg90NZgTqMqnLP2ytR\nhfvHDbQl3I0xBs/9UGt3lrGnvCpg72kF6jAfrNzJf9bu5len9aZrqi3hbowx8N9pj+YGcLi5XwuU\niEwWkd0isrKB/SIij4nIRhFZLiLD6+27RkQ2eD+u8WfOQ0oPepZwH9g5mWtzugfiLY0xJiQM6tya\npPiYgN4P5e8W1BRgbCP7zwB6eT8mAU8CiEgqcC8wCsgG7hWRFL8mBf784VqKy6t48PzBtoS7McbU\nExMdxaieqQG9DuXX38Kq+hXQ2LjEccCL6jEPaCMiHYHTgU9Uda+qlgCf0HihO24Lt+xl2vwCrs/t\nwaAurf35VsYYE5JyMlxsLa6gsKQiIO/ndDOhM7Ct3vNC77aGtn+PiEwSkTwRySsqKjrmIE98vpHO\nbVrwi1N7H/PnMMaYcJaT6bkONW9TYO6Hcnq98iMNkdNGtn9/o+ozwDMAWVlZRzzGF09cMYJtJRUk\n2hLuxhhzRH3aJ/H+bSfQr0NyQN7P6RZUIdC13vMuwPZGtvtNi7hoettM5cYY0yARYUCn1gFbcsjp\nAjUTuNo7mm80UKqqO4CPgNNEJMU7OOI07zZjjDERwq/9WSIyHTgZcIlIIZ6RebEAqvoUMAs4E9gI\nVADXefftFZH7gYXeT3WfqgZ2EihjjDGO8muBUtXxTexX4JYG9k0GJvsjlzHGmODndBefMcYYc0RW\noIwxxgQlK1DGGGOCkhUoY4wxQckKlDHGmKAknoF04UFEioCtx/EpXEBgV+QKTnYePOw8eNh58LDz\n8F/Hey66qWrbpg4KqwJ1vEQkT1WznM7hNDsPHnYePOw8eNh5+K9AnQvr4jPGGBOUrEAZY4wJSlag\n/tczTgcIEnYePOw8eNh58LDz8F8BORd2DcoYY0xQshaUMcaYoGQFyhhjTFCKyAIlImNFZJ2IbBSR\nO4+wP15EXvXuny8i3QOf0v98OA+/FJHVIrJcRP4jIt2cyOlvTZ2HesddJCIqImE51NiX8yAil3i/\nJ1aJyLRAZwwEH34u0kXkcxFZ4v3ZONOJnP4mIpNFZLeIrGxgv4jIY97ztFxEhjd7CFWNqA8gGsgH\negJxwDKg/2HH/AR4yvv4MuBVp3M7dB5OAVp6H98cqefBe1wS8BUwD8hyOrdD3w+9gCVAivd5O6dz\nO3QengFu9j7uD2xxOrefzsVJwHBgZQP7zwQ+AAQYDcxv7gyR2ILKBjaq6iZVrQZeAcYddsw44AXv\n4zeAH4lIYNY4Dpwmz4Oqfq6qFd6n84AuAc4YCL58PwDcDzwMVAYyXAD5ch5uAP6lqiUAqro7wBkD\nwZfzoECy93FrYHsA8wWMqn4FNLZQ7DjgRfWYB7QRkY7NmSESC1RnYFu954XebUc8RlVrgVIgLSDp\nAseX81DfBDx/LYWbJs+DiAwDuqrqe4EMFmC+fD/0BnqLyGwRmSciYwOWLnB8OQ+/B670rhI+C/hp\nYKIFnaP9HXLU/LqibpA6Ukvo8LH2vhwT6nz+GkXkSiAL+IFfEzmj0fMgIlHA34FrAxXIIb58P8Tg\n6eY7GU9r+msRGaiq+/ycLZB8OQ/jgSmq+lcRGQNM9Z4Ht//jBRW//56MxBZUIdC13vMufL+J/t0x\nIhKDpxnfWFM3FPlyHhCRHwO/A85V1aoAZQukps5DEjAQ+EJEtuDpa58ZhgMlfP25eEdVa1R1M7AO\nT8EKJ76chwnAawCqOhdIwDN5aqTx6XfI8YjEArUQ6CUiPUQkDs8giJmHHTMTuMb7+CLgM/VeFQwj\nTZ4Hb9fW03iKUzheb4AmzoOqlqqqS1W7q2p3PNfizlXVPGfi+o0vPxdv4xk4g4i48HT5bQpoSv/z\n5TwUAD8CEJF+eApUUUBTBoeZwNXe0XyjgVJV3dGcbxBxXXyqWisitwIf4RmxM1lVV4nIfUCeqs4E\nnsfTbN+Ip+V0mXOJ/cPH8/AI0Ap43TtGpEBVz3UstB/4eB7Cno/n4SPgNBFZDdQBt6tqsXOpm5+P\n5+FXwLMi8gs8XVrXhuEfsIjIdDzduS7v9bZ7gVgAVX0Kz/W3M4GNQAVwXbNnCMPzaowxJgxEYhef\nMcaYEGAFyhhjTFCyAmWMMSYoWYEyxhgTlKxAGWOMCUpWoIwxxgQlK1DGBBkReU5E+jeyf4SIrPAu\nc/BYGE5kbAxg90EZE3JEZAHwMzyzWswCHlPVcJzI10Q4a0EZ4xAR6S4ia0XkBe+Cb2+ISEsR+aKh\nuf68yxkkq+pc7+wFLwLnBTS4MQFiBcoYZ/UBnlHVwcB+PItlNqYznkk6D2n2JQ6MCRZWoIxx1jZV\nne19/BJwQhPHR8JSMMYAVqCMcdrhxaWpYlPI/65s3OxLHBgTLKxAGeOsdO+id+BZCO+bxg72LmdQ\nJiKjvaP3rgbe8XNGYxxhBcoYZ60BrhGR5UAq8KQPr7kZeA7PMgf5gI3gM2HJhpkb4xAR6Q68p6oD\nHY5iTFCyFpQxxpigZC0oY4KUiMwH4g/bfJWqrnAijzGBZgXKGGNMULIuPmOMMUHJCpQxxpigZAXK\nGGNMULICZYwxJij9P5MjMQTJeWLeAAAAAElFTkSuQmCC\n",
197 |       "text/plain": [
198 |        "<matplotlib.figure.Figure at 0x1a0d05c0c18>"
199 |       ]
200 |      },
201 |      "metadata": {},
202 |      "output_type": "display_data"
203 |     }
204 |    ],
205 |    "source": [
206 |     "NUM_SAMPLES = 5000\n",
207 |     "N = 10\n",
208 |     "pi_0s = np.linspace(0,1,10)\n",
209 |     "\n",
210 |     "R = np.zeros(pi_0s.size)\n",
211 |     "\n",
212 |     "for i in range(0,pi_0s.size):\n",
213 |     "    this_pi0 = pi_0s[i]\n",
214 |     "    this_pi1 = 1- this_pi0\n",
215 |     "    threshold = np.log(((C_10-C_00)*this_pi0)/((C_01-C_11)*this_pi1))\n",
216 |     "\n",
217 |     "    for k in range(0,NUM_SAMPLES):\n",
218 |     "        #draw a sample from H, and then from p(y|H)\n",
219 |     "        H = np.random.choice(2,p=[this_pi0,this_pi1])\n",
220 |     "        if(H==0):\n",
221 |     "            y = np.random.multinomial(N,p_0)\n",
222 |     "            L = 0\n",
223 |     "            for j in range(0,len(dictionary)):\n",
224 |     "                L += y[j]*np.log(p_1[j]/p_0[j])\n",
225 |     "            if(L<threshold):\n",
226 |     "                R[i] += C_00/NUM_SAMPLES\n",
227 |     "            else:\n",
228 |     "                R[i] += C_10/NUM_SAMPLES\n",
229 |     "        else:\n",
230 |     "            y = np.random.multinomial(N,p_1)\n",
231 |     "            L = 0\n",
232 |     "            for j in range(0,len(dictionary)):\n",
233 |     "                L += y[j]*np.log(p_1[j]/p_0[j])\n",
234 |     "            if(L<threshold):\n",
235 |     "                R[i] += C_01/NUM_SAMPLES\n",
236 |     "            else:\n",
237 |     "                R[i] += C_11/NUM_SAMPLES  \n",
238 |     "\n",
239 |     "plt.plot(pi_0s,R)\n",
240 |     "\n",
241 |     "plt.xlabel(\"pi_0\")\n",
242 |     "plt.ylabel(\"Bayes Risk R\")\n",
243 |     "\n",
244 |     "plt.tight_layout()\n",
245 |     "plt.show()"
246 |    ]
247 |   },
248 |   {
249 |    "cell_type": "markdown",
250 |    "metadata": {},
251 |    "source": [
252 |     "It's worth fiddling both with the costs and the number of words N per sample. How do the affect the Bayes risk? Which prior is the worst-case prior, corresponding with the minimax detector? How does changing the costs impact this prior?"
253 |    ]
254 |   }
255 |  ],
256 |  "metadata": {
257 |   "kernelspec": {
258 |    "display_name": "Python 3",
259 |    "language": "python",
260 |    "name": "python3"
261 |   },
262 |   "language_info": {
263 |    "codemirror_mode": {
264 |     "name": "ipython",
265 |     "version": 3
266 |    },
267 |    "file_extension": ".py",
268 |    "mimetype": "text/x-python",
269 |    "name": "python",
270 |    "nbconvert_exporter": "python",
271 |    "pygments_lexer": "ipython3",
272 |    "version": "3.6.0"
273 |   }
274 |  },
275 |  "nbformat": 4,
276 |  "nbformat_minor": 2
277 | }
278 | 


--------------------------------------------------------------------------------
/ch.4/.ipynb_checkpoints/Bayesian Inference with Exponentials-checkpoint.ipynb:
--------------------------------------------------------------------------------
1 | {
2 |  "cells": [],
3 |  "metadata": {},
4 |  "nbformat": 4,
5 |  "nbformat_minor": 1
6 | }
7 | 


--------------------------------------------------------------------------------
/ch.4/.ipynb_checkpoints/Bias, Variance, and the Bias-variance Trade-off-checkpoint.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# Bias, Variance, and the Bias-variance Trade-off\n",
  8 |     "\n",
  9 |     "Here we'll spend time visualizing what it means for an estimator to be biased, what the variance of an estimator represents, and why there is an inherent trade-off between bias and variance. We'll do this with the simple example of estimating the mean and variance of a scalar Gaussian distribution."
 10 |    ]
 11 |   },
 12 |   {
 13 |    "cell_type": "code",
 14 |    "execution_count": 3,
 15 |    "metadata": {
 16 |     "collapsed": true
 17 |    },
 18 |    "outputs": [],
 19 |    "source": [
 20 |     "import numpy as np\n",
 21 |     "import matplotlib.pyplot as plt\n",
 22 |     "\n",
 23 |     "#Standard normal distribution\n",
 24 |     "mu = 0\n",
 25 |     "sigma_2 = 1"
 26 |    ]
 27 |   },
 28 |   {
 29 |    "cell_type": "markdown",
 30 |    "metadata": {},
 31 |    "source": [
 32 |     "First, suppose the mean is unknown but the variance known. Let's use the sample mean estimator, taken from n i.i.d. samples. To understand the estimator as a random variable, we'll loop over many *different* realizations of the data and its estimator.\n",
 33 |     "\n",
 34 |     "The histogram shows us that mu_hat is a Gaussian distribution. Its mean is clearly zero, which means that it's unbiased! And its variance is proportional to 1/n"
 35 |    ]
 36 |   },
 37 |   {
 38 |    "cell_type": "code",
 39 |    "execution_count": 18,
 40 |    "metadata": {
 41 |     "collapsed": false
 42 |    },
 43 |    "outputs": [
 44 |     {
 45 |      "data": {
 46 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADixJREFUeJzt3W2Ipeddx/Hvz01iJVaTZafTtek6FZZoFPLAEKItpbpJ\nDUnprqBLBHUoC4uiUkGQVcG+3fpCfMAHljQ6YlsTonGXJo1s1oYg1JBJXds8lY1hQzfs7kzT9CEq\nlrR/X8wdnCRz9pyZOWfOnGu+H1jO/XTm/Ofee35zzXWu+zqpKiRJk+97xl2AJGk4DHRJaoSBLkmN\nMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIy7bzBfbtWtXzczMbOZLStLEe/LJJ79aVVP9jtvU\nQJ+ZmWFhYWEzX1KSJl6SFwc5zi4XSWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMM\ndElqxKbeKSpNmpkjD/bcd/bonZtYidSfLXRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhph\noEtSIwYK9CRXJbk/yXNJnk3yk0l2JjmZ5Ez3ePWoi5Uk9TZoC/1PgIer6keB64FngSPAqaraC5zq\n1iVJY9I30JP8IPB+4BMAVfXtqvo6sB+Y7w6bBw6MqkhJUn+DtNDfAywBf53k35PcneRKYLqqznfH\nXACmR1WkJKm/QQL9MuAm4C+r6kbgv3hT90pVFVCrPTnJ4SQLSRaWlpY2Wq8kqYdBAv0ccK6qHu/W\n72c54C8m2Q3QPS6u9uSqOlZVs1U1OzU1NYyaJUmr6BvoVXUB+EqSa7tN+4BngBPAXLdtDjg+kgol\nSQMZdD703wQ+meQK4AXgIyz/MrgvySHgReDgaEqUJA1ioECvqtPA7Cq79g23HEnSenmnqCQ1wkCX\npEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElq\nxKDzoUsTZebIg2s6/uzRO0dUibR5bKFLUiMMdElqhF0uEmvvopG2IlvoktQIA12SGmGgS1IjDHRJ\naoSBLkmNGGiUS5KzwLeA7wCvVdVskp3AvcAMcBY4WFWvjKZMSVI/a2mh/3RV3VBVs936EeBUVe0F\nTnXrkqQx2UiXy35gvlueBw5svBxJ0noNGugFPJLkySSHu23TVXW+W74ATA+9OknSwAa9U/R9VfVS\nkncAJ5M8t3JnVVWSWu2J3S+AwwB79uzZULGSpN4GaqFX1Uvd4yLwAHAzcDHJboDucbHHc49V1WxV\nzU5NTQ2naknSW/QN9CRXJnn768vAB4GngBPAXHfYHHB8VEVKkvobpMtlGnggyevHf6qqHk7yBHBf\nkkPAi8DB0ZUpSeqnb6BX1QvA9atsfxnYN4qiJElr552iktQIA12SGmGgS1IjDHRJaoSBLkmNMNAl\nqREGuiQ1wkCXpEYMOjmXNFYzRx5cdfvZo3duciXS1mULXZIaYaBLUiPsctGW0qtrRVJ/ttAlqREG\nuiQ1wi4XjZRdKNLmsYUuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGuGwRU00h0VK/88WuiQ1wkCX\npEYM3OWSZAewALxUVR9KshO4F5gBzgIHq+qVURQpTZK1zt3uXO8alrW00D8KPLti/Qhwqqr2Aqe6\ndUnSmAwU6EmuAe4E7l6xeT8w3y3PAweGW5okaS0GbaH/MfA7wHdXbJuuqvPd8gVgerUnJjmcZCHJ\nwtLS0vorlSRdUt9AT/IhYLGqnux1TFUVUD32Hauq2aqanZqaWn+lkqRLGuRN0fcCH05yB/A24AeS\n/B1wMcnuqjqfZDewOMpCJUmX1reFXlW/W1XXVNUMcBfwL1X1S8AJYK47bA44PrIqJUl9bWQc+lHg\ntiRngFu7dUnSmKzp1v+qehR4tFt+Gdg3/JIkSevhnaKS1Agn59JQOEmWNH620CWpEQa6JDXCLhe9\nxaW6T5wwStq6bKFLUiMMdElqhF0u0jo5skdbjS10SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgD\nXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGtF3PvQkbwMeA763O/7+qvpY\nkp3AvcAMcBY4WFWvjK5UDdt65vN2DvDx6/V/4McDapAW+v8CP1NV1wM3ALcnuQU4Apyqqr3AqW5d\nkjQmfQO9lr3arV7e/StgPzDfbZ8HDoykQknSQAbqQ0+yI8lpYBE4WVWPA9NVdb475AIwPaIaJUkD\nGCjQq+o7VXUDcA1wc5KfeNP+YrnV/hZJDidZSLKwtLS04YIlSatb0yiXqvo68DngduBikt0A3eNi\nj+ccq6rZqpqdmpraaL2SpB76BnqSqSRXdcvfB9wGPAecAOa6w+aA46MqUpLUX99hi8BuYD7JDpZ/\nAdxXVZ9J8nngviSHgBeBgyOsU5p4DvnUqPUN9Kr6InDjKttfBvaNoihJ0tp5p6gkNcJAl6RGGOiS\n1AgDXZIaMcgoF0lj4KgYrZUtdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJ\naoSBLkmNMNAlqREGuiQ1wsm5tgEneZK2B1voktQIA12SGmGgS1IjDHRJaoSBLkmN6BvoSd6d5HNJ\nnknydJKPdtt3JjmZ5Ez3ePXoy5Uk9TJIC/014Ler6jrgFuDXk1wHHAFOVdVe4FS3Lkkak76BXlXn\nq+oL3fK3gGeBdwH7gfnusHngwKiKlCT1t6Y+9CQzwI3A48B0VZ3vdl0ApodamSRpTQYO9CTfD/wD\n8FtV9c2V+6qqgOrxvMNJFpIsLC0tbahYSVJvAwV6kstZDvNPVtU/dpsvJtnd7d8NLK723Ko6VlWz\nVTU7NTU1jJolSasYZJRLgE8Az1bVH63YdQKY65bngOPDL0+SNKhBJud6L/DLwJeSnO62/R5wFLgv\nySHgReDgaEqUJA2ib6BX1b8C6bF733DLkSStl3eKSlIjDHRJaoQfcNEQP8hie+v1/3/26J2bXInG\nxRa6JDXCQJekRtjlMoHsWpG0GlvoktQIA12SGmGXi9Q4R79sH7bQJakRBrokNcIuly3M0SyS1sIW\nuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEwxYlvYF3lk4uW+iS1AgDXZIaYZeLtE15J3J7bKFL\nUiMMdElqRN9AT3JPksUkT63YtjPJySRnuserR1umJKmfQVrofwPc/qZtR4BTVbUXONWtS5LGqG+g\nV9VjwNfetHk/MN8tzwMHhlyXJGmN1jvKZbqqznfLF4DpXgcmOQwcBtizZ886X64NjiqQNEobflO0\nqgqoS+w/VlWzVTU7NTW10ZeTJPWw3kC/mGQ3QPe4OLySJEnrsd5APwHMdctzwPHhlCNJWq9Bhi1+\nGvg8cG2Sc0kOAUeB25KcAW7t1iVJY9T3TdGq+sUeu/YNuRZJ0gZ4p6gkNcJAl6RGGOiS1AgDXZIa\n4XzokjbkUndA+7F1m8sWuiQ1wkCXpEY02+UyzE8u91PQJU0CW+iS1AgDXZIaYaBLUiOa7UNfj2F9\nAIUfZCFtDt/feiNb6JLUCANdkhqx7bpc7A6R1mc9Pztr7RKxC2VjbKFLUiMMdElqxMR3udiFIknL\nbKFLUiMMdElqxMR0uWzFrpWtWJOk7csWuiQ1wkCXpEZsKNCT3J7ky0meT3JkWEVJktZu3X3oSXYA\nfw7cBpwDnkhyoqqeGVZxkto06vefxnXH6bg/X3UjLfSbgeer6oWq+jbw98D+4ZQlSVqrjQT6u4Cv\nrFg/122TJI3ByIctJjkMHO5WX03y5QGetgv46uiqaoLnqD/PUX8TcY7y8bF+naGcow1+Dz88yEEb\nCfSXgHevWL+m2/YGVXUMOLaWL5xkoapmN1Bb8zxH/XmO+vMc9TdJ52gjXS5PAHuTvCfJFcBdwInh\nlCVJWqt1t9Cr6rUkvwH8M7ADuKeqnh5aZZKkNdlQH3pVPQQ8NKRaVlpTF8025Tnqz3PUn+eov4k5\nR6mqcdcgSRoCb/2XpEZsiUBP8gtJnk7y3SQ9303ezlMNJNmZ5GSSM93j1T2OO5vkS0lOJ1nY7DrH\nod91kWV/2u3/YpKbxlHnOA1wjj6Q5BvddXM6yR+Mo85xSXJPksUkT/XYPxnXUFWN/R/wY8C1wKPA\nbI9jdgD/CfwIcAXwH8B14659E8/RHwJHuuUjwMd7HHcW2DXuejfxvPS9LoA7gM8CAW4BHh933Vvw\nHH0A+My4ax3jOXo/cBPwVI/9E3ENbYkWelU9W1X9bjja7lMN7Afmu+V54MAYa9lKBrku9gN/W8v+\nDbgqye7NLnSMtvvPTl9V9RjwtUscMhHX0JYI9AFt96kGpqvqfLd8AZjucVwBjyR5srtLt3WDXBfb\n/doZ9Pv/qa474bNJfnxzSpsYE3ENbdonFiV5BHjnKrt+v6qOb1YdW9mlztHKlaqqJL2GJ72vql5K\n8g7gZJLnutaHdClfAPZU1atJ7gD+Cdg75pq0RpsW6FV16wa/xEBTDUyyS52jJBeT7K6q892feos9\nvsZL3eNikgdY/nO75UAf5Lpo/trpo+/3X1XfXLH8UJK/SLKrqrb8PC+bZCKuoUnqctnuUw2cAOa6\n5TngLX/VJLkyydtfXwY+CKz6rn1DBrkuTgC/0o1UuAX4xoruq+2g7zlK8s4k6ZZvZjkbXt70Sreu\nibiGtsSHRCf5OeDPgCngwSSnq+pnk/wQcHdV3VFONXAUuC/JIeBF4CDAynPEcr/6A93P5WXAp6rq\n4THVuyl6XRdJfrXb/1cs3818B/A88N/AR8ZV7zgMeI5+Hvi1JK8B/wPcVd3wju0gyadZHumzK8k5\n4GPA5TBZ15B3ikpSIyapy0WSdAkGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5Jjfg//SGq\nW4gp+soAAAAASUVORK5CYII=\n",
 47 |       "text/plain": [
 48 |        "<matplotlib.figure.Figure at 0x10dd66748>"
 49 |       ]
 50 |      },
 51 |      "metadata": {},
 52 |      "output_type": "display_data"
 53 |     }
 54 |    ],
 55 |    "source": [
 56 |     "NUM_SAMPLES = 1000\n",
 57 |     "n = 10\n",
 58 |     "\n",
 59 |     "mu_hat = np.zeros(NUM_SAMPLES)\n",
 60 |     "for i in range(0,NUM_SAMPLES):\n",
 61 |     "    y = np.random.randn(n) #n i.i.d. samples\n",
 62 |     "    mu_hat[i] = np.mean(y)\n",
 63 |     "\n",
 64 |     "plt.hist(mu_hat,50)\n",
 65 |     "plt.show()\n",
 66 |     "    "
 67 |    ]
 68 |   },
 69 |   {
 70 |    "cell_type": "markdown",
 71 |    "metadata": {},
 72 |    "source": [
 73 |     "Next, let's try using the sample *variance* estimator, supposing that the mean and variance are both unknown. We want to see that this estimator is in fact biased!"
 74 |    ]
 75 |   },
 76 |   {
 77 |    "cell_type": "code",
 78 |    "execution_count": 24,
 79 |    "metadata": {
 80 |     "collapsed": false
 81 |    },
 82 |    "outputs": [
 83 |     {
 84 |      "data": {
 85 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEe9JREFUeJzt3WGMZWddx/Hvj3UtxJJQ0nHddrduSRaTluiWTFYixlQI\ntqkvFgxplhe1muqiKSiJL9jyQsBkk5pAiRrBLKFhMUDdCMgGiqRtaggJtkybUrpbKqtt090s3RFF\n2miqu/x9cU/rZd3Ze2buvTN3nvl+ksmc+5xz5v73ofzmmec859xUFZKkdr1srQuQJE2XQS9JjTPo\nJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklq3E+sdQEAl156ae3YsWOty5CkdeWhhx7616qa\nG3XcTAT9jh07WFhYWOsyJGldSfJ0n+OcupGkxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEG\nvSQ1zqCXpMbNxJ2xGm3H/i+/tP3U7b++hpVIWm8c0UtS4wx6SWqcQS9JjTPoJalxIy/GJnk58DXg\nou74v62q9yf5APC7wGJ36Puq6u7unNuAW4CzwB9U1VenULv48Yu04IVaSf9fn1U3LwBvqqrnk2wG\nvp7kK92+j1TVh4YPTnIVsBe4GrgMuDfJa6vq7CQLX+9cRSNptYycuqmB57uXm7uvusApe4C7quqF\nqnoSOA7sHrtSSdKK9JqjT7IpySPAaeCeqnqg2/XuJI8muTPJJV3b5cAzQ6ef6NokSWugV9BX1dmq\n2gVsA3YneR3wMeA1wC7gFPDh5bxxkn1JFpIsLC4ujj5BkrQiy1p1U1U/AO4Hrq+qZ7tfAD8CPs7/\nTc+cBLYPnbatazv3Zx2sqvmqmp+bG/nZtpKkFRoZ9Enmkryq234F8BbgO0m2Dh32NuCxbvsIsDfJ\nRUmuBHYCD062bElSX31W3WwFDiXZxOAXw+Gq+lKSv06yi8GF2aeAdwJU1dEkh4FjwBngVlfcDJy7\nFFKSVsPIoK+qR4FrztN+0wXOOQAcGK80SdIk+PTKGeZfAJImwUcgSFLjDHpJapxBL0mNM+glqXEG\nvSQ1zlU3jfGpmJLO5YhekhrniH4dcn29pOUw6GeA0y2SpsmpG0lqnEEvSY1z6mbGOP8uadIc0UtS\n4wx6SWqcQS9JjXOOfsqcc5e01hzRS1LjDHpJatzIoE/y8iQPJvlWkqNJPti1vzrJPUm+232/ZOic\n25IcT/JEkuum+Q+QJF1Ynzn6F4A3VdXzSTYDX0/yFeA3gPuq6vYk+4H9wHuTXAXsBa4GLgPuTfLa\nqjo7pX+DluCjFSRBjxF9DTzfvdzcfRWwBzjUtR8C3tpt7wHuqqoXqupJ4Diwe6JVS5J66zVHn2RT\nkkeA08A9VfUAsKWqTnWHfA/Y0m1fDjwzdPqJrk2StAZ6BX1Vna2qXcA2YHeS152zvxiM8ntLsi/J\nQpKFxcXF5ZwqSVqGZa26qaofAPcD1wPPJtkK0H0/3R12Etg+dNq2ru3cn3Wwquaran5ubm4ltUuS\nehh5MTbJHPA/VfWDJK8A3gL8KXAEuBm4vfv+xe6UI8BnktzB4GLsTuDBKdQ+s7xJStIs6bPqZitw\nKMkmBn8BHK6qLyX5BnA4yS3A08CNAFV1NMlh4BhwBrjVFTeStHZGBn1VPQpcc5727wNvXuKcA8CB\nsauTJI3NZ91sEK6plzYug34DMvSljcVn3UhS4xzR6yWO9KU2OaKXpMYZ9JLUOINekhpn0EtS4wx6\nSWqcQS9JjTPoJalxBr0kNc6gl6TGeWfsBuez86X2GfRj8JEBktYDg35CHBlLmlXO0UtS4wx6SWqc\nQS9JjTPoJalxI4M+yfYk9yc5luRokj/s2j+Q5GSSR7qvG4bOuS3J8SRPJLlumv8ASdKF9Vl1cwb4\no6p6OMkrgYeS3NPt+0hVfWj44CRXAXuBq4HLgHuTvLaqzk6ycElSPyNH9FV1qqoe7rafAx4HLr/A\nKXuAu6rqhap6EjgO7J5EsZKk5VvWHH2SHcA1wANd07uTPJrkziSXdG2XA88MnXaC8/xiSLIvyUKS\nhcXFxWUXLknqp3fQJ7kY+Bzwnqr6IfAx4DXALuAU8OHlvHFVHayq+aqan5ubW86pkqRl6BX0STYz\nCPlPV9XnAarq2ao6W1U/Aj7O/03PnAS2D52+rWuTJK2BPqtuAnwCeLyq7hhq3zp02NuAx7rtI8De\nJBcluRLYCTw4uZIlScvRZ9XNG4GbgG8neaRrex/wjiS7gAKeAt4JUFVHkxwGjjFYsXOrK24kae2M\nDPqq+jqQ8+y6+wLnHAAOjFGXJGlCvDNWkhrnY4p1Xj5rX2qHQa+RDH1pfXPqRpIaZ9BLUuMMeklq\nnEEvSY0z6CWpca66WabhFSiStB44opekxhn0ktQ4g16SGmfQS1LjDHpJapyrbrQsPvdGWn8c0UtS\n4wx6SWqcQS9JjTPoJalxIy/GJtkOfArYwuCDwA9W1Z8leTXwN8AOBh8OfmNV/Xt3zm3ALcBZ4A+q\n6qtTqX6V+NgDSetZn1U3Z4A/qqqHk7wSeCjJPcBvAfdV1e1J9gP7gfcmuQrYC1wNXAbcm+S1VXV2\nOv8EzQJX40iza+TUTVWdqqqHu+3ngMeBy4E9wKHusEPAW7vtPcBdVfVCVT0JHAd2T7pwSVI/y5qj\nT7IDuAZ4ANhSVae6Xd9jMLUDg18CzwyddqJrkyStgd5Bn+Ri4HPAe6rqh8P7qqoYzN/3lmRfkoUk\nC4uLi8s5VZK0DL2CPslmBiH/6ar6fNf8bJKt3f6twOmu/SSwfej0bV3bj6mqg1U1X1Xzc3NzK61f\nkjRCn1U3AT4BPF5VdwztOgLcDNzeff/iUPtnktzB4GLsTuDBSRat2eBqJGl96LPq5o3ATcC3kzzS\ntb2PQcAfTnIL8DRwI0BVHU1yGDjGYMXOra64kaS1MzLoq+rrQJbY/eYlzjkAHBijLknShPj0Sk2c\na+ql2eIjECSpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjXF65BO/6nAyXWkprzxG9JDXOoJekxhn0\nktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUuJFBn+TOJKeTPDbU9oEkJ5M8\n0n3dMLTvtiTHkzyR5LppFS5J6qfPiP6TwPXnaf9IVe3qvu4GSHIVsBe4ujvno0k2TapYSdLyjXx6\nZVV9LcmOnj9vD3BXVb0APJnkOLAb+MaKK1xFPrFSUovGmaN/d5JHu6mdS7q2y4Fnho450bVJktbI\nSoP+Y8BrgF3AKeDDy/0BSfYlWUiysLi4uMIyJEmjrCjoq+rZqjpbVT8CPs5gegbgJLB96NBtXdv5\nfsbBqpqvqvm5ubmVlCFJ6mFFQZ9k69DLtwEvrsg5AuxNclGSK4GdwIPjlShJGsfIi7FJPgtcC1ya\n5ATwfuDaJLuAAp4C3glQVUeTHAaOAWeAW6vq7HRKlyT10WfVzTvO0/yJCxx/ADgwTlGSpMnxzlhJ\natzIEb00KcP3KTx1+6+vYSXSxmLQa00Y+tLqcepGkhpn0EtS4wx6SWqcQS9JjfNirGaKF2mlyXNE\nL0mNM+glqXEGvSQ1zjl6rTk/2UuaLkf0ktQ4g16SGmfQS1LjnKPXzHJNvTQZjuglqXEGvSQ1bsNP\n3bi0T1LrHNFLUuNGBn2SO5OcTvLYUNurk9yT5Lvd90uG9t2W5HiSJ5JcN63CtXHt2P/ll74kjdZn\nRP9J4Ppz2vYD91XVTuC+7jVJrgL2Ald353w0yaaJVStJWraRc/RV9bUkO85p3gNc220fAv4BeG/X\nfldVvQA8meQ4sBv4xmTKnQxHguuP/5tJK7fSOfotVXWq2/4esKXbvhx4Zui4E13b/5NkX5KFJAuL\ni4srLEOSNMrYF2OrqoBawXkHq2q+qubn5ubGLUOStISVBv2zSbYCdN9Pd+0nge1Dx23r2iRJa2Sl\nQX8EuLnbvhn44lD73iQXJbkS2Ak8OF6JkqRxjLwYm+SzDC68XprkBPB+4HbgcJJbgKeBGwGq6miS\nw8Ax4Axwa1WdnVLtkqQe+qy6eccSu968xPEHgAPjFCVJmhzvjJWkxhn0ktQ4g16SGrfhn16pdlzo\n7lk/uEQbmSN6SWqcQS9JjXPqRhuCnz+rjcwRvSQ1zqCXpMYZ9JLUOINekhrnxVita37ylDSaQa8N\nxxU42micupGkxjmilzqO9NUqg14bmnP82gicupGkxm2YEb0jN0kblSN6SWrcWCP6JE8BzwFngTNV\nNZ/k1cDfADuAp4Abq+rfxytTkrRSkxjR/2pV7aqq+e71fuC+qtoJ3Ne9liStkWlM3ewBDnXbh4C3\nTuE9JEk9jRv0Bdyb5KEk+7q2LVV1qtv+HrBlzPeQJI1h3FU3v1xVJ5P8NHBPku8M76yqSlLnO7H7\nxbAP4IorrhizDEnSUsYa0VfVye77aeALwG7g2SRbAbrvp5c492BVzVfV/Nzc3DhlSJIuYMUj+iQ/\nBbysqp7rtn8N+BPgCHAzcHv3/YuTKHQlXDsvSeNN3WwBvpDkxZ/zmar6+yTfBA4nuQV4Grhx/DIl\nSSu14qCvqn8BfuE87d8H3jxOUZKkyfHOWElq3IZ51o20HD6yWC0x6KUJ8BeDZplTN5LUOEf00jI4\nctd65IhekhrniF5aIW/I03ph0EsjGOha7wx6acKcx9escY5ekhrX3IjeP7Ml6cc1F/TSeuNUj6bN\nqRtJapwjemkNOMWo1ZSq837S36qan5+vhYWFFZ/v/2nUuuEpHad69KIkD1XV/KjjHNFL65ihrz6c\no5ekxjmil9YBpyc1DoNeasRS0zh9p3ecBmrX1KZuklyf5Ikkx5Psn9b7SJIubCoj+iSbgL8E3gKc\nAL6Z5EhVHZvG+0nqZ9wpoHFG/f7FsHamNXWzGzheVf8CkOQuYA9g0EvrjNcH1r9pBf3lwDNDr08A\nvzil95J0jkmO3KfxHkudO879AuNco7jQv6XPOeO832r8dTOVG6aSvB24vqp+p3t9E/CLVfWuoWP2\nAfu6lz8HPLHEj7sU+NeJFzld1rw6rHl1WPPqWEnNP1tVc6MOmtaI/iSwfej1tq7tJVV1EDg46gcl\nWehz59cssebVYc2rw5pXxzRrntaqm28CO5NcmeQngb3AkSm9lyTpAqYyoq+qM0neBXwV2ATcWVVH\np/FekqQLm9oNU1V1N3D3BH7UyOmdGWTNq8OaV4c1r46p1TwTT6+UJE2PDzWTpMbNRNCPelxCBv68\n2/9oktevRZ3n1DSq5muT/EeSR7qvP16LOs+p6c4kp5M8tsT+WeznUTXPVD8n2Z7k/iTHkhxN8ofn\nOWam+rlnzbPWzy9P8mCSb3U1f/A8x8xaP/epeTr9XFVr+sXgYu0/A68BfhL4FnDVOcfcAHwFCPAG\n4IF1UPO1wJfWun/PqelXgNcDjy2xf6b6uWfNM9XPwFbg9d32K4F/Wgf/Pfepedb6OcDF3fZm4AHg\nDTPez31qnko/z8KI/qXHJVTVfwMvPi5h2B7gUzXwj8Crkmxd7UKH9Kl55lTV14B/u8Ahs9bPfWqe\nKVV1qqoe7rafAx5ncKf4sJnq5541z5Su757vXm7uvs694Dhr/dyn5qmYhaA/3+MSzv2PrM8xq6lv\nPb/U/cn4lSRXr05pY5m1fu5rJvs5yQ7gGgYjt2Ez288XqBlmrJ+TbEryCHAauKeqZr6fe9QMU+jn\nWQj6Vj0MXFFVPw/8BfB3a1xPq2ayn5NcDHwOeE9V/XCt6+ljRM0z189VdbaqdjG48353ktetdU2j\n9Kh5Kv08C0E/8nEJPY9ZTX0e8fDDF/9Mq8E9BZuTXLp6Ja7IrPXzSLPYz0k2MwjMT1fV589zyMz1\n86iaZ7GfX1RVPwDuB64/Z9fM9fOLlqp5Wv08C0Hf53EJR4Df7K6ivwH4j6o6tdqFDhlZc5KfSZJu\nezeDvv7+qle6PLPWzyPNWj93tXwCeLyq7ljisJnq5z41z2A/zyV5Vbf9CgafffGdcw6btX4eWfO0\n+nnNP0qwlnhcQpLf6/b/FYM7bG8AjgP/Cfz2WtXb1dSn5rcDv5/kDPBfwN7qLquvlSSfZXBV/9Ik\nJ4D3M7ggNJP9DL1qnrV+fiNwE/Dtbi4W4H3AFTCz/dyn5lnr563AoQw+5OhlwOGq+tIs5wb9ap5K\nP3tnrCQ1bhambiRJU2TQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUuP8FZiUGSrNVMcAA\nAAAASUVORK5CYII=\n",
 86 |       "text/plain": [
 87 |        "<matplotlib.figure.Figure at 0x10db9bfd0>"
 88 |       ]
 89 |      },
 90 |      "metadata": {},
 91 |      "output_type": "display_data"
 92 |     }
 93 |    ],
 94 |    "source": [
 95 |     "NUM_SAMPLES = 10000\n",
 96 |     "n = 10\n",
 97 |     "\n",
 98 |     "sigma_2_hat = np.zeros(NUM_SAMPLES)\n",
 99 |     "for i in range(0,NUM_SAMPLES):\n",
100 |     "    y = np.random.randn(n) #n i.i.d. samples\n",
101 |     "    mu_hat = np.mean(y)\n",
102 |     "    sigma_2_hat[i] = np.mean((y - mu_hat)**2)\n",
103 |     "\n",
104 |     "plt.hist(sigma_2_hat,100)\n",
105 |     "plt.show()"
106 |    ]
107 |   },
108 |   {
109 |    "cell_type": "markdown",
110 |    "metadata": {},
111 |    "source": [
112 |     "Notice a few things. First, the sample variance is *not* Gaussian. In fact, it is distributed according to a *chi-square* distribution, which is roughly the distribution of the *square* of a Gaussian random variable. Second, and more important, the mean of this distribution is clearly not one. This is because the sample variance is biased by the constant $(n-1)/n)$. If we increase $n$, we get a distribution that is closer to being unbiased.\n",
113 |     "\n",
114 |     "Finally, let's look at the bias variance trade-off by altering the sample mean estimator from above. We'll see that by \"overdoing\" the averaging of samples, we can reduce the variance of the estimator, but we add bias."
115 |    ]
116 |   },
117 |   {
118 |    "cell_type": "code",
119 |    "execution_count": 39,
120 |    "metadata": {
121 |     "collapsed": false
122 |    },
123 |    "outputs": [
124 |     {
125 |      "data": {
126 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGBZJREFUeJzt3X2QVPW95/H3Bxwca/GKDIRFRnYwwU2YTAg4xSbGBwJl\n4EayaEys2aA7q1SZ3WKNRuMyPtRqtkIVqVSlkpunXUrU0cVrEK9KGfUWEW8Z4wN3IChPGlmByxCU\ncRASsguXId/9Yw7cBmeYnpnu6Z7ffF5VU336d87p/p7j8dOHX5/za0UEZmaWrmGlLsDMzIrLQW9m\nljgHvZlZ4hz0ZmaJc9CbmSXOQW9mljgHvZlZ4hz0ZmaJc9CbmSXujFIXADBmzJioqakpdRlmZoPK\n+vXrP4iIsT0tVxZBX1NTQ0tLS6nLMDMbVCTtyme5vLpuJI2StErSW5K2Sfq8pNGS1kh6J3s8N2f5\nOyVtl/S2pDl93QgzM+u/fPvofww8HxGfBKYC24Am4IWImAy8kD1H0hSgAagF5gI/lzS80IWbmVl+\negx6SecAlwHLASLinyPiADAfaM4WawauyqbnA49FxJGI2AFsB2YUunAzM8tPPn30k4A24EFJU4H1\nwC3AuIjYmy3zHjAum54AvJazfmvWdhJJNwE3AUycOLFPxZtZeTt69Citra0cPny41KUMapWVlVRX\nV1NRUdGn9fMJ+jOA6cDNEfG6pB+TddMcFxEhqVcD20fEMmAZQH19vQfFN0tQa2srZ599NjU1NUgq\ndTmDUkTQ3t5Oa2srkyZN6tNr5NNH3wq0RsTr2fNVdAb/+5LGA2SP+7L5e4Dzc9avztrMbIg5fPgw\nVVVVDvl+kERVVVW//lXUY9BHxHvAbkn/NmuaDWwFVgONWVsj8HQ2vRpokHSmpEnAZGBdnys0s0HN\nId9//d2H+V5HfzOwQtII4F3gBjo/JFZKWgjsAq4FiIgtklbS+WHQASyKiGP9qtLMzPosr6CPiI1A\nfRezZnez/BJgST/qMrME1TT9qqCvt3PplT0vs3Mn8+bNY/PmzSfa7rvvPkaOHMl3vvOdLtd56KGH\naGlp4ac//elH5l188cW88sorfS+6l7UWQlncGWunV9dcx6bGTaUuIz/3nZMzfbB0dZRQd2GWTyhZ\n+StWyBeTBzUzsyFr5syZLF68mBkzZnDhhRfym9/85sS83bt3M3PmTCZPnsx3v/vdE+0jR44E4NCh\nQ8yePZvp06dTV1fH0093fk355z//mSuvvJKpU6fy6U9/ml/+8pcArF+/nssvv5yLLrqIOXPmsHfv\n3hPtU6dOZerUqfzsZz8rynY66M1sSOvo6GDdunX86Ec/OinQ161bxxNPPMGbb77J448//pHxuCor\nK3nyySfZsGEDL774IrfffjsRwfPPP895553HG2+8webNm5k7dy5Hjx7l5ptvZtWqVaxfv54bb7yR\nu+++G4AbbriBn/zkJ7zxxhtF20Z33Qwig6oL51Tu0umWu3qKq7srVo63f/WrXwXgoosuYufOnSfm\nX3HFFVRVVZ1Y5uWXX6a+/l++qowI7rrrLl566SWGDRvGnj17eP/996mrq+P2229n8eLFzJs3j0sv\nvZTNmzezefNmrrjiCgCOHTvG+PHjOXDgAAcOHOCyyy4D4Prrr+e5554r+D5w0Jepsgz1xMPagZum\nqqoqPvzww5Pa9u/ff+LmozPPPBOA4cOH09HRcWKZUz8gTn2+YsUK2traWL9+PRUVFdTU1HD48GEu\nvPBCNmzYwLPPPss999zD7Nmzufrqq6mtreXVV1896TUOHDhQsO08HQd9Gatrrit1Cd3LDf3EFfpK\nERtYI0eOZPz48axdu5ZZs2axf/9+nn/+eW655RYefPDBbtdbs2YN+/fv56yzzuKpp57igQceOGn+\nwYMH+djHPkZFRQUvvvgiu3Z1jhj8hz/8gdGjR3PdddcxatQo7r//fpqammhra+PVV1/l85//PEeP\nHuX3v/89tbW1jBo1ipdffplLLrmEFStWFGUfOOgHibIO/e4MoQ+DfPgDo3T/Onr44YdZtGgRt912\nGwD33nsvH//4x0+7zowZM7jmmmtobW3luuuuO6nbBmDBggV85Stfoa6ujvr6ej75yU8CsGnTJu64\n4w6GDRtGRUUFv/jFLxgxYgSrVq3iW9/6FgcPHqSjo4Nbb72V2tpaHnzwQW688UYk8aUvfako26+I\n0g8zU19fH/7hkZN1F+wl7c4pVHCXabdPuQVxCl1G27Zt41Of+lSpy0hCV/tS0vqI6Ooep5P4qhsz\ns8Q56M3MEuegNzNLnIPezCxxDnozs8Q56M3MEufr6M1s4BT63ooiXqo7d+5cXnvtNS655BKeeeaZ\nE+07duygoaGB9vZ2LrroIh555BFGjBhRtDoKwWf0ZmZduOOOO3jkkUc+0r548WK+/e1vs337ds49\n91yWL19egup6x0FvZsl7+OGH+cxnPsPUqVO5/vrr81pn9uzZnH322Se1RQRr167la1/7GgCNjY08\n9dRTBa+30Nx1Y2ZJ27JlC9/73vd45ZVXGDNmDPv372fFihX84Ac/+Miyn/jEJ1i1alW3r9Xe3s6o\nUaM444zO6KyurmbPnj1Fq71QHPRmlrS1a9fy9a9/nTFjxgAwevRoFixYwIIFC0pc2cBx0JvZkNPX\nM/qqqioOHDhAR0cHZ5xxBq2trUyYMKGYpRaEg97MkjZr1iyuvvpqbrvtNqqqqti/f3+fz+gl8cUv\nfpFVq1bR0NBAc3Mz8+fPL0LVheWgN7OBU4KRS2tra7n77ru5/PLLGT58ONOmTeOhhx7qcb1LL72U\nt956i0OHDlFdXc3y5cuZM2cO3//+92loaOCee+5h2rRpLFy4sPgb0U8OejNLXmNjI42Njb1aJ/eH\nwnNdcMEFrFu3rhBlDRhfXmlmljgHvZlZ4hz0ZmaJy6uPXtJO4E/AMaAjIuoljQZ+CdQAO4FrI+LD\nbPk7gYXZ8t+KiL8veOVDVF1zXWl/TrAQcsc7KdOfFSwH3f20YQo/MWgDqzdfxn4xIj7Ied4EvBAR\nSyU1Zc8XS5oCNAC1wHnAryVdGBHHClZ14srqh8AT/IHvcvttWLNi60/XzXygOZtuBq7KaX8sIo5E\nxA5gOzCjH+9jZmb9kO8ZfdB5Zn4M+F8RsQwYFxF7s/nvAeOy6QnAaznrtmZtZjbEFfpfq8Xsxhw+\nfDh1dZ31Tpw4kdWrVwNpD1N8SUR8FvhrYJGky3JnRkTQ+WGQN0k3SWqR1NLW1tabVc3Miu6ss85i\n48aNbNy48UTIQ8LDFEfEnuxxH/AknV0x70saD5A97ssW3wOcn7N6ddZ26msui4j6iKgfO3Zs37fA\nzKwHfRmmuCvJDlMs6V8BwyLiT9n0l4D/AawGGoGl2ePT2SqrgUcl/ZDOL2MnA4PrNjIzS0Zfhyk+\nfPgw06dPZ8SIETQ1NXHVVVclPUzxOOBJSceXfzQinpf0j8BKSQuBXcC1ABGxRdJKYCvQASzyFTdm\nVip9HaZ4165dTJgwgXfffZdZs2ZRV1fHOecMzqvQegz6iHgXmNpFezswu5t1lgBL+l2dmVkR5HNG\nf3z44QsuuICZM2fyu9/9jmuuuWZQDlPsO2PNLGmzZs3i8ccfp729HeDEMMXHv2jN/Tse8h9++CFH\njhwB4IMPPuC3v/0tU6ZMOWmYYsDDFJuZnaoUd3X3ZZjibdu28c1vfpNhw4bxl7/8haamJqZMmQLg\nYYrNzMpRb4cpvvjii9m0qesPJQ9TbGZmZcdBb2aWOAe9mRVV543z1h/93Yfuox+Ekhiq+LhTR8f0\nsMU9GkzDF1dWVtLe3k5VVRXZvTjWSxFBe3s7lZWVfX4NB72ZFU11dTWtra14PKv+qayspLq6us/r\nO+jNrGgqKiqYNGlSqcsY8txHb2aWOAe9mVniHPRmZolz0JuZJc5Bb2aWOAe9mVniHPRmZolz0JuZ\nJc43TJWZuua6Upfw0WEJBqHuhgkwG4p8Rm9mljgHvZlZ4hz0ZmaJcx/9IJXUUMW5cr8f8JDFvTKY\nhi+2geUzejOzxPmMvoyU9IqbQXqlja+uMeuZz+jNzBLnoDczS5yD3swscXkHvaThkn4n6Zns+WhJ\nayS9kz2em7PsnZK2S3pb0pxiFG5mZvnpzZextwDbgL/KnjcBL0TEUklN2fPFkqYADUAtcB7wa0kX\nRsSxAtZtJHyJ5XG+1LIgTveFtS+9HBryOqOXVA1cCdyf0zwfaM6mm4Grctofi4gjEbED2A7MKEy5\nZmbWW/l23fwI+G/AX3LaxkXE3mz6PWBcNj0B2J2zXGvWdhJJN0lqkdTS1tbWu6rNzCxvPQa9pHnA\nvohY390yERFA9OaNI2JZRNRHRP3YsWN7s6qZmfVCPn30XwD+vaQvA5XAX0n638D7ksZHxF5J44F9\n2fJ7gPNz1q/O2szMrAR6PKOPiDsjojoiauj8knVtRFwHrAYas8Uagaez6dVAg6QzJU0CJgPrCl65\nmZnlpT9DICwFVkpaCOwCrgWIiC2SVgJbgQ5gka+4MTMrnV4FfUT8A/AP2XQ7MLub5ZYAS/pZm5mZ\nFYAHNSsTJRnQbBANZObBy8z6zkMgmJklzkFvZpY4B72ZWeIc9GZmiXPQm5klzkFvZpY4B72ZWeIc\n9GZmiXPQm5klzkFvZpY4B72ZWeIc9GZmiXPQm5klzkFvZpY4B72ZWeIc9GZmiXPQm5klzkFvZpY4\n/5TgIFfXXMemxk2lLqPodlZ+48R0zeFHS1hJWrr7icadS68c4EqsmHxGb2aWOAe9mVniHPRmZolz\n0JuZJc5BX2J1zXXFf5P7zvmXPzMbchz0ZmaJ6zHoJVVKWifpDUlbJH03ax8taY2kd7LHc3PWuVPS\ndklvS5pTzA0wM7PTy+eM/ggwKyKmAp8F5kr6HNAEvBARk4EXsudImgI0ALXAXODnkoYXo3gzM+tZ\nj0EfnQ5lTyuyvwDmA81ZezNwVTY9H3gsIo5ExA5gOzCjoFWbmVne8uqjlzRc0kZgH7AmIl4HxkXE\n3myR94Bx2fQEYHfO6q1Zm5mZlUBeQR8RxyLis0A1MEPSp0+ZH3Se5edN0k2SWiS1tLW19WZVMzPr\nhV5ddRMRB4AX6ex7f1/SeIDscV+22B7g/JzVqrO2U19rWUTUR0T92LFj+1K7mZnlIZ+rbsZKGpVN\nnwVcAbwFrAYas8Uagaez6dVAg6QzJU0CJgPrCl24mZnlJ5/RK8cDzdmVM8OAlRHxjKRXgZWSFgK7\ngGsBImKLpJXAVqADWBQRx4pTfhoG5Kap43zTlNmQ02PQR8SbwLQu2tuB2d2sswRY0u/qzMys3zwe\nfQKGypj0x3ls+uLzOPVp8RAIZmaJc9CbmSXOQW9mljgHvZlZ4hz0ZmaJc9CbmSXOQW9mljgHvZlZ\n4hz0ZmaJc9CbmSXOQW9mljgHvZlZ4hz0ZmaJc9CbmSXOQW9mljgHvZlZ4hz0ZmaJc9CbmSXOQW9m\nljgHvZlZ4hz0ZmaJc9CbmSXujFIXMJTVNdcV78XvO6d4r21mg4rP6M3MEuegNzNLXI9BL+l8SS9K\n2ippi6RbsvbRktZIeid7PDdnnTslbZf0tqQ5xdwAMzM7vXz66DuA2yNig6SzgfWS1gD/CXghIpZK\nagKagMWSpgANQC1wHvBrSRdGxLHibIJBZ3//psZNpS5jwO2s/MaJ6ZrDj5awkqGhpulXXbbvXHrl\nAFdivdHjGX1E7I2IDdn0n4BtwARgPtCcLdYMXJVNzwcei4gjEbED2A7MKHThZmaWn1710UuqAaYB\nrwPjImJvNus9YFw2PQHYnbNaa9ZmZmYlkHfQSxoJPAHcGhF/zJ0XEQFEb95Y0k2SWiS1tLW19WZV\nMzPrhbyCXlIFnSG/IiL+Lmt+X9L4bP54YF/Wvgc4P2f16qztJBGxLCLqI6J+7Nixfa3fzMx6kM9V\nNwKWA9si4oc5s1YDjdl0I/B0TnuDpDMlTQImA+sKV7KZmfVGPlfdfAG4HtgkaWPWdhewFFgpaSGw\nC7gWICK2SFoJbKXzip1FvuLGzKx0egz6iHgZUDezZ3ezzhJgST/qMjOzAvGdsWZmiXPQm5klzkFv\nZpY4B72ZWeIc9GZmifMPjyRkqA5sdpwHOCsdD3ZW3hz0JVK0X5fyL0uZ2SncdWNmljgHvZlZ4tx1\nk5i6SRPZtOOfSl1Gybm/vjy47748+IzezCxxDnozs8Q56EugaFfcmJl1wUFvZpY4B72ZWeIc9GZm\niXPQm5klzkFvZpY4B72ZWeIc9GZmiXPQm5klzkFvZpY4B72ZWeIc9GZmiXPQm5klzkFvZpY4B72Z\nWeJ6DHpJD0jaJ2lzTttoSWskvZM9npsz705J2yW9LWlOsQo3M7P85HNG/xAw95S2JuCFiJgMvJA9\nR9IUoAGozdb5uaThBavWzMx6rcegj4iXgP2nNM8HmrPpZuCqnPbHIuJIROwAtgMzClSrmZn1QV/7\n6MdFxN5s+j1gXDY9Adids1xr1vYRkm6S1CKppa2trY9lDD7+dSkzG2j9/jI2IgKIPqy3LCLqI6J+\n7Nix/S3DzMy60degf1/SeIDscV/Wvgc4P2e56qzNzMxKpK9BvxpozKYbgadz2hsknSlpEjAZWNe/\nEs3MrD/O6GkBSX8LzATGSGoF7gWWAislLQR2AdcCRMQWSSuBrUAHsCgijhWpdjMzy4M6u9hLq76+\nPlpaWkpdxoAYqC9jN+34pwF5n8Gm5vCjpS7BTmPn0itLXcKgIml9RNT3tJzvjDUzS5yD3swscT32\n0dvgVDdportvurCz8hsnpt2NU35qmn7VZbu7dPrHZ/QDyDdLmVkpOOjNzBLnoDczS5yD3swscf4y\nNmH+Qvb0/MXs4OEvafvHZ/RmZolz0JuZJc5Bb2aWOAf9APE19GZWKg56M7PEOejNzBLnoDczS5yD\n3swscb5hKnG+aSo/vnlqcPKNVPnxGf0A8BU3ZlZKDnozs8Q56M3MEucfBy+ycuq2cV9977m/Pi2p\n9d37x8HNzAxw0JuZJc9Bb2aWOPfRF1E59c8f5376vnN/fboGa999vn30vmFqiPENVH3nm6rS1d2N\nVzB4PwRyFe2MXtJc4MfAcOD+iFja3bIpntGX49l8Lod94Tj0h6Zy+AAo6VU3koYDPwP+GpgC/AdJ\nU4rxXuWo3EPezIaWYnXdzAC2R8S7AJIeA+YDW4v0fkXV1T/rcv8ZD51dIoOJu3AKx106Q9NgGmen\nWEE/Adid87wV+HdFeq9e7/Cegjv3f9adld9gZ+VHX+N4UA62gM91vHYHfuGcegJwnD8Aho7T9fd3\nZSA+GEr2Zaykm4CbsqeHJL1d8Pf4fi+WPenZvG7ac23O5m0+dcYY4IP837n0ut/GPht0+6DAutj+\neV0umLChfgxAnvugNznVhX+Tz0LFCvo9wPk5z6uzthMiYhmwrEjvXzKSWvL5ciRlQ30fDPXtB+8D\nKK99UKwbpv4RmCxpkqQRQAOwukjvZWZmp1GUM/qI6JD0X4G/p/PyygciYksx3svMzE6vaH30EfEs\n8GyxXr+MJdcd1QdDfR8M9e0H7wMoo31QFkMgmJlZ8XhQMzOzxDno+0DSXElvS9ouqamL+ZL0N9n8\nNyVNL0WdxZTHPpgp6aCkjdnffy9FncUk6QFJ+yR95BrbbH7Sx0Ee2z8UjoHzJb0oaaukLZJu6WKZ\n0h8HEeG/XvzR+eXy/wEuAEYAbwBTTlnmy8BzdF6i/jng9VLXXYJ9MBN4ptS1Fnk/XAZMBzZ3Mz/1\n46Cn7R8Kx8B4YHo2fTbw+3LMA5/R996J4R0i4p+B48M75JoPPBydXgNGSRo/0IUWUT77IHkR8RKw\n/zSLJH0c5LH9yYuIvRGxIZv+E7CNzpEBcpX8OHDQ915Xwzuc+h82n2UGs3y37+Lsn6rPSaodmNLK\nSurHQT6GzDEgqQaYBrx+yqySHwcej96KZQMwMSIOSfoy8BQwucQ12cAaMseApJHAE8CtEfHHUtdz\nKp/R916Pwzvkucxgls8QF3+MiEPZ9LNAhaQxA1diWUj9ODitoXIMSKqgM+RXRMTfdbFIyY8DB33v\n5TO8w2rgP2bftn8OOBgRewe60CLqcR9I+teSlE3PoPNYax/wSksr9ePgtIbCMZBt33JgW0T8sJvF\nSn4cuOuml6Kb4R0k/eds/v+k847gLwPbgf8L3FCqeoshz33wNeC/SOoA/h/QENklCKmQ9Ld0Xlky\nRlIrcC9QAUPjOMhj+5M/BoAvANcDmyRtzNruAiZC+RwHvjPWzCxx7roxM0ucg97MLHEOejOzxDno\nzcwS56A3M0ucg97MLHEOejOzxDnozcwS9/8BqYzVnUDac4kAAAAASUVORK5CYII=\n",
127 |       "text/plain": [
128 |        "<matplotlib.figure.Figure at 0x10dbae588>"
129 |       ]
130 |      },
131 |      "metadata": {},
132 |      "output_type": "display_data"
133 |     }
134 |    ],
135 |    "source": [
136 |     "NUM_SAMPLES = 10000\n",
137 |     "n = 10\n",
138 |     "c1 = 10\n",
139 |     "c2 = 50\n",
140 |     "\n",
141 |     "mu_hat_ub = np.zeros(NUM_SAMPLES) #unbiased estimate\n",
142 |     "mu_hat_b1 = np.zeros(NUM_SAMPLES) #first biased estimate\n",
143 |     "mu_hat_b2 = np.zeros(NUM_SAMPLES) #second biased estimate\n",
144 |     "for i in range(0,NUM_SAMPLES):\n",
145 |     "    y = np.random.randn(n) + 1#n i.i.d. samples where the mean is 1!\n",
146 |     "    mu_hat_ub[i] = np.mean(y)\n",
147 |     "    mu_hat_b1[i] = n/(n+c1)*np.mean(y)\n",
148 |     "    mu_hat_b2[i] = n/(n+c2)*np.mean(y)\n",
149 |     "\n",
150 |     "plt.hist(mu_hat_ub,50)\n",
151 |     "plt.hist(mu_hat_b1,50)\n",
152 |     "plt.hist(mu_hat_b2,50)\n",
153 |     "plt.gca().legend(['Unbiased','c='+str(c1),'c='+str(c2)])\n",
154 |     "plt.show()"
155 |    ]
156 |   },
157 |   {
158 |    "cell_type": "code",
159 |    "execution_count": null,
160 |    "metadata": {
161 |     "collapsed": true
162 |    },
163 |    "outputs": [],
164 |    "source": []
165 |   }
166 |  ],
167 |  "metadata": {
168 |   "kernelspec": {
169 |    "display_name": "Python 3",
170 |    "language": "python",
171 |    "name": "python3"
172 |   },
173 |   "language_info": {
174 |    "codemirror_mode": {
175 |     "name": "ipython",
176 |     "version": 3
177 |    },
178 |    "file_extension": ".py",
179 |    "mimetype": "text/x-python",
180 |    "name": "python",
181 |    "nbconvert_exporter": "python",
182 |    "pygments_lexer": "ipython3",
183 |    "version": "3.6.0"
184 |   }
185 |  },
186 |  "nbformat": 4,
187 |  "nbformat_minor": 2
188 | }
189 | 


--------------------------------------------------------------------------------
/ch.4/.ipynb_checkpoints/The Cramer-Rao Bound and Asymptotic Efficiency-checkpoint.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# The Cramer-Rao Bound and Asymptotic Efficiency\n",
  8 |     "\n",
  9 |     "Here we'll see how the ML estimator is asymptotically efficient, even if it is biased. We'll do this with the exponential distribution, looping over many instances of its ML estimator, which is not biased and does not achieve the CRLB for any finite n."
 10 |    ]
 11 |   },
 12 |   {
 13 |    "cell_type": "code",
 14 |    "execution_count": 10,
 15 |    "metadata": {},
 16 |    "outputs": [
 17 |     {
 18 |      "data": {
 19 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAEqVJREFUeJzt3X9sndV9x/H3dyHFNGWBmgSFOJkz\nGrH+gtC6kA11yqCb+KWESmWCjjZQpKgS28jaitL+MypNNJWmAlMRWko6KCuhiJYSAeqKgAj4I1kc\nCCGQVgQWyCVAUhdSWGsg47s//DjzyE18bd/rax+/X5Ll5znPufd+75X98fG5z3NuZCaSpHL9QbsL\nkCS1lkEvSYUz6CWpcAa9JBXOoJekwhn0klQ4g16SCmfQS1LhDHpJKtwR7S4A4Ljjjsvu7u52lyFJ\nk8rmzZt/nZmzhus3IYK+u7ub3t7edpchSZNKRLzQSD+nbiSpcAa9JBXOoJekwk2IOXpJGot33nmH\nWq1Gf39/u0tpiY6ODrq6upg+ffqobm/QS5r0arUaRx99NN3d3UREu8tpqsykr6+PWq3GggULRnUf\nTt1ImvT6+/vp7OwsLuQBIoLOzs4x/bdi0EsqQokhP2isz82gl6TCOUcvqTjdV9/X1Pvbueq8YftE\nBJdccgm33XYbAPv372fOnDmcfvrp3Hvvvdxyyy309vbyve99r6m1NcKgL0S9H+xGfjglNceMGTPY\ntm0bv//97znqqKN44IEHmDt3brvLApy6kaSmOeecc7jvvoFB19q1a7n44ovbXNEAg16SmuSiiy7i\njjvuoL+/n61bt3L66ae3uyTAoJekpjn55JPZuXMna9eu5dxzz213OQc4Ry9JTbR06VK+9rWvsX79\nevr6+tpdDmDQS1JTfelLX2LmzJl8/OMfZ/369e0uBzDoJRWonWecdXV1ceWVV9Y9dsstt/Czn/3s\nwP6GDRvo6upqeU0GvSQ1wZtvvnlQ25IlS1iyZAkAl156KZdeeun4FlXxzVhJKpxBL0mFM+glqXAG\nvSQVzqCXpMIZ9JJUOE+vlFSea2Y2+f72NdTtlVdeYeXKlWzatIkjjzyS7u5urr/+ek455RROOukk\n3n77bXp6elizZg3Tp09n/fr1LFu2jAULFvDuu+8ye/Zsbr/9dmbPnt3U8h3RS1ITZCaf/exnWbJk\nCc899xzPPPMM1157La+++ionnngiW7Zs4amnnqJWq3HnnXceuN2nP/1ptmzZwtatW/nUpz7FjTfe\n2PTaDHpJaoKHH36Y6dOn8+Uvf/lA26JFi5g3b96B/WnTpnHaaafx0ksvHXT7zOSNN97g2GOPbXpt\nTt1IUhNs27aNT37yk4ft09/fz8aNG7nhhhsOtD366KMsWrSIvr4+ZsyYwbXXXtv02hzRS1KLPffc\ncyxatIjOzk7mz5/PySeffODY4NTNrl27uOyyy7jqqqua/vgNBX1E7IyIpyJiS0T0Vm0fjIgHIuLZ\n6vuxVXtExL9ExI6I2BoRn2h61ZI0wXz0ox9l8+bNdY8NztHv2LGDDRs2sG7durr9li5dyiOPPNL0\n2kYyov+LzFyUmT3V/tXAg5m5EHiw2gc4B1hYfa0AbmpWsRqZ7qvvq/slqfnOPPNM3nrrLb7//e8f\naNu0aRMvvPDCgf05c+awatUqvv3tb9e9j8cee4wTTzyx6bWNZY5+GbCk2r4VWA98vWr/YWYmsCEi\njomIOZn58lgK1QCDWmpAg6dDNlNEcPfdd7Ny5UpWrVpFR0fHgdMrh7rgggu45pprePTRR4H/m6PP\nTGbOnMnNN9/c9NoaDfoEfhERCfxrZq4Gjh8M78x8OSIGT/ycC+wactta1WbQSyraCSec8P9OnRy0\nbdu2A9sRwZNPPnlgf9++1v9RajToz8jM3VWYPxARvzxM36jTlgd1iljBwNQO8+fPb7AMSdJINTRH\nn5m7q+97gLuB04BXI2IOQPV9T9W9BswbcvMuYHed+1ydmT2Z2TNr1qzRPwNJ0mENG/QRMSMijh7c\nBv4K2AasA5ZX3ZYD91Tb64AvVmffLAb2OT8vqdUG3hYs01ifWyNTN8cDd0fEYP/bM/PnEbEJuDMi\nLgdeBC6s+t8PnAvsAH4HXDamCiVpGB0dHfT19dHZ2UmVVcXITPr6+ujo6Bj1fQwb9Jn5PHBKnfY+\n4Kw67QlcMeqKJGmEurq6qNVq7N27t92ltERHR8eYPkTcJRAkTXrTp09nwYIF7S5jwnIJBEkqnEEv\nSYUz6CWpcAa9JBXOoJekwhn0klQ4T6+cguqtgLlz1XltqETSeHBEL0mFM+glqXAGvSQVzqCXpMIZ\n9JJUOINekgpn0EtS4Qx6SSqcQS9JhTPoJalwBr0kFc6gl6TCGfSSVDhXr5zA6q0yKUkj5Yhekgpn\n0EtS4Qx6SSqcQS9JhTPoJalwBr0kFa7hoI+IaRHxRETcW+0viIiNEfFsRPw4It5XtR9Z7e+ojne3\npnRJUiNGMqK/Etg+ZP87wHWZuRB4Dbi8ar8ceC0zPwRcV/WTJLVJQ0EfEV3AecDN1X4AZwJ3VV1u\nBS6otpdV+1THz6r6S5LaoNER/fXAVcC71X4n8Hpm7q/2a8DcansusAugOr6v6i9JaoNhgz4izgf2\nZObmoc11umYDx4be74qI6I2I3r179zZUrCRp5BoZ0Z8BLI2IncAdDEzZXA8cExGDa+V0Abur7Row\nD6A6PhP4zXvvNDNXZ2ZPZvbMmjVrTE9CknRowwZ9Zn4jM7sysxu4CHgoM/8GeBj4XNVtOXBPtb2u\n2qc6/lBmHjSilySNj7GcR/914CsRsYOBOfg1VfsaoLNq/wpw9dhKlCSNxYiWKc7M9cD6avt54LQ6\nffqBC5tQmySpCVyPXkD9te93rjqvDZVIajaXQJCkwhn0klQ4g16SCmfQS1LhDHpJKpxBL0mFM+gl\nqXAGvSQVzqCXpMIZ9JJUOINekgpn0EtS4Qx6SSqcQS9JhTPoJalwBr0kFc6gl6TC+QlTOqR6nzoF\nfvKUNNk4opekwhn0klQ4g16SCmfQS1LhDHpJKpxBL0mFM+glqXAGvSQVzqCXpMING/QR0RER/xkR\nT0bE0xHxrap9QURsjIhnI+LHEfG+qv3Ian9Hdby7tU9BknQ4jYzo3wLOzMxTgEXA2RGxGPgOcF1m\nLgReAy6v+l8OvJaZHwKuq/pJktpk2KDPAW9Wu9OrrwTOBO6q2m8FLqi2l1X7VMfPiohoWsWSpBFp\naI4+IqZFxBZgD/AA8Bzwembur7rUgLnV9lxgF0B1fB/QWec+V0REb0T07t27d2zPQpJ0SA2tXpmZ\n/wMsiohjgLuBD9frVn2vN3rPgxoyVwOrAXp6eg46romr3qqWrmgpTVwjOusmM18H1gOLgWMiYvAP\nRRewu9quAfMAquMzgd80o1hJ0sg1ctbNrGokT0QcBXwG2A48DHyu6rYcuKfaXlftUx1/KDMdsU9A\nOzs+f+BLUrkambqZA9waEdMY+MNwZ2beGxHPAHdExD8BTwBrqv5rgNsiYgcDI/mLWlD3lDc0nLv7\nb2+4r6SpZ9igz8ytwKl12p8HTqvT3g9c2JTqJElj5kcJTgCH+si+sXAUL2mQSyBIUuEMekkqnEEv\nSYUz6CWpcL4ZWxjfhJX0Xo7oJalwBr0kFc6gl6TCOUcvYGRLKkiaXBzRS1LhDHpJKpxBL0mFM+gl\nqXC+GVsAL5KSdDiO6CWpcAa9JBXOoJekwhn0klQ434zVYXnFrDT5OaKXpMI5oldTHOoDzneuOm+c\nK5H0Xo7oJalwjugnkfG6MMoLsKSyOKKXpMIZ9JJUOKduJjinUSSNlSN6SSrcsEEfEfMi4uGI2B4R\nT0fElVX7ByPigYh4tvp+bNUeEfEvEbEjIrZGxCda/SQkSYfWyIh+P/DVzPwwsBi4IiI+AlwNPJiZ\nC4EHq32Ac4CF1dcK4KamVy1JatiwQZ+ZL2fm49X2G8B2YC6wDLi16nYrcEG1vQz4YQ7YABwTEXOa\nXrkkqSEjmqOPiG7gVGAjcHxmvgwDfwyA2VW3ucCuITerVW3vva8VEdEbEb179+4deeWSpIY0HPQR\n8QHgJ8DKzPzt4brWacuDGjJXZ2ZPZvbMmjWr0TIkSSPUUNBHxHQGQv5HmfnTqvnVwSmZ6vueqr0G\nzBty8y5gd3PKlSSNVCNn3QSwBtiemd8dcmgdsLzaXg7cM6T9i9XZN4uBfYNTPJrcdnZ8/sCXpMmj\nkQumzgC+ADwVEVuqtm8Cq4A7I+Jy4EXgwurY/cC5wA7gd8BlTa1YkjQiwwZ9Zj5G/Xl3gLPq9E/g\nijHWJUlqEq+MlaTCGfSSVDiDXpIK5+qVaql6HzHoxwtK48sRvSQVzqCXpMIZ9JJUOINekgpn0EtS\n4TzrZgJyLRlJzeSIXpIKZ9BLUuEMekkqnHP046zelaKS1EqO6CWpcAa9JBXOoJekwjlHP0F47ryk\nVnFEL0mFc0SvMRv630h3/+3D9j/UmUeuUy+1hkGvUXGqSZo8nLqRpMIZ9JJUOINekgpn0EtS4Qx6\nSSqcZ92Ms5GeijjZlP78pMlo2KCPiB8A5wN7MvNjVdsHgR8D3cBO4K8z87WICOAG4Fzgd8Clmfl4\na0qf/DxFUdJ4aGTq5hbg7Pe0XQ08mJkLgQerfYBzgIXV1wrgpuaUKUkarWFH9Jn5SER0v6d5GbCk\n2r4VWA98vWr/YWYmsCEijomIOZn5crMKVrnqXTHr1bLS2I32zdjjB8O7+j67ap8L7BrSr1a1SZLa\npNlvxkadtqzbMWIFA9M7zJ8/v8llTDDXzGx3BZKmsNEG/auDUzIRMQfYU7XXgHlD+nUBu+vdQWau\nBlYD9PT01P1joMnNM3CkiWG0UzfrgOXV9nLgniHtX4wBi4F9zs9LUns1cnrlWgbeeD0uImrAPwKr\ngDsj4nLgReDCqvv9DJxauYOB0ysva0HNkqQRaOSsm4sPceisOn0TuGKsRak8g9M4TuFI488lECSp\ncC6BoAnNc+ulsXNEL0mFM+glqXAGvSQVzjl6TTr15u3BuXvpUBzRS1LhHNG3iuvbSJogHNFLUuEM\nekkqnEEvSYVzjl5t4zLG0vhwRC9JhXNE30yeaTOsoaP4ZnNdHKk+R/SSVDiDXpIK59TNWDldI2mC\nc0QvSYVzRD9ajuQlTRIGvYrmSpeSQa8JwounpNYx6DXhGPpScxn0I+G8fDG8uEpTiWfdSFLhHNFr\nQnMaRxo7R/SSVDhH9MNxXn7KONSpmPXUm8/3VE5NVI7oJalwkZnNv9OIs4EbgGnAzZm56nD9e3p6\nsre3t+l1jJqj+EllMs7dO8pXM0TE5szsGa5f06duImIacCPwl0AN2BQR6zLzmWY/VlMZ7pOWb9hK\nh9eKOfrTgB2Z+TxARNwBLAMmRtAb6JoAWjWf7/UBqqcVQT8X2DVkvwac3oLHGTA0uK/ZV79dU8Zo\nP8FqovwnMNY3hMd6n63iH5v2vlnfiqCPOm0HvREQESuAFdXumxHxqzE/8rfqPTQAxwG/HvP9T26+\nBod9Dc4f10KaIb4zqpu15edglLW20oT5fRjja/NHjXRqRdDXgHlD9ruA3e/tlJmrgdUtePyDRERv\nI29YlMzXwNcAfA0GTbXXoRWnV24CFkbEgoh4H3ARsK4FjyNJakDTR/SZuT8i/hb4DwZOr/xBZj7d\n7MeRJDWmJVfGZub9wP2tuO9RGpcpognO18DXAHwNBk2p16ElF0xJkiYOl0CQpMIVHfQRcXZE/Coi\ndkTE1e2upx0iYl5EPBwR2yPi6Yi4st01tUtETIuIJyLi3nbX0g4RcUxE3BURv6x+Hv603TWNt4j4\nh+r3YFtErI2IjnbXNB6KDfohSzGcA3wEuDgiPtLeqtpiP/DVzPwwsBi4Yoq+DgBXAtvbXUQb3QD8\nPDP/BDiFKfZaRMRc4O+Bnsz8GAMni1zU3qrGR7FBz5ClGDLzbWBwKYYpJTNfzszHq+03GPjlntve\nqsZfRHQB5wE3t7uWdoiIPwT+HFgDkJlvZ+br7a2qLY4AjoqII4D3U+canxKVHPT1lmKYcgE3VER0\nA6cCG9tbSVtcD1wFvNvuQtrkj4G9wL9V01c3R8SMdhc1njLzJeCfgReBl4F9mfmL9lY1PkoO+oaW\nYpgqIuIDwE+AlZn523bXM54i4nxgT2ZubnctbXQE8Angpsw8FfhvYEq9bxURxzLwX/0C4ARgRkRc\n0t6qxkfJQd/QUgxTQURMZyDkf5SZP213PW1wBrA0InYyMIV3ZkT8e3tLGnc1oJaZg//N3cVA8E8l\nnwH+KzP3ZuY7wE+BP2tzTeOi5KB3KQYgIoKBedntmfnddtfTDpn5jczsysxuBn4OHsrMKTGSG5SZ\nrwC7IuKkquksJsrS4ePnRWBxRLy/+r04iynyhnSxnxnrUgwHnAF8AXgqIrZUbd+srl7W1PJ3wI+q\ngc/zwGVtrmdcZebGiLgLeJyBs9GeYIpcIeuVsZJUuJKnbiRJGPSSVDyDXpIKZ9BLUuEMekkqnEEv\nSYUz6CWpcAa9JBXufwF6q/2JAqERkwAAAABJRU5ErkJggg==\n",
 20 |       "text/plain": [
 21 |        "<matplotlib.figure.Figure at 0x1f481550400>"
 22 |       ]
 23 |      },
 24 |      "metadata": {},
 25 |      "output_type": "display_data"
 26 |     },
 27 |     {
 28 |      "name": "stdout",
 29 |      "output_type": "stream",
 30 |      "text": [
 31 |       "mean: 2.22446642413\n",
 32 |       "variance: 0.579662332574\n",
 33 |       "CRB variance: 0.4\n"
 34 |      ]
 35 |     }
 36 |    ],
 37 |    "source": [
 38 |     "import numpy as np\n",
 39 |     "import matplotlib.pyplot as plt\n",
 40 |     "\n",
 41 |     "NUM_SAMPLES = 5000 #Number of samples to loop over\n",
 42 |     "rate = 2 #Set the rate of the exponential distribution\n",
 43 |     "\n",
 44 |     "n = 10\n",
 45 |     "crb_variance = rate**2/n\n",
 46 |     "rate_hat = np.zeros(NUM_SAMPLES)\n",
 47 |     "rate_hat_crb = np.zeros(NUM_SAMPLES)\n",
 48 |     "for i in range(0,NUM_SAMPLES):\n",
 49 |     "    y = np.random.exponential(1/rate,n)\n",
 50 |     "    rate_hat_crb[i] = rate + np.sqrt(crb_variance)*np.random.randn() #This is what an efficient estimator would look like\n",
 51 |     "    rate_hat[i] = 1/np.mean(y) #The ML estimate is the reciprocal of the sample mean\n",
 52 |     "    \n",
 53 |     "plt.hist(rate_hat,50)\n",
 54 |     "plt.hist(rate_hat_crb,50)\n",
 55 |     "plt.gca().legend(['ML','CRB'])\n",
 56 |     "plt.show()\n",
 57 |     "\n",
 58 |     "print('mean: '+str(np.mean(rate_hat)))\n",
 59 |     "print('variance: '+str(np.var(rate_hat)))\n",
 60 |     "print('CRB variance: '+str(crb_variance))"
 61 |    ]
 62 |   },
 63 |   {
 64 |    "cell_type": "markdown",
 65 |    "metadata": {},
 66 |    "source": [
 67 |     "It's straightforward to see that the distribution of the ML estimator is not that predicted by the CRB: It is biased, and its variance is not as predicted. Let's try a larger sample size!"
 68 |    ]
 69 |   },
 70 |   {
 71 |    "cell_type": "code",
 72 |    "execution_count": 13,
 73 |    "metadata": {},
 74 |    "outputs": [
 75 |     {
 76 |      "data": {
 77 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAFctJREFUeJzt3X+s3XWd5/Hna2qHEnSKlmJqb7u3\nwa47OmLVK5gQd7vozvDDUEzGXdzVFSTpmEACcVzF+Udmd8WazIhOdM1WcEFX6BAVbYCdnS7SAMkU\nuWVKKXSMxalyaaWdigjRosB7/7jfdq71tPfce8+55/bb5yO5Od/v5/v5fs/7+ON1Pv2c749UFZKk\n9vqdQRcgSeovg16SWs6gl6SWM+glqeUMeklqOYNeklrOoJekljPoJanlDHpJarmXDboAgNNOO62G\nh4cHXYYkHVe2bt36T1W1eLJ+cyLoh4eHGR0dHXQZknRcSfKjbvo5dSNJLWfQS1LLGfSS1HJzYo5e\nkmbi17/+NWNjYxw8eHDQpfTFggULGBoaYv78+dPav+ugTzIPGAWerKp3J1kBbABeBTwEfKCqfpXk\nJOCrwFuBA8B/qKrd06pOkrowNjbGK17xCoaHh0ky6HJ6qqo4cOAAY2NjrFixYlrHmMrUzVXAzgnr\nnwGur6qVwNPA5U375cDTVfVa4PqmnyT1zcGDB1m0aFHrQh4gCYsWLZrRv1a6CvokQ8CFwA3NeoBz\ngW80XW4GLm6W1zTrNNvfmTb+py9pTmlzzMz0s3U7ov8c8DHgpWZ9EfCzqnqhWR8DljbLS4EnAJrt\nzzT9JUkDMOkcfZJ3A/uqamuS1YeaO3StLrZNPO5aYC3A8uXLuypWkroxfM2dPT3e7nUXTtonCe9/\n//v52te+BsALL7zAkiVLOPvss7njjju46aabGB0d5Qtf+EJPa+tGNz/GngNclOQCYAHwe4yP8E9N\n8rJm1D4E7Gn6jwHLgLEkLwMWAj898qBVtR5YDzAyMuITyue4Tv/H6eZ//NKJ4pRTTmHHjh388pe/\n5OSTT2bTpk0sXbp08h1nwaRTN1X1iaoaqqph4BLgu1X1n4B7gD9uun0Q+E6zvLFZp9n+3aoyyCW1\n3vnnn8+dd44Pim699Vbe9773DbiicTO5YOrjwEeS7GJ8Dv7Gpv1GYFHT/hHgmpmVKEnHh0suuYQN\nGzZw8OBBtm/fztlnnz3okoApXjBVVZuBzc3yD4GzOvQ5CLy3B7VpjnM6R/pNZ555Jrt37+bWW2/l\nggsuGHQ5h3llrCT10EUXXcRHP/pRNm/ezIEDBwZdDmDQS1JPfehDH2LhwoW88Y1vZPPmzYMuBzDo\nJbXQIKcQh4aGuOqqqzpuu+mmm/j2t799eH3Lli0MDQ31vSaDXpJ64LnnnvutttWrV7N69WoALr30\nUi699NLZLarhbYolqeUMeklqOYNeklrOoJekljPoJanlDHpJajlPr5TUPtcu7PHxnumq209+8hOu\nvvpqHnzwQU466SSGh4f53Oc+x5ve9CZe97rX8atf/YqRkRFuvPFG5s+fz+bNm1mzZg0rVqzgpZde\n4vTTT+eWW27h9NNP72n5juglqQeqive85z2sXr2axx9/nMcee4zrrruOp556ijPOOINt27bxyCOP\nMDY2xm233XZ4v3e84x1s27aN7du387a3vY0vfvGLPa/NEf0JrNcPZzjaMb3RmU4E99xzD/Pnz+fD\nH/7w4bZVq1axe/fuw+vz5s3jrLPO4sknn/yt/auKZ599lte+9rU9r82gl6Qe2LFjB29961uP2efg\nwYM88MADfP7znz/cdt9997Fq1SoOHDjAKaecwnXXXdfz2py6kaQ+e/zxx1m1ahWLFi1i+fLlnHnm\nmYe3HZq6eeKJJ7jsssv42Mc+1vP3N+glqQfe8IY3sHXr1o7bDs3R79q1iy1btrBx48aO/S666CLu\nvffentdm0EtSD5x77rk8//zzfPnLXz7c9uCDD/KjH/3o8PqSJUtYt24dn/70pzse4/777+eMM87o\neW2TztEnWQDcC5zU9P9GVX0yyU3AvwEOnXd0aVVtSxLg88AFwC+a9od6XrkkHU2Xp0P2UhJuv/12\nrr76atatW8eCBQsOn1450cUXX8y1117LfffdB/zzHH1VsXDhQm644Yae19bNj7HPA+dW1XNJ5gP3\nJ/k/zbb/UlXfOKL/+cDK5u9s4EvNqyS12mte85rfOHXykB07dhxeTsLDDz98eP2ZZ/r/pTTp1E2N\nO3Sj5fnNXx1jlzXAV5v9tgCnJlky81IlSdPR1Rx9knlJtgH7gE1V9UCz6VNJtie5PslJTdtS4IkJ\nu481bZKkAegq6KvqxapaBQwBZyX5A+ATwL8C3ga8Cvh40z2dDnFkQ5K1SUaTjO7fv39axUvSIVXH\nmmg4vs30s03pgqmq+lmSzcB5VfUXTfPzSf4X8NFmfQxYNmG3IWBPh2OtB9YDjIyMtPe/IR31Clyv\nmFWvLFiwgAMHDrBo0SLGzwdpj6riwIEDLFiwYNrH6Oasm8XAr5uQPxl4F/CZJEuqam9zls3FwKFf\nGzYCVybZwPiPsM9U1d5pVyhJkxgaGmJsbIy2zg4sWLBgRg8R72ZEvwS4Ock8xqd6bquqO5J8t/kS\nCLANOHSDh7sYP7VyF+OnV1427eokqQvz589nxYoVgy5jzpo06KtqO/DmDu3nHqV/AVfMvDRJUi94\nZawktZxBL0ktZ9BLUssZ9JLUcga9JLWcQS9JLWfQS1LLGfSS1HIGvSS1nEEvSS03pbtXSr3U6a6W\n3tFS6j1H9JLUcga9JLWcQS9JLWfQS1LL+WPsCeJoj/OT1H6O6CWp5SYN+iQLknwvycNJHk3y5037\niiQPJPlBkr9O8rtN+0nN+q5m+3B/P4Ik6Vi6GdE/D5xbVW8CVgHnJXk78Bng+qpaCTwNXN70vxx4\nuqpeC1zf9JMkDcikQV/jnmtW5zd/BZwLfKNpvxm4uFle06zTbH9nkvSsYknSlHQ1R59kXpJtwD5g\nE/A48LOqeqHpMgYsbZaXAk8ANNufARb1smhJUve6CvqqerGqVgFDwFnA73fq1rx2Gr3XkQ1J1iYZ\nTTK6f//+buuVJE3RlM66qaqfAZuBtwOnJjl0euYQsKdZHgOWATTbFwI/7XCs9VU1UlUjixcvnl71\nkqRJdXPWzeIkpzbLJwPvAnYC9wB/3HT7IPCdZnljs06z/btV9VsjeknS7OjmgqklwM1J5jH+xXBb\nVd2R5DFgQ5L/Dvw9cGPT/0bga0l2MT6Sv6QPdUuSujRp0FfVduDNHdp/yPh8/ZHtB4H39qQ6SdKM\neQsE9cXuBf/xN9aHD94yoEokeQsESWo5g16SWs6gl6SWc45eHR05xz4Z5+ClucsRvSS1nCN6zYqJ\n/0Jw9C/NLkf0ktRyjujVE1Od05c0ewx6AQa11GZO3UhSyxn0ktRyBr0ktZxz9Jp13vBMml2O6CWp\n5Qx6SWo5g16SWq6bZ8YuS3JPkp1JHk1yVdN+bZInk2xr/i6YsM8nkuxK8v0kf9TPDyBJOrZufox9\nAfjTqnooySuArUk2Nduur6q/mNg5yesZf07sG4DXAP8vyb+sqhd7WbiObviaOwddgqQ5ZNIRfVXt\nraqHmuVngZ3A0mPssgbYUFXPV9U/Arvo8GxZSdLsmNLplUmGGX9Q+APAOcCVSf4zMMr4qP9pxr8E\ntkzYbYxjfzFIx9TpXyi71104gEqk41PXQZ/k5cA3gaur6udJvgT8N6Ca178EPgSkw+7V4XhrgbUA\ny5cvn3rlaiWnnaTe6+qsmyTzGQ/5r1fVtwCq6qmqerGqXgK+zD9Pz4wByybsPgTsOfKYVbW+qkaq\namTx4sUz+QySpGPo5qybADcCO6vqsxPal0zo9h5gR7O8EbgkyUlJVgArge/1rmRJ0lR0M3VzDvAB\n4JEk25q2PwPel2QV49Myu4E/AaiqR5PcBjzG+Bk7V3jGjSQNzqRBX1X303ne/a5j7PMp4FMzqEuS\n1CNeGStJLWfQS1LLeZviE5SPDpROHI7oJanlHNFr4HwQidRfjuglqeUMeklqOYNeklrOOXrNec7h\nSzPjiF6SWs6gl6SWM+glqeUMeklqOYNeklrOoJekljPoJanlPI9ex6WjPUR897oLZ7kSae7r5pmx\ny5Lck2RnkkeTXNW0vyrJpiQ/aF5f2bQnyV8l2ZVke5K39PtDSJKOrpupmxeAP62q3wfeDlyR5PXA\nNcDdVbUSuLtZBzif8QeCrwTWAl/qedWSpK5NGvRVtbeqHmqWnwV2AkuBNcDNTbebgYub5TXAV2vc\nFuDUJEt6XrkkqStT+jE2yTDwZuAB4NVVtRfGvwyA05tuS4EnJuw21rRJkgag66BP8nLgm8DVVfXz\nY3Xt0FYdjrc2yWiS0f3793dbhiRpiro66ybJfMZD/utV9a2m+akkS6pqbzM1s69pHwOWTdh9CNhz\n5DGraj2wHmBkZOS3vgh04vJ5tlJvTRr0SQLcCOysqs9O2LQR+CCwrnn9zoT2K5NsAM4Gnjk0xaPB\nMTylE1c3I/pzgA8AjyTZ1rT9GeMBf1uSy4EfA+9ttt0FXADsAn4BXNbTitU1w10SdBH0VXU/nefd\nAd7ZoX8BV8ywLnXhaBcNSdJE3gJBklrOoJekljPoJanlDHpJajmDXpJazqCXpJYz6CWp5Qx6SWo5\ng16SWs6gl6SWM+glqeUMeklqOYNeklrOoJekluvqCVPSXHLkffaHD94yoEqk44MjeklqOYNeklpu\n0qBP8pUk+5LsmNB2bZInk2xr/i6YsO0TSXYl+X6SP+pX4ZKk7nQzor8JOK9D+/VVtar5uwsgyeuB\nS4A3NPv8jyTzelWsJGnqunlm7L1Jhrs83hpgQ1U9D/xjkl3AWcDfTbtCdc2HgUvqZCZz9Fcm2d5M\n7byyaVsKPDGhz1jTJkkakOkG/ZeAM4BVwF7gL5v2dOhbnQ6QZG2S0SSj+/fvn2YZkqTJTCvoq+qp\nqnqxql4Cvsz49AyMj+CXTeg6BOw5yjHWV9VIVY0sXrx4OmVIkrowraBPsmTC6nuAQ2fkbAQuSXJS\nkhXASuB7MytRkjQTk/4Ym+RWYDVwWpIx4JPA6iSrGJ+W2Q38CUBVPZrkNuAx4AXgiqp6sT+lS5K6\nkaqOU+izamRkpEZHRwddxvHn2oWDrmBOmOwWCLvXXThLlUizK8nWqhqZrJ/3ulHrDV9z52+1Gf46\nkXgLBElqOYNeklrOoJekljPoJanl/DFWxz0fRCIdmyN6SWo5g16SWs6gl6SWM+glqeX8MfZ44i0P\nJE2DI3pJajmDXpJazqA/TnS6MZckdcM5erWOF1BJv8kRvSS1nEEvSS03adAn+UqSfUl2TGh7VZJN\nSX7QvL6yaU+Sv0qyK8n2JG/pZ/GSpMl1M6K/CTjviLZrgLuraiVwd7MOcD7jDwRfCawFvtSbMiVJ\n0zVp0FfVvcBPj2heA9zcLN8MXDyh/as1bgtwapIlvSpWkjR1052jf3VV7QVoXk9v2pcCT0zoN9a0\nSZIGpNc/xqZDW3XsmKxNMppkdP/+/T0uQ5J0yHTPo38qyZKq2ttMzexr2seAZRP6DQF7Oh2gqtYD\n6wFGRkY6fhlIveB59TrRTTfoNwIfBNY1r9+Z0H5lkg3A2cAzh6Z4pLmk05XGu9ddOIBKpP6bNOiT\n3AqsBk5LMgZ8kvGAvy3J5cCPgfc23e8CLgB2Ab8ALutDzZKkKZg06KvqfUfZ9M4OfQu4YqZFSZJ6\nxytjJanlDHpJajmDXpJazqCXpJYz6CWp5XzwiNQ42lO8PL9exztH9JLUco7opUl4Fa2Od47oJanl\nHNHPZdcuPLy4e8EA62gZb3KmE40jeklqOYNeklrOoJekljPoJanlDHpJajmDXpJazqCXpJab0Xn0\nSXYDzwIvAi9U1UiSVwF/DQwDu4F/X1VPz6xMSdJ09WJE/2+ralVVjTTr1wB3V9VK4O5mXZI0IP24\nMnYN4w8TB7gZ2Ax8vA/v0z4TroTV7Jl4paxXyaqNZhr0BfxtkgL+Z1WtB15dVXsBqmpvktNnWmRr\nGeySZsFMg/6cqtrThPmmJP/Q7Y5J1gJrAZYvXz7DMiRJRzOjOfqq2tO87gNuB84CnkqyBKB53XeU\nfddX1UhVjSxevHgmZUiSjmHaI/okpwC/U1XPNst/CPxXYCPwQWBd8/qdXhQqzXXet15z1Uymbl4N\n3J7k0HFuqaq/SfIgcFuSy4EfA++deZkt4Zz8nOctjNVG0w76qvoh8KYO7QeAd86kKElS73hlrCS1\nnE+Ykqah03z8VPo5d6/Z5IheklrOoJekljPoJanlnKOXjsHTLdUGjuglqeUc0feTF0hJmgMc0UtS\nyzmil6agV3P23Z6H37EGz8HXFDmil6SWM+glqeWcupGOM95WQVPliF6SWs4R/Ux5CuUJzQuqdDww\n6KfKYJd0nDHopR5yhK+5qG9Bn+Q84PPAPOCGqlrXr/fqK0fwmgGDX3NBX4I+yTzgi8C/A8aAB5Ns\nrKrH+vF+0vGin8Hf7UVYnp1z4unXiP4sYFfzXFmSbADWAHM/6B3BaxYNYsQ/laty/VJoh34F/VLg\niQnrY8DZfXkng1ktcmTwT0U/viRmcquGmZrKl0w/binRj38hdTrmbHyZ9ivo06GtfqNDshZY26w+\nl+T7faplkE4D/mnQRcwSP+vAvbtfBx7I581nBvI+U/6sM61zhvv/i2469Svox4BlE9aHgD0TO1TV\nemB9n95/TkgyWlUjg65jNvhZ2+tE+rxt/az9ujL2QWBlkhVJfhe4BNjYp/eSJB1DX0b0VfVCkiuB\n/8v46ZVfqapH+/FekqRj69t59FV1F3BXv45/nGj11NQR/KztdSJ93lZ+1lTV5L0kScct714pSS1n\n0PdBkq8k2Zdkx6Br6acky5Lck2RnkkeTXDXomvopyYIk30vycPN5/3zQNfVbknlJ/j7JHYOupd+S\n7E7ySJJtSUYHXU8vOXXTB0n+NfAc8NWq+oNB19MvSZYAS6rqoSSvALYCF7f1VhdJApxSVc8lmQ/c\nD1xVVVsGXFrfJPkIMAL8XlX17UT9uSDJbmCkqubgNRIz44i+D6rqXuCng66j36pqb1U91Cw/C+xk\n/KroVqpxzzWr85u/1o6UkgwBFwI3DLoWzYxBr55IMgy8GXhgsJX0VzOVsQ3YB2yqqjZ/3s8BHwNe\nGnQhs6SAv02ytblyvzUMes1YkpcD3wSurqqfD7qefqqqF6tqFeNXe5+VpJVTc0neDeyrqq2DrmUW\nnVNVbwHOB65opmBbwaDXjDRz1d8Evl5V3xp0PbOlqn4GbAbOG3Ap/XIOcFEzb70BODfJ/x5sSf1V\nVXua133A7YzfhbcVDHpNW/Pj5I3Azqr67KDr6bcki5Oc2iyfDLwL+IfBVtUfVfWJqhqqqmHGb2Hy\n3ap6/4DL6pskpzQnFJDkFOAPgdacNWfQ90GSW4G/A16XZCzJ5YOuqU/OAT7A+GhvW/N3waCL6qMl\nwD1JtjN+P6dNVdX60w5PEK8G7k/yMPA94M6q+psB19Qznl4pSS3niF6SWs6gl6SWM+glqeUMeklq\nOYNeklrOoJekljPoJanlDHpJarn/D7RfShYwToeHAAAAAElFTkSuQmCC\n",
 78 |       "text/plain": [
 79 |        "<matplotlib.figure.Figure at 0x1f48184b6a0>"
 80 |       ]
 81 |      },
 82 |      "metadata": {},
 83 |      "output_type": "display_data"
 84 |     },
 85 |     {
 86 |      "name": "stdout",
 87 |      "output_type": "stream",
 88 |      "text": [
 89 |       "mean: 2.10682095363\n",
 90 |       "variance: 0.243610781895\n",
 91 |       "CRB variance: 0.2\n"
 92 |      ]
 93 |     }
 94 |    ],
 95 |    "source": [
 96 |     "n = 20\n",
 97 |     "crb_variance = rate**2/n\n",
 98 |     "rate_hat = np.zeros(NUM_SAMPLES)\n",
 99 |     "rate_hat_crb = np.zeros(NUM_SAMPLES)\n",
100 |     "for i in range(0,NUM_SAMPLES):\n",
101 |     "    y = np.random.exponential(1/rate,n)\n",
102 |     "    rate_hat_crb[i] = rate + np.sqrt(crb_variance)*np.random.randn() #This is what an efficient estimator would look like\n",
103 |     "    rate_hat[i] = 1/np.mean(y) #The ML estimate is the reciprocal of the sample mean\n",
104 |     "    \n",
105 |     "plt.hist(rate_hat,50)\n",
106 |     "plt.hist(rate_hat_crb,50)\n",
107 |     "plt.gca().legend(['ML','CRB'])\n",
108 |     "plt.show()\n",
109 |     "\n",
110 |     "print('mean: '+str(np.mean(rate_hat)))\n",
111 |     "print('variance: '+str(np.var(rate_hat)))\n",
112 |     "print('CRB variance: '+str(crb_variance))"
113 |    ]
114 |   },
115 |   {
116 |    "cell_type": "markdown",
117 |    "metadata": {},
118 |    "source": [
119 |     "Sure enough, the mean is closer to 2, and the variance is closer to that predicted by the CRB. Let's go even higher!"
120 |    ]
121 |   },
122 |   {
123 |    "cell_type": "code",
124 |    "execution_count": 14,
125 |    "metadata": {},
126 |    "outputs": [
127 |     {
128 |      "data": {
129 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAFOFJREFUeJzt3X+Q3HV9x/Hnu/TKMcgEDRc9c6GX\noalTqXDKGegwdq7QH/zoJDBiGxyVIG20xdFMdWzqHzXtTDGdaVVaqZ0gDuCYYAZFU4K1lJIRZgrm\nYGKIYNugp5xEiCdGGD008O4f+714XDa5ze3u7d3nno+Zm9v97md335/5zr32c5/v5/vdyEwkSeX6\npU4XIElqL4Nekgpn0EtS4Qx6SSqcQS9JhTPoJalwBr0kFc6gl6TCGfSSVLhf7nQBAKeddlr29/d3\nugxJmlceeuihH2Rmz3Tt5kTQ9/f3Mzw83OkyJGleiYjvNNLOqRtJKpxBL0mFM+glqXBzYo5ekprx\n85//nNHRUcbHxztdSlt0d3fT19dHV1fXjJ5v0Eua90ZHRznllFPo7+8nIjpdTktlJmNjY4yOjrJ8\n+fIZvYZTN5LmvfHxcRYvXlxcyANEBIsXL27qvxWDXlIRSgz5Cc32zaCXpMI5Ry+pOP0bdrT09UY2\nXTptm4jgbW97G5/5zGcAOHToEL29vZx77rnceeed3HzzzQwPD/OJT3yipbU1wqDXrDraH2Ajf0jS\nXHbyySezd+9efvrTn3LSSSdx9913s3Tp0k6XBRj0mof8sNBcdfHFF7Njxw6uuOIKtm7dypVXXsl9\n993X6bKco5ekVlmzZg233XYb4+Pj7Nmzh3PPPbfTJQEGvSS1zFlnncXIyAhbt27lkksu6XQ5hzl1\nI0kttGrVKj7wgQ+wc+dOxsbGOl0OYNBrjmv16gmp3d75zneyaNEiXve617Fz585OlwMY9JIK1MkD\n8319fbzvfe+r+9jNN9/MF7/4xcP3H3jgAfr6+tpek0EvSS3w3HPPHbFtaGiIoaEhANauXcvatWtn\nt6iKQa+mudxRmttcdSNJhTPoJalw007dREQ38FXgxKr97Zn54YhYDtwGvAJ4GHh7Zv4sIk4EbgXO\nAcaAP87MkTbVrznMFTPS3NDIiP554ILMPBsYAC6KiPOAvwc+lpkrgGeAa6r21wDPZOavAR+r2kmS\nOmTaoM+aicPJXdVPAhcAt1fbbwEuq26vru5TPX5hlHyhaEma4xpadRMRJwAPAb8G3AA8DvwoMw9V\nTUaBicu0LQWeAMjMQxFxEFgM/KCFdUtHqDdV5MqfBWrjoha/3sGGmn3/+99n/fr17Nq1ixNPPJH+\n/n4+/vGPc/bZZ/Oa17yGn/3sZwwODnLTTTfR1dXFzp07Wb16NcuXL+fFF19kyZIlbNmyhSVLlrS0\n/IYOxmbmC5k5APQBK4HfqNes+l1v9J5TN0TEuogYjojhAwcONFqvJM1Jmcnll1/O0NAQjz/+OI8+\n+ijXXXcdTz31FGeccQa7d+/mkUceYXR0lG3bth1+3pve9CZ2797Nnj17eOMb38gNN9zQ8tqOa9VN\nZv4I2AmcB5waERP/EfQBT1a3R4FlANXji4Af1nmtzZk5mJmDPT09M6tekuaIe++9l66uLt797ncf\n3jYwMMCyZcsO3z/hhBNYuXIl3/ve9454fmby7LPP8vKXv7zltTWy6qYH+Hlm/igiTgJ+l9oB1nuB\nK6itvLkK+FL1lO3V/f+uHv+vzDxiRC/NBk/m0mzZu3cv55xzzjHbjI+P8+CDD3L99dcf3nbfffcx\nMDDA2NgYJ598Mtddd13La2tkRN8L3BsRe4BdwN2ZeSfwl8BfRMQ+anPwN1XtbwIWV9v/AtjQ8qol\naR55/PHHGRgYYPHixZx++umcddZZhx+bmLp54oknuPrqq/ngBz/Y8vefdkSfmXuA19fZ/i1q8/VT\nt48Db2lJdVowXHOv+e7MM8/k9ttvr/vYxBz9/v37GRoaYvv27axateqIdqtWreLNb35zy2vzzFhJ\naoELLriA559/nhtvvPHwtl27dvGd73zn8P3e3l42bdrERz7ykbqvcf/993PGGWe0vDYvaiapPA0u\nh2yliOCOO+5g/fr1bNq0ie7u7sPLKye77LLL2Lhx4+Hvkp2Yo89MFi1axKc+9amW12bQS1KLvPrV\nr37J0skJe/fuPXw7Ivj6179++P7Bg+3/UHLqRpIKZ9BLUuEMeklFKPl0nWb7ZtBLmve6u7sZGxsr\nMuwzk7GxMbq7u2f8Gh6MlTTv9fX1MTo6SqnXzeru7m7qS8QNenXESPdbj9jWP76lA5WoBF1dXSxf\nvrzTZcxZTt1IUuEc0Wve878D6dgc0UtS4RzRa16pN3qXdGwGvRaMl3xIbKQj10OROsGpG0kqnEEv\nSYUz6CWpcM7Ra07z4KvUPEf0klQ4g16SCmfQS1LhDHpJKpxBL0mFm3bVTUQsA24FXgW8CGzOzOsj\nYiPwp8DEBaA/lJl3Vc/5K+Aa4AXgvZn5lTbULjWlf8OOI7aNbLq0A5VI7dXI8spDwPsz8+GIOAV4\nKCLurh77WGb+w+TGEfFaYA1wJvBq4D8j4tcz84VWFq7yuJRSao9pp24yc39mPlzdfhZ4DFh6jKes\nBm7LzOcz89vAPmBlK4qVJB2/4zphKiL6gdcDDwLnA++JiHcAw9RG/c9Q+xB4YNLTRjn2B4PmkXrT\nHdNxpC51VsMHYyPiZcDngfWZ+WPgk8AZwACwH/jHiaZ1nn7EN/ZGxLqIGI6I4VK/51GS5oKGgj4i\nuqiF/Gcz8wsAmflUZr6QmS8CN/KL6ZlRYNmkp/cBT059zczcnJmDmTnY09PTTB8kSccwbdBHRAA3\nAY9l5kcnbe+d1OxyYG91ezuwJiJOjIjlwArga60rWZJ0PBqZoz8feDvwSETsrrZ9CLgyIgaoTcuM\nAO8CyMxvRMQ24FFqK3audcWNJHXOtEGfmfdTf979rmM85++Av2uiLklSi3hmrCQVzqCXpMIZ9JJU\nOL9hSi3nCVLS3OKIXpIKZ9BLUuEMekkqnHP0KlIjxwnqtznY+mKkDnNEL0mFM+glqXAGvSQVzqCX\npMIZ9JJUOFfdqHEbFzHS/dJN/eNbOlOLpIY5opekwhn0klQ4g16SCmfQS1LhDHpJKpxBL0mFM+gl\nqXCuo1dT/DYpae4z6KVJ+jfsOGLbyKZLO1CJ1DrTBn1ELANuBV4FvAhszszrI+IVwOeAfmAE+KPM\nfCYiArgeuAT4CbA2Mx9uT/lql7qB112noaQ5r5E5+kPA+zPzN4DzgGsj4rXABuCezFwB3FPdB7gY\nWFH9rAM+2fKqJUkNmzboM3P/xIg8M58FHgOWAquBW6pmtwCXVbdXA7dmzQPAqRHR2/LKJUkNOa5V\nNxHRD7weeBB4ZWbuh9qHAbCkarYUeGLS00arbZKkDmg46CPiZcDngfWZ+eNjNa2zLeu83rqIGI6I\n4QMHDjRahiTpODUU9BHRRS3kP5uZX6g2PzUxJVP9frraPgosm/T0PuDJqa+ZmZszczAzB3t6emZa\nvyRpGtMGfbWK5ibgscz86KSHtgNXVbevAr40afs7ouY84ODEFI8kafY1so7+fODtwCMRsbva9iFg\nE7AtIq4Bvgu8pXrsLmpLK/dRW155dUsrliQdl2mDPjPvp/68O8CFddoncG2TdUmSWsQzYxe4eidG\nSSqLFzWTpMIZ9JJUOINekgpn0EtS4Qx6SSqcQS9JhXN5pTSdjYuO2NQ/vsUvJNG84Yhekgpn0EtS\n4Qx6SSqcQS9JhfNgrOoa6X5rp0uQ1CKO6CWpcI7o5ehdKpwjekkqnEEvSYUz6CWpcM7RS5N4vEIl\nckQvSYUz6CWpcAa9JBXOOXqpVaZeznjjwc7UIU0x7Yg+Ij4dEU9HxN5J2zZGxPciYnf1c8mkx/4q\nIvZFxP9ExB+0q3BJUmMambq5GbiozvaPZeZA9XMXQES8FlgDnFk9518i4oRWFStJOn7TBn1mfhX4\nYYOvtxq4LTOfz8xvA/uAlU3UJ0lqUjNz9O+JiHcAw8D7M/MZYCnwwKQ2o9U2qTj9G3a85P5Id4cK\nkaYx01U3nwTOAAaA/cA/VtujTtus9wIRsS4ihiNi+MCBAzMsQ5I0nRkFfWY+lZkvZOaLwI38Ynpm\nFFg2qWkf8ORRXmNzZg5m5mBPT89MypAkNWBGQR8RvZPuXg5MrMjZDqyJiBMjYjmwAvhacyVKkpox\n7Rx9RGwFhoDTImIU+DAwFBED1KZlRoB3AWTmNyJiG/AocAi4NjNfaE/pkqRGRGbdKfRZNTg4mMPD\nw50uY+GYemKP2qJ/fMvh2yObLu1gJSpVRDyUmYPTtfMSCJJUOC+BsIBMLAd0GaC0sDiil6TCGfSS\nVDiDXpIKZ9BLUuEMekkqnEEvSYUz6CWpcAa9JBXOoJekwhn0klQ4g16SCmfQS1LhDHpJKpxXryzd\npGvPe9VKaWFyRC9JhTPoJalwBr0kFc6gl6TCGfSSVDiDXpIKZ9BLUuGmDfqI+HREPB0Reydte0VE\n3B0R/1f9fnm1PSLinyJiX0TsiYg3tLN4SdL0GhnR3wxcNGXbBuCezFwB3FPdB7gYWFH9rAM+2Zoy\nJUkzNW3QZ+ZXgR9O2bwauKW6fQtw2aTtt2bNA8CpEdHbqmIlScdvppdAeGVm7gfIzP0RsaTavhR4\nYlK70Wrb/pmXKM1PI91v/cWdjRO/D3aiFC1wrT4YG3W2Zd2GEesiYjgihg8cONDiMiRJE2Y6on8q\nInqr0Xwv8HS1fRRYNqldH/BkvRfIzM3AZoDBwcG6HwZSafo37HjJ/ZFNl3aoEi0kMw367cBVwKbq\n95cmbX9PRNwGnAscnJji0eyZHCZesVLStEEfEVuBIeC0iBgFPkwt4LdFxDXAd4G3VM3vAi4B9gE/\nAa5uQ81SMaaO8MFRvlpv2qDPzCuP8tCFddomcG2zRUmSWsczYyWpcAa9JBXOoJekwhn0klQ4g16S\nCmfQS1LhDHpJKpxBL0mFm+klEDQXbVwEeNkDSS9l0Euz6CWXLq70j2/pQCVaSJy6kaTCGfSSVDin\nbqQ5pt4VLcGrWmrmHNFLUuEMekkqnFM389gRX0vnssp5yZU4ajdH9JJUOINekgpn0EtS4Qx6SSqc\nB2OlecL19ZopR/SSVDhH9PPVxkUup5TUkKaCPiJGgGeBF4BDmTkYEa8APgf0AyPAH2XmM82VKUma\nqVZM3fxOZg5k5mB1fwNwT2auAO6p7kuSOqQdUzergaHq9i3ATuAv2/A+C0v1pSKSdLyaHdEn8B8R\n8VBErKu2vTIz9wNUv5c0+R6SpCY0O6I/PzOfjIglwN0R8c1Gn1h9MKwDOP3005ssQ5J0NE2N6DPz\nyer308AdwErgqYjoBah+P32U527OzMHMHOzp6WmmDEnSMcw46CPi5Ig4ZeI28PvAXmA7cFXV7Crg\nS80WKUmauWambl4J3BERE6+zJTP/PSJ2Adsi4hrgu8Bbmi9TkjRTMw76zPwWcHad7WPAhc0UJUlq\nHS+BIEmF8xII0jzgt1CpGQa9NAfVC3Zppgz6ucizYHUcjnb54nq8pPHC5By9JBXOoJekwhn0klQ4\ng16SCufBWGmecsmlGmXQSwUx/FWPQS8tIC7FXJgM+g7q37DDE2MktZ0HYyWpcI7oZ9OUM15HujtU\nh9SAo03zOKUz/ziil6TCGfSSVDiDXpIK5xx9O3kVSs0BU1d2tWNdvfP5c5tBLy0wzZ5UdTxr8TU3\nOHUjSYUz6CWpcAa9JBXOOfpW8cCrdIR68/keoJ19bQv6iLgIuB44AfhUZm5q13tJas5cuOqlK3fa\npy1BHxEnADcAvweMArsiYntmPtqO95PUeu0K/1as2vFD4fi0a0S/EtiXmd8CiIjbgNXA3Ar6qdMt\nGw/O7HnSAtFI+Lf6A8LlnM1rV9AvBZ6YdH8UOLct79TK0DXApePWyKW2Z2tqqF0j/eN93bl23f92\nBX3U2ZYvaRCxDlhX3X0uIv6nBe97GvCDFrzOfLCQ+goLq78LpK9/CLPU1/j7OfG6dfvaZG2/2kij\ndgX9KLBs0v0+4MnJDTJzM7C5lW8aEcOZOdjK15yrFlJfYWH1176WqZN9bdc6+l3AiohYHhG/AqwB\ntrfpvSRJx9CWEX1mHoqI9wBfoba88tOZ+Y12vJck6djato4+M+8C7mrX6x9FS6eC5riF1FdYWP21\nr2XqWF8jM6dvJUmat7zWjSQVbt4FfUR8OiKejoi9R3k8IuKfImJfROyJiDfMdo2t1EB/hyLiYETs\nrn7+erZrbJWIWBYR90bEYxHxjYh4X502RezfBvtaxL6NiO6I+FpEfL3q69/UaXNiRHyu2q8PRkT/\n7FfavAb7ujYiDkzar3/S9sIyc179AL8NvAHYe5THLwG+TG0t/3nAg52uuc39HQLu7HSdLeprL/CG\n6vYpwP8Cry1x/zbY1yL2bbWvXlbd7gIeBM6b0ubPgX+tbq8BPtfputvY17XAJ2azrnk3os/MrwI/\nPEaT1cCtWfMAcGpE9M5Oda3XQH+LkZn7M/Ph6vazwGPUzrKerIj922Bfi1Dtq+equ13Vz9SDg6uB\nW6rbtwMXRkS9Ey/ntAb7OuvmXdA3oN7lF4r8A5rkt6p/Fb8cEWd2uphWqP51fz21EdFkxe3fY/QV\nCtm3EXFCROwGngbuzsyj7tfMPAQcBBbPbpWt0UBfAd5cTT3eHhHL6jzeUiUG/bSXXyjMw8CvZubZ\nwD8DX+xwPU2LiJcBnwfWZ+aPpz5c5ynzdv9O09di9m1mvpCZA9TOkl8ZEb85pUkx+7WBvv4b0J+Z\nZwH/yS/+k2mbEoN+2ssvlCQzfzzxr2LWzl3oiojTOlzWjEVEF7Xg+2xmfqFOk2L273R9LW3fAmTm\nj4CdwEVTHjq8XyPil4FFzPMpy6P1NTPHMvP56u6NwDntrqXEoN8OvKNanXEecDAz93e6qHaJiFdN\nzGVGxEpq+3Sss1XNTNWPm4DHMvOjR2lWxP5tpK+l7NuI6ImIU6vbJwG/C3xzSrPtwFXV7SuA/8rq\nyOV80khfpxxTWkXt+ExbzbuvEoyIrdRWI5wWEaPAh6kd8CAz/5Xa2biXAPuAnwBXd6bS1migv1cA\nfxYRh4CfAmvm4x9I5Xzg7cAj1RwnwIeA06G4/dtIX0vZt73ALVH7QqJfArZl5p0R8bfAcGZup/ah\n95mI2EdtJL+mc+U2pZG+vjciVgGHqPV1bbuL8sxYSSpciVM3kqRJDHpJKpxBL0mFM+glqXAGvSQV\nzqCXpMIZ9JJUOINekgr3/8a9NlLTVf5FAAAAAElFTkSuQmCC\n",
130 |       "text/plain": [
131 |        "<matplotlib.figure.Figure at 0x1f4ffae59e8>"
132 |       ]
133 |      },
134 |      "metadata": {},
135 |      "output_type": "display_data"
136 |     },
137 |     {
138 |      "name": "stdout",
139 |      "output_type": "stream",
140 |      "text": [
141 |       "mean: 2.03633487197\n",
142 |       "variance: 0.0889654309787\n",
143 |       "CRB variance: 0.08\n"
144 |      ]
145 |     }
146 |    ],
147 |    "source": [
148 |     "n = 50\n",
149 |     "crb_variance = rate**2/n\n",
150 |     "rate_hat = np.zeros(NUM_SAMPLES)\n",
151 |     "rate_hat_crb = np.zeros(NUM_SAMPLES)\n",
152 |     "for i in range(0,NUM_SAMPLES):\n",
153 |     "    y = np.random.exponential(1/rate,n)\n",
154 |     "    rate_hat_crb[i] = rate + np.sqrt(crb_variance)*np.random.randn() #This is what an efficient estimator would look like\n",
155 |     "    rate_hat[i] = 1/np.mean(y) #The ML estimate is the reciprocal of the sample mean\n",
156 |     "    \n",
157 |     "plt.hist(rate_hat,50)\n",
158 |     "plt.hist(rate_hat_crb,50)\n",
159 |     "plt.gca().legend(['ML','CRB'])\n",
160 |     "plt.show()\n",
161 |     "\n",
162 |     "print('mean: '+str(np.mean(rate_hat)))\n",
163 |     "print('variance: '+str(np.var(rate_hat)))\n",
164 |     "print('CRB variance: '+str(crb_variance))"
165 |    ]
166 |   },
167 |   {
168 |    "cell_type": "markdown",
169 |    "metadata": {},
170 |    "source": [
171 |     "At this point we're doing quite well. Note that the distribution of the ML estimator is looking more and more Gaussian, a consequence of the central limit theorem. Once more, at an even higher number of samples."
172 |    ]
173 |   },
174 |   {
175 |    "cell_type": "code",
176 |    "execution_count": 17,
177 |    "metadata": {},
178 |    "outputs": [
179 |     {
180 |      "data": {
181 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAFUpJREFUeJzt3X+Q3HWd5/Hn+8Is4yIVID+4mAk3\nOeSslRMijoFbzqss3p2AuwmUy21gRUTusl7hHanT87L+sU7dnZirul1/lC5bQSzAkrApFE0B7ppj\nSQlrBRnYEAKsZ9BRxkQSAxvJacDA+/7o74Qm6Znpmeme7vnk+ajq6m9/vp/ufqfrm1d/5vP90ZGZ\nSJLK9Y86XYAkqb0MekkqnEEvSYUz6CWpcAa9JBXOoJekwhn0klQ4g16SCmfQS1LhTuh0AQDz58/P\n/v7+TpchSbPKo48++vPMXDBRv64I+v7+foaGhjpdhiTNKhHx42b6OXUjSYUz6CWpcAa9JBWuK+bo\nJWk6fv3rXzMyMsKhQ4c6XUpb9Pb20tfXR09Pz5Seb9BLmvVGRkY4+eST6e/vJyI6XU5LZSb79+9n\nZGSEpUuXTuk1nLqRNOsdOnSIefPmFRfyABHBvHnzpvXXikEvqQglhvyo6f7bDHpJKpxz9JKK07/u\n3pa+3vD6907YJyJ4//vfz1e+8hUADh8+zKJFizj//PO55557uPXWWxkaGuILX/hCS2trhkEvAQzO\nHaP9wMzWoVnrpJNOYufOnfzqV7/iDW94A1u2bGHx4sWdLgtw6kaSWuaSSy7h3ntrf01s3LiRK6+8\nssMV1Rj0ktQiq1ev5s477+TQoUPs2LGD888/v9MlAQa9JLXMOeecw/DwMBs3buTSSy/tdDlHOEcv\nSS20cuVKPvaxj7F161b279/f6XKAJoI+InqB7wAnVv3vysxPRsRS4E7gNOAx4OrMfDkiTgRuB94B\n7Af+IDOH21S/VNNoZ6o7UtUBH/rQh5g7dy5ve9vb2Lp1a6fLAZob0b8EXJSZByOiB3goIr4F/Bfg\nM5l5Z0T8BXAdcFN1/0JmvjkiVgP/C/iDNtUvScdo5nDIdunr6+OGG25ouO7WW2/lG9/4xpHH27Zt\no6+vr+01TThHnzUHq4c91S2Bi4C7qvbbgMuq5VXVY6r1746ST1mTJODgwYPHtK1YsYJ77rkHgA9+\n8IMcPHiQkZGRI7eZCHlocmdsRMyJiO3AXmAL8AzwD5l5uOoyAoweMLoYeBagWn8AmNfKoiVJzWsq\n6DPzlcxcBvQBy4HfatStum80es+jGyJiTUQMRcTQvn37mq1XkjRJkzrqJjP/ISK2AhcAp0TECdWo\nvQ/YXXUbAZYAIxFxAjAXeL7Ba20ANgAMDAwc80UgTVej0+A7OXcrdcqEI/qIWBARp1TLbwD+NfA0\n8ADw+1W3a4BvVsubq8dU6/8mMw1ySeqQZkb0i4DbImIOtS+GTZl5T0Q8BdwZEf8T+Dvglqr/LcBX\nImIXtZH86jbULUlq0oRBn5k7gLc3aP8htfn6o9sPAVe0pDqpHca6gJlUKM+MlVSeVn+ZN3ny3c9+\n9jPWrl3LI488woknnkh/fz+f/exnOffcc3nLW97Cyy+/zMDAALfccgs9PT1s3bqVVatWsXTpUl59\n9VUWLlzIHXfcwcKFC1tavte6kaQWyEwuv/xyVqxYwTPPPMNTTz3FjTfeyHPPPceZZ57J9u3beeKJ\nJxgZGWHTpk1Hnveud72L7du3s2PHDt75znfyxS9+seW1OaKXJstr16uBBx54gJ6eHj784Q8faVu2\nbBnDw8NHHs+ZM4fly5fz05/+9JjnZyYvvvgib37zm1tem0EvSS2wc+dO3vGOd4zb59ChQzz88MN8\n7nOfO9L24IMPsmzZMvbv389JJ53EjTfe2PLanLqRpDZ75plnWLZsGfPmzeOMM87gnHPOObJudOrm\n2Wef5dprr+XjH/94y9/fEb2KNdx71bGNg5N8EY/QUZPOPvts7rrrrobrRufo9+zZw4oVK9i8eTMr\nV648pt/KlSt53/ve1/LaHNFLUgtcdNFFvPTSS9x8881H2h555BF+/OMfH3m8aNEi1q9fz6c//emG\nr/HQQw9x5plntrw2R/SSytOBHeMRwd13383atWtZv349vb29Rw6vrHfZZZcxODjIgw8+CLw2R5+Z\nzJ07ly996Ustr82gl6QWedOb3vS6QydH7dy588hyRPD4448feXzgQPu/lJy6kaTCGfSSVDiDXlIR\nSr5I7nT/bQa9pFmvt7eX/fv3Fxn2mcn+/fvp7e2d8mu4M1bSrNfX18fIyAil/lpdb2/vtH5f1qCX\nNOv19PSwdOnSTpfRtZy6kaTCGfSSVDiDXpIKZ9BLUuEMekkqnEEvSYUz6CWpcAa9JBXOE6Y0q/Sv\nu7dh+/DUzw6XijfhiD4ilkTEAxHxdEQ8GRE3VO2DEfHTiNhe3S6te84fR8SuiPh+RLynnf8ASdL4\nmhnRHwY+mpmPRcTJwKMRsaVa95nM/N/1nSPircBq4GzgTcD/iYh/lpmvtLJwSVJzJhzRZ+aezHys\nWn4ReBpYPM5TVgF3ZuZLmfkjYBewvBXFSpImb1Jz9BHRD7wdeBi4EPhIRHwAGKI26n+B2pfAtrqn\njTD+F4NUhPr9B8Pr39vBSqTXa/qom4h4I/A1YG1m/gK4CTgTWAbsAf50tGuDpx9zkeiIWBMRQxEx\nVOqlRSWpGzQV9BHRQy3kv5qZXwfIzOcy85XMfBW4mdemZ0aAJXVP7wN2H/2ambkhMwcyc2DBggXT\n+TdIksbRzFE3AdwCPJ2Zf1bXvqiu2+XA6M+cbwZWR8SJEbEUOAv4XutKliRNRjNz9BcCVwNPRMT2\nqu0TwJURsYzatMww8EcAmflkRGwCnqJ2xM71HnEjSZ0zYdBn5kM0nne/b5znfAr41DTqkiS1iJdA\nkKTCGfSSVDiDXpIK50XN1L0G5x7TNNwL/Yfu6EAx0uxl0GvWGe69qtMlSLOKUzeSVDhH9FI7NZh+\nYvDAzNeh45ojekkqnEEvSYUz6CWpcAa9JBXOnbFSi7zusM/BjpUhHcMRvSQVzhG9ukOjwxAltYQj\nekkqnEEvSYUz6CWpcAa9JBXOoJekwhn0klQ4g16SCmfQS1LhDHpJKpxnxmrG9a+795i24d4OFCId\nJyYc0UfEkoh4ICKejognI+KGqv20iNgSET+o7k+t2iMiPh8RuyJiR0Sc1+5/hCRpbM1M3RwGPpqZ\nvwVcAFwfEW8F1gH3Z+ZZwP3VY4BLgLOq2xrgppZXLUlq2oRBn5l7MvOxavlF4GlgMbAKuK3qdhtw\nWbW8Crg9a7YBp0TEopZXLklqyqR2xkZEP/B24GHg9MzcA7UvA2Bh1W0x8Gzd00aqNklSBzQd9BHx\nRuBrwNrM/MV4XRu0ZYPXWxMRQxExtG/fvmbLkCRNUlNH3URED7WQ/2pmfr1qfi4iFmXmnmpqZm/V\nPgIsqXt6H7D76NfMzA3ABoCBgYFjvgikYo117f3BAzNbh44bzRx1E8AtwNOZ+Wd1qzYD11TL1wDf\nrGv/QHX0zQXAgdEpHknSzGtmRH8hcDXwRERsr9o+AawHNkXEdcBPgCuqdfcBlwK7gF8C17a0YknS\npEwY9Jn5EI3n3QHe3aB/AtdPsy5JUot4Zqxm1uBcz4KVZpjXupGkwhn0klQ4g16SCmfQS1LhDHpJ\nKpxBL0mFM+glqXAGvSQVzhOm1FZH/2ygJ0tJM88RvSQVzqCXukT/unsb/nC6NF0GvSQVzjl6qUsM\n915VWxisa/THSNQCjuglqXAGvSQVzqCXpMIZ9JJUOINekgrnUTdqGY8Bl7qTQa+2OXK4oKSOcupG\nkgpn0EtS4Qx6SSrchHP0EfFl4HeBvZn5z6u2QeA/APuqbp/IzPuqdX8MXAe8AvznzPzrNtQtHd8G\n5zZo83IJaqyZEf2twMUN2j+Tmcuq22jIvxVYDZxdPefPI2JOq4qVJE3ehEGfmd8Bnm/y9VYBd2bm\nS5n5I2AXsHwa9UnHNS9drFaYzuGVH4mIDwBDwEcz8wVgMbCtrs9I1abSDc7116OkLjXVnbE3AWcC\ny4A9wJ9W7dGgbzZ6gYhYExFDETG0b9++Rl0kSS0wpaDPzOcy85XMfBW4mdemZ0aAJXVd+4DdY7zG\nhswcyMyBBQsWTKUMSVITphT0EbGo7uHlwM5qeTOwOiJOjIilwFnA96ZXoiRpOpo5vHIjsAKYHxEj\nwCeBFRGxjNq0zDDwRwCZ+WREbAKeAg4D12fmK+0pXZLUjAmDPjOvbNB8yzj9PwV8ajpFSXq9o4+8\ncce3JsMzYyWpcAa9JBXOoJekwnk9eqmLeU1/tYIjekkqnEEvSYUz6CWpcAa9JBXOoJekwhn0klQ4\ng16SCmfQS1LhPGFKk9bop+28yJbUvRzRS1LhDHpJKpxBL0mFM+glqXAGvSQVzqCXpMIZ9JJUOINe\nkgpn0EtS4Qx6SSqcl0DQ5AzO9XIH0iwz4Yg+Ir4cEXsjYmdd22kRsSUiflDdn1q1R0R8PiJ2RcSO\niDivncVLkibWzNTNrcDFR7WtA+7PzLOA+6vHAJcAZ1W3NcBNrSlTkjRVEwZ9Zn4HeP6o5lXAbdXy\nbcBlde23Z8024JSIWNSqYiVJkzfVnbGnZ+YegOp+YdW+GHi2rt9I1SZJ6pBWH3UTDdqyYceINREx\nFBFD+/bta3EZkqRRUw3650anZKr7vVX7CLCkrl8fsLvRC2TmhswcyMyBBQsWTLEMSdJEphr0m4Fr\nquVrgG/WtX+gOvrmAuDA6BSPJKkzJjyOPiI2AiuA+RExAnwSWA9siojrgJ8AV1Td7wMuBXYBvwSu\nbUPNkhoZnNug7cDM16GuM2HQZ+aVY6x6d4O+CVw/3aIkSa3jmbEaW6MRoqRZx2vdSAXrX3cv/evu\n7XQZ6jCDXpIKZ9BLUuEMekkqnEEvSYUz6CWpcAa9JBXO4+ilgg33XlVbGDxqhWfMHlcc0UtS4Qx6\nSSqcQS9JhTPoJalw7oyVFy+TCueIXpIK54heOh75IyXHFUf0klQ4g14SgNetL5hBL0mFM+glqXAG\nvSQVzqNujjceMy8ddxzRS1LhDHpJKty0pm4iYhh4EXgFOJyZAxFxGvCXQD8wDPy7zHxhemVKkqaq\nFXP0v5OZP697vA64PzPXR8S66vF/a8H7aLKcj5dEe6ZuVgG3Vcu3AZe14T0kSU2abtAn8O2IeDQi\n1lRtp2fmHoDqfuE030OSNA3Tnbq5MDN3R8RCYEtE/H2zT6y+GNYAnHHGGdMsQ1IrNLoMwvD693ag\nErXStEb0mbm7ut8L3A0sB56LiEUA1f3eMZ67ITMHMnNgwYIF0ylDkjSOKQd9RJwUESePLgP/FtgJ\nbAauqbpdA3xzukVKkqZuOlM3pwN3R8To69yRmX8VEY8AmyLiOuAnwBXTL1OSNFVTDvrM/CFwboP2\n/cC7p1OUpJk33HvVGGv8QZLZzjNjJalwBr0kFc6rV5bAM2AljcMRvSQVzqCXpMI5dSNpfA2mBvsP\n3XFMm2fQdi9H9JJUOEf0ktrK6+d0niN6SSqcQS9JhTPoJalwztFLmrTG18XxmjjdyhG9JBXOEb2k\nlmh0dI26gyN6SSqcI/rZxguYaZZxPr/zDPpuZaBLahGnbiSpcI7oJc04L4swswx6SS0x9m/OqtMM\n+m7gfLzUkCP/1jDoJc24hqP/QWDQo3HawZ2xklQ4R/TtMtZ0jCMWaUzNnF07mTNwneapaduIPiIu\njojvR8SuiFjXrveRJI2vLSP6iJgDfBH4N8AI8EhEbM7Mp9rxfh03mZ2p7niVxtTskTuNfrO2U2bD\nDuN2Td0sB3Zl5g8BIuJOYBUwu4PekJa62jFfFIPjdJ5gGrWki7S1K+gXA8/WPR4Bzm/LO7VrLtxQ\nl4pWH+QzNQLv1Og/MrP1LxpxBfCezPz31eOrgeWZ+Z/q+qwB1lQP3wJ8v+WF1MwHft6m126n2Vj3\nbKwZZmfd1jxzurnuf5KZCybq1K4R/QiwpO5xH7C7vkNmbgA2tOn9j4iIocwcaPf7tNpsrHs21gyz\ns25rnjmzte567Trq5hHgrIhYGhG/AawGNrfpvSRJ42jLiD4zD0fER4C/BuYAX87MJ9vxXpKk8bXt\nhKnMvA+4r12vPwltnx5qk9lY92ysGWZn3dY8c2Zr3Ue0ZWesJKl7eK0bSSrcrA36iPhyROyNiJ1j\nrP/DiNhR3b4bEefWrRuOiCciYntEDM1c1U3VvSIiDlS1bY+IP6lb15HLSjRR83+tq3dnRLwSEadV\n6zryWUfEkoh4ICKejognI+KGBn0iIj5ffZ47IuK8unXXRMQPqts1XVRz123XTdbdVdt1kzV33XY9\nZZk5K2/AvwLOA3aOsf63gVOr5UuAh+vWDQPzu7TuFcA9DdrnAM8A/xT4DeBx4K3dUPNRfX8P+JtO\nf9bAIuC8avlk4P8e/XkBlwLfAgK4YHQbAU4Dfljdn1otn9olNXfddt1k3V21XTdT81H9u2K7nupt\n1o7oM/M7wPPjrP9uZr5QPdxG7Vj+jpuo7nEcuaxEZr4MjF5Wou0mWfOVwMY2ltOUzNyTmY9Vyy8C\nT1M7Y7veKuD2rNkGnBIRi4D3AFsy8/lqG9oCXNwNNXfjdt3kZz2WjmzXU6i5K7brqZq1QT9J11Eb\nuY1K4NsR8Wh1hm63+RcR8XhEfCsizq7aGl1Wotn/TDMiIn6TWiB+ra654591RPQDbwcePmrVWJ9p\nxz/rcWqu13Xb9QR1d+V2PdFn3a3b9WQUfz36iPgdav8h/mVd84WZuTsiFgJbIuLvq1FrN3iM2mnN\nByPiUuAbwFnUpheO1m2HTP0e8LeZWT/67+hnHRFvpPYfdG1m/uLo1Q2ekuO0z4gJah7t03Xb9QR1\nd+V23cxnTRdu15NV9Ig+Is4BvgSsysz9o+2Zubu63wvcTe3Px66Qmb/IzIPV8n1AT0TMp4nLSnSB\n1Rz1520nP+uI6KH2n/irmfn1Bl3G+kw79lk3UXNXbtcT1d2N23Uzn3Wlq7brKen0ToLp3IB+xt6p\neQawC/jto9pPAk6uW/4ucHEX1f2Pee38huXAT6iNek6gtlNwKa/ttDq7G2qu1s+lNo9/Ujd81tVn\ndjvw2XH6vJfX74z9XtV+GvAjajtiT62WT+uSmrtuu26y7q7arpupuerXVdv1VG+zduomIjZS25M/\nPyJGgE8CPQCZ+RfAnwDzgD+PCIDDWbsw0enA3VXbCcAdmflXXVT37wP/MSIOA78CVmdti+rYZSWa\nqBngcuDbmfn/6p7ayc/6QuBq4ImI2F61fYJaUI7WfR+1I292Ab8Erq3WPR8R/4PaNZsA/nu+/s/2\nTtbcjdt1M3V323bdTM3Qfdv1lHhmrCQVrug5ekmSQS9JxTPoJalwBr0kFc6gl6TCGfSSVDiDXpIK\nZ9BLUuH+PyRp+ed07bmKAAAAAElFTkSuQmCC\n",
182 |       "text/plain": [
183 |        "<matplotlib.figure.Figure at 0x1f481a45dd8>"
184 |       ]
185 |      },
186 |      "metadata": {},
187 |      "output_type": "display_data"
188 |     },
189 |     {
190 |      "name": "stdout",
191 |      "output_type": "stream",
192 |      "text": [
193 |       "mean: 2.01289078955\n",
194 |       "variance: 0.0405462254784\n",
195 |       "CRB variance: 0.04\n"
196 |      ]
197 |     }
198 |    ],
199 |    "source": [
200 |     "n = 100\n",
201 |     "crb_variance = rate**2/n\n",
202 |     "rate_hat = np.zeros(NUM_SAMPLES)\n",
203 |     "rate_hat_crb = np.zeros(NUM_SAMPLES)\n",
204 |     "for i in range(0,NUM_SAMPLES):\n",
205 |     "    y = np.random.exponential(1/rate,n)\n",
206 |     "    rate_hat_crb[i] = rate + np.sqrt(crb_variance)*np.random.randn() #This is what an efficient estimator would look like\n",
207 |     "    rate_hat[i] = 1/np.mean(y) #The ML estimate is the reciprocal of the sample mean\n",
208 |     "    \n",
209 |     "plt.hist(rate_hat,50)\n",
210 |     "plt.hist(rate_hat_crb,50)\n",
211 |     "plt.gca().legend(['ML','CRB'])\n",
212 |     "plt.show()\n",
213 |     "\n",
214 |     "print('mean: '+str(np.mean(rate_hat)))\n",
215 |     "print('variance: '+str(np.var(rate_hat)))\n",
216 |     "print('CRB variance: '+str(crb_variance))"
217 |    ]
218 |   },
219 |   {
220 |    "cell_type": "markdown",
221 |    "metadata": {},
222 |    "source": [
223 |     "At this point the distributions are difficult to distinguish. Asymptotic efficiency, indeed!"
224 |    ]
225 |   },
226 |   {
227 |    "cell_type": "code",
228 |    "execution_count": null,
229 |    "metadata": {
230 |     "collapsed": true
231 |    },
232 |    "outputs": [],
233 |    "source": []
234 |   }
235 |  ],
236 |  "metadata": {
237 |   "kernelspec": {
238 |    "display_name": "Python 3",
239 |    "language": "python",
240 |    "name": "python3"
241 |   },
242 |   "language_info": {
243 |    "codemirror_mode": {
244 |     "name": "ipython",
245 |     "version": 3
246 |    },
247 |    "file_extension": ".py",
248 |    "mimetype": "text/x-python",
249 |    "name": "python",
250 |    "nbconvert_exporter": "python",
251 |    "pygments_lexer": "ipython3",
252 |    "version": "3.6.3"
253 |   }
254 |  },
255 |  "nbformat": 4,
256 |  "nbformat_minor": 2
257 | }
258 | 


--------------------------------------------------------------------------------
/ch.4/Bias, Variance, and the Bias-variance Trade-off.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# Bias, Variance, and the Bias-variance Trade-off\n",
  8 |     "\n",
  9 |     "Here we'll spend time visualizing what it means for an estimator to be biased, what the variance of an estimator represents, and why there is an inherent trade-off between bias and variance. We'll do this with the simple example of estimating the mean and variance of a scalar Gaussian distribution."
 10 |    ]
 11 |   },
 12 |   {
 13 |    "cell_type": "code",
 14 |    "execution_count": 2,
 15 |    "metadata": {
 16 |     "collapsed": true
 17 |    },
 18 |    "outputs": [],
 19 |    "source": [
 20 |     "import numpy as np\n",
 21 |     "import matplotlib.pyplot as plt\n",
 22 |     "\n",
 23 |     "#Standard normal distribution\n",
 24 |     "mu = 0\n",
 25 |     "sigma_2 = 1"
 26 |    ]
 27 |   },
 28 |   {
 29 |    "cell_type": "markdown",
 30 |    "metadata": {},
 31 |    "source": [
 32 |     "First, suppose the mean is unknown but the variance known. Let's use the sample mean estimator, taken from n i.i.d. samples. To understand the estimator as a random variable, we'll loop over many *different* realizations of the data and its estimator.\n",
 33 |     "\n",
 34 |     "The histogram shows us that mu_hat is a Gaussian distribution. Its mean is clearly zero, which means that it's unbiased! And its variance is proportional to 1/n"
 35 |    ]
 36 |   },
 37 |   {
 38 |    "cell_type": "code",
 39 |    "execution_count": 5,
 40 |    "metadata": {},
 41 |    "outputs": [
 42 |     {
 43 |      "data": {
 44 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAADcVJREFUeJzt3V2MXedVh/Fn1WlaRBOC47FjxRkm\nEhZqqKgjRlZQVCrygQyuYl+koVVbppKluSgIUPgyHxIS7YUDgrQSXGAlFVMExKkhspUEaHCTVkhN\nVLuJCqkpTiPTGltxSp3SqKKV28XFbFPXmfHec+Z8rnl+knXO3uc9c5Ze2f95vWbvdyIzkSRNvteN\nugBJUn8Y6JJUhIEuSUUY6JJUhIEuSUUY6JJUhIEuSUUY6JJUhIEuSUVcMcwP27BhQ87MzAzzIyVp\n4h07duyrmTnVNm6ogT4zM8PRo0eH+ZGSNPEi4j+7jLPlIklFGOiSVISBLklFGOiSVISBLklFGOiS\nVISBLklFGOiSVISBLklFDPVOUWlYZvY+tuT5k/t2DrkSaXhcoUtSEQa6JBVhoEtSEQa6JBVhoEtS\nEQa6JBVhoEtSEQa6JBVhoEtSEQa6JBVhoEtSEQa6JBVhoEtSEQa6JBXRafvciDgJfAP4DnA+M2cj\nYj1wAJgBTgL3ZOa5wZQpSWqzkhX6z2TmtsycbY73AkcycytwpDmWJI3Ialouu4CF5vkCsHv15UiS\netU10BP4REQci4j55tymzDwD0DxuHESBkqRuuv4Kulsz83REbASeiIh/7/oBzTeAeYDp6ekeSpQk\nddFphZ6Zp5vHs8AjwHbgpYjYDNA8nl3mvfszczYzZ6empvpTtSTpNVoDPSJ+MCKuuvAc+Fng34DD\nwFwzbA44NKgiJUnturRcNgGPRMSF8X+Tmf8YEZ8FHo6IPcCXgXcOrkxpvMzsfWzJ8yf37RxyJdL3\ntAZ6Zr4IvHWJ8/8N3D6IoiRJK+edopJUhIEuSUUY6JJUhIEuSUUY6JJUhIEuSUV0vfVf6onXa0vD\n4wpdkoow0CWpCANdkoqwh66J0K9e/HJfZzn2+jVJXKFLUhEGuiQVYctFY2WlLRFJ3+MKXZKKMNAl\nqQgDXZKKMNAlqQgDXZKKMNAlqQgDXZKKMNAlqQgDXZKKMNAlqQgDXZKKcC8XjcRa3LPFX8enQXOF\nLklFGOiSVIQtF020yq0bWzRaqc4r9IhYFxHPRsSjzfGNEfFMRJyIiAMRceXgypQktVlJy+VXgeMX\nHd8H3J+ZW4FzwJ5+FiZJWplOgR4RW4CdwAPNcQC3AQebIQvA7kEUKEnqpmsP/cPAbwFXNcfXAq9k\n5vnm+BRw/VJvjIh5YB5genq690qlCVC5p6/x17pCj4h3AGcz89jFp5cYmku9PzP3Z+ZsZs5OTU31\nWKYkqU2XFfqtwF0R8fPAG4GrWVyxXxMRVzSr9C3A6cGVKUlq07pCz8zfycwtmTkDvAv4ZGa+B3gS\nuLsZNgccGliVkqRWq7mx6LeBeyPiBRZ76g/2pyRJUi9WdGNRZj4FPNU8fxHY3v+SJEm98NZ/SSrC\nQJekItzLRboMryvXJHGFLklFGOiSVIQtF2nEbOuoX1yhS1IRBrokFWGgS1IRBrokFWGgS1IRBrok\nFWGgS1IRBrokFWGgS1IRBrokFWGgS1IRBrokFWGgS1IRBrokFWGgS1IRBrokFWGgS1IRBrokFWGg\nS1IRBrokFWGgS1IRBrokFXFF24CIeCPwaeANzfiDmfkHEXEj8BCwHvgc8L7M/PYgi9X4mtn72KhL\nWDOWm+uT+3YOuRKNmy4r9G8Bt2XmW4FtwI6IuAW4D7g/M7cC54A9gytTktSmNdBz0avN4eubPwnc\nBhxszi8AuwdSoSSpk0499IhYFxHPAWeBJ4AvAa9k5vlmyCng+sGUKEnqolOgZ+Z3MnMbsAXYDrx5\nqWFLvTci5iPiaEQcffnll3uvVJJ0WSu6yiUzXwGeAm4BromICz9U3QKcXuY9+zNzNjNnp6amVlOr\nJOkyWgM9IqYi4prm+Q8AdwDHgSeBu5thc8ChQRUpSWrXetkisBlYiIh1LH4DeDgzH42ILwAPRcSH\ngGeBBwdYpySpRWugZ+bngZuXOP8ii/10SdIY8E5RSSrCQJekIgx0SSrCQJekIgx0SSrCQJekIgx0\nSSrCQJekIgx0SSrCQJekIrrs5SJpAvir6eQKXZKKMNAlqQgDXZKKsIeu17AXK00mV+iSVISBLklF\n2HJZw5ZrragWW2hrhyt0SSrCQJekIgx0SSrCQJekIgx0SSrCQJekIgx0SSrC69AlfR+vW59crtAl\nqQgDXZKKMNAlqYjWHnpE3AB8DLgO+C6wPzM/EhHrgQPADHASuCczzw2uVPXKPVuktaHLCv088OuZ\n+WbgFuCXIuImYC9wJDO3AkeaY0nSiLQGemaeyczPNc+/ARwHrgd2AQvNsAVg96CKlCS1W9FlixEx\nA9wMPANsyswzsBj6EbFxmffMA/MA09PTq6lVLWytSGtb5x+KRsSbgL8Dfi0z/6fr+zJzf2bOZubs\n1NRULzVKkjroFOgR8XoWw/yvM/Pvm9MvRcTm5vXNwNnBlChJ6qI10CMigAeB45n5pxe9dBiYa57P\nAYf6X54kqasuPfRbgfcB/xoRzzXnfhfYBzwcEXuALwPvHEyJkqQuWgM9M/8FiGVevr2/5UiSeuWd\nopJUhIEuSUW4fa60RnnfQj2u0CWpCANdkoow0CWpCHvo6syeqzTeXKFLUhEGuiQVYaBLUhEGuiQV\nYaBLUhEGuiQV4WWLkjpZ7rLVk/t2DrkSLccVuiQVYaBLUhEGuiQVYaBLUhEGuiQVYaBLUhEGuiQV\nYaBLUhEGuiQVYaBLUhEGuiQV4V4uE8hfBSdpKa7QJakIA12SimhtuUTER4F3AGcz8y3NufXAAWAG\nOAnck5nnBldmbW5LKqkfuqzQ/xLYccm5vcCRzNwKHGmOJUkj1Bromflp4GuXnN4FLDTPF4Ddfa5L\nkrRCvfbQN2XmGYDmcWP/SpIk9WLgly1GxDwwDzA9PT3oj5M0Afy50WD0ukJ/KSI2AzSPZ5cbmJn7\nM3M2M2enpqZ6/DhJUpteA/0wMNc8nwMO9accSVKvWgM9Iv4W+AzwYxFxKiL2APuAOyPiBHBncyxJ\nGqHWHnpmvnuZl27vcy2SCunnFhX23LvxTlFJKsJAl6QiDHRJKsLtc8eY2+RKWglX6JJUhIEuSUXY\ncpG0KrYGx4crdEkqwkCXpCIMdEkqwh76ENlrlPrLLQG+nyt0SSrCQJekIgx0SSrCHvoA2CuXeuO/\nndVxhS5JRRjoklSEgS5JRdhDb/TSu1ur17pKGk+u0CWpCANdkoow0CWpCHvoq+A1s9JkWem/2Un7\nOZkrdEkqwkCXpCJsuUgqZ622Q12hS1IRBrokFWGgS1IRq+qhR8QO4CPAOuCBzNzXl6qW0K+e2KRd\nhiRpcoz6V+L1vEKPiHXAnwM/B9wEvDsibupXYZKklVlNy2U78EJmvpiZ3wYeAnb1pyxJ0kqtJtCv\nB75y0fGp5pwkaQRW00OPJc7lawZFzAPzzeGrEfHFVXzmpTYAX13JG+K+Pn76ZFjxHK1RzlO7NTdH\nPeTFknPUh9z5kS6DVhPop4AbLjreApy+dFBm7gf2r+JzlhURRzNzdhBfuwrnqBvnqZ1z1G7Uc7Sa\nlstnga0RcWNEXAm8Czjcn7IkSSvV8wo9M89HxC8D/8TiZYsfzczn+1aZJGlFVnUdemY+Djzep1p6\nMZBWTjHOUTfOUzvnqN1I5ygyX/NzTEnSBPLWf0kqYqICPSLWR8QTEXGiefzhy4y9OiL+KyL+bJg1\njlqXOYqIbRHxmYh4PiI+HxG/MIpahy0idkTEFyPihYjYu8Trb4iIA83rz0TEzPCrHL0O83RvRHyh\n+btzJCI6XVJXSdscXTTu7ojIiBjKlS8TFejAXuBIZm4FjjTHy/kg8KmhVDVeuszRN4FfzMwfB3YA\nH46Ia4ZY49B13KpiD3AuM38UuB9Yc3ctdJynZ4HZzPwJ4CDwR8OtcrS6bnsSEVcBvwI8M6zaJi3Q\ndwELzfMFYPdSgyLiJ4FNwCeGVNc4aZ2jzPyPzDzRPD8NnAWmhlbhaHTZquLiuTsI3B4RS91AV1nr\nPGXmk5n5zebwaRbvQVlLum578kEWv9n977AKm7RA35SZZwCax42XDoiI1wF/AvzmkGsbF61zdLGI\n2A5cCXxpCLWNUpetKv5/TGaeB74OXDuU6sbHSrf02AP8w0ArGj+tcxQRNwM3ZOajwyxs7H4FXUT8\nM3DdEi/9Xscv8QHg8cz8StXFVR/m6MLX2Qz8FTCXmd/tR21jrMtWFZ22syiu8xxExHuBWeDtA61o\n/Fx2jppF5f3A+4dV0AVjF+iZecdyr0XESxGxOTPPNGF0dolhPwW8LSI+ALwJuDIiXs3My/XbJ0of\n5oiIuBp4DPj9zHx6QKWOky5bVVwYcyoirgB+CPjacMobG5229IiIO1hcQLw9M781pNrGRdscXQW8\nBXiqWVReBxyOiLsy8+ggC5u0lsthYK55PgccunRAZr4nM6czcwb4DeBjlcK8g9Y5arZqeITFufn4\nEGsbpS5bVVw8d3cDn8y1d6NG6zw17YS/AO7KzCUXDMVddo4y8+uZuSEzZ5oceprFuRpomMPkBfo+\n4M6IOAHc2RwTEbMR8cBIKxsfXeboHuCngfdHxHPNn22jKXc4mp74ha0qjgMPZ+bzEfGHEXFXM+xB\n4NqIeAG4l8tfRVVSx3n6Yxb/9/vx5u/OmtrDqeMcjYR3ikpSEZO2QpckLcNAl6QiDHRJKsJAl6Qi\nDHRJKsJAl6QiDHRJKsJAl6Qi/g+QY4wWD6d6tgAAAABJRU5ErkJggg==\n",
 45 |       "text/plain": [
 46 |        "<matplotlib.figure.Figure at 0x1b5dc45a1d0>"
 47 |       ]
 48 |      },
 49 |      "metadata": {},
 50 |      "output_type": "display_data"
 51 |     }
 52 |    ],
 53 |    "source": [
 54 |     "NUM_SAMPLES = 1000\n",
 55 |     "n = 50\n",
 56 |     "\n",
 57 |     "mu_hat = np.zeros(NUM_SAMPLES)\n",
 58 |     "for i in range(0,NUM_SAMPLES):\n",
 59 |     "    y = np.random.randn(n) #n i.i.d. samples\n",
 60 |     "    mu_hat[i] = np.mean(y)\n",
 61 |     "\n",
 62 |     "plt.hist(mu_hat,50)\n",
 63 |     "plt.show()\n",
 64 |     "    "
 65 |    ]
 66 |   },
 67 |   {
 68 |    "cell_type": "markdown",
 69 |    "metadata": {},
 70 |    "source": [
 71 |     "Next, let's try using the sample *variance* estimator, supposing that the mean and variance are both unknown. We want to see that this estimator is in fact biased!"
 72 |    ]
 73 |   },
 74 |   {
 75 |    "cell_type": "code",
 76 |    "execution_count": 7,
 77 |    "metadata": {},
 78 |    "outputs": [
 79 |     {
 80 |      "data": {
 81 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAEi5JREFUeJzt3X+sZGddx/H3h98oSIG91Lrdeitu\nlUKk4FqrqClU+VGMW5JCigqFNC7RQsAQQ+UPQZSkGLFiQHChDS0BSsOvrlBBrCAgFNhCKbS1usDa\nLt10FykFJaDdfv1jztZxuffOuffO3Jl59v1Kbu6ZZ56Z+d7dmc8885znnElVIUlq172mXYAkabIM\neklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1Lj7jPtAgA2bdpUi4uL0y5DkubKtdde\n+42qWhjVbyaCfnFxkd27d0+7DEmaK0n+vU8/p24kqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6\nSWqcQS9JjTPoJalxM3FkrDTK4gUfvGd774VPn2Il0vxxRC9JjTPoJalxBr0kNc6gl6TGGfSS1DiD\nXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDVuZNAneUCSzyb5YpIbkvxx135iks8k+bck70pyv679\n/t3lPd31i5P9EyRJK+kzov8+8KSqeixwCvDUJKcBrwEuqqqtwB3AeV3/84A7quongYu6fpKkKRkZ\n9DXwn93F+3Y/BTwJeHfXfilwVre9vbtMd/0ZSTK2iiVJq9Jrjj7JvZNcBxwAPgJ8BfhWVd3VddkH\nbO62NwO3AnTX3wk8fJxFS5L66xX0VXWoqk4BjgdOBR61VLfu91Kj9zqyIcmOJLuT7D548GDfeiVJ\nq7SqVTdV9S3gY8BpwDFJDn9xyfHAbd32PmALQHf9Q4BvLnFfO6tqW1VtW1hYWFv1kqSR+qy6WUhy\nTLf9QOBXgZuAjwJnd93OBa7stnd1l+mu/8eq+oERvSRpY/T5KsHjgEuT3JvBG8MVVfWBJDcClyf5\nU+ALwMVd/4uBtyXZw2Akf84E6pYk9TQy6KvqeuBxS7R/lcF8/ZHt3wOeOZbqJEnr5pGxktS4PlM3\n0kQtXvDBe7b3Xvj0KVYitcmg11QMh7ukyXLqRpIaZ9BLUuMMeklqnEEvSY1zZ6zGzlU00mwx6DXX\nfFORRjPoNVEbGcSGvrQ05+glqXGO6DVTHJVL4+eIXpIaZ9BLUuOcutGG8fw20nQ4opekxjmi18xa\n7hOAnwyk1XFEL0mNM+glqXFO3WjNXPMuzQdH9JLUOINekhpn0EtS40YGfZItST6a5KYkNyR5cdf+\nyiRfT3Jd93Pm0G3+MMmeJDcnecok/wBJ0sr67Iy9C3hpVX0+yYOBa5N8pLvuoqr68+HOSU4GzgEe\nDfwY8A9JTqqqQ+MsXJLUz8igr6r9wP5u+ztJbgI2r3CT7cDlVfV94GtJ9gCnAp8eQ72aUR7EJM2u\nVc3RJ1kEHgd8pmt6YZLrk1yS5KFd22bg1qGb7WPlNwZJ0gT1DvokDwLeA7ykqr4NvBF4JHAKgxH/\naw93XeLmtcT97UiyO8nugwcPrrpwSVI/vYI+yX0ZhPzbq+q9AFV1e1Udqqq7gTczmJ6BwQh+y9DN\njwduO/I+q2pnVW2rqm0LCwvr+RskSSvos+omwMXATVX1F0Ptxw11ewbw5W57F3BOkvsnORHYCnx2\nfCVLklajz6qbJwDPAb6U5Lqu7eXAs5OcwmBaZi/wAoCquiHJFcCNDFbsnO+Km/nj6Q2kdvRZdfNJ\nlp53v2qF27waePU66tKMmvfVNb6B6WjkkbGS1DiDXpIaZ9BLUuM8H72a5Fy89H8c0UtS4wx6SWqc\nQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY3zXDfSEjxXjlriiF6SGueI\nXiPN+7dKSUc7R/SS1DhH9LrH0TZyP/LvdS5erTLo1byj7Q1MOpJTN5LUOINekho3cuomyRbgMuBH\ngbuBnVX1uiQPA94FLAJ7gWdV1R1JArwOOBP4LvC8qvr8ZMqXxscpHrWqz4j+LuClVfUo4DTg/CQn\nAxcAV1fVVuDq7jLA04Ct3c8O4I1jr1qS1NvIoK+q/YdH5FX1HeAmYDOwHbi063YpcFa3vR24rAau\nAY5JctzYK5ck9bKqOfoki8DjgM8Ax1bVfhi8GQCP6LptBm4dutm+ru3I+9qRZHeS3QcPHlx95ZKk\nXnoHfZIHAe8BXlJV316p6xJt9QMNVTuraltVbVtYWOhbhiRplXoFfZL7Mgj5t1fVe7vm2w9PyXS/\nD3Tt+4AtQzc/HrhtPOVKklZrZNB3q2guBm6qqr8YumoXcG63fS5w5VD7czNwGnDn4SkeSdLG63Nk\n7BOA5wBfSnJd1/Zy4ELgiiTnAbcAz+yuu4rB0so9DJZXPn+sFWvdPAWvdHQZGfRV9UmWnncHOGOJ\n/gWcv866tEFcOy61zyNjJalxntRMWgWnvTSPHNFLUuMMeklqnEEvSY0z6CWpce6MPQq4hFI6ujmi\nl6TGGfSS1DiDXpIaZ9BLUuMMeklqnKtupDXydAiaF47oJalxBr0kNc6pG2kEDzjTvHNEL0mNM+gl\nqXEGvSQ1zqCXpMYZ9JLUOINekhrn8spGuSRQ0mEjR/RJLklyIMmXh9pemeTrSa7rfs4cuu4Pk+xJ\ncnOSp0yqcElSP31G9G8FXg9cdkT7RVX158MNSU4GzgEeDfwY8A9JTqqqQ2OoVZpZnvdGs2xk0FfV\nx5Ms9ry/7cDlVfV94GtJ9gCnAp9ec4VakQEjaZT17Ix9YZLru6mdh3Ztm4Fbh/rs69p+QJIdSXYn\n2X3w4MF1lCFJWslag/6NwCOBU4D9wGu79izRt5a6g6raWVXbqmrbwsLCGsuQJI2ypqCvqtur6lBV\n3Q28mcH0DAxG8FuGuh4P3La+EiVJ67GmoE9y3NDFZwCHV+TsAs5Jcv8kJwJbgc+ur0RJ0nqM3Bmb\n5J3A6cCmJPuAVwCnJzmFwbTMXuAFAFV1Q5IrgBuBu4DzXXEjSdPVZ9XNs5dovniF/q8GXr2eoqQW\nuUJK0+KRsdKYGeiaNZ7rRpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4D5iSJsiv\ndNQscEQvSY1zRN8QR4+SluKIXpIa54hemiGeEE2TYNBLU2CgayM5dSNJjTPoJalxTt3MCT/qS1or\nR/SS1DiDXpIaZ9BLUuMMeklq3MidsUkuAX4dOFBVj+naHga8C1gE9gLPqqo7kgR4HXAm8F3geVX1\n+cmUfvTyVAeSVqPPiP6twFOPaLsAuLqqtgJXd5cBngZs7X52AG8cT5mSpLUaGfRV9XHgm0c0bwcu\n7bYvBc4aar+sBq4Bjkly3LiKlSSt3lrX0R9bVfsBqmp/kkd07ZuBW4f67eva9q+9RKltTsVp0sa9\nMzZLtNWSHZMdSXYn2X3w4MExlyFJOmytQX/74SmZ7veBrn0fsGWo3/HAbUvdQVXtrKptVbVtYWFh\njWVIkkZZa9DvAs7tts8Frhxqf24GTgPuPDzFI0majj7LK98JnA5sSrIPeAVwIXBFkvOAW4Bndt2v\nYrC0cg+D5ZXPn0DNkqRVGBn0VfXsZa46Y4m+BZy/3qIkSePjkbGS1DiDXpIaZ9BLUuP84pEZ5oE0\nksbBEb0kNc6gl6TGOXUzY5yukTRuBr00o/p8IbxfGq8+nLqRpMYZ9JLUOKduZoDz8pImyRG9JDXO\noJekxjl1I80Zp/q0Wga9NAcMd62HUzeS1DiDXpIaZ9BLUuOco5ca5KkRNMyg30C++CRNg1M3ktQ4\ng16SGmfQS1Lj1jVHn2Qv8B3gEHBXVW1L8jDgXcAisBd4VlXdsb4yJUlrNY6dsU+sqm8MXb4AuLqq\nLkxyQXf5ZWN4nKZ4pKPGzeeUljOJqZvtwKXd9qXAWRN4DElST+sd0Rfw90kK+Juq2gkcW1X7Aapq\nf5JHrLdISePhEt+j03qD/glVdVsX5h9J8i99b5hkB7AD4IQTTlhnGZKk5axr6qaqbut+HwDeB5wK\n3J7kOIDu94FlbruzqrZV1baFhYX1lCFJWsGaR/RJfhi4V1V9p9t+MvAqYBdwLnBh9/vKcRQqaW2W\n20m7XLtTOu1Zz9TNscD7khy+n3dU1YeSfA64Isl5wC3AM9dfpiRprdYc9FX1VeCxS7T/B3DGeoqS\nJI2PR8ZKUuM8e+WYeLCKpFnliF6SGmfQS1LjDHpJapxBL0mNc2espBV5fpz554hekhrniH4dXFKp\nFvm8bo9BL2lNnNKZH07dSFLjHNFL6s1pnflk0EtaN6dxZptTN5LUOEf0ksbK0f3sMehXyTlKabx8\nY5g8p24kqXGO6JfhKENaP19Hs8Ggl7QhDP3pMeh7cF5eGi9fUxvLoJfUBD8xLM+gH+IoQ5ody70e\nDfHVazro+7zDG+6SWjexoE/yVOB1wL2Bt1TVhZN6rNUy3KX55RTN6k0k6JPcG3gD8GvAPuBzSXZV\n1Y3jfqy+/+mGu9SeSb+uV7r/eXqTmdSI/lRgT1V9FSDJ5cB2YOxBL0lrMc19ABv9qWRSQb8ZuHXo\n8j7g5yf0WJIaMa4R+nrup+9tlwvrWZxamlTQZ4m2+n8dkh3Aju7ifya5eYX72wR8Y+SDvqZ3fRuh\nV80zZN7qhfmred7qhfmreSr1Lpc9fTIpr1lXzT/ep9Okgn4fsGXo8vHAbcMdqmonsLPPnSXZXVXb\nxlfe5M1bzfNWL8xfzfNWL8xfzfNWL2xMzZM6qdnngK1JTkxyP+AcYNeEHkuStIKJjOir6q4kLwQ+\nzGB55SVVdcMkHkuStLKJraOvqquAq8Z0d72meGbMvNU8b/XC/NU8b/XC/NU8b/XCBtScqhrdS5I0\nt/ziEUlq3EwFfZKnJrk5yZ4kF6zQ7+wklWSqe9f71JvkWUluTHJDkndsdI1L1LNizUlOSPLRJF9I\ncn2SM6dR51A9lyQ5kOTLy1yfJH/V/T3XJ3n8Rte4RE2jav6trtbrk3wqyWM3usYj6lmx3qF+P5fk\nUJKzN6q2FWoZWXOS05Nc1732/mkj61uillHPiYck+dskX+zqff5YC6iqmfhhsNP2K8BPAPcDvgic\nvES/BwMfB64Bts1yvcBW4AvAQ7vLj5j1f2MG84W/222fDOydcs2/Ajwe+PIy158J/B2DYzdOAz4z\nzXp71vyLQ8+Jp0275lH1Dj13/pHBfrez5+Df+BgGR+Kf0F2e9mtvVL0vB17TbS8A3wTuN67Hn6UR\n/T2nTaiq/wYOnzbhSH8C/BnwvY0sbgl96v0d4A1VdQdAVR3Y4BqP1KfmAn6k234IRxz/sNGq6uMM\nnvTL2Q5cVgPXAMckOW5jqlvaqJqr6lOHnxMMBizHb0hhy9cz6t8Y4EXAe4BpP4eBXjX/JvDeqrql\n6z/VunvUW8CDkwR4UNf3rnE9/iwF/VKnTdg83CHJ44AtVfWBjSxsGSPrBU4CTkryz0mu6c7oOU19\nan4l8NtJ9jEYvb1oY0pbsz5/0yw7j8EnkpmVZDPwDOBN065lFU4CHprkY0muTfLcaRc0wuuBRzEY\nWH0JeHFV3T2uO5+l89GveNqEJPcCLgKet1EFjTDyNA8M/n23AqczGLV9IsljqupbE65tOX1qfjbw\n1qp6bZJfAN7W1Ty2J92Y9fmbZlKSJzII+l+adi0j/CXwsqo6NBhwzoX7AD8LnAE8EPh0kmuq6l+n\nW9ayngJcBzwJeCTwkSSfqKpvj+POZ2lEP+q0CQ8GHgN8LMleBvOxu6a4Q3bkaR66PldW1f9U1deA\nmxkE/7T0qfk84AqAqvo08AAG5w+ZVX3+ppmT5GeAtwDbq+o/pl3PCNuAy7vX3dnAXyc5a7oljbQP\n+FBV/VdVfYPBfr2p7vQe4fkMppqqqvYAXwN+elx3PktBv+JpE6rqzqraVFWLVbXIYG7zN6pq93TK\n7XWah/cDTwRIsonBx8mvbmiV/1+fmm9hMAoiyaMYBP3BDa1ydXYBz+1W35wG3FlV+6dd1EqSnAC8\nF3jODI8w71FVJw697t4N/F5VvX/KZY1yJfDLSe6T5IcYnD33pinXtJLh192xwE8xxqyYmambWua0\nCUleBeyuqpk6V07Pej8MPDnJjcAh4A+mOXrrWfNLgTcn+X0GUyDPq24pwDQkeSeDqa9N3X6DVwD3\nBaiqNzHYj3AmsAf4LoOR0VT1qPmPgIczGBkD3FVTPBFXj3pnzqiaq+qmJB8CrgfuZvAtdysuH51m\nvQwWmbw1yZcYTEe+rPskMp7Hn+JrWJK0AWZp6kaSNAEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6\nSWqcQS9JjftfS8MsvwXE3SkAAAAASUVORK5CYII=\n",
 82 |       "text/plain": [
 83 |        "<matplotlib.figure.Figure at 0x1b5db6efb00>"
 84 |       ]
 85 |      },
 86 |      "metadata": {},
 87 |      "output_type": "display_data"
 88 |     }
 89 |    ],
 90 |    "source": [
 91 |     "NUM_SAMPLES = 10000\n",
 92 |     "n = 50\n",
 93 |     "\n",
 94 |     "sigma_2_hat = np.zeros(NUM_SAMPLES)\n",
 95 |     "for i in range(0,NUM_SAMPLES):\n",
 96 |     "    y = np.random.randn(n) #n i.i.d. samples\n",
 97 |     "    mu_hat = np.mean(y)\n",
 98 |     "    sigma_2_hat[i] = np.mean((y - mu_hat)**2)\n",
 99 |     "\n",
100 |     "plt.hist(sigma_2_hat,100)\n",
101 |     "plt.show()"
102 |    ]
103 |   },
104 |   {
105 |    "cell_type": "markdown",
106 |    "metadata": {},
107 |    "source": [
108 |     "Notice a few things. First, the sample variance is *not* Gaussian. In fact, it is distributed according to a *chi-square* distribution, which is roughly the distribution of the *square* of a Gaussian random variable. Second, and more important, the mean of this distribution is clearly not one. This is because the sample variance is biased by the constant $(n-1)/n)$. If we increase $n$, we get a distribution that is closer to being unbiased.\n",
109 |     "\n",
110 |     "Finally, let's look at the bias variance trade-off by altering the sample mean estimator from above. We'll see that by \"overdoing\" the averaging of samples, we can reduce the variance of the estimator, but we add bias."
111 |    ]
112 |   },
113 |   {
114 |    "cell_type": "code",
115 |    "execution_count": 8,
116 |    "metadata": {},
117 |    "outputs": [
118 |     {
119 |      "data": {
120 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAGmFJREFUeJzt3XF01OWd7/H3VwhGFwqSgAsENlDR\nFowg5FiqdKFGC4otaLXLSjEge+jey9rqqpeoPVe6pz0Hb8/ZVk973ctdWoOHtlB2FVZZdy3i0doi\nTSgUKHWlNoURLsQE0lKLJfV7/5hf0iFMyG+SmUzmyed1zpz5zfN7ZvL9ZeAzvzzzzDPm7oiISLgu\nyHcBIiKSWwp6EZHAKehFRAKnoBcRCZyCXkQkcAp6EZHAdRn0ZnaFme1OufzGzO41s+Fm9qKZvRld\nXxL1NzN7wswOmtnPzGxa7g9DREQ602XQu/sb7j7V3acC04F3gWeAGmCbu08EtkW3AW4CJkaX5cCT\nuShcRETiyXTopgr4pbv/GpgP1EbttcCCaHs+sM6TdgDDzGxUVqoVEZGMDcyw/0Lgu9H2pe5+FMDd\nj5rZyKh9DHA45T6JqO1oZw9aWlrq5eXlGZYiItK/1dfXv+PuI7rqFzvozWwQ8Cngoa66pmk7Z50F\nM1tOcmiHcePGUVdXF7cUEREBzOzXcfplMnRzE7DL3Y9Ft4+1DclE18ej9gQwNuV+ZcCRjg/m7mvc\nvdLdK0eM6PIFSUREuimToP9r/jRsA7AFqI62q4HNKe13RbNvZgAtbUM8IiLS+2IN3ZjZxcCNwOdS\nmlcDG81sGXAIuCNq3wrcDBwkOUNnadaqFRGRjMUKend/Fyjp0NZEchZOx74OrMhKdSJS0M6cOUMi\nkeD06dP5LqWgFRcXU1ZWRlFRUbfun+msGxGR2BKJBEOGDKG8vByzdPM0pCvuTlNTE4lEgvHjx3fr\nMbQEgojkzOnTpykpKVHI94CZUVJS0qO/ihT0IpJTCvme6+nvUEEvIhI4jdGLSK8pr3k+q4/XsHpe\n130aGrjlllvYt29fe9uqVasYPHgwDzzwQNr7PPXUU9TV1fGNb3zjnH3XXnstP/rRj7pfdIa1ZoOC\nvh+qqK1gb/XefJfRuVVDU7Zb8ldHL+ks/OKEmPS+XIV8LmnoRkT6rdmzZ7Ny5UquueYaLr/8cl59\n9dX2fYcPH2bu3LlcccUVfOlLX2pvHzx4MACnTp2iqqqKadOmUVFRwebNyc+M/u53v2PevHlMmTKF\nK6+8kg0bNgBQX1/PrFmzmD59OnPmzOHo0aPt7VOmTOGjH/0o3/zmN3NynAp6EenXWltb2blzJ1//\n+tfPCvSdO3eyfv16du/ezfe///1z1uMqLi7mmWeeYdeuXWzfvp37778fd+eFF15g9OjR7Nmzh337\n9jF37lzOnDnDPffcw6ZNm6ivr+fuu+/mkUceAWDp0qU88cQT/PjHP87ZMSroRSRonc1YaWu/7bbb\nAJg+fToNDQ3t+2+88UZKSkq46KKLuO222/jhD3941v3dnYcffpirrrqKG264gbfffptjx45RUVHB\nD37wA1auXMmrr77K0KFDeeONN9i3bx833ngjU6dO5ctf/jKJRIKWlhZOnjzJrFmzAFi8eHEOfgMa\noxeRwJWUlHDixImz2pqbm9s/fHThhRcCMGDAAFpbW9v7dHyB6Hh7/fr1NDY2Ul9fT1FREeXl5Zw+\nfZrLL7+c+vp6tm7dykMPPcQnPvEJbr31ViZPnnzOWfvJkyd7Zfqpzuj7iYrainyXkLRq6J8uBa68\n5vm0F+lbBg8ezKhRo9i2bRuQDPkXXniBmTNnnvd+L774Is3Nzfz+97/n2Wef5brrrjtrf0tLCyNH\njqSoqIjt27fz618nVww+cuQIF198MZ/97Gd54IEH2LVrF1dccQWNjY3tQX/mzBn279/PsGHDGDp0\naPtfC+vXr8/24QM6o+93+kzgdxRA8HdFLwL5m0m0bt06VqxYwf333w/Ao48+ygc/+MHz3mfmzJks\nXryYgwcPcuedd1JZWXnW/kWLFvHJT36SyspKpk6dyoc+9CEA9u7dy4MPPsgFF1xAUVERTz75JIMG\nDWLTpk18/vOfp6WlhdbWVu69914mT57Mt7/9be6++24uvvhi5syZk5PjV9CLSPAmTZrE9u3bz2l/\n+eWX27dLS0vbx+iXLFnCkiVL0j7WqVOn2vunewO1vLw8bWBPnTqVV1555Zz26dOns2fPnvbbq1at\nOs+RdI+Cvp/q83Pp2/SzOfWpNL9eskVB34/0uWGbQIdrNEQjfY3ejBURCZyCXkQkcAp6EZHAKehF\nRAKnN2NFpPdk+w34HM7Emjt3Ljt27GDmzJk899xz7e2/+tWvWLhwIc3NzUybNo2nn36aQYMG5ayO\nbNAZvYhIGg8++CBPP/30Oe0rV67kvvvu48033+SSSy5h7dq1eaguMwp6EQneunXruOqqq5gyZUrs\nhcOqqqoYMmTIWW3uzksvvcTtt98OQHV1Nc8++2zW6822WEM3ZjYM+GfgSsCBu4E3gA1AOdAAfMbd\nT1hyhZ7HgZuBd4El7r4r65WLiMSwf/9+vvKVr/Daa69RWlpKc3Mz69ev56tf/eo5fS+77DI2bdrU\n6WM1NTUxbNgwBg5MRmdZWRlvv/12zmrPlrhj9I8DL7j77WY2CLgYeBjY5u6rzawGqAFWAjcBE6PL\nR4Ano2sRkV7XdgZeWloKwPDhw1m0aBGLFi3K+LHc/Zy2Qvjy8y6D3sw+APwlsATA3f8A/MHM5gOz\no261wMskg34+sM6Tv5EdZjbMzEa5+9GsVy8i0gV3T7vEcHfO6EtLSzl58iStra0MHDiQRCLB6NGj\ns15ztsU5o58ANALfNrMpQD3wBeDStvB296NmNjLqPwY4nHL/RNSmoBeRXldVVcWtt97KfffdR0lJ\nCc3Nzd0+ozczPv7xj7Np0yYWLlxIbW0t8+fPz0HV2RUn6AcC04B73P11M3uc5DBNZ9L9HXPO3ztm\nthxYDjBu3LgYZYhIwcvDwnSTJ0/mkUceYdasWQwYMICrr76ap556qsv7fexjH+MXv/gFp06doqys\njLVr1zJnzhwee+wxFi5cyBe/+EWuvvpqli1blvuD6KE4QZ8AEu7+enR7E8mgP9Y2JGNmo4DjKf3H\npty/DDjS8UHdfQ2wBqCysvLcgS8RkSyprq6muro6o/ukflF4qgkTJrBz585slNVrupxe6e7/Dzhs\nZldETVXAz4EtQNtvrhrYHG1vAe6ypBlAi8bnRUTyJ+6sm3uA9dGMm7eApSRfJDaa2TLgEHBH1Hcr\nyamVB0lOr1ya1YpFRCQjsYLe3XcDlWl2VaXp68CKHtYlWdTZOvS99uUj2frYex6+hKQvri2vLySR\nTOmTsSIigVPQi4gETkEvIhI4LVMsIr0m299bnMv3mAYMGEBFRbLecePGsWXLFkDLFIuIBOOiiy5i\n9+7d7N69uz3kQcsUi4j0Sd1ZpjidoJcpFhEpVN1dpvj06dNUVlYycOBAampqWLBgQfDLFIuIFKTu\nLlN86NAhRo8ezVtvvcX1119PRUUFH/jAB87pF8QyxSIihay7yxS3LT88YcIEZs+ezU9/+lM+/elP\nF+QyxRqjF5GgVVVVsXHjRpqamgDalylue6M19dIW8idOnOC9994D4J133uG1115j0qRJZy1TDAS1\nTLGISFb0ypIbHXRnmeIDBw7wuc99jgsuuID333+fmpoaJk2aBBDsMsUiIgUt02WKr732WvbuTf+i\nVIjLFCvoA5ftD6jEkq1FzHpRX1y8TCRbNEYvIhI4Bb2I5FRy5XLpiZ7+DhX0IpIzxcXFNDU1Kex7\nwN1pamqiuLi424+hMXoRyZmysjISiQSNjY35LqWgFRcXU1ZW1u37K+hFJGeKiooYP358vsvo9zR0\nIyISOAW9iEjgFPQiIoFT0IuIBC7Wm7Fm1gD8Fvgj0OrulWY2HNgAlAMNwGfc/YQll4l7HLgZeBdY\n4u67sl+6ZENFbUVe1h/psdRP365qyV8dfUhnn+5tWD2vlyuRviaTM/qPu/tUd6+MbtcA29x9IrAt\nug1wEzAxuiwHnsxWsSIikrmeDN3MB2qj7VpgQUr7Ok/aAQwzs1E9+DkiItIDcYPegf80s3ozWx61\nXeruRwGi65FR+xjgcMp9E1GbiIjkQdwPTF3n7kfMbCTwopn94jx9032v1jmff45eMJYDjBs3LmYZ\nIiKSqVhn9O5+JLo+DjwDXAMcaxuSia6PR90TwNiUu5cBR9I85hp3r3T3yhEjRnT/CERE5Ly6DHoz\n+zMzG9K2DXwC2AdsAdpW8q8GNkfbW4C7LGkG0NI2xCMiIr0vztDNpcAz0ZfrDgS+4+4vmNlPgI1m\ntgw4BNwR9d9KcmrlQZLTK5dmvWoREYmty6B397eAKWnam4CqNO0OrMhKdSIi0mP6ZKyISOAU9CIi\ngVPQi4gETkEvIhI4Bb2ISOAU9CIigVPQi4gETkEvIhI4Bb2ISOAU9CIigVPQi4gETkEvIhI4Bb2I\nSOAU9CIigYv7VYJSgCpqK3rvh60a2ns/qwfKa57PdwkivU5n9CIigVPQi4gETkEvIhI4jdELFbUV\n7K3em+8yui/1/YFVLfmro4/q7H2JhtXzerkSyRed0YuIBE5BLyISOAW9iEjgYo/Rm9kAoA54291v\nMbPxwPeA4cAuYLG7/8HMLgTWAdOBJuCv3L0h65XLefXKHPo+PHde8+VF/iSTM/ovAAdSbj8GfM3d\nJwIngGVR+zLghLtfBnwt6iciInkSK+jNrAyYB/xzdNuA64FNUZdaYEG0PT+6TbS/KuovIiJ5EPeM\n/uvA/wDej26XACfdvTW6nQDGRNtjgMMA0f6WqL+IiORBl0FvZrcAx929PrU5TVePsS/1cZebWZ2Z\n1TU2NsYqVkREMhfnjP464FNm1kDyzdfrSZ7hDzOztjdzy4Aj0XYCGAsQ7R8KNHd8UHdf4+6V7l45\nYsSIHh2EiIh0rsugd/eH3L3M3cuBhcBL7r4I2A7cHnWrBjZH21ui20T7X3L3c87oRUSkd/RkHv1K\n4O/N7CDJMfi1UftaoCRq/3ugpmcliohIT2S01o27vwy8HG2/BVyTps9p4I4s1CYiIlmgRc0ECGBh\nszbRh7gaiqH89HfyXEzfpsXO+g8tgSAiEjgFvYhI4BT0IiKBU9CLiAROQS8iEjgFvYhI4BT0IiKB\nU9CLiAROQS8iEjh9MjZAOf0awT789YEikp6CXtoFswxCpKH4zvZtLYcQn5ZGCI+GbkREAqegFxEJ\nnIJeRCRwCnoRkcAp6EVEAqegFxEJnIJeRCRwCnoRkcAp6EVEAqegFxEJXJdBb2bFZrbTzPaY2X4z\n+1LUPt7MXjezN81sg5kNitovjG4fjPaX5/YQRETkfOKc0b8HXO/uU4CpwFwzmwE8BnzN3ScCJ4Bl\nUf9lwAl3vwz4WtRPRETypMug96RT0c2i6OLA9cCmqL0WWBBtz49uE+2vMjPLWsUiIpKRWGP0ZjbA\nzHYDx4EXgV8CJ929NeqSAMZE22OAwwDR/hagJJtFi4hIfLGC3t3/6O5TgTLgGuDD6bpF1+nO3r1j\ng5ktN7M6M6trbGyMW6+IiGQoo/Xo3f2kmb0MzACGmdnA6Ky9DDgSdUsAY4GEmQ0EhgLNaR5rDbAG\noLKy8pwXAumenHzpiL5sRKSgxZl1M8LMhkXbFwE3AAeA7cDtUbdqYHO0vSW6TbT/JXdXkIuI5Emc\nM/pRQK2ZDSD5wrDR3Z8zs58D3zOzLwM/BdZG/dcCT5vZQZJn8gtzULeIiMTUZdC7+8+Aq9O0v0Vy\nvL5j+2ngjqxUJyIiPaZPxoqIBE5BLyISOAW9iEjgFPQiIoFT0IuIBE5BLyISOAW9iEjgFPQiIoFT\n0IuIBE5BLyISuIxWr5TwVdRWsLd6b77LyLqG4jvbt8tPfyePlRSu8prnO93XsHpeL1YimdIZvYhI\n4BT0IiKBU9CLiAROQS8iEjgFvYhI4DTrRs4R6sybNpqBk32dzcjRbJy+QWf0IiKB0xl9QCpqK7L3\nYKuGZu+xRCSvdEYvIhI4Bb2ISOAU9CIigesy6M1srJltN7MDZrbfzL4QtQ83sxfN7M3o+pKo3czs\nCTM7aGY/M7NpuT4IERHpXJwz+lbgfnf/MDADWGFmk4AaYJu7TwS2RbcBbgImRpflwJNZr1pERGLr\nMujd/ai774q2fwscAMYA84HaqFstsCDang+s86QdwDAzG5X1ykVEJJaMxujNrBy4GngduNTdj0Ly\nxQAYGXUbAxxOuVsiahMRkTyIHfRmNhj4F+Bed//N+bqmafM0j7fczOrMrK6xsTFuGSIikqFYQW9m\nRSRDfr27/2vUfKxtSCa6Ph61J4CxKXcvA450fEx3X+Pule5eOWLEiO7WLyIiXYgz68aAtcABd//H\nlF1bgOpouxrYnNJ+VzT7ZgbQ0jbEIyIivS/OEgjXAYuBvWa2O2p7GFgNbDSzZcAh4I5o31bgZuAg\n8C6wNKsVi4hIRroMenf/IenH3QGq0vR3YEUP6xIRkSzRJ2NFRAKnoBcRCZyCXkQkcAp6EZHA6YtH\nApDVLxwRkeDojF5EJHAKehGRwCnoRUQCp6AXEQmc3oyVtCrGj2Pvrw7lu4ycayi+s327/PR38lhJ\nmMprnk/b3rB6Xi9X0r/pjF5EJHAKehGRwCnoRUQCp6AXEQmcgl5EJHAKehGRwCnoRUQCp6AXEQmc\ngl5EJHAKehGRwCnoRUQCp6AvcPrSERHpSpeLmpnZt4BbgOPufmXUNhzYAJQDDcBn3P2EmRnwOHAz\n8C6wxN135aZ0ybX+srBZGy1w1nu02FnvinNG/xQwt0NbDbDN3ScC26LbADcBE6PLcuDJ7JQpIiLd\n1WXQu/srQHOH5vlAbbRdCyxIaV/nSTuAYWY2KlvFiohI5ro7Rn+pux8FiK5HRu1jgMMp/RJRm4iI\n5Em234y1NG2etqPZcjOrM7O6xsbGLJfRP+iNWBGJo7tBf6xtSCa6Ph61J4CxKf3KgCPpHsDd17h7\npbtXjhgxoptliIhIV7r7VYJbgGpgdXS9OaX978zse8BHgJa2IR4pTP1t5k0bzcDJD83GyY040yu/\nC8wGSs0sATxKMuA3mtky4BBwR9R9K8mplQdJTq9cmoOaRUQkA10Gvbv/dSe7qtL0dWBFT4sSEZHs\n0SdjRUQCp6AXEQmcgr5AaWqliMSloBcRCZyCXkQkcN2dRy/9SH+dS99Gc+rzT/Pre0Zn9CIigVPQ\ni4gETkEvIhI4Bb2ISOAU9CIigbPk8jT5VVlZ6XV1dfkuo2Dk68NS/XnmTTqagdN39ZfZOGZW7+6V\nXfXTGb2ISOAU9CIigVPQi4gETkEvIhI4Bb2ISOC01o3E1t/XvOlIa+D0XVob52wKesmIwj49hX5h\n6K8vABq6KTD6whERyZSCvoAo5EWkOxT0BUIhLyLdlZOgN7O5ZvaGmR00s5pc/Iz+oLzm+WTArxqa\n71JEpIBl/c1YMxsAfBO4EUgAPzGzLe7+82z/rP6iYvy4fJdwFr0he356Y7bwhP4mbS5m3VwDHHT3\ntwDM7HvAfEBBHxCFfTypoQ8K/kITygtALoJ+DHA45XYC+EgOfg7QO09E6s/oztlax//sbTqGZccz\n9yEfzqTK3pdar0I/ns7+LaTSi0Hf11nudEdvvGjkIugtTds5ayGb2XJgeXTzlJm9keHPKQXe6bSI\nxzJ8tJjOPrhbunGfVPs67NvXWcfzHmtf0PkxZqTPH2eWdHGc8f5dFQg9p13oYVb9RZxOuQj6BDA2\n5XYZcKRjJ3dfA6zp7g8xs7o46zCHoL8cq44zPP3lWPv6ceZi1s1PgIlmNt7MBgELgS05+DkiIhJD\n1s/o3b3VzP4O+A9gAPAtd9+f7Z8jIiLx5GStG3ffCmzNxWOn6PawTwHqL8eq4wxPfznWPn2cfeI7\nY0VEJHe0BIKISOAKJujNbLiZvWhmb0bXl3TS749mtju6FMybwF0tG2FmF5rZhmj/62ZW3vtVZkeM\nY11iZo0pz+Pf5KPOnjCzb5nZcTNLO2fWkp6Ifgc/M7NpvV1jtsQ41tlm1pLyfP7P3q4xG8xsrJlt\nN7MDZrbfzL6Qpk/ffF7dvSAuwP8CaqLtGuCxTvqdynet3Ti2AcAvgQnAIGAPMKlDn/8O/FO0vRDY\nkO+6c3isS4Bv5LvWHh7nXwLTgH2d7L8Z+HeSH0GYAbye75pzeKyzgefyXWcWjnMUMC3aHgL8V5p/\nu33yeS2YM3qSyyjURtu1wII81pJt7ctGuPsfgLZlI1KlHv8moMrMsvQ5pV4V51gLnru/AjSfp8t8\nYJ0n7QCGmdmo3qkuu2IcaxDc/ai774q2fwscILkSQKo++bwWUtBf6u5HIfkLB0Z20q/YzOrMbIeZ\nFcqLQbplIzr+A2rv4+6tQAtQ0ivVZVecYwX4dPSn7yYzG5tmf6GL+3sIxUfNbI+Z/buZTc53MT0V\nDZ1eDbzeYVeffF771FcJmtkPgD9Ps+uRDB5mnLsfMbMJwEtmttfdf5mdCnMmzrIRsZaWKABxjuPf\ngO+6+3tm9rck/5K5PueV9a5Qns84dgF/4e6nzOxm4FlgYp5r6jYzGwz8C3Cvu/+m4+40d8n789qn\ngt7db+hsn5kdM7NR7n40+lPoeCePcSS6fsvMXib5qtvXgz7OshFtfRJmNhAYSmH+udzlsbp7U8rN\n/wvkaOWivIq1VEgIUsPQ3bea2f82s1J3L7g1cMysiGTIr3f3f03TpU8+r4U0dLMFqI62q4HNHTuY\n2SVmdmG0XQpcR2Esjxxn2YjU478deMmjd38KTJfH2mFM81Mkx0JDswW4K5qlMQNoaRuaDI2Z/Xnb\n+0lmdg3J3Gk6/736nugY1gIH3P0fO+nWJ5/XPnVG34XVwEYzWwYcAu4AMLNK4G/d/W+ADwP/x8ze\nJ/mPabUXwBeeeCfLRpjZPwB17r6F5D+wp83sIMkz+YX5q7j7Yh7r583sU0AryWNdkreCu8nMvkty\ntkmpmSWAR4EiAHf/J5KfHL8ZOAi8CyzNT6U9F+NYbwf+m5m1Ar8HFhboScp1wGJgr5ntjtoeBsZB\n335e9clYEZHAFdLQjYiIdIOCXkQkcAp6EZHAKehFRAKnoBcRCZyCXkQkcAp6EZHAKehFRAL3/wEy\nsEE6AnSoDwAAAABJRU5ErkJggg==\n",
121 |       "text/plain": [
122 |        "<matplotlib.figure.Figure at 0x1b5dca00780>"
123 |       ]
124 |      },
125 |      "metadata": {},
126 |      "output_type": "display_data"
127 |     }
128 |    ],
129 |    "source": [
130 |     "NUM_SAMPLES = 10000\n",
131 |     "n = 10\n",
132 |     "c1 = 10\n",
133 |     "c2 = 50\n",
134 |     "\n",
135 |     "mu_hat_ub = np.zeros(NUM_SAMPLES) #unbiased estimate\n",
136 |     "mu_hat_b1 = np.zeros(NUM_SAMPLES) #first biased estimate\n",
137 |     "mu_hat_b2 = np.zeros(NUM_SAMPLES) #second biased estimate\n",
138 |     "for i in range(0,NUM_SAMPLES):\n",
139 |     "    y = np.random.randn(n) + 1#n i.i.d. samples where the mean is 1!\n",
140 |     "    mu_hat_ub[i] = np.mean(y)\n",
141 |     "    mu_hat_b1[i] = n/(n+c1)*np.mean(y)\n",
142 |     "    mu_hat_b2[i] = n/(n+c2)*np.mean(y)\n",
143 |     "\n",
144 |     "plt.hist(mu_hat_ub,50)\n",
145 |     "plt.hist(mu_hat_b1,50)\n",
146 |     "plt.hist(mu_hat_b2,50)\n",
147 |     "plt.gca().legend(['Unbiased','c='+str(c1),'c='+str(c2)])\n",
148 |     "plt.show()"
149 |    ]
150 |   },
151 |   {
152 |    "cell_type": "code",
153 |    "execution_count": null,
154 |    "metadata": {
155 |     "collapsed": true
156 |    },
157 |    "outputs": [],
158 |    "source": []
159 |   }
160 |  ],
161 |  "metadata": {
162 |   "kernelspec": {
163 |    "display_name": "Python 3",
164 |    "language": "python",
165 |    "name": "python3"
166 |   },
167 |   "language_info": {
168 |    "codemirror_mode": {
169 |     "name": "ipython",
170 |     "version": 3
171 |    },
172 |    "file_extension": ".py",
173 |    "mimetype": "text/x-python",
174 |    "name": "python",
175 |    "nbconvert_exporter": "python",
176 |    "pygments_lexer": "ipython3",
177 |    "version": "3.6.3"
178 |   }
179 |  },
180 |  "nbformat": 4,
181 |  "nbformat_minor": 2
182 | }
183 | 


--------------------------------------------------------------------------------
/ch.4/The Cramer-Rao Bound and Asymptotic Efficiency.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# The Cramer-Rao Bound and Asymptotic Efficiency\n",
  8 |     "\n",
  9 |     "Here we'll see how the ML estimator is asymptotically efficient, even if it is biased. We'll do this with the exponential distribution, looping over many instances of its ML estimator, which is not biased and does not achieve the CRLB for any finite n."
 10 |    ]
 11 |   },
 12 |   {
 13 |    "cell_type": "code",
 14 |    "execution_count": 21,
 15 |    "metadata": {
 16 |     "collapsed": false
 17 |    },
 18 |    "outputs": [
 19 |     {
 20 |      "data": {
 21 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAFHBJREFUeJzt3X+MXeV95/H3t+Aw1E0NGBs5HmfH\nJRZN0oAhE3AXpfLidpcfEXaksIJuEkNcWZFoCm0j6vafstKKuNKqgSgIycEpTjaYWDQEy6BskYMF\nkdZejxPHODgVhjr4xgZPHHChiQNevvvHnHGm5pq54znXZ+aZ90sa3XOe89x7v0e2PvPMc59zbmQm\nkqRy/UbTBUiSusugl6TCGfSSVDiDXpIKZ9BLUuEMekkqnEEvSYUz6CWpcAa9JBXuzKYLADj//POz\nr6+v6TIkaVLZsWPHzzJz1mj9JkTQ9/X1MTAw0HQZkjSpRMRPOunn1I0kFc6gl6TCGfSSVLgJMUcv\nSePx5ptv0mq1OHr0aNOldEVPTw+9vb1MmzbtlJ5v0Eua9FqtFu9+97vp6+sjIpoup1aZyeHDh2m1\nWsyfP/+UXsOpG0mT3tGjR5k5c2ZxIQ8QEcycOXNcf60Y9JKKUGLIDxvvuRn0klQ45+glFadv1WO1\nvt6+1deN2ici+OQnP8nXv/51AI4dO8acOXO44oor2LRpEw888AADAwN8+ctfrrW2Thj0k9zJ/kN3\n8h9TUn2mT5/O7t27+eUvf8nZZ5/NE088wdy5c5suC3DqRpJqc8011/DYY0ODr/Xr13PTTTc1XNEQ\ng16SanLjjTfy0EMPcfToUXbt2sUVV1zRdElAh0EfEedExMMR8eOI2BMRvx8R50XEExHxXPV4btU3\nIuJLEbE3InZFxGXdPQVJmhguvvhi9u3bx/r167n22mubLue4Tkf09wDfyczfBS4B9gCrgM2ZuQDY\nXO0DXAMsqH5WAvfVWrEkTWDXX389n//85yfMtA108GFsRPw28AfAzQCZ+QbwRkQsBRZX3dYBW4C/\nApYCX8vMBLZWfw3MycyDtVcvSRPMZz7zGWbMmMGHPvQhtmzZ0nQ5QGerbn4HGAT+ISIuAXYAtwEX\nDId3Zh6MiNlV/7nA/hHPb1VtBr2k06LJVWe9vb3cdtttbY898MADfPvb3z6+v3XrVnp7e7teUydT\nN2cClwH3ZealwL/x62madtpdwpVv6xSxMiIGImJgcHCwo2IlaaJ6/fXX39a2ePFiNm3aBMDNN9/M\n66+/TqvVOv5zOkIeOgv6FtDKzG3V/sMMBf/LETEHoHo8NKL/vBHP7wUOnPiimbkmM/szs3/WrFG/\nCUuSdIpGDfrMfAnYHxEXVU1LgGeBjcDyqm058Gi1vRH4dLX6ZhFwxPl5SWpOp1fGfg74RkS8C3gB\nuIWhXxIbImIF8CJwQ9X3ceBaYC/wi6qvalD3Zd2SpoaOgj4zdwL9bQ4tadM3gVvHWZckqSZeGStJ\nhTPoJalw3r1SUnnunFHz6x3pqNtLL73E7bffzvbt2znrrLPo6+vj7rvv5pJLLuGiiy7ijTfeoL+/\nn7Vr1zJt2jS2bNnC0qVLmT9/Pm+99RazZ8/mwQcfZPbs2aO/2Rg4opekGmQmH//4x1m8eDHPP/88\nzz77LHfddRcvv/wyF154ITt37uSZZ56h1WqxYcOG48/76Ec/ys6dO9m1axcf+chHuPfee2uvzaCX\npBo8+eSTTJs2jc9+9rPH2xYuXMi8eb++rOiMM87g8ssv56c//enbnp+ZvPbaa5x77rm11+bUjSTV\nYPfu3Xz4wx9+xz5Hjx5l27Zt3HPPPcfbnn76aRYuXMjhw4eZPn06d911V+21OaKXpC57/vnnWbhw\nITNnzuS9730vF1988fFjw1M3+/fv55ZbbuGOO+6o/f0NekmqwQc/+EF27NjR9tjwHP3evXvZunUr\nGzdubNvv+uuv56mnnqq9NoO+UH2rHnvbj6Tuueqqq/jVr37FV77yleNt27dv5yc/+cnx/Tlz5rB6\n9Wq+8IUvtH2N733ve1x44YW11+YcvaTydLgcsk4RwSOPPMLtt9/O6tWr6enpOb68cqRly5Zx5513\n8vTTTwO/nqPPTGbMmMH9999fe20GvSTV5D3vec+/Wzo5bPfu3ce3I4If/vCHx/ePHOn+LyWnbiSp\ncAa9JBXOoJdUhKEb55ZpvOdm0Eua9Hp6ejh8+HCRYZ+ZHD58mJ6enlN+DT+MlTTp9fb20mq1KPX7\np3t6esb1/bIGvaRJb9q0acyfP7/pMiYsp24kqXAGvSQVzqCXpMIZ9JJUOINekgpn0EtS4VxeOUF5\nW2FJdXFEL0mF6yjoI2JfRDwTETsjYqBqOy8inoiI56rHc6v2iIgvRcTeiNgVEZd18wQkSe9sLCP6\n/5SZCzOzv9pfBWzOzAXA5mof4BpgQfWzErivrmIlSWM3nqmbpcC6ansdsGxE+9dyyFbgnIiYM473\nkSSNQ6dBn8A/RcSOiFhZtV2QmQcBqsfZVftcYP+I57aqtn8nIlZGxEBEDJR6IyJJmgg6XXVzZWYe\niIjZwBMR8eN36Btt2t5279DMXAOsAejv7y/v3qKSNEF0NKLPzAPV4yHgEeBy4OXhKZnq8VDVvQXM\nG/H0XuBAXQVLksZm1KCPiOkR8e7hbeA/A7uBjcDyqtty4NFqeyPw6Wr1zSLgyPAUjyTp9Otk6uYC\n4JGIGO7/YGZ+JyK2AxsiYgXwInBD1f9x4FpgL/AL4Jbaq5YkdWzUoM/MF4BL2rQfBpa0aU/g1lqq\nkySNm1fGSlLhvNfNFNLu/jn7Vl/XQCWSTidH9JJUOINekgpn0EtS4Qx6SSqcQS9JhTPoJalwBr0k\nFc6gl6TCGfSSVDiDXpIKZ9BLUuEMekkqnEEvSYUz6CWpcAa9JBXOoJekwhn0klQ4g16SCmfQS1Lh\nDHpJKpxfDj7FtfvCcPBLw6WSOKKXpMJ1HPQRcUZE/CAiNlX78yNiW0Q8FxHfjIh3Ve1nVft7q+N9\n3SldktSJsYzobwP2jNj/O+CLmbkAeAVYUbWvAF7JzPcBX6z6SZIa0lHQR0QvcB1wf7UfwFXAw1WX\ndcCyantptU91fEnVX5LUgE5H9HcDdwBvVfszgVcz81i13wLmVttzgf0A1fEjVX9JUgNGDfqI+Bhw\nKDN3jGxu0zU7ODbydVdGxEBEDAwODnZUrCRp7DoZ0V8JXB8R+4CHGJqyuRs4JyKGl2f2Ageq7RYw\nD6A6PgP4+YkvmplrMrM/M/tnzZo1rpOQJJ3cqEGfmX+dmb2Z2QfcCHw3M/8b8CTwiarbcuDRantj\ntU91/LuZ+bYRvSTp9BjPOvq/Av4iIvYyNAe/tmpfC8ys2v8CWDW+EiVJ4zGmK2Mzcwuwpdp+Abi8\nTZ+jwA011CZJqoFXxkpS4Qx6SSqcQS9JhTPoJalwBr0kFc6gl6TCGfSSVDiDXpIKZ9BLUuEMekkq\nnEEvSYUz6CWpcAa9JBXOoJekwhn0klQ4g16SCmfQS1LhDHpJKpxBL0mFM+glqXBj+nJw1a9v1WNN\nlyCpcI7oJalwBr0kFc6gl6TCjRr0EdETEf83In4YET+KiP9etc+PiG0R8VxEfDMi3lW1n1Xt762O\n93X3FCRJ76STEf2vgKsy8xJgIXB1RCwC/g74YmYuAF4BVlT9VwCvZOb7gC9W/dQF+3r+mH09f9x0\nGZImuFGDPoe8Xu1Oq34SuAp4uGpfByyrtpdW+1THl0RE1FaxJGlMOlpeGRFnADuA9wH3As8Dr2bm\nsapLC5hbbc8F9gNk5rGIOALMBH5WY90aYeSovu/ogw1WImki6ujD2Mz8f5m5EOgFLgfe365b9dhu\n9J4nNkTEyogYiIiBwcHBTuuVJI3RmFbdZOarwBZgEXBORAz/RdALHKi2W8A8gOr4DODnbV5rTWb2\nZ2b/rFmzTq16SdKoRp26iYhZwJuZ+WpEnA38IUMfsD4JfAJ4CFgOPFo9ZWO1/3+q49/NzLeN6NV9\nw1M6pzKd0+6K3X2rrxt3TZJOv07m6OcA66p5+t8ANmTmpoh4FngoIv4H8ANgbdV/LfD1iNjL0Ej+\nxi7UrZNwFY6kE40a9Jm5C7i0TfsLDM3Xn9h+FLihluokSePmlbGSVDjvXjkFuPxSmtoc0UtS4Qx6\nSSqcQS9JhTPopxhvhCZNPQa9JBXOoJekwrm8cpJx2kXSWBn06pj3v5EmJ6duJKlwjujllbNS4RzR\nS1LhDHpJKpxBL0mFc45+inKZpjR1OKKXpMIZ9JJUOINekgpn0EtS4fwwdpLww1NJp8oRvSQVzhH9\nBOYoXlIdHNFLUuEMekkq3KhBHxHzIuLJiNgTET+KiNuq9vMi4omIeK56PLdqj4j4UkTsjYhdEXFZ\nt09C9fE7ZaXydDKiPwb8ZWa+H1gE3BoRHwBWAZszcwGwudoHuAZYUP2sBO6rvWpJUsdGDfrMPJiZ\n36+2XwP2AHOBpcC6qts6YFm1vRT4Wg7ZCpwTEXNqr1yS1JExzdFHRB9wKbANuCAzD8LQLwNgdtVt\nLrB/xNNaVduJr7UyIgYiYmBwcHDslUuSOtJx0EfEbwH/CNyemf/6Tl3btOXbGjLXZGZ/ZvbPmjWr\n0zIkSWPUUdBHxDSGQv4bmfmtqvnl4SmZ6vFQ1d4C5o14ei9woJ5yJUlj1cmqmwDWAnsy8+9HHNoI\nLK+2lwOPjmj/dLX6ZhFwZHiKR5J0+nVyZeyVwKeAZyJiZ9X2N8BqYENErABeBG6ojj0OXAvsBX4B\n3FJrxYVzaaOkuo0a9Jn5PdrPuwMsadM/gVvHWZckqSZeGStJhTPoJalwBr0kFc6gl6TCeT/6CWKy\nrrbpW/VY2/Z9q687zZVIOhmD/jQ6WShKUjc5dSNJhXNEr7ZGTiX1HX2wwUokjZcjekkqnEEvSYUz\n6DUqv15QmtwMekkqnEEvSYUz6CWpcC6vbJDz3pJOB0f0klQ4g16SCufUjbqi3X19vNGZ1AxH9JJU\nOINekgpn0EtS4Zyjb8BkXVbpHS2lyckRvSQVzqCXpMKNOnUTEV8FPgYcyszfq9rOA74J9AH7gP+a\nma9ERAD3ANcCvwBuzszvd6d0TTYuuZSa0cmI/gHg6hPaVgGbM3MBsLnaB7gGWFD9rATuq6fMAtw5\nw9v9SmrEqEGfmU8BPz+heSmwrtpeBywb0f61HLIVOCci5tRVrCRp7E51jv6CzDwIUD3OrtrnAvtH\n9GtVbZKkhtT9YWy0acu2HSNWRsRARAwMDg7WXIYkadiprqN/OSLmZObBamrmUNXeAuaN6NcLHGj3\nApm5BlgD0N/f3/aXgSYu19RLk8epjug3Asur7eXAoyPaPx1DFgFHhqd4JEnN6GR55XpgMXB+RLSA\nvwVWAxsiYgXwInBD1f1xhpZW7mVoeeUtXahZkjQGowZ9Zt50kkNL2vRN4NbxFiVJqo9XxkpS4byp\nmRrV7mpZ8IpZqU6O6CWpcI7ou+nOGU1XcFoML7V0maU0MTmil6TCGfSSVDiDXpIK5xy9auNtEaSJ\nyaDXhOSXlEj1cepGXeGXrEgTh0EvSYUz6CWpcM7Rd8MUuVBK0uRg0NfFcJc0QTl1I0mFc0SvScMl\nl9KpMejVVV5EJTXPoB8v5+Y7ZuhLzTDoNan5xSXS6Az6U+EoXtIkYtCrSCcb6bfj6F+lM+g75Si+\nVn4rlXT6uI5ekgrniF5qwzX7KklXRvQRcXVE/HNE7I2IVd14D5Vh+HbG3tJY6p7aR/QRcQZwL/BH\nQAvYHhEbM/PZut+ra0bOx995pLk6ppim1tmP5YNbaTLqxtTN5cDezHwBICIeApYCEyfox/LBqh/C\nNqLdCL/pD27Hu5JnLGv+nTpSnSIz633BiE8AV2fmn1T7nwKuyMw/Pdlz+vv7c2Bg4NTecLTRt0Fd\nnJGB79W23TGWX1Tjec3TqemL67rx/hGxIzP7R+3XhaC/AfgvJwT95Zn5uRP6rQRWVrsXAf9cayGj\nOx/42Wl+z9Op5PMr+dyg7PMr+dzg9J/ff8jMWaN16sbUTQuYN2K/FzhwYqfMXAOs6cL7dyQiBjr5\nTThZlXx+JZ8blH1+JZ8bTNzz68aqm+3AgoiYHxHvAm4ENnbhfSRJHah9RJ+ZxyLiT4H/DZwBfDUz\nf1T3+0iSOtOVC6Yy83Hg8W68do0amzY6TUo+v5LPDco+v5LPDSbo+dX+YawkaWLxXjeSVLgpGfSl\n3qIhIuZFxJMRsScifhQRtzVdUzdExBkR8YOI2NR0LXWKiHMi4uGI+HH1b/j7TddUp4j48+r/5e6I\nWB8RPU3XNB4R8dWIOBQRu0e0nRcRT0TEc9XjuU3WOGzKBf2IWzRcA3wAuCkiPtBsVbU5BvxlZr4f\nWATcWtC5jXQbsKfpIrrgHuA7mfm7wCUUdI4RMRf4M6A/M3+PoYUaNzZb1bg9AFx9QtsqYHNmLgA2\nV/uNm3JBz4hbNGTmG8DwLRomvcw8mJnfr7ZfYygo5jZbVb0iohe4Dri/6VrqFBG/DfwBsBYgM9/I\nzFebrap2ZwJnR8SZwG/S5vqaySQznwJ+fkLzUmBdtb0OWHZaizqJqRj0c4H9I/ZbFBaGABHRB1wK\nbGu2ktrdDdwBvNV0ITX7HWAQ+IdqWur+iJjedFF1ycyfAv8TeBE4CBzJzH9qtqquuCAzD8LQwAuY\n3XA9wNQM+mjTVtTSo4j4LeAfgdsz81+brqcuEfEx4FBm7mi6li44E7gMuC8zLwX+jQnyZ38dqrnq\npcB84D3A9Ij4ZLNVTR1TMeg7ukXDZBUR0xgK+W9k5rearqdmVwLXR8Q+hqbcroqI/9VsSbVpAa3M\nHP4L7GGGgr8Ufwj8S2YOZuabwLeA/9hwTd3wckTMAageDzVcDzA1g77YWzRERDA0x7snM/++6Xrq\nlpl/nZm9mdnH0L/bdzOziFFhZr4E7I+Ii6qmJUykW3uP34vAooj4zer/6RIK+rB5hI3A8mp7OfBo\ng7UcN+W+SrDwWzRcCXwKeCYidlZtf1NdqayJ73PAN6oByAvALQ3XU5vM3BYRDwPfZ2h12A+YoFeR\ndioi1gOLgfMjogX8LbAa2BARKxj65XZDcxX+mlfGSlLhpuLUjSRNKQa9JBXOoJekwhn0klQ4g16S\nCmfQS1LhDHpJKpxBL0mF+//lOMJf3mec0wAAAABJRU5ErkJggg==\n",
 22 |       "text/plain": [
 23 |        "<matplotlib.figure.Figure at 0x1f4812d7390>"
 24 |       ]
 25 |      },
 26 |      "metadata": {},
 27 |      "output_type": "display_data"
 28 |     },
 29 |     {
 30 |      "name": "stdout",
 31 |      "output_type": "stream",
 32 |      "text": [
 33 |       "mean: 2.23339653997\n",
 34 |       "variance: 0.63759510832\n",
 35 |       "CRB variance: 0.4\n"
 36 |      ]
 37 |     }
 38 |    ],
 39 |    "source": [
 40 |     "import numpy as np\n",
 41 |     "import matplotlib.pyplot as plt\n",
 42 |     "\n",
 43 |     "NUM_SAMPLES = 5000 #Number of samples to loop over\n",
 44 |     "rate = 2 #Set the rate of the exponential distribution\n",
 45 |     "\n",
 46 |     "n = 10\n",
 47 |     "crb_variance = rate**2/n\n",
 48 |     "rate_hat = np.zeros(NUM_SAMPLES)\n",
 49 |     "rate_hat_crb = np.zeros(NUM_SAMPLES)\n",
 50 |     "for i in range(0,NUM_SAMPLES):\n",
 51 |     "    y = np.random.exponential(1/rate,n)\n",
 52 |     "    rate_hat_crb[i] = rate + np.sqrt(crb_variance)*np.random.randn() #This is what an efficient estimator would look like\n",
 53 |     "    rate_hat[i] = 1/np.mean(y) #The ML estimate is the reciprocal of the sample mean\n",
 54 |     "    \n",
 55 |     "plt.hist(rate_hat,50)\n",
 56 |     "plt.hist(rate_hat_crb,50)\n",
 57 |     "plt.gca().legend(['ML','CRB'])\n",
 58 |     "plt.show()\n",
 59 |     "\n",
 60 |     "print('mean: '+str(np.mean(rate_hat)))\n",
 61 |     "print('variance: '+str(np.var(rate_hat)))\n",
 62 |     "print('CRB variance: '+str(crb_variance))"
 63 |    ]
 64 |   },
 65 |   {
 66 |    "cell_type": "markdown",
 67 |    "metadata": {},
 68 |    "source": [
 69 |     "It's straightforward to see that the distribution of the ML estimator is not that predicted by the CRB: It is biased, and its variance is not as predicted. Let's try a larger sample size!"
 70 |    ]
 71 |   },
 72 |   {
 73 |    "cell_type": "code",
 74 |    "execution_count": 22,
 75 |    "metadata": {
 76 |     "collapsed": false
 77 |    },
 78 |    "outputs": [
 79 |     {
 80 |      "data": {
 81 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAEc5JREFUeJzt3X+MXWWdx/H31zIybMWCQ2tqp810\nkRDXFQqOLRuiacBd+WHamkgCu7iAbBoTNDSuYdF/xGSD9R8BIzHB1qWy0kpQpAGyuwRogD+KbbWU\nQt1QSLFXfhQHqLA6YOW7f8xpHae3zJ2Zc3tnnnm/ksnc85zn3vs9hH7uM8957jmRmUiSyvWuThcg\nSWovg16SCmfQS1LhDHpJKpxBL0mFM+glqXAGvSQVzqCXpMIZ9JJUuGM6XQDASSedlH19fZ0uQ5Km\nlG3btv02M2eP1m9SBH1fXx9bt27tdBmSNKVExHOt9HPqRpIKZ9BLUuEMekkq3KSYo5ekifjjH/9I\no9FgcHCw06W0RXd3N729vXR1dY3r+Qa9pCmv0Whw/PHH09fXR0R0upxaZSYDAwM0Gg0WLlw4rtdw\n6kbSlDc4OEhPT09xIQ8QEfT09EzorxWDXlIRSgz5gyZ6bAa9JBXOOXpJxem79t5aX2/P6gtH7RMR\nXHrppdx2220AHDhwgLlz57JkyRLuuecebr31VrZu3cp3v/vdWmtrhUE/TR3pH0Ir/0NLOtzMmTPZ\nuXMnf/jDHzjuuOO4//77mTdvXqfLApy6kaTanH/++dx779Agav369VxyySUdrmiIQS9JNbn44ovZ\nsGEDg4OD7NixgyVLlnS6JMCgl6TanHbaaezZs4f169dzwQUXdLqcQ5yjl6QaLVu2jK985Sts2rSJ\ngYGBTpcDGPSagGYndD2Zq+nu85//PLNmzeIjH/kImzZt6nQ5gEEvqUCdHHD09vZy9dVXN9136623\n8rOf/ezQ9ubNm+nt7W17TQa9JNXgjTfeOKxt6dKlLF26FIDLL7+cyy+//OgWVTHo9Rfq/qKJpM5z\n1Y0kFc6gl6TCGfSSVDiDXpIKZ9BLUuFcdSOpPNfNqvn19rfU7cUXX2TVqlVs2bKFY489lr6+Pm68\n8UZOP/10Tj31VN566y36+/tZu3YtXV1dbNq0ieXLl7Nw4ULefvtt5syZw+23386cOXNqLd8RvSTV\nIDP5zGc+w9KlS3nmmWd46qmnuP7663nppZc4+eST2b59O0888QSNRoM77rjj0PM+/vGPs337dnbs\n2MHHPvYxbr755tprM+glqQYPPfQQXV1dfOELXzjUtmjRIubPn39oe8aMGSxevJjf/OY3hz0/M3n9\n9dc58cQTa6/NqRtJqsHOnTv56Ec/+o59BgcHeeyxx7jpppsOtT3yyCMsWrSIgYEBZs6cyfXXX197\nbS2P6CNiRkT8MiLuqbYXRsRjEfF0RPw4It5dtR9bbe+u9vfVXrUkTSHPPPMMixYtoqenhwULFnDa\naacd2ndw6mbv3r1cccUVXHPNNbW//1imbq4Gdg3b/hZwQ2aeArwKXFm1Xwm8mpkfBG6o+klS0T78\n4Q+zbdu2pvsOztHv3r2bzZs3s3Hjxqb9li1bxsMPP1x7bS0FfUT0AhcCa6rtAM4B7qy6rANWVI+X\nV9tU+8+t+ktSsc455xzefPNNvv/97x9q27JlC88999yh7blz57J69Wq++c1vNn2NRx99lJNPPrn2\n2lqdo78RuAY4vtruAV7LzAPVdgM4eBfcecBegMw8EBH7q/6/Hf6CEbESWAmwYMGC8dYvSYdrcTlk\nnSKCu+66i1WrVrF69Wq6u7sPLa8cbsWKFVx33XU88sgjwJ/n6DOTWbNmsWbNmtprGzXoI+LTwL7M\n3BYRSw82N+maLez7c0PmLcAtAP39/Yftl6Sp5gMf+MBfLJ08aOfOnYceRwSPP/74oe39+9v/odTK\niP5sYFlEXAB0A+9laIR/QkQcU43qe4Hnq/4NYD7QiIhjgFnAK7VXLklqyahz9Jn51czszcw+4GLg\nwcz8J+Ah4LNVt8uAu6vHG6ttqv0PZqYjdknqkIl8YerfgC9HxG6G5uDXVu1rgZ6q/cvAtRMrUZJG\nV/J4cqLHNqYvTGXmJmBT9fhZYHGTPoPARROqSpLGoLu7m4GBAXp6eihtkV9mMjAwQHd397hfw2/G\nSpryent7aTQavPzyy50upS26u7sndBNxg17SlNfV1cXChQs7Xcak5UXNJKlwBr0kFc6gl6TCGfSS\nVDiDXpIK56qbaaDv2ns7XYKkDnJEL0mFc0SvWh3pr4c9qy88ypVIOsgRvSQVzqCXpMIZ9JJUOINe\nkgpn0EtS4Qx6SSqcQS9JhTPoJalwBr0kFc6gl6TCGfSSVDiDXpIKZ9BLUuEMekkqnEEvSYUz6CWp\ncAa9JBXOoJekwhn0klQ47xmro6LZvWS9j6x0dDiil6TCGfSSVDiDXpIKZ9BLUuE8GVuQZic8JckR\nvSQVzqCXpMKNGvQR0R0RP4+IxyPiyYj4RtW+MCIei4inI+LHEfHuqv3Yant3tb+vvYcgSXonrYzo\n3wTOyczTgUXAeRFxFvAt4IbMPAV4Fbiy6n8l8GpmfhC4oeonSeqQUYM+h7xRbXZVPwmcA9xZta8D\nVlSPl1fbVPvPjYiorWJJ0pi0NEcfETMiYjuwD7gfeAZ4LTMPVF0awLzq8TxgL0C1fz/QU2fRkqTW\ntbS8MjP/BCyKiBOAu4APNetW/W42es+RDRGxElgJsGDBgpaK1dGzp/sfDz3uG7y9g5VImqgxrbrJ\nzNeATcBZwAkRcfCDohd4vnrcAOYDVPtnAa80ea1bMrM/M/tnz549vuolSaNqZdXN7GokT0QcB3wS\n2AU8BHy26nYZcHf1eGO1TbX/wcw8bEQvSTo6Wpm6mQusi4gZDH0w3JGZ90TEU8CGiPh34JfA2qr/\nWuC2iNjN0Ej+4jbULUlq0ahBn5k7gDOatD8LLG7SPghcVEt1kqQJ81o3GtXwE7PgyVlpqvESCJJU\nOINekgrn1I1q5fp7afJxRC9JhXNEL+DwE66SymHQa0L8gJAmP6duJKlwBr0kFc6gl6TCGfSSVDhP\nxmrMPAErTS2O6CWpcAa9JBXOoJekwjlHr47pu/bepu17Vl94lCuRyuaIXpIKZ9BLUuEMekkqnEEv\nSYUz6CWpcAa9JBXO5ZVT1JGWJkrSSAa92mbkNXG8h6zUGQa9jhqDX+oM5+glqXAGvSQVzqmbacSp\nE2l6ckQvSYUz6CWpcE7dFGy0W/55S0BpenBEL0mFM+glqXAGvSQVzqCXpMJ5MlaTTrMLtnkfWWn8\nHNFLUuFGDfqImB8RD0XEroh4MiKurtrfFxH3R8TT1e8Tq/aIiO9ExO6I2BERZ7b7ICRJR9bKiP4A\n8K+Z+SHgLOCqiPgb4Frggcw8BXig2gY4Hzil+lkJfK/2qiVJLRs16DPzhcz8RfX4dWAXMA9YDqyr\nuq0DVlSPlwM/zCGbgRMiYm7tlUuSWjKmOfqI6APOAB4D3p+ZL8DQhwEwp+o2D9g77GmNqk2S1AEt\nB31EvAf4CbAqM3/3Tl2btGWT11sZEVsjYuvLL7/cahmSpDFqKegjoouhkP9RZv60an7p4JRM9Xtf\n1d4A5g97ei/w/MjXzMxbMrM/M/tnz5493volSaNoZdVNAGuBXZn57WG7NgKXVY8vA+4e1v7P1eqb\ns4D9B6d4JElHXytfmDob+BzwRERsr9q+BqwG7oiIK4FfAxdV++4DLgB2A78Hrqi1YknSmIwa9Jn5\nKM3n3QHObdI/gasmWJemGe9+JbWP34yVpMIZ9JJUOC9qVhjvGiVpJINeU0KzK1qCV7WUWuHUjSQV\nzqCXpMIZ9JJUOINekgpn0EtS4Qx6SSqcQS9JhTPoJalwBr0kFc5vxk5yR/pGqCS1yqBXx3hdHuno\ncOpGkgpn0EtS4Zy6meKc/pA0Gkf0klQ4R/RT0HQYxQ8/Ru8fK02MI3pJKpxBL0mFc+pGU5q3GJRG\n54hekgpn0EtS4Qx6SSqcQS9JhfNkrCa9kd8bcF29NDaO6CWpcAa9JBXOoJekwhn0klQ4g16SCmfQ\nS1LhXF6pKcdLGEtj44hekgpn0EtS4Zy6UZGaXb7YSxdruhp1RB8RP4iIfRGxc1jb+yLi/oh4uvp9\nYtUeEfGdiNgdETsi4sx2Fi9JGl0rUze3AueNaLsWeCAzTwEeqLYBzgdOqX5WAt+rp0xJ0niNGvSZ\n+TDwyojm5cC66vE6YMWw9h/mkM3ACRExt65iJUljN945+vdn5gsAmflCRMyp2ucBe4f1a1RtL4x8\ngYhYydConwULFoyzjOlh5NUbJWks6l51E03aslnHzLwlM/szs3/27Nk1lyFJOmi8I/qXImJuNZqf\nC+yr2hvA/GH9eoHnJ1KgVBdvJK7parxBvxG4DFhd/b57WPsXI2IDsATYf3CKR2N03SwA9nR3uI5J\nzpuSSKMbNegjYj2wFDgpIhrA1xkK+Dsi4krg18BFVff7gAuA3cDvgSvaULMkaQxGDfrMvOQIu85t\n0jeBqyZalCSpPl4CQZIKZ9BLUuEMekkqnBc1myyqVTaSVDdH9JJUOINekgrn1I2K4m0GpcM5opek\nwjmi17Tn3ahUOoNeasILoKkkTt1IUuEc0XeSa+clHQWO6CWpcAa9JBXOoJekwjlHr2J59ylpiEF/\nNHnyVVIHOHUjSYUz6CWpcAa9JBXOOXpNG56c1XRl0EtjcKRr4DTjdXE0WTh1I0mFc0SvacublGi6\ncEQvSYVzRN9OfkFK0iTgiF6SCueIXsKllyqbQV83p2skTTJO3UhS4Qx6SSqcUzdSE66xV0kMeqlN\nml0uwcsiqBMM+ony5KukSc6gHw/DfVqpc+mlF0VTJxj00iR1pA8FPwA0Vq66kaTCtWVEHxHnATcB\nM4A1mbm6He/TVk7P6Agm44ocR/96J7UHfUTMAG4G/h5oAFsiYmNmPlX3e9XKYFcNjsaHwFjm+dvF\nFUVTSztG9IuB3Zn5LEBEbACWA5M76KVxGHmittV9I02GD4VmQd2uDxX/Ajm62hH084C9w7YbwJI2\nvM+QkSPx6/YfeZ+kI5poqBve76yT/30iM+t9wYiLgE9l5r9U258DFmfml0b0WwmsrDZPBf631kI6\n5yTgt50uos1KP8bSjw/KP8bSjw+GjnFmZs4erWM7RvQNYP6w7V7g+ZGdMvMW4JY2vH9HRcTWzOzv\ndB3tVPoxln58UP4xln58cOgY+1rp247llVuAUyJiYUS8G7gY2NiG95EktaD2EX1mHoiILwL/zdDy\nyh9k5pN1v48kqTVtWUefmfcB97XjtaeA4qajmij9GEs/Pij/GEs/PhjDMdZ+MlaSNLl4CQRJKpxB\nX5OI+EFE7IuInZ2upR0iYn5EPBQRuyLiyYi4utM11S0iuiPi5xHxeHWM3+h0Te0QETMi4pcRcU+n\na2mHiNgTEU9ExPaI2NrpeuoWESdExJ0R8avq3+Pfjfocp27qERGfAN4AfpiZf9vpeuoWEXOBuZn5\ni4g4HtgGrJj0l7YYg4gIhtYlvxERXcCjwNWZubnDpdUqIr4M9APvzcxPd7qeukXEHqA/M4tcRx8R\n64BHMnNNtbLxrzLztXd6jiP6mmTmw8Arna6jXTLzhcz8RfX4dWAXQ9+CLkYOeaPa7Kp+ihoJRUQv\ncCGwptO1aOwi4r3AJ4C1AJn51mghDwa9xiEi+oAzgMc6W0n9qmmN7cA+4P7MLO0YbwSuAd7udCFt\nlMD/RMS26hv4Jflr4GXgP6rptzURMXO0Jxn0GpOIeA/wE2BVZv6u0/XULTP/lJmLGPpG9+KIKGYa\nLiI+DezLzG2drqXNzs7MM4HzgauqadVSHAOcCXwvM88A/g+4drQnGfRqWTVv/RPgR5n5007X007V\nn8ObgPM6XEqdzgaWVXPYG4BzIuI/O1tS/TLz+er3PuAuhq6oW4oG0Bj2l+adDAX/OzLo1ZLqROVa\nYFdmfrvT9bRDRMyOiBOqx8cBnwR+1dmq6pOZX83M3ur6KBcDD2bmpR0uq1YRMbNaLEA1pfEPQDEr\n4TLzRWBvRJxaNZ1LC5eA956xNYmI9cBS4KSIaABfz8y1na2qVmcDnwOeqOawAb5WfQu6FHOBddXN\nc94F3JGZRS5BLNj7gbuGxiUcA9yemf/V2ZJq9yXgR9WKm2eBK0Z7gssrJalwTt1IUuEMekkqnEEv\nSYUz6CWpcAa9JBXOoJekwhn0klQ4g16SCvf/Aj20ujcz3oEAAAAASUVORK5CYII=\n",
 82 |       "text/plain": [
 83 |        "<matplotlib.figure.Figure at 0x1f4815a5e80>"
 84 |       ]
 85 |      },
 86 |      "metadata": {},
 87 |      "output_type": "display_data"
 88 |     },
 89 |     {
 90 |      "name": "stdout",
 91 |      "output_type": "stream",
 92 |      "text": [
 93 |       "mean: 2.09662727849\n",
 94 |       "variance: 0.246962915703\n",
 95 |       "CRB variance: 0.2\n"
 96 |      ]
 97 |     }
 98 |    ],
 99 |    "source": [
100 |     "n = 20\n",
101 |     "crb_variance = rate**2/n\n",
102 |     "rate_hat = np.zeros(NUM_SAMPLES)\n",
103 |     "rate_hat_crb = np.zeros(NUM_SAMPLES)\n",
104 |     "for i in range(0,NUM_SAMPLES):\n",
105 |     "    y = np.random.exponential(1/rate,n)\n",
106 |     "    rate_hat_crb[i] = rate + np.sqrt(crb_variance)*np.random.randn() #This is what an efficient estimator would look like\n",
107 |     "    rate_hat[i] = 1/np.mean(y) #The ML estimate is the reciprocal of the sample mean\n",
108 |     "    \n",
109 |     "plt.hist(rate_hat,50)\n",
110 |     "plt.hist(rate_hat_crb,50)\n",
111 |     "plt.gca().legend(['ML','CRB'])\n",
112 |     "plt.show()\n",
113 |     "\n",
114 |     "print('mean: '+str(np.mean(rate_hat)))\n",
115 |     "print('variance: '+str(np.var(rate_hat)))\n",
116 |     "print('CRB variance: '+str(crb_variance))"
117 |    ]
118 |   },
119 |   {
120 |    "cell_type": "markdown",
121 |    "metadata": {},
122 |    "source": [
123 |     "Sure enough, the mean is closer to 2, and the variance is closer to that predicted by the CRB. Let's go even higher!"
124 |    ]
125 |   },
126 |   {
127 |    "cell_type": "code",
128 |    "execution_count": 23,
129 |    "metadata": {
130 |     "collapsed": false
131 |    },
132 |    "outputs": [
133 |     {
134 |      "data": {
135 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAFBVJREFUeJzt3X+Q3HV9x/Hnu3hyDDJBQ2JjLull\ngDqVChHPhI5jJwXbSuwQmNE2OCqgM6mtzpBRR6N/1LMzYjrT+qtSnGAY0JFgpgpkANumSgb4IzEJ\nE0IgtR56moMI8dQIo0GD7/6x30uXZO9uc7t7m/vc8zGzs9/9fD+7+75vvnnd5z773e83MhNJUrl+\nr9sFSJI6y6CXpMIZ9JJUOINekgpn0EtS4Qx6SSqcQS9JhTPoJalwBr0kFe4l3S4A4Jxzzsn+/v5u\nlyFJM8ru3bt/mpnzJut3SgR9f38/u3bt6nYZkjSjRMSPmunn1I0kFc6gl6TCGfSSVLhTYo5eklrx\n29/+lpGREY4cOdLtUjqit7eXvr4+enp6pvR8g17SjDcyMsJZZ51Ff38/EdHtctoqMxkdHWVkZIQl\nS5ZM6TWcupE04x05coS5c+cWF/IAEcHcuXNb+mvFoJdUhBJDfkyrP5tBL0mFc45eUnH6193b1tcb\nXv/WSftEBO985zv56le/CsDRo0dZsGABy5cv55577uHWW29l165dfPGLX2xrbc0w6DXjNPpP3Mx/\nRKmTzjzzTPbt28evf/1rzjjjDLZu3crChQu7XRbg1I0ktc3ll1/OvffWBiKbNm3i6quv7nJFNQa9\nJLXJ6tWrueOOOzhy5Ah79+5l+fLl3S4JMOglqW0uvPBChoeH2bRpEytXrux2Occ4Ry9JbXTFFVfw\n4Q9/mG3btjE6OtrtcgCDXpLa6j3veQ9z5szhta99Ldu2bet2OYBBL6lA3TwKq6+vj+uvv77hultv\nvZW77rrr2OPt27fT19fX8ZoMeklqg+eee+6EthUrVrBixQoArr32Wq699trpLapi0KsMg3PGaT88\nvXVIpyCPupGkwhn0klQ4p240OznVo1nEEb0kFW7SoI+I3oj4bkQ8EhGPRcQnq/YlEbEjIr4fEV+P\niJdW7adXj4eq9f2d/REkSRNpZurmeeDSzHwuInqAhyLiW8AHgc9m5h0R8SXgvcBN1f3PM/O8iFgN\n/BPwNx2qX5JONN7U3JRfr7kpvZ/85CesXbuWnTt3cvrpp9Pf38/nPvc5LrroIl796lfzm9/8hoGB\nATZu3EhPTw/btm1j1apVLFmyhN/97nfMnz+f22+/nfnz57e1/ElH9FkzdoBoT3VL4FLg36v224Ar\nq+VV1WOq9ZdFyZd+kSRq13a96qqrWLFiBU888QSPP/44N9xwA08//TTnnnsue/bs4dFHH2VkZITN\nmzcfe96b3vQm9uzZw969e3nDG97AjTfe2PbamvowNiJOA3YD5wE3Ak8Av8jMo1WXEWDsxMsLgQMA\nmXk0Ig4Dc4GfHveaa4A1AIsXL27tp9DM1mj05YeimmHuv/9+enp6eN/73nesbenSpQwPDx97fNpp\np7Fs2TKefPLJE56fmTz77LOcd955ba+tqaDPzBeApRFxNnAn8EeNulX3jUbveUJD5gZgA8DAwMAJ\n66W2aPef8NI49u3bx+tf//oJ+xw5coQdO3bw+c9//ljbgw8+yNKlSxkdHeXMM8/khhtuaHttJ3XU\nTWb+AtgGXAKcHRFjvyj6gKeq5RFgEUC1fg7ws3YUK3Xc4JzGN6kFTzzxBEuXLmXu3LksXryYCy+8\n8Ni6sambAwcOcN111/GRj3yk7e/fzFE386qRPBFxBvBmYD9wP/C2qts1wN3V8pbqMdX672SmI3ZJ\nRbvgggvYvXt3w3Vjc/RDQ0Ns376dLVu2NOx3xRVX8MADD7S9tmZG9AuA+yNiL7AT2JqZ9wAfBT4Y\nEUPU5uA3Vv03AnOr9g8C69petcrnyFozzKWXXsrzzz/PzTfffKxt586d/OhHPzr2eMGCBaxfv55P\nf/rTDV/joYce4txzz217bZPO0WfmXuB1Ddp/ACxr0H4EeHtbqlNxGl7Yu7cLhahsXfgwPyK48847\nWbt2LevXr6e3t/fY4ZX1rrzySgYHB3nwwQeB/5+jz0zmzJnDl7/85bbX5ikQJKlNXvWqV73o0Mkx\n+/btO7YcETzyyCPHHh8+3PlfSp4CQZIKZ9BLUuEMeklFKPngvlZ/NoNe0ozX29vL6OhokWGfmYyO\njtLbO/WjFvwwVjPL4ByP0tEJ+vr6GBkZ4dChQ90upSN6e3tbuoi4QS9pxuvp6WHJkiXdLuOU5dSN\nJBXOoJekwjl1o45o9A1YSd3hiF6SCmfQS1LhDHpJKpxBL0mFM+glqXAGvSQVzqCXpMIZ9JJUOL8w\nJTVjvOvVduGSddLJckQvSYUz6CWpcAa9JBVu0qCPiEURcX9E7I+IxyLi+qp9MCKejIg91W1l3XM+\nFhFDEfG9iPjLTv4AkqSJNfNh7FHgQ5n5cEScBeyOiK3Vus9m5j/Xd46I1wCrgQuAVwH/HRF/mJkv\ntLNwzTzDve/odgnSrDTpiD4zD2bmw9Xys8B+YOEET1kF3JGZz2fmD4EhYFk7ipUknbyTmqOPiH7g\ndcCOqukDEbE3Im6JiJdXbQuBA3VPG2HiXwySpA5qOugj4mXAN4C1mflL4CbgXGApcBD4l7GuDZ5+\nwqXZI2JNROyKiF2lXtBXkk4FTQV9RPRQC/mvZeY3ATLz6cx8ITN/B9zM/0/PjACL6p7eBzx1/Gtm\n5obMHMjMgXnz5rXyM0iSJtDMUTcBbAT2Z+Zn6toX1HW7CthXLW8BVkfE6RGxBDgf+G77SpYknYxm\njrp5I/Au4NGI2FO1fRy4OiKWUpuWGQb+FiAzH4uIzcDj1I7Yeb9H3EhS90wa9Jn5EI3n3e+b4Dmf\nAj7VQl2SpDbxpGZqv8E5DPd2uwhJYzwFgiQVzqCXpMIZ9JJUOOfo1ZL+dfee0Ob8vHRqcUQvSYUz\n6CWpcAa9JBXOoJekwhn0klQ4g16SCmfQS1LhDHpJKpxfmJJacPwXxobXv7VLlUjjM+ilThicM077\n4emtQ8KpG0kqnkEvSYUz6CWpcAa9JBXOoJekwhn0klQ4g16SCudx9FILhnvf8eKGwa6UIU1o0hF9\nRCyKiPsjYn9EPBYR11ftr4iIrRHx/er+5VV7RMQXImIoIvZGxMWd/iEkSeNrZkR/FPhQZj4cEWcB\nuyNiK3At8O3MXB8R64B1wEeBy4Hzq9ty4KbqXqUZnOP1YaUZYNIRfWYezMyHq+Vngf3AQmAVcFvV\n7Tbgymp5FfCVrNkOnB0RC9peuSSpKSf1YWxE9AOvA3YAr8zMg1D7ZQDMr7otBA7UPW2kapMkdUHT\nH8ZGxMuAbwBrM/OXETFu1wZt2eD11gBrABYvXtxsGeqG8U7QJWlGaGpEHxE91EL+a5n5zar56bEp\nmer+map9BFhU9/Q+4KnjXzMzN2TmQGYOzJs3b6r1S5Im0cxRNwFsBPZn5mfqVm0BrqmWrwHurmt/\nd3X0zSXA4bEpHknS9Gtm6uaNwLuARyNiT9X2cWA9sDki3gv8GHh7te4+YCUwBPwKuK6tFUuSTsqk\nQZ+ZD9F43h3gsgb9E3h/i3VJktrEUyBIUuEMekkqnEEvSYUz6CWpcAa9JBXOoJekwhn0klQ4g16S\nCmfQS1LhDHpJKpxBL0mF8+LgOkH/untf9NjLBUozmyN6SSqcI3ppOjW6Wtfg4emvQ7OKI3pJKpxB\nL0mFM+glqXAGvSQVzqCXpMIZ9JJUOINekgpn0EtS4Qx6SSrcpEEfEbdExDMRsa+ubTAinoyIPdVt\nZd26j0XEUER8LyL+slOFS5Ka08yI/lbgLQ3aP5uZS6vbfQAR8RpgNXBB9Zx/i4jT2lWsJOnkTRr0\nmfkA8LMmX28VcEdmPp+ZPwSGgGUt1CdJalErc/QfiIi91dTOy6u2hcCBuj4jVZskqUumGvQ3AecC\nS4GDwL9U7dGgbzZ6gYhYExG7ImLXoUOHpliGJGkyUzpNcWY+PbYcETcD91QPR4BFdV37gKfGeY0N\nwAaAgYGBhr8MNM2qU+h6oRGpLFMa0UfEgrqHVwFjR+RsAVZHxOkRsQQ4H/huayVKklox6Yg+IjYB\nK4BzImIE+ASwIiKWUpuWGQb+FiAzH4uIzcDjwFHg/Zn5QmdKlyQ1Y9Kgz8yrGzRvnKD/p4BPtVKU\nJKl9/GasJBXOoJekwhn0klQ4g16SCmfQS1LhDHpJKpxBL0mFM+glqXBTOteNpPbpX3fvix4Pr39r\nlypRqRzRS1LhHNFLp5jjR/hjHOlrqgx6qcuGe9/RsL3/yO3TXIlK5dSNJBXOoJekwhn0klQ45+hn\no+qSgZJmB0f0klQ4g16SCmfQS1LhDHpJKpxBL0mFM+glqXAGvSQVbtKgj4hbIuKZiNhX1/aKiNga\nEd+v7l9etUdEfCEihiJib0Rc3MniJUmTa2ZEfyvwluPa1gHfzszzgW9XjwEuB86vbmuAm9pTpiRp\nqiYN+sx8APjZcc2rgNuq5duAK+vav5I124GzI2JBu4qVJJ28qc7RvzIzDwJU9/Or9oXAgbp+I1Xb\nCSJiTUTsiohdhw4dmmIZkqTJtPvD2GjQlo06ZuaGzBzIzIF58+a1uQxJ0pipBv3TY1My1f0zVfsI\nsKiuXx/w1NTLkyS1aqpnr9wCXAOsr+7vrmv/QETcASwHDo9N8UhqTaNLDHp5QTVj0qCPiE3ACuCc\niBgBPkEt4DdHxHuBHwNvr7rfB6wEhoBfAdd1oGZpVnvRpQcH61YMHp7uUjRDTBr0mXn1OKsua9A3\ngfe3WpSk8a8lK50svxkrSYUz6CWpcAa9JBXOoJekwnlx8JJ5EXBJOKKXpOIZ9JJUOINekgrnHL1U\niPpTJHhqBNVzRC9JhTPoJalwBr0kFc6gl6TCGfSSVDiDXpIK5+GVJfBUB2L8C5L0H7m9tt5DLmct\nR/SSVDiDXpIKZ9BLUuEMekkqnEEvSYUz6CWpcC0dXhkRw8CzwAvA0cwciIhXAF8H+oFh4K8z8+et\nlSmpU+rPelnPwzHL0Y7j6P8sM39a93gd8O3MXB8R66rHH23D+8jj5SVNQSe+MLUKWFEt3wZsw6CX\nuubYF6kGj1sxeHi6S1GXtDpHn8B/RcTuiFhTtb0yMw8CVPfzW3wPSVILWh3RvzEzn4qI+cDWiPif\nZp9Y/WJYA7B48eIWy5B0ssabm1d5WhrRZ+ZT1f0zwJ3AMuDpiFgAUN0/M85zN2TmQGYOzJs3r5Uy\nJEkTmHLQR8SZEXHW2DLwF8A+YAtwTdXtGuDuVouUJE1dK1M3rwTujIix17k9M/8jInYCmyPivcCP\ngbe3XqYkaaqmHPSZ+QPgogbto8BlrRQlSWofvxkrSYXzwiOnKr8cJalNDHpplnrRFanqjF2RSuVw\n6kaSCmfQS1LhDHpJKpxz9N3mh646RTU6RYKnLp6ZHNFLUuEMekkqnEEvSYUz6CWpcH4YK6llXnf2\n1GbQS2qaFyuZmQz6dhvvcEmvz6kZwlMjlMegl9QUfwHMXH4YK0mFc0Qv6ZTgN3E7x6CX1DGG96nB\noJc0rTxyZ/oZ9FN1sicj8+RlkrrEoJd0yvKLWO1h0Esqhp8JNGbQS+qI8Y67b6STx+L7V0EHj6OP\niLdExPciYigi1nXqfSRJE+vIiD4iTgNuBP4cGAF2RsSWzHy8E+/XUX6IKk3oZEbu7eKROyenU1M3\ny4ChzPwBQETcAawCuh/0BrekCZQ4z9+poF8IHKh7PAIs78g7GdzSjNeNvwoYbP6zgcn+gmhUf7Ov\nPR2/RDoV9NGgLV/UIWINsKZ6+FxEfK+F9zsH+GkLzy+B26DG7VDjdqiZZDv8VVvepFHgNfva8U8t\nvfUfNNOpU0E/Aiyqe9wHPFXfITM3ABva8WYRsSszB9rxWjOV26DG7VDjdqhxO9R06qibncD5EbEk\nIl4KrAa2dOi9JEkT6MiIPjOPRsQHgP8ETgNuyczHOvFekqSJdewLU5l5H3Bfp17/OG2ZAprh3AY1\nbocat0ON2wGIzJy8lyRpxvIKU5JUuBkT9BFxS0Q8ExH7xlkfEfGF6pQLeyPi4umucTo0sR1WRMTh\niNhT3f5humvstIhYFBH3R8T+iHgsIq5v0Kf4/aHJ7TAb9ofeiPhuRDxSbYdPNuhzekR8vdofdkRE\n//RX2kWZOSNuwJ8CFwP7xlm/EvgWtUNaLwF2dLvmLm2HFcA93a6zw9tgAXBxtXwW8L/Aa2bb/tDk\ndpgN+0MAL6uWe4AdwCXH9fl74EvV8mrg692uezpvM2ZEn5kPAD+boMsq4CtZsx04OyIWTE9106eJ\n7VC8zDyYmQ9Xy88C+6l9G7te8ftDk9uheNW/8XPVw57qdvyHj6uA26rlfwcui4jG33Mq0IwJ+iY0\nOu3CrNvpK39S/Rn7rYi4oNvFdFL1J/jrqI3i6s2q/WGC7QCzYH+IiNMiYg/wDLA1M8fdHzLzKHAY\nmDu9VXZPSUE/6WkXZomHgT/IzIuAfwXu6nI9HRMRLwO+AazNzF8ev7rBU4rcHybZDrNif8jMFzJz\nKbVv4S+LiD8+rsus2R8aKSnoJz3twmyQmb8c+zM2a99l6ImIc7pcVttFRA+1cPtaZn6zQZdZsT9M\nth1my/4wJjN/AWwD3nLcqmP7Q0S8BJjDLJoCLSnotwDvro62uAQ4nJkHu13UdIuI3x+be4yIZdT+\njUe7W1V7VT/fRmB/Zn5mnG7F7w/NbIdZsj/Mi4izq+UzgDcD/3Ncty3ANdXy24DvZPXJ7GwwYy4l\nGBGbqB1BcE5EjACfoPahC5n5JWrfwl0JDAG/Aq7rTqWd1cR2eBvwdxFxFPg1sLrAHfqNwLuAR6t5\nWYCPA4thVu0PzWyH2bA/LABuqy549HvA5sy8JyL+EdiVmVuo/UL8akQMURvJr+5eudPPb8ZKUuFK\nmrqRJDVg0EtS4Qx6SSqcQS9JhTPoJalwBr0kFc6gl6TCGfSSVLj/A+XWjrKWfSmpAAAAAElFTkSu\nQmCC\n",
136 |       "text/plain": [
137 |        "<matplotlib.figure.Figure at 0x1f480ff2a20>"
138 |       ]
139 |      },
140 |      "metadata": {},
141 |      "output_type": "display_data"
142 |     },
143 |     {
144 |      "name": "stdout",
145 |      "output_type": "stream",
146 |      "text": [
147 |       "mean: 2.03346433764\n",
148 |       "variance: 0.0856195027526\n",
149 |       "CRB variance: 0.08\n"
150 |      ]
151 |     }
152 |    ],
153 |    "source": [
154 |     "n = 50\n",
155 |     "crb_variance = rate**2/n\n",
156 |     "rate_hat = np.zeros(NUM_SAMPLES)\n",
157 |     "rate_hat_crb = np.zeros(NUM_SAMPLES)\n",
158 |     "for i in range(0,NUM_SAMPLES):\n",
159 |     "    y = np.random.exponential(1/rate,n)\n",
160 |     "    rate_hat_crb[i] = rate + np.sqrt(crb_variance)*np.random.randn() #This is what an efficient estimator would look like\n",
161 |     "    rate_hat[i] = 1/np.mean(y) #The ML estimate is the reciprocal of the sample mean\n",
162 |     "    \n",
163 |     "plt.hist(rate_hat,50)\n",
164 |     "plt.hist(rate_hat_crb,50)\n",
165 |     "plt.gca().legend(['ML','CRB'])\n",
166 |     "plt.show()\n",
167 |     "\n",
168 |     "print('mean: '+str(np.mean(rate_hat)))\n",
169 |     "print('variance: '+str(np.var(rate_hat)))\n",
170 |     "print('CRB variance: '+str(crb_variance))"
171 |    ]
172 |   },
173 |   {
174 |    "cell_type": "markdown",
175 |    "metadata": {},
176 |    "source": [
177 |     "At this point we're doing quite well. Note that the distribution of the ML estimator is looking more and more Gaussian, a consequence of the central limit theorem. Once more, at an even higher number of samples."
178 |    ]
179 |   },
180 |   {
181 |    "cell_type": "code",
182 |    "execution_count": 24,
183 |    "metadata": {
184 |     "collapsed": false
185 |    },
186 |    "outputs": [
187 |     {
188 |      "data": {
189 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAF3FJREFUeJzt3XGQnGWd4PHvjzAyXKRCSCZczCQ7\nOeS8E4UBx8At51UWdk9Al0DVuhX2VEDqsu5hHVStq+gfGr0Ds1W3q1iybAXCJVASSKloCnHXHJIC\nzkog4UIIoGeAaMZEEgNGOE004Xd/9DuhCT0zPdPd0z1vvp+qrn77eZ+3+zdv3vz66ed93ueNzESS\nVF7HtTsASVJrmeglqeRM9JJUciZ6SSo5E70klZyJXpJKzkQvSSU3aqKPiO6IeCwinoyIpyPiC0X5\nyoh4ISK2FI/+ojwi4qsRsT0itkbEOa3+IyRJwzu+jjoHgQsy89WI6AIejYjvFev+JjO/cVT9i4HT\ni8e5wK3FsySpDUZN9Fm5dPbV4mVX8RjpctpFwJ3Fdhsi4uSImJ2Zu4fbYObMmdnX11d/1JIkNm/e\n/MvM7BmtXj0teiJiCrAZeDtwS2ZujIi/Am6MiM8BDwI3ZOZBYA6ws2rzwaJs91HvuQRYAjBv3jw2\nbdpUTyiSpEJE/LSeenWdjM3Mw5nZD/QCCyLiXcBngH8DvBc4Bfj00GfXeosa77k8Mwcyc6CnZ9Qv\nJEnSOI1p1E1m/gpYD1yUmbuz4iDwP4EFRbVBYG7VZr3AribEKkkah3pG3fRExMnF8onAHwM/iojZ\nRVkAlwHbik3WAh8tRt+cB+wfqX9ektRa9fTRzwZWFf30xwFrMvP+iPhBRPRQ6arZAny8qP8AcAmw\nHfgNcHXzw5ak1/3+979ncHCQAwcOtDuUluju7qa3t5eurq5xbV/PqJutwNk1yi8Ypn4C144rGkka\nh8HBQU466ST6+vqodDKUR2ayb98+BgcHmT9//rjewytjJU16Bw4cYMaMGaVL8gARwYwZMxr6tWKi\nl1QKZUzyQxr920z0klRydV0wJUmTSd8N323q++1Y9oFR60QEH/7wh7nrrrsAOHToELNnz+bcc8/l\n/vvvZ+XKlWzatImvfe1rTY2tHiZ6TQ5Lp9Uo2z/xcUjDmDp1Ktu2beO3v/0tJ554IuvWrWPOnDnt\nDguw60aSmubiiy/mu9+t/JpYvXo1V1xxRZsjqjDRS1KTLF68mHvuuYcDBw6wdetWzj23MybuNdFL\nUpOceeaZ7Nixg9WrV3PJJZe0O5wj7KOXpCa69NJL+eQnP8n69evZt29fu8MBTPSS1FQf+9jHmDZt\nGu9+97tZv359u8MBTPSSSqie4ZCt0tvby3XXXVdz3cqVK/n2t7995PWGDRvo7e1teUwmeklqgldf\nffVNZQsXLmThwoUAXHXVVVx11VUTG1TBRK/SGO4imXa27qRO4KgbSSo5E70klZyJXpJKzj56lV6t\nvnv77XUssUUvSSVni15S+dSa7bSh96tvptRf/OIXXH/99Tz++OOccMIJ9PX18ZWvfIWzzjqLd7zj\nHfzud79jYGCAFStW0NXVxfr161m0aBHz58/ntddeY9asWdx9993MmjWrqeHbotfktXTaGx47uv+i\n3RHpGJaZXH755SxcuJDnnnuOZ555hptuuokXX3yR0047jS1btvDUU08xODjImjVrjmz3vve9jy1b\ntrB161be+973cssttzQ9NhO9JDXBQw89RFdXFx//+MePlPX39zN37twjr6dMmcKCBQv4+c9//qbt\nM5NXXnmF6dOnNz02u24kqQm2bdvGe97znhHrHDhwgI0bN3LzzTcfKXvkkUfo7+9n3759TJ06lZtu\nuqnpsY3aoo+I7oh4LCKejIinI+ILRfn8iNgYET+JiHsj4i1F+QnF6+3F+r6mRy1Jk8hzzz1Hf38/\nM2bMYN68eZx55plH1g113ezcuZOrr76aT33qU03//Hq6bg4CF2TmWUA/cFFEnAf8LfDlzDwdeBm4\npqh/DfByZr4d+HJRT5oQO7r/ouZDarUzzjiDzZs311w31Ee/fft2NmzYwNq1a2vWu/TSS3n44Yeb\nHtuoiT4rhmbr6SoeCVwAfKMoXwVcViwvKl5TrL8wIqJpEUtSB7rgggs4ePAgt91225Gyxx9/nJ/+\n9KdHXs+ePZtly5bxpS99qeZ7PProo5x22mlNj62uPvqImAJsBt4O3AI8B/wqMw8VVQaBobvgzgF2\nAmTmoYjYD8wAfnnUey4BlgDMmzevsb9Ckqq14cbxEcF9993H9ddfz7Jly+ju7j4yvLLaZZddxtKl\nS3nkkUeA1/voM5Np06Zx++23Nz22uhJ9Zh4G+iPiZOA+4N/WqlY812q955sKMpcDywEGBgbetF6S\nJpu3ve1tbxg6OWTbtm1HliOCJ5988sjr/ftb/6U0plE3mfmriFgPnAecHBHHF636XmBXUW0QmAsM\nRsTxwDTgpeaFrGPJ0PQFO7rbHIg0idUz6qanaMkTEScCfww8CzwE/FlR7UrgO8Xy2uI1xfofZKYt\ndklqk3pa9LOBVUU//XHAmsy8PyKeAe6JiP8O/B9gRVF/BXBXRGyn0pJf3IK4pYZ4k5LyyUzKOu6j\n0bbyqIk+M7cCZ9cofx5YUKP8APChhqKSmqzWEMu+A3e3IRK1Qnd3N/v27WPGjBmlS/aZyb59++ju\nHn//pVfGSpr0ent7GRwcZO/eve0OpSW6u7sbuom4iV7SpNfV1cX8+fPbHUbHclIzSSo5E70klZyJ\nXpJKzkQvSSVnopekknPUjTrGcBcxSWqMLXpJKjlb9Oo43ihEai5b9JJUcrbopWpLp9Uom/ibWEjN\nZKJXW3jiVZo4dt1IUsmZ6CWp5Ez0klRy9tHrmOUwTh0rbNFLUsnZoldLObpGaj9b9JJUciZ6SSo5\nE70kldyoiT4i5kbEQxHxbEQ8HRHXFeVLI+LnEbGleFxStc1nImJ7RPw4It7fyj9AkjSyek7GHgL+\nOjOfiIiTgM0Rsa5Y9+XM/B/VlSPincBi4AzgbcD/ioh/nZmHmxm4JKk+o7boM3N3Zj5RLL8CPAvM\nGWGTRcA9mXkwM18AtgMLmhGsJGnsxtRHHxF9wNnAxqLoExGxNSLuiIjpRdkcYGfVZoPU+GKIiCUR\nsSkiNu3du3fMgUuS6lN3oo+ItwLfBK7PzF8DtwKnAf3AbuDvhqrW2DzfVJC5PDMHMnOgp6dnzIFL\nkupTV6KPiC4qSf7rmfktgMx8MTMPZ+ZrwG283j0zCMyt2rwX2NW8kCVJY1HPqJsAVgDPZubfV5XP\nrqp2ObCtWF4LLI6IEyJiPnA68FjzQpYkjUU9o27OBz4CPBURW4qyzwJXREQ/lW6ZHcBfAmTm0xGx\nBniGyoidax1xI0ntM2qiz8xHqd3v/sAI29wI3NhAXJKkJvHKWEkqOWevlEbjDcM1yZnopXEYbvrl\nHcs+MMGRSKOz60aSSs4WvdrK2/lJrWei14QxqUvtYdeNJJWciV6SSs5EL0klZ6KXpJIz0UtSyZno\nJankTPSSVHImekkqORO9JJWciV6SSs5EL0klZ6KXpJJzUjNpHGpN0NZ34O42RCKNzha9JJWciV6S\nSs5EL0klN2qij4i5EfFQRDwbEU9HxHVF+SkRsS4iflI8Ty/KIyK+GhHbI2JrRJzT6j9CkjS8ek7G\nHgL+OjOfiIiTgM0RsQ64CngwM5dFxA3ADcCngYuB04vHucCtxbNUerVuGu4Nw9Vuo7boM3N3Zj5R\nLL8CPAvMARYBq4pqq4DLiuVFwJ1ZsQE4OSJmNz1ySVJdxtRHHxF9wNnARuDUzNwNlS8DYFZRbQ6w\ns2qzwaLs6PdaEhGbImLT3r17xx65JKkudSf6iHgr8E3g+sz89UhVa5Tlmwoyl2fmQGYO9PT01BuG\nJGmM6kr0EdFFJcl/PTO/VRS/ONQlUzzvKcoHgblVm/cCu5oTriRprOoZdRPACuDZzPz7qlVrgSuL\n5SuB71SVf7QYfXMesH+oi0eSNPHqGXVzPvAR4KmI2FKUfRZYBqyJiGuAnwEfKtY9AFwCbAd+A1zd\n1IjVsWqNOJHUfqMm+sx8lNr97gAX1qifwLUNxiVNOs5/o07llbGSVHImekkqOacpVkvU6saQ1B62\n6CWp5Ez0klRyJnpJKjkTvSSVnIlekkrOUTdSiw13xbDz1Gui2KKXpJIz0UtSyZnoJank7KNX45ZO\nA2BHd5vjkFSTLXpJKjkTvSSVnF03GrOjhwvaZSN1Nlv0klRyJnpJKjm7bqQW8vaC6gS26CWp5Ez0\nklRyo3bdRMQdwAeBPZn5rqJsKfCfgb1Ftc9m5gPFus8A1wCHgf+amf/cgrjVLkunOcpGmmTqadGv\nBC6qUf7lzOwvHkNJ/p3AYuCMYpt/iIgpzQpWkjR2oyb6zHwYeKnO91sE3JOZBzPzBWA7sKCB+CRJ\nDWqkj/4TEbE1Iu6IiOlF2RxgZ1WdwaJMktQm4030twKnAf3AbuDvivKoUTdrvUFELImITRGxae/e\nvbWqSJKaYFyJPjNfzMzDmfkacBuvd88MAnOrqvYCu4Z5j+WZOZCZAz09PeMJQ5JUh3El+oiYXfXy\ncmBbsbwWWBwRJ0TEfOB04LHGQpQkNaKe4ZWrgYXAzIgYBD4PLIyIfirdMjuAvwTIzKcjYg3wDHAI\nuDYzD7cmdElSPUZN9Jl5RY3iFSPUvxG4sZGgJEnN45WxklRyTmqmETn3vDT52aKXpJIz0UtSyZno\nJank7KOX2uTo8x8AO5Z9oA2RqOxs0UtSydmilyaYtxfURDPRa3jeZEQqBbtuJKnkTPSSVHImekkq\nORO9JJWciV6SSs5EL0kl5/BKqQMcGVu/tKpw6f52hKISskUvSSVnopekkjPRS1LJmeglqeQ8GauK\npdPaHYGkFrFFL0klN2qLPiLuAD4I7MnMdxVlpwD3An3ADuDPM/PliAjgZuAS4DfAVZn5RGtCl44N\ntW5QAt6kRPWrp0W/ErjoqLIbgAcz83TgweI1wMXA6cVjCXBrc8KUJI3XqIk+Mx8GXjqqeBGwqlhe\nBVxWVX5nVmwATo6I2c0KVpI0duM9GXtqZu4GyMzdETGrKJ8D7KyqN1iU7R5/iNKxabguG2msmn0y\nNmqUZc2KEUsiYlNEbNq7d2+Tw5AkDRlvon9xqEumeN5TlA8Cc6vq9QK7ar1BZi7PzIHMHOjp6Rln\nGJKk0Yw30a8FriyWrwS+U1X+0ag4D9g/1MUjSWqPeoZXrgYWAjMjYhD4PLAMWBMR1wA/Az5UVH+A\nytDK7VSGV17dgpglSWMwaqLPzCuGWXVhjboJXNtoUJKk5vHKWEkqORO9JJWciV6SSs7ZK6VJqtYF\nVc5/o1pM9FKHOnIf2Sp9B+5uQySa7Oy6kaSSM9FLUsmZ6CWp5Ez0klRynow9BtUcrdHdhkAkTQgT\nvTSJOBJH42HXjSSVnIlekkrOrptjUK2f/5LKyxa9JJWciV6SSs6um7JbOq3dEUhqM1v0klRyJnpJ\nKjkTvSSVnIlekkrOk7FSidSaxwi889SxrqFEHxE7gFeAw8ChzByIiFOAe4E+YAfw55n5cmNhSpLG\nqxkt+j/KzF9Wvb4BeDAzl0XEDcXrTzfhcyTV4ERnGk0rum4WAQuL5VXAekz0Ult5I/FjW6MnYxP4\nfkRsjoglRdmpmbkboHie1eBnSJIa0GiL/vzM3BURs4B1EfGjejcsvhiWAMybN6/BMCRJw2moRZ+Z\nu4rnPcB9wALgxYiYDVA87xlm2+WZOZCZAz09PY2EIUkawbhb9BExFTguM18plv8j8EVgLXAlsKx4\n/k4zAlWVWvPXLN0/8XFImhQa6bo5FbgvIobe5+7M/KeIeBxYExHXAD8DPtR4mJKk8Rp3os/M54Gz\napTvAy5sJCiN3bAXynjTbw3Di6uOHV4ZK5XQcHcRc3z9scm5biSp5GzRl4T3gZU0HFv0klRyJnpJ\nKjm7bqRjiBOgHZts0UtSydmilzQqZ7+c3Ez0kt5guAupNHnZdSNJJWeil6SSM9FLUsnZR9/pak1J\nLEljYKLvJCZ1TSLOfjl52HUjSSVni146xtU7IZ5X0E5eJvp2sZtG0gQx0UtqKq+i7Tz20UtSydmi\nnwh200hqIxO9pLq0Yopju3kmhole0rjVPWLnhvq/EByf33wtS/QRcRFwMzAFuD0zl7Xqs9rGLhlp\nwtj6H7+WJPqImALcAvwJMAg8HhFrM/OZVnyepM7mna3aq1Ut+gXA9sx8HiAi7gEWAZMj0ddqqS/d\nP/FxSCVWb/L3S6JxrUr0c4CdVa8HgXNb8kmNJuV6u1/sppE6TqtuktJol9BY4pqI7qdWJfqoUZZv\nqBCxBFhSvHw1In7cpM+eyRfil016r1abCRhr8xlra0xArB98U0mtZFKr3lEaijX+drxbjuuzGon1\nD+qp1KpEPwjMrXrdC+yqrpCZy4Hlzf7giNiUmQPNft9WMNbWMNbWMNbWmIhYW3Vl7OPA6RExPyLe\nAiwG1rbosyRJI2hJiz4zD0XEJ4B/pjK88o7MfLoVnyVJGlnLxtFn5gPAA616/xE0vTuohYy1NYy1\nNYy1NVoea2Tm6LUkSZOWs1dKUslNmkQfEXdExJ6I2DbM+v8UEVuLxw8j4qyqdTsi4qmI2BIRmzog\n1oURsb+IZ0tEfK5q3UUR8eOI2B4RN3RArH9TFee2iDgcEacU6yZ6v86NiIci4tmIeDoirqtRJyLi\nq8X+2xoR51StuzIiflI8ruyAWDvimK0z1o44ZuuMtSOO2YjojojHIuLJItYv1KhzQkTcW+y7jRHR\nV7XuM0X5jyPi/Q0Fk5mT4gH8B+AcYNsw6/8QmF4sXwxsrFq3A5jZQbEuBO6vUT4FeA74V8BbgCeB\nd7Yz1qPq/inwgzbu19nAOcXyScD/PXr/AJcA36My/Pq8oeMAOAV4vnieXixPb3OsHXHM1hlrRxyz\n9cTaKcdscQy+tVjuAjYC5x1V578A/1gsLwbuLZbfWezLE4D5xT6eMt5YJk2LPjMfBl4aYf0PM/Pl\n4uUGKmP322K0WEdwZOqIzPwdMDR1RMuMMdYrgNUtDGdEmbk7M58oll8BnqVyFXa1RcCdWbEBODki\nZgPvB9Zl5kvFcbIOuKidsXbKMVvnfh3OhB6z44i1bcdscQy+WrzsKh5HnxRdBKwqlr8BXBgRUZTf\nk5kHM/MFYDuVfT0ukybRj9E1VFp1QxL4fkRsjsoVuZ3g3xU/6b4XEWcUZbWmjqj3P1xLRcS/oJIY\nv1lV3Lb9WvzEPZtKK6nacPuwbft2hFirdcQxO0qsHXXMjrZfO+GYjYgpEbEF2EOloTHs8ZqZh4D9\nwAyavF9LNx99RPwRlf80/76q+PzM3BURs4B1EfGjoiXbLk8Af5CZr0bEJcC3gdOpY+qINvpT4H9n\nZnXrvy37NSLeSuU/7/WZ+eujV9fYJEcob6lRYh2q0xHH7CixdtQxW89+pQOO2cw8DPRHxMnAfRHx\nrsysPh82IcdrqVr0EXEmcDuwKDP3DZVn5q7ieQ9wHw38BGqGzPz10E+6rFxv0BURM6lj6og2WsxR\nP4HbsV8joovKf/CvZ+a3alQZbh9O+L6tI9aOOWZHi7WTjtl69muhI47Z4vN+Baznzd2FR/ZfRBwP\nTKPSldrc/ToRJyWa9QD6GP4E5zwq/Vh/eFT5VOCkquUfAhe1OdZ/yevXMCwAfkblG/x4KicJ5/P6\nia0z2hlrsX7o4Jvazv1a7KM7ga+MUOcDvPFk7GNF+SnAC1ROxE4vlk9pc6wdcczWGWtHHLP1xNop\nxyzQA5xcLJ8IPAJ88Kg61/LGk7FriuUzeOPJ2Odp4GTspOm6iYjVVM78z4yIQeDzVE5ukJn/CHyO\nSt/WP1TOZXAoKxMFnUrlJxNUDsq7M/Of2hzrnwF/FRGHgN8Ci7PyrzvhU0fUESvA5cD3M/P/VW06\n4fsVOB/4CPBU0e8J8FkqCXMo3geojLzZDvwGuLpY91JE/Dcq8zABfDHf+JO+HbF2yjFbT6ydcszW\nEyt0xjE7G1gVlRsxHUclid8fEV8ENmXmWmAFcFdEbKfyxbS4+Duejog1VO7hcQi4NivdQOPilbGS\nVHKl6qOXJL2ZiV6SSs5EL0klZ6KXpJIz0UtSyZnoJankTPSSVHImekkquf8PDWWRbDFnqioAAAAA\nSUVORK5CYII=\n",
190 |       "text/plain": [
191 |        "<matplotlib.figure.Figure at 0x1f481769240>"
192 |       ]
193 |      },
194 |      "metadata": {},
195 |      "output_type": "display_data"
196 |     },
197 |     {
198 |      "name": "stdout",
199 |      "output_type": "stream",
200 |      "text": [
201 |       "mean: 2.02602114868\n",
202 |       "variance: 0.0416227008759\n",
203 |       "CRB variance: 0.04\n"
204 |      ]
205 |     }
206 |    ],
207 |    "source": [
208 |     "n = 100\n",
209 |     "crb_variance = rate**2/n\n",
210 |     "rate_hat = np.zeros(NUM_SAMPLES)\n",
211 |     "rate_hat_crb = np.zeros(NUM_SAMPLES)\n",
212 |     "for i in range(0,NUM_SAMPLES):\n",
213 |     "    y = np.random.exponential(1/rate,n)\n",
214 |     "    rate_hat_crb[i] = rate + np.sqrt(crb_variance)*np.random.randn() #This is what an efficient estimator would look like\n",
215 |     "    rate_hat[i] = 1/np.mean(y) #The ML estimate is the reciprocal of the sample mean\n",
216 |     "    \n",
217 |     "plt.hist(rate_hat,50)\n",
218 |     "plt.hist(rate_hat_crb,50)\n",
219 |     "plt.gca().legend(['ML','CRB'])\n",
220 |     "plt.show()\n",
221 |     "\n",
222 |     "print('mean: '+str(np.mean(rate_hat)))\n",
223 |     "print('variance: '+str(np.var(rate_hat)))\n",
224 |     "print('CRB variance: '+str(crb_variance))"
225 |    ]
226 |   },
227 |   {
228 |    "cell_type": "markdown",
229 |    "metadata": {},
230 |    "source": [
231 |     "At this point the distributions are difficult to distinguish. Asymptotic efficiency, indeed!"
232 |    ]
233 |   }
234 |  ],
235 |  "metadata": {
236 |   "kernelspec": {
237 |    "display_name": "Python [default]",
238 |    "language": "python",
239 |    "name": "python3"
240 |   },
241 |   "language_info": {
242 |    "codemirror_mode": {
243 |     "name": "ipython",
244 |     "version": 3
245 |    },
246 |    "file_extension": ".py",
247 |    "mimetype": "text/x-python",
248 |    "name": "python",
249 |    "nbconvert_exporter": "python",
250 |    "pygments_lexer": "ipython3",
251 |    "version": "3.5.2"
252 |   }
253 |  },
254 |  "nbformat": 4,
255 |  "nbformat_minor": 2
256 | }
257 | 


--------------------------------------------------------------------------------
/ch.6/empty:
--------------------------------------------------------------------------------
1 | 
2 | 


--------------------------------------------------------------------------------
/course-notes.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/docnok/detection-estimation-learning/9c2e629c164ef246c9f8f670dcc10a868bdf8bbc/course-notes.pdf


--------------------------------------------------------------------------------