├── data_to_train_on.npy ├── gifs ├── hopfield_4.gif └── hopfield_4_p2.gif ├── README.md └── hopfield.ipynb /data_to_train_on.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TarinZ/hopfield-nets/HEAD/data_to_train_on.npy -------------------------------------------------------------------------------- /gifs/hopfield_4.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TarinZ/hopfield-nets/HEAD/gifs/hopfield_4.gif -------------------------------------------------------------------------------- /gifs/hopfield_4_p2.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TarinZ/hopfield-nets/HEAD/gifs/hopfield_4_p2.gif -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # hopfield-nets 2 | 3 | 4 | 5 | Hopfield Nets were one of the first success stories of the Connectionists, and they operate on a somewhat different principle than the DNNs that are so popular today. In essense, Hopfield networks are a specific class of Connectionist _energy_ based neural networks, whereby relationships between neurons/pixels are impressed into the weights as minimum energy states. 6 | 7 | This allows us to store patterns and later recover them from their corrupted versions - the so called, "content addressable networks", which are somewhat akin to how we are so easily able to recall memories by only being given certain cues. 8 | 9 | The above animation shows a hopfield network at work, completely recovering a corrupted pattern. 10 | 11 | This repo will walk you through the process of building a Hopfield network. To do so, simply run hopfield.ipynb via: 12 | 13 | `ipython notebook hopfield.ipynb` 14 | -------------------------------------------------------------------------------- /hopfield.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 90, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "# First, we will load up some dependencies. Make sure you can get through this part first. \n", 12 | "import numpy as np\n", 13 | "import matplotlib\n", 14 | "matplotlib.use('TkAgg')\n", 15 | "import matplotlib.pyplot as plt\n", 16 | "from IPython import display\n", 17 | "%matplotlib inline " 18 | ] 19 | }, 20 | { 21 | "cell_type": "markdown", 22 | "metadata": {}, 23 | "source": [ 24 | "Now we are going to load the data in. I have already saved off two images, a '9' and a '4', to keep things simple for us. We also reshape them into their native [28 x 28] size, so that we can later corrupt them in this domain, and then reconstruct them." 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": 35, 30 | "metadata": { 31 | "collapsed": false 32 | }, 33 | "outputs": [], 34 | "source": [ 35 | "data = (np.load('data_to_train_on.npy')).astype(np.float32).reshape(2,28,28)" 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": {}, 41 | "source": [ 42 | "What does our data look like? Well they are just two binary images, showing a figure '9', and a figure '4'. Let's image them just to take a peek at what they look like. " 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 36, 48 | "metadata": { 49 | "collapsed": false 50 | }, 51 | "outputs": [ 52 | { 53 | "data": { 54 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2oAAAGfCAYAAAAu6yGIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFHJJREFUeJzt3U+IrQeZ5/HfM5a9URcRMyGk05MWZEAGJsIlDLQMDj3d\n2G6iG+ksmgw0xEWPKPSixU3XZkCG1p7NIEQMZsC2aVDHLGSGtAhOwyDeSNBopkeRSCdck4gL7VWj\nPrO4J5M6ya0/t06dep9T5/OBS1WdPzkP7z2p537ve+rc6u4AAAAwxz9begAAAADWCTUAAIBhhBoA\nAMAwQg0AAGAYoQYAADCMUAMAABhGqAEAAAwj1AAAAIYRagAAAMMcXOaDVb2tk/su8yEBWMRz6f5p\nLT3FrrAfAfbJUz/t7jtPu9WlhtrNJXT9ch8SgAVcW3qAHXNf7EeAfVE/PsutvPQRAABgmI1Crare\nW1V/X1U/rKqPXdRQALDr7EgANnHuUKuqNyT5r0n+IMk7kzxUVe+8qMEAYFfZkQBsapMzag8k+WF3\n/6i7/ynJXyd58GLGAoCdZkcCsJFNQu2eJP9w5OvnV5cBwL6zIwHYyNbfTKSqHqmq61V1PXl52w8H\nADvBfgTgJJuE2gtJ7j3y9W+uLlvT3Y9297Xuvpac+s8FAMBVcOqOtB8BOMkmofatJO+oqt+uqt9I\n8odJnriYsQBgp9mRAGzk3P/gdXf/sqr+Y5L/meQNSR7r7u9d2GQAsKPsSAA2de5QS5Lu/mqSr17Q\nLABwZdiRAGxi628mAgAAwO0RagAAAMMINQAAgGGEGgAAwDBCDQAAYBihBgAAMIxQAwAAGEaoAQAA\nDCPUAAAAhhFqAAAAwwg1AACAYYQaAADAMEINAABgGKEGAAAwjFADAAAYRqgBAAAMI9QAAACGEWoA\nAADDCDUAAIBhhBoAAMAwQg0AAGAYoQYAADCMUAMAABhGqAEAAAwj1AAAAIYRagAAAMMINQAAgGGE\nGgAAwDBCDQAAYBihBgAAMIxQAwAAGEaoAQAADCPUAAAAhhFqAAAAwwg1AACAYYQaAADAMEINAABg\nGKEGAAAwjFADAAAYRqgBAAAMI9QAAACGEWoAAADDCDUAAIBhhBoAAMAwQg0AAGAYoQYAADCMUAMA\nABhGqAEAAAwj1AAAAIYRagAAAMMINQAAgGGEGgAAwDBCDQAAYBihBgAAMIxQAwAAGEaoAQAADHOw\n9AAAAMA59eHSE5yuDpeeYCc5owYAADCMUAMAABhmo5c+VtVzSX6R5FdJftnd1y5iKADYdXYkAJu4\niJ9R+3fd/dML+O8AwFVjRwJwLl76CAAAMMymodZJ/raqnqqqRy5iIAC4IuxIAM5t05c+vru7X6iq\nf57kyar6P939jaM3WC2n1YL6rQ0fDgB2xok70n4E4CQbnVHr7hdWH19K8uUkD9ziNo9297WbP0R9\n5yYPBwA747QdaT8CcJJzh1pVvamq3vLK50l+P8kzFzUYAOwqOxKATW3y0se7kny5ql757/xVd/+P\nC5kKAHabHQnARs4dat39oyT/+gJnAYArwY4EYFPenh8AAGAYoQYAADDMpm/PDwBwa314uY9Xl/x4\nAFvkjBoAAMAwQg0AAGAYoQYAADCMUAMAABhGqAEAAAwj1AAAAIYRagAAAMMINQAAgGGEGgAAwDBC\nDQAAYBihBgAAMIxQAwAAGOZg6QFgEX249ARcljpcegIAOBt/PuEIZ9QAAACGEWoAAADDCDUAAIBh\nhBoAAMAwQg0AAGAYoQYAADCMUAMAABhGqAEAAAwj1AAAAIYRagAAAMMINQAAgGGEGgAAwDBCDQAA\nYJiDpQeAjfXh0hMAAFx9dbj0BHvFGTUAAIBhhBoAAMAwQg0AAGAYoQYAADCMUAMAABhGqAEAAAwj\n1AAAAIYRagAAAMMINQAAgGGEGgAAwDBCDQAAYBihBgAAMMzB0gNwBfXh0hMAAMBOc0YNAABgGKEG\nAAAwjFADAAAYRqgBAAAMI9QAAACGEWoAAADDCDUAAIBhhBoAAMAwQg0AAGAYoQYAADCMUAMAABhG\nqAEAAAwj1AAAAIY5WHoAuPLq8HIfry/58YD94HsLwKVyRg0AAGAYoQYAADDMqaFWVY9V1UtV9cyR\ny95aVU9W1Q9WH+/Y7pgAMI8dCcC2nOWM2ueSvPc1l30syde6+x1Jvrb6GgD2zediRwKwBaeGWnd/\nI8nPXnPxg0keX33+eJL3X/BcADCeHQnAtpz3Z9Tu6u4bq89/kuSuC5oHAHadHQnAxjZ+M5Hu7iR9\n3PVV9UhVXa+q68nLmz4cAOyMk3ak/QjASc4bai9W1d1Jsvr40nE37O5Hu/tad19L7jznwwHAzjjT\njrQfATjJeUPtiSQPrz5/OMlXLmYcANh5diQAGzvL2/N/Icn/TvIvq+r5qvrjJJ9I8ntV9YMk/371\nNQDsFTsSgG05OO0G3f3QMVf97gXPAgA7xY4EYFs2fjMRAAAALpZQAwAAGObUlz6y5/pw6QlmqcOl\nJ9hfjj0Au8Kfn7gAzqgBAAAMI9QAAACGEWoAAADDCDUAAIBhhBoAAMAwQg0AAGAYoQYAADCMUAMA\nABhGqAEAAAwj1AAAAIYRagAAAMMINQAAgGEOlh6AS9KHS08wSx1e3mM59q93mccfAGAHOaMGAAAw\njFADAAAYRqgBAAAMI9QAAACGEWoAAADDCDUAAIBhhBoAAMAwQg0AAGAYoQYAADCMUAMAABhGqAEA\nAAwj1AAAAIYRagAAAMMcLD0A59CHS0+wHXW49ASnu6rHHgB2hV18MXbhz117zhk1AACAYYQaAADA\nMEINAABgGKEGAAAwjFADAAAYRqgBAAAMI9QAAACGEWoAAADDCDUAAIBhhBoAAMAwQg0AAGAYoQYA\nADDMwdIDwP/Xh0tPwO2qw6UnANgd9hxwG5xRAwAAGEaoAQAADCPUAAAAhhFqAAAAwwg1AACAYYQa\nAADAMEINAABgGKEGAAAwjFADAAAYRqgBAAAMI9QAAACGEWoAAADDCDUAAIBhDpYeABigDpeeAGBz\nfbj0BHD57PAryxk1AACAYYQaAADAMKeGWlU9VlUvVdUzRy47rKoXqurp1a/3bXdMAJjHjgRgW85y\nRu1zSd57i8v/srvvX/366sWOBQA74XOxIwHYglNDrbu/keRnlzALAOwUOxKAbdnkZ9Q+XFXfWb3s\n444LmwgAdp8dCcBGzhtqn07y9iT3J7mR5JPH3bCqHqmq61V1PXn5nA8HADvjTDvSfgTgJOcKte5+\nsbt/1d2/TvKZJA+ccNtHu/tad19L7jzvnACwE866I+1HAE5yrlCrqruPfPmBJM8cd1sA2Cd2JAAX\n4eC0G1TVF5K8J8nbqur5JH+e5D1VdX+STvJckg9tcUYAGMmOBGBbTg217n7oFhd/dguzAMBOsSMB\n2JZN3vURAACALRBqAAAAw5z60kcGqsPbv0+f4z4A8Aq7Zz+c5/f5KvMcZkHOqAEAAAwj1AAAAIYR\nagAAAMMINQAAgGGEGgAAwDBCDQAAYBihBgAAMIxQAwAAGEaoAQAADCPUAAAAhhFqAAAAwwg1AACA\nYQ6WHoBLUodLT3A19OHSEwDsDrsH4NycUQMAABhGqAEAAAwj1AAAAIYRagAAAMMINQAAgGGEGgAA\nwDBCDQAAYBihBgAAMIxQAwAAGEaoAQAADCPUAAAAhhFqAAAAwwg1AACAYYQaAADAMEINAABgGKEG\nAAAwjFADAAAYRqgBAAAMI9QAAACGEWoAAADDCDUAAIBhhBoAAMAwQg0AAGAYoQYAADCMUAMAABhG\nqAEAAAwj1AAAAIY5WHoA2Cl1ePv36XPcBwCAveaMGgAAwDBCDQAAYBihBgAAMIxQAwAAGEaoAQAA\nDCPUAAAAhhFqAAAAwwg1AACAYYQaAADAMEINAABgGKEGAAAwjFADAAAY5mDpAWCn9OHSEwAAsAec\nUQMAABhGqAEAAAxzaqhV1b1V9fWq+n5Vfa+qPrK6/K1V9WRV/WD18Y7tjwsAM9iPAGzTWc6o/TLJ\nn3b3O5P8myR/UlXvTPKxJF/r7nck+drqawDYF/YjAFtzaqh1943u/vbq818keTbJPUkeTPL46maP\nJ3n/toYEgGnsRwC26bZ+Rq2q7kvyriTfTHJXd99YXfWTJHdd6GQAsCPsRwAu2plDrarenOSLST7a\n3T8/el13d5I+5n6PVNX1qrqevLzRsAAwjf0IwDacKdSq6o25uYQ+391fWl38YlXdvbr+7iQv3eq+\n3f1od1/r7mvJnRcxMwCMYD8CsC1nedfHSvLZJM9296eOXPVEkodXnz+c5CsXPx4AzGQ/ArBNB2e4\nze8k+aMk362qp1eXfTzJJ5L8TVX9cZIfJ/ngdkYEgJHsRwC25tRQ6+6/S1LHXP27FzsOAOwG+xGA\nbbqtd30EAABg+4QaAADAMEINAABgGKEGAAAwjFADAAAYRqgBAAAMI9QAAACGEWoAAADDCDUAAIBh\nhBoAAMAwQg0AAGAYoQYAADCMUAMAABhGqAEAAAwj1AAAAIYRagAAAMMINQAAgGGEGgAAwDBCDQAA\nYBihBgAAMIxQAwAAGEaoAQAADCPUAAAAhhFqAAAAwwg1AACAYYQaAADAMEINAABgGKEGAAAwjFAD\nAAAY5mDpAWCn1OHt36fPcR8AAPaaM2oAAADDCDUAAIBhhBoAAMAwQg0AAGAYoQYAADCMUAMAABhG\nqAEAAAwj1AAAAIYRagAAAMMINQAAgGGEGgAAwDBCDQAAYBihBgAAMMzB0gMAA/Th+e5X57wfAAAn\nckYNAABgGKEGAAAwjFADAAAYRqgBAAAMI9QAAACGEWoAAADDCDUAAIBhhBoAAMAwQg0AAGAYoQYA\nADCMUAMAABhGqAEAAAxzsPQAwA7rw/Pdr855PwC4TPYVC3JGDQAAYBihBgAAMMypoVZV91bV16vq\n+1X1var6yOryw6p6oaqeXv163/bHBYAZ7EcAtuksP6P2yyR/2t3frqq3JHmqqp5cXfeX3f0X2xsP\nAMayHwHYmlNDrbtvJLmx+vwXVfVsknu2PRgATGY/ArBNt/UzalV1X5J3Jfnm6qIPV9V3quqxqrrj\ngmcDgJ1gPwJw0c4calX15iRfTPLR7v55kk8neXuS+3PzbxQ/ecz9Hqmq61V1PXn5AkYGgDnsRwC2\n4UyhVlVvzM0l9Pnu/lKSdPeL3f2r7v51ks8keeBW9+3uR7v7WndfS+68qLkBYHH2IwDbcpZ3fawk\nn03ybHd/6sjldx+52QeSPHPx4wHATPYjANt0lnd9/J0kf5Tku1X19Oqyjyd5qKruT9JJnkvyoa1M\nCAAz2Y8AbM1Z3vXx75LULa766sWPAwC7wX4EYJtu610fAQAA2D6hBgAAMMxZfkYN2EQdnu9+fc77\nAQCw85xRAwAAGEaoAQAADCPUAAAAhhFqAAAAwwg1AACAYYQaAADAMEINAABgGKEGAAAwjFADAAAY\nRqgBAAAMI9QAAACGEWoAAADDHCw9AHCMOlx6AgAAFuKMGgAAwDBCDQAAYBihBgAAMIxQAwAAGEao\nAQAADCPUAAAAhhFqAAAAwwg1AACAYYQaAADAMEINAABgGKEGAAAwjFADAAAYRqgBAAAMU919eQ9W\n9XKSHx9z9duS/PTShpnP8VjneKxzPF7lWKybcjz+RXffufQQu8J+vC2OxzrHY53jsc7xWDfleJxp\nR15qqJ2kqq5397Wl55jC8VjneKxzPF7lWKxzPK4ev6frHI91jsc6x2Od47Fu146Hlz4CAAAMI9QA\nAACGmRRqjy49wDCOxzrHY53j8SrHYp3jcfX4PV3neKxzPNY5Huscj3U7dTzG/IwaAAAAN006owYA\nAEAGhFpVvbeq/r6qflhVH1t6nqVV1XNV9d2qerqqri89z2Wrqseq6qWqeubIZW+tqier6gerj3cs\nOeNlOuZ4HFbVC6vnyNNV9b4lZ7xMVXVvVX29qr5fVd+rqo+sLt/L58gJx2NvnyNXjR25zo60I4+y\nI19lP667Kvtx0Zc+VtUbkvzfJL+X5Pkk30ryUHd/f7GhFlZVzyW51t0T/o2HS1dV/zbJPyb5b939\nr1aX/eckP+vuT6z+oHJHd//ZknNelmOOx2GSf+zuv1hytiVU1d1J7u7ub1fVW5I8leT9Sf5D9vA5\ncsLx+GD29DlyldiRr2dH2pFH2ZGvsh/XXZX9uPQZtQeS/LC7f9Td/5Tkr5M8uPBMLKi7v5HkZ6+5\n+MEkj68+fzw3/0fbC8ccj73V3Te6+9urz3+R5Nkk92RPnyMnHA+uBjuSNXbkOjvyVfbjuquyH5cO\ntXuS/MORr5/PDh7EC9ZJ/raqnqqqR5YeZoi7uvvG6vOfJLlryWGG+HBVfWf1so+9eBnDa1XVfUne\nleSb8Rx57fFIPEeuAjvy9ezI19v773+3sNff/+zHdbu8H5cONV7v3d19f5I/SPInq9P6rPTN1+ru\n+1uVfjrJ25Pcn+RGkk8uO87lq6o3J/liko9298+PXrePz5FbHI+9f45wZdmRJ9jH73+3sNff/+zH\ndbu+H5cOtReS3Hvk699cXba3uvuF1ceXknw5N1/6su9eXL3W+JXXHL+08DyL6u4Xu/tX3f3rJJ/J\nnj1HquqNuflN9/Pd/aXVxXv7HLnV8dj358gVYke+hh15S3v7/e9W9vn7n/247irsx6VD7VtJ3lFV\nv11Vv5HkD5M8sfBMi6mqN61+4DFV9aYkv5/kmZPvtReeSPLw6vOHk3xlwVkW98o33JUPZI+eI1VV\nST6b5Nnu/tSRq/byOXLc8djn58gVY0ceYUceay+//x1nX7//2Y/rrsp+XPwfvF69LeZ/SfKGJI91\n939adKAFVdXbc/NvCJPkIMlf7dvxqKovJHlPkrcleTHJnyf570n+JslvJflxkg9291788PAxx+M9\nuXnKvpM8l+RDR15/fqVV1buT/K8k303y69XFH8/N153v3XPkhOPxUPb0OXLV2JGvsiPtyNeyI19l\nP667Kvtx8VADAABg3dIvfQQAAOA1hBoAAMAwQg0AAGAYoQYAADCMUAMAABhGqAEAAAwj1AAAAIYR\nagAAAMP8PxJHEGV7AAN2AAAAAElFTkSuQmCC\n", 55 | "text/plain": [ 56 | "" 57 | ] 58 | }, 59 | "metadata": {}, 60 | "output_type": "display_data" 61 | } 62 | ], 63 | "source": [ 64 | "# Show the training data - it's just two different samples that we are going to want the Hopfield net to store.\n", 65 | "fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(15,7.5))\n", 66 | "ax[0].imshow(data[0], interpolation='None', cmap='winter')\n", 67 | "ax[1].imshow(data[1], interpolation='None', cmap='winter')\n", 68 | "plt.show()" 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": 46, 74 | "metadata": { 75 | "collapsed": false 76 | }, 77 | "outputs": [], 78 | "source": [ 79 | "# Here, we will set some initial parameters, as well as define some helper functions used in the training and fielding\n", 80 | "# of the Hopfield net\n", 81 | "\n", 82 | "# This is a cool way to make it so that we can store ad-hoc members in a struct. I find it handy, where I use \n", 83 | "# \"N\" for \"number of things\", and \"I\" to keep track of various indices. Personal preference.\n", 84 | "N = lambda:0\n", 85 | "I = lambda:0\n", 86 | "\n", 87 | "# Train function for the hopfield net. (By updating the weights)\n", 88 | "def train_hopfield_weights(data, weights):\n", 89 | " # data : A [d x rows x cols] numpy array of data. 'd' is the number of data samples used to train. (In this case 2)\n", 90 | " # weights : A [rows*cols x rows*cols] matrix of the hopfield weights, initialized with 0s.\n", 91 | "\n", 92 | " # First, we vectorize our data:\n", 93 | " data = data.reshape((-1, 1, data.shape[1]*data.shape[2]))\n", 94 | " \n", 95 | " # Number of patterns to store/train with.\n", 96 | " N.data = data.shape[0] \n", 97 | "\n", 98 | " # Loop through all data samples.\n", 99 | " for dd in xrange(N.data):\n", 100 | "\n", 101 | " # Data cross-product gives neural hopfield update rule.\n", 102 | " w_update = data[dd].T * data[dd]\n", 103 | " \n", 104 | " # Sum all pattern cross-products.\n", 105 | " weights += w_update\n", 106 | "\n", 107 | " # Hopfield nets are a form of RNNs, albeit without self-connections, and so, we need to make sure that the \n", 108 | " # diagonal elements of the final weight matrix are zero. \n", 109 | " np.fill_diagonal(weights, 0)\n", 110 | " \n", 111 | " # Return our new weights\n", 112 | " return (weights.astype(np.float32))\n", 113 | "\n", 114 | "\n", 115 | "# This function computes the Hopfield nets' energy.\n", 116 | "def energy(x, w):\n", 117 | " # x: [1 x N] data vector\n", 118 | " # w: [N x N] hopfield net weight matrix.\n", 119 | " energy = -x.dot(w).dot(x.T)\n", 120 | " return energy\n", 121 | "\n", 122 | "# We will use this function to take in a good image, and corrupt it, based on two types of noise, either bernouli noise,\n", 123 | "# (where every pixel is randomly flipped in sign), or masking noise, where we blot out entire sections of the image. Our\n", 124 | "# Hopfield net will be able to recover the original image via energy minimization as we will see.\n", 125 | "def corrupt_input(image_to_corrupt, method, bernouli_noise_probability = 0.1, mask_value = -1, mask_matrix=np.zeros((2,2))):\n", 126 | " if method == 'bernouli':\n", 127 | " noise_image = (-1*np.random.binomial(1, bernouli_noise_probability, (image_to_corrupt.shape[0], image_to_corrupt.shape[1]))).astype(np.float32)\n", 128 | " noise_image[noise_image==0] = 1\n", 129 | " corrupted_image = noise_image * image_to_corrupt\n", 130 | " elif method == 'mask':\n", 131 | " corrupted_image = np.copy(image_to_corrupt)\n", 132 | " corrupted_image[mask_matrix[0,0]:mask_matrix[0,1], \n", 133 | " mask_matrix[1,0]:mask_matrix[1,1]] = mask_value\n", 134 | "\n", 135 | " return corrupted_image " 136 | ] 137 | }, 138 | { 139 | "cell_type": "markdown", 140 | "metadata": {}, 141 | "source": [ 142 | "\n", 143 | "\n" 144 | ] 145 | }, 146 | { 147 | "cell_type": "markdown", 148 | "metadata": {}, 149 | "source": [ 150 | "OK this is where the fun begins! We are now going to train our Hopfield net. The objective is simple: We want it to store those two patterns we are feeding it, such that a faithful reconstruction can be performed, when _corrupted_ versions of either of those inputs is given later in the future. " 151 | ] 152 | }, 153 | { 154 | "cell_type": "code", 155 | "execution_count": 64, 156 | "metadata": { 157 | "collapsed": false 158 | }, 159 | "outputs": [], 160 | "source": [ 161 | "# Train the Hopfield net! \n", 162 | "\n", 163 | "# So there are as many neurons as pixels per pattern, and this is what this computes.\n", 164 | "N.neurons = data.shape[1] * data.shape[2]\n", 165 | "\n", 166 | "# The weight matrix encodes the relationships between neurons, in a Hebbian sense.\n", 167 | "weights = np.zeros((N.neurons, N.neurons)).astype(np.float32)\n", 168 | "\n", 169 | "# Great - we are all set - now we train! \n", 170 | "weights = train_hopfield_weights(data, weights)\n" 171 | ] 172 | }, 173 | { 174 | "cell_type": "markdown", 175 | "metadata": {}, 176 | "source": [ 177 | "Now at this point, we have a trained weight matrix, encoded with the pattern '4' and pattern '9'. (We will see later on that Hopfield nets aren't that memory efficient, but it's still amazing what they are capable of). Now remember - the objective here is simple: We want it to store those two patterns we are feeding it, such that faithful reconstruction can later be performed, when _corrupted_ versions of either of those inputs are presented. But does it work? Let's find out! \n", 178 | "\n", 179 | "We can corrupt our data in many ways - and here, we can add two types of noise: Bernouli noise, (where every pixel is flipped in sign with a certain probability) and/or masking noise, (where we simply blot out certain regions with one particular sign).\n" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": 72, 185 | "metadata": { 186 | "collapsed": false 187 | }, 188 | "outputs": [ 189 | { 190 | "data": { 191 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2oAAAGfCAYAAAAu6yGIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFplJREFUeJzt3V+orXd5J/DvM257URUaSSYkaTqpIANSaMRDGKgMDp0W\n6030RpqLkoFCvOiIQi9GvOmaiwEZqu1NESIGM2AtBXXMhcyQiuAUBvEcCRrNdBSJNDEmES+0zEVR\nf3NxlmSvk7PP3vtdf95nrfX5wGGv/a71y/us3/qt9znfvO9ap8YYAQAAoI9/MXcBAAAArBLUAAAA\nmhHUAAAAmhHUAAAAmhHUAAAAmhHUAAAAmhHUAAAAmhHUAAAAmhHUAAAAmjnZ5c6qbh/JfbvcJQCz\neDZj/KjmrmJf1O2/OnLfr11+4LW7N18MTPG2H1x+zNT1O2Vf6+zvUO3yNeMG1340xrjjvEftNKhd\nD2lXd7tLAGZwZe4C9st9v5ZcfeTy42qx8VJgkquLy4+Zun6n7Gud/R2qXb5m3KC+f5FHufQRAACg\nmbWCWlW9s6r+oaq+W1Uf2lRRALDv9EgA1jE5qFXVa5L8VZI/SPKWJA9V1Vs2VRgA7Cs9EoB1rXNG\n7YEk3x1jfG+M8c9J/ibJg5spCwD2mh4JwFrWCWr3JPnHU78/t9wGAMdOjwRgLVv/MpGqeqSqrlbV\n1eTlbe8OAPbCSn98+f/NXQ4AzawT1J5Pcu+p3399uW3FGOPRMcaVMcaV5Nx/LgAADsG5PXKlP97x\nqzstDoD+1glqX0vy5qr6zar6lSR/mOSJzZQFAHtNjwRgLZP/wesxxs+q6j8m+Z9JXpPksTHGtzZW\nGQDsKT0SgHVNDmpJMsb4YpIvbqgWADgYeiQA69j6l4kAAABwOYIaAABAM2td+ggAbMC1u5NazF0F\n7Afvlc0wj+05owYAANCMoAYAANCMoAYAANCMoAYAANCMoAYAANCMoAYAANCMoAYAANCMoAYAANCM\noAYAANCMoAYAANCMoAYAANCMoAYAANDMydwFAACw52oxdwVwcJxRAwAAaEZQAwAAaEZQAwAAaEZQ\nAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAA\naOZk7gIAAPbOWFx+TE0Ys2tTnleyH89tl3Y5j16zg+WMGgAAQDOCGgAAQDOCGgAAQDOCGgAAQDOC\nGgAAQDOCGgAAQDOCGgAAQDOCGgAAQDOCGgAAQDOCGgAAQDOCGgAAQDOCGgAAQDMncxcAAHAUxmLa\nuJo4jlVT5n/q3O/yNdvlvqzhnXJGDQAAoBlBDQAAoBlBDQAAoBlBDQAAoBlBDQAAoBlBDQAAoBlB\nDQAAoBlBDQAAoBlBDQAAoBlBDQAAoBlBDQAAoBlBDQAAoBlBDQAAoJmTuQsAuJCxmLuC89Vi7gqA\nXZnyfnccm9ehPrd9WFdM4owaAABAM4IaAABAM2td+lhVzyb5aZKfJ/nZGOPKJooCgH2nRwKwjk18\nRu3fjTF+tIH/DgAcGj0SgElc+ggAANDMukFtJPm7qrpWVY9soiAAOBB6JACTrXvp49vHGM9X1b9M\n8mRV/Z8xxldOP2DZnJYN6jfW3B0A7I1b9kj9EYBbWeuM2hjj+eXPl5J8PskDN3nMo2OMK9c/RH3H\nOrsDgL1xXo/UHwG4lclBrapeV1Vv+OXtJL+f5OlNFQYA+0qPBGBd61z6eGeSz1fVL/87fz3G+B8b\nqQoA9pseCcBaJge1Mcb3kvz2BmsBgIOgRwKwLl/PDwAA0IygBgAA0My6X89PkozFbvdXO94fANv1\nth8kVxeXH7fLfrDrXjdF9/7YvT6Oy5T1uA/HgQPijBoAAEAzghoAAEAzghoAAEAzghoAAEAzghoA\nAEAzghoAAEAzghoAAEAzghoAAEAzghoAAEAzghoAAEAzghoAAEAzghoAAEAzJ3MXAByhsZi7Aujl\n2t1JLeau4tam1Oe9Dpezy/eM92d7zqgBAAA0I6gBAAA0I6gBAAA0I6gBAAA0I6gBAAA0I6gBAAA0\nI6gBAAA0I6gBAAA0I6gBAAA0I6gBAAA0I6gBAAA0I6gBAAA0I6gBAAA0czJ3AQDt1GLuCgA4y1hM\nG+fYvj5zuFPOqAEAADQjqAEAADQjqAEAADQjqAEAADQjqAEAADQjqAEAADQjqAEAADQjqAEAADQj\nqAEAADQjqAEAADQjqAEAADQjqAEAADRzMncBAAArajF3BXS2D+tjLOauoJep87EPr/UWOaMGAADQ\njKAGAADQjKAGAADQjKAGAADQjKAGAADQjKAGAADQjKAGAADQjKAGAADQjKAGAADQjKAGAADQjKAG\nAADQjKAGAADQjKAGAADQzMncBbQzFnNXAAD97LI/Tt1XTRwHZ9n1WjzUv4d6b07ijBoAAEAzghoA\nAEAz5wa1qnqsql6qqqdPbXtjVT1ZVd9Z/rxtu2UCQD96JADbcpEzap9K8s4btn0oyZfGGG9O8qXl\n7wBwbD4VPRKALTg3qI0xvpLkxzdsfjDJ48vbjyd594brAoD29EgAtmXqZ9TuHGO8sLz9wyR3bqge\nANh3eiQAa1v7y0TGGCPJOOv+qnqkqq5W1dXk5XV3BwB741Y9Un8E4FamBrUXq+quJFn+fOmsB44x\nHh1jXBljXEnumLg7ANgbF+qR+iMAtzI1qD2R5OHl7YeTfGEz5QDA3tMjAVjbRb6e/zNJ/neSf11V\nz1XVHyf5SJLfq6rvJPn3y98B4KjokQBsy8l5DxhjPHTGXb+74VoAYK/okQBsy9pfJgIAAMBmCWoA\nAADNnHvpI8CZxmLuCgBge2oxbZz+yAY4owYAANCMoAYAANCMoAYAANCMoAYAANCMoAYAANCMoAYA\nANCMoAYAANCMoAYAANCMoAYAANCMoAYAANCMoAYAANCMoAYAANDMydwFAAATjcXlx9SEMVPHTamP\n4zF1fUxdw7t8vxyqXb9m3U2ej4s9zBk1AACAZgQ1AACAZgQ1AACAZgQ1AACAZgQ1AACAZgQ1AACA\nZgQ1AACAZgQ1AACAZgQ1AACAZgQ1AACAZgQ1AACAZgQ1AACAZgQ1AACAZk7mLgA4w1jMXcFhqMXc\nFcD53vaD5Ori8uN2ub6nHJOm1uf4RxeHvBa7Hz+SaTXucl+T5/A/X+hRzqgBAAA0I6gBAAA0I6gB\nAAA0I6gBAAA0I6gBAAA0I6gBAAA0I6gBAAA0I6gBAAA0I6gBAAA0I6gBAAA0I6gBAAA0I6gBAAA0\nczJ3ARygsZi7AoD9cu3upBZzV7F5+sEq87Fq6po3j5sxZR6nvma7PL4d0LHUGTUAAIBmBDUAAIBm\nBDUAAIBmBDUAAIBmBDUAAIBmBDUAAIBmBDUAAIBmBDUAAIBmBDUAAIBmBDUAAIBmBDUAAIBmBDUA\nAIBmBDUAAIBmTuYugAnGYu4KYPdqMXcFQHf6437xer3aLnud+W/PGTUAAIBmBDUAAIBmzg1qVfVY\nVb1UVU+f2raoquer6qnln3dtt0wA6EePBGBbLnJG7VNJ3nmT7X8xxrh/+eeLmy0LAPbCp6JHArAF\n5wa1McZXkvx4B7UAwF7RIwHYlnU+o/b+qvrG8rKP2zZWEQDsPz0SgLVMDWofT/KmJPcneSHJR896\nYFU9UlVXq+pq8vLE3QHA3rhQj9QfAbiVSUFtjPHiGOPnY4xfJPlEkgdu8dhHxxhXxhhXkjum1gkA\ne+GiPVJ/BOBWJgW1qrrr1K/vSfL0WY8FgGOiRwKwCSfnPaCqPpPkHUlur6rnkvxZkndU1f1JRpJn\nk7xvizUCQEt6JADbcm5QG2M8dJPNn9xCLQCwV/RIALZlnW99BAAAYAsENQAAgGbOvfTx6NTi8mPG\nhDHMa8rrfMisYeA8+9Af96HGXdplr5syj/vQi3e9PnY5j/sw/0fOGTUAAIBmBDUAAIBmBDUAAIBm\nBDUAAIBmBDUAAIBmBDUAAIBmBDUAAIBmBDUAAIBmBDUAAIBmBDUAAIBmBDUAAIBmBDUAAIBmTuYu\n4CDUYu4KANhnb/tBcnVx+XHd+0/3+pLpNY4J4/ZhPqaa8tymzOHUfbGfjvx95owaAABAM4IaAABA\nM4IaAABAM4IaAABAM4IaAABAM4IaAABAM4IaAABAM4IaAABAM4IaAABAM4IaAABAM4IaAABAM4Ia\nAABAM4IaAABAMydzFwAAR+/a3UktLj9uTBgzZT/7YpfzsQ/z2H197MMc7oMpr3OyH/O/DzVukTNq\nAAAAzQhqAAAAzQhqAAAAzQhqAAAAzQhqAAAAzQhqAAAAzQhqAAAAzQhqAAAAzQhqAAAAzQhqAAAA\nzQhqAAAAzQhqAAAAzQhqAAAAzZzMXQAAMFEt5q7g1sZi2ripz6v7fOya+dg/h/qa7fpYcCCcUQMA\nAGhGUAMAAGhGUAMAAGhGUAMAAGhGUAMAAGhGUAMAAGhGUAMAAGhGUAMAAGhGUAMAAGhGUAMAAGhG\nUAMAAGhGUAMAAGjmZO4CAIADVYu5K9iesbj8mEOeDzbDuuIUZ9QAAACaEdQAAACaOTeoVdW9VfXl\nqvp2VX2rqj6w3P7Gqnqyqr6z/Hnb9ssFgB70RwC26SJn1H6W5E/HGG9J8m+S/ElVvSXJh5J8aYzx\n5iRfWv4OAMdCfwRga84NamOMF8YYX1/e/mmSZ5Lck+TBJI8vH/Z4kndvq0gA6EZ/BGCbLvUZtaq6\nL8lbk3w1yZ1jjBeWd/0wyZ0brQwA9oT+CMCmXTioVdXrk3w2yQfHGD85fd8YYyQZZ4x7pKquVtXV\n5OW1igWAbvRHALbhQkGtql6b603o02OMzy03v1hVdy3vvyvJSzcbO8Z4dIxxZYxxJbljEzUDQAv6\nIwDbcpFvfawkn0zyzBjjY6fueiLJw8vbDyf5wubLA4Ce9EcAtunkAo/5nSR/lOSbVfXUctuHk3wk\nyd9W1R8n+X6S926nRABoSX8EYGvODWpjjL9PUmfc/bubLQcA9oP+CMA2XepbHwEAANg+QQ0AAKCZ\ni3xGDQCA02oxdwXHayymjduH12wfapziUJ/XljmjBgAA0IygBgAA0IygBgAA0IygBgAA0IygBgAA\n0IygBgAA0IygBgAA0IygBgAA0IygBgAA0IygBgAA0IygBgAA0IygBgAA0MzJ3AUAAHtgLC4/piaM\ngfMc8rryPuMUZ9QAAACaEdQAAACaEdQAAACaEdQAAACaEdQAAACaEdQAAACaEdQAAACaEdQAAACa\nEdQAAACaEdQAAACaEdQAAACaEdQAAACaEdQAAACaOZm7AABgD9Ri7gpg96au+zFxXHdTn5fjxyTO\nqAEAADQjqAEAADQjqAEAADQjqAEAADQjqAEAADQjqAEAADQjqAEAADQjqAEAADQjqAEAADQjqAEA\nADQjqAEAADQjqAEAADQjqAEAADRzMncBAAA0MRbTxtXEcd1NnY99cMjP7UA4owYAANCMoAYAANCM\noAYAANCMoAYAANCMoAYAANCMoAYAANCMoAYAANCMoAYAANCMoAYAANCMoAYAANCMoAYAANCMoAYA\nANDMydwFAA3UYu4KAOjgkPvBWFx+zCHPxyE/twPhjBoAAEAzghoAAEAz5wa1qrq3qr5cVd+uqm9V\n1QeW2xdV9XxVPbX8867tlwsAPeiPAGzTRT6j9rMkfzrG+HpVvSHJtap6cnnfX4wx/nx75QFAW/oj\nAFtzblAbY7yQ5IXl7Z9W1TNJ7tl2YQDQmf4IwDZd6jNqVXVfkrcm+epy0/ur6htV9VhV3bbh2gBg\nL+iPAGzahYNaVb0+yWeTfHCM8ZMkH0/ypiT35/r/UfzoGeMeqaqrVXU1eXkDJQNAH/ojANtwoaBW\nVa/N9Sb06THG55JkjPHiGOPnY4xfJPlEkgduNnaM8egY48oY40pyx6bqBoDZ6Y8AbMtFvvWxknwy\nyTNjjI+d2n7XqYe9J8nTmy8PAHrSHwHYpot86+PvJPmjJN+sqqeW2z6c5KGquj/JSPJskvdtpUIA\n6El/BGBrLvKtj3+fpG5y1xc3Xw4A7Af9EYBtutS3PgIAALB9ghoAAEAzF/mMGgBw7Mbi8mNqwhg4\nz5S1mExbj1P3NdWhvmd2+ZodEGfUAAAAmhHUAAAAmhHUAAAAmhHUAAAAmhHUAAAAmhHUAAAAmhHU\nAAAAmhHUAAAAmhHUAAAAmhHUAAAAmhHUAAAAmhHUAAAAmjmZuwAAYKKxuPyYmjBmnXGHasrcT3Wo\ncz91Dnc5H4c697s2dR53eYxryBk1AACAZgQ1AACAZgQ1AACAZgQ1AACAZgQ1AACAZgQ1AACAZgQ1\nAACAZgQ1AACAZgQ1AACAZgQ1AACAZgQ1AACAZgQ1AACAZgQ1AACAZmqMsbudVb2c5Ptn3H17kh/t\nrJj+zMcq87HKfLzCXKzqMh//aoxxx9xF7Av98VLMxyrzscp8rDIfq7rMx4V65E6D2q1U1dUxxpW5\n6+jCfKwyH6vMxyvMxSrzcXi8pqvMxyrzscp8rDIfq/ZtPlz6CAAA0IygBgAA0EynoPbo3AU0Yz5W\nmY9V5uMV5mKV+Tg8XtNV5mOV+VhlPlaZj1V7NR9tPqMGAADAdZ3OqAEAAJAGQa2q3llV/1BV362q\nD81dz9yq6tmq+mZVPVVVV+euZ9eq6rGqeqmqnj617Y1V9WRVfWf587Y5a9ylM+ZjUVXPL9fIU1X1\nrjlr3KWqureqvlxV366qb1XVB5bbj3KN3GI+jnaNHBo9cpUeqUeepke+Qn9cdSj9cdZLH6vqNUn+\nb5LfS/Jckq8leWiM8e3ZippZVT2b5MoYo8O/8bBzVfVvk/xTkv82xvit5bb/muTHY4yPLP+ictsY\n4z/NWeeunDEfiyT/NMb48zlrm0NV3ZXkrjHG16vqDUmuJXl3kv+QI1wjt5iP9+ZI18gh0SNfTY/U\nI0/TI1+hP646lP449xm1B5J8d4zxvTHGPyf5myQPzlwTMxpjfCXJj2/Y/GCSx5e3H8/1N9pROGM+\njtYY44UxxteXt3+a5Jkk9+RI18gt5oPDoEeyQo9cpUe+Qn9cdSj9ce6gdk+Sfzz1+3PZw0ncsJHk\n76rqWlU9MncxTdw5xnhhefuHSe6cs5gm3l9V31he9nEUlzHcqKruS/LWJF+NNXLjfCTWyCHQI19N\nj3y1oz/+3cRRH//0x1X73B/nDmq82tvHGPcn+YMkf7I8rc/SuH6t7rF/VenHk7wpyf1JXkjy0XnL\n2b2qen2Szyb54BjjJ6fvO8Y1cpP5OPo1wsHSI2/hGI9/N3HUxz/9cdW+98e5g9rzSe499fuvL7cd\nrTHG88ufLyX5fK5f+nLsXlxea/zLa45fmrmeWY0xXhxj/HyM8Yskn8iRrZGqem2uH3Q/Pcb43HLz\n0a6Rm83Hsa+RA6JH3kCPvKmjPf7dzDEf//THVYfQH+cOal9L8uaq+s2q+pUkf5jkiZlrmk1VvW75\ngcdU1euS/H6Sp2896ig8keTh5e2Hk3xhxlpm98sD7tJ7ckRrpKoqySeTPDPG+Nipu45yjZw1H8e8\nRg6MHnmKHnmmozz+neVYj3/646pD6Y+z/4PXy6/F/Mskr0ny2Bjjv8xa0Iyq6k25/n8Ik+QkyV8f\n23xU1WeSvCPJ7UleTPJnSf57kr9N8htJvp/kvWOMo/jw8Bnz8Y5cP2U/kjyb5H2nrj8/aFX19iT/\nK8k3k/xiufnDuX7d+dGtkVvMx0M50jVyaPTIV+iReuSN9MhX6I+rDqU/zh7UAAAAWDX3pY8AAADc\nQFADAABoRlADAABoRlADAABoRlADAABoRlADAABoRlADAABoRlADAABo5v8DBRanvVwq8H8AAAAA\nSUVORK5CYII=\n", 192 | "text/plain": [ 193 | "" 194 | ] 195 | }, 196 | "metadata": {}, 197 | "output_type": "display_data" 198 | } 199 | ], 200 | "source": [ 201 | "# Let's pick one the image '4', and corrupt it.\n", 202 | "image = data[1]\n", 203 | "\n", 204 | "# Corrupt the image with bernouli noise.\n", 205 | "corrupted_image = corrupt_input(image, method='bernouli', bernouli_noise_probability=0.1)\n", 206 | "\n", 207 | "# Let's see the original image, as well as the corrupted version.\n", 208 | "fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(15,7.5))\n", 209 | "ax[0].imshow(image, interpolation='None', cmap='winter')\n", 210 | "ax[1].imshow(corrupted_image, interpolation='None', cmap='winter')\n", 211 | "plt.show()\n" 212 | ] 213 | }, 214 | { 215 | "cell_type": "markdown", 216 | "metadata": {}, 217 | "source": [ 218 | "This is the moment of truth: With a corrupted input now at hand, we then present this to our trained net, and hope that it denoises it. We do so by picking pixels at random, and then evaluating their activation functions, based on the current input configuration, (corrupted image), and the set weights, (that do not change here, as the net has already been trained). \n", 219 | "\n", 220 | "Since the Hopfield net stores patterns as minimal enery states, every iteration where we update a neuron/pixel should place the net into a lower energy configuration - that is - closer to the original pattern we stored - the weights are designed to do so after all." 221 | ] 222 | }, 223 | { 224 | "cell_type": "code", 225 | "execution_count": 83, 226 | "metadata": { 227 | "collapsed": false 228 | }, 229 | "outputs": [ 230 | { 231 | "data": { 232 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2oAAAGrCAYAAACxAGQzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X2QbHdZJ/DvYwZESBaCiTEJIRFFMbqSLHejlizLiiCw\n6yaoi0ZXg6JhqwCxtFyQtXR2RUSLAFu1iCRFTFx58QWQrAVqQApEUbnJRgiJSNBkSXLzggEJ+IIJ\nv/3jnGt6bu6dO9Mz3f3r6c+nqut2n5c+T//mTD/32+ecnmqtBQAAgH58waILAAAAYCNBDQAAoDOC\nGgAAQGcENQAAgM4IagAAAJ0R1AAAADojqMECVdVlVfWSRdcBAD2qqmdV1fsWXQcsgqBG16rqe6pq\nf1V9pqoOVNU7qurxi67roKpqVfUVM3puzQmADarqxqr6+7Ev3jZ+4HfsoutahKo6Y+zDa4uuBWZB\nUKNbVfVjSV6V5KVJTkryyCSvTvIfp3iu+72Je2MHYEl9W2vt2CRnJTk7yU8uuJ4t0XdhewQ1ulRV\nD03yP5I8t7X2ltbaZ1tr/9Ra+53W2n8dl/nCqnpVVd063l5VVV84zntiVd1cVS+sqtuS/MoRpt3v\nqNXkUbLxk8pfrqorq+ruqnpPVZ0+znvvuMqfj59sftc4/T9U1TVV9amq+uOq+rqJ5z67qq4en+vX\nkzxoG2NyY1X9RFV9sKo+W1Wvq6qTxqOMd1fVO6vq+Inlf3P8tPVvq+q9VfU1E/O+uKr+T1V9uqo+\nUFUvmRyHqnrM+JrvqqqPVNUzt1onAPPRWrstye9lCGxJ/rk3vryq/l9V3T72sC+amH/u2KM+XVUf\nq6qnjtNPqaorxvf9G6rqhyem/31VPXziOc6uqk9U1QPGxz9YVddX1Ser6vcO9slxXquq51bVR5N8\ndJx2xB4z9qcrxvr+LMmXb3U8xp79S2Nf/ExV/VFVfen4/4NPVtVfVNXZE8u/aByDu6vquqp6xsS8\nY6rqovF1/nVVPa8mjt5V1UPHPnygqm4Z++gxW60VtkJQo1ffmCHEvHWTZf5bkm/I0KAem+ScJD81\nMf9Lkzw8yelJLtxk2tF8b5KfTXJCkmuSvD5JWmtPGOc/trV2bGvt18cGcGmS5yT54iSvTXLF2Dgf\nmOS3k/zvsYbfTPIdW6zhoO9I8uQkX5nk25K8I8mLk5yY4ff5RyaWfUeSRyf5kiRXH6x79Ookn80w\nHheMtyRJVT0kyZVJ3jCu+91JfqmqztxmrQDMUFU9IsnTktwwMfllGXrEWUm+IsmpSX56XP6cJL+a\n5CeSPCzJE5LcOK73piQ3JzklyXcmeWlVfXNr7dYk78/GfvU9SX6rtfZPVXVuhj707Rl60R8meeMh\npZ6X5OuTnLmFHvPqJP+Q5OQkPzjetuOZGf4vcEKSfxxrv3p8/FtJXjGx7MeS/JskD03y35P8WlWd\nPM774Qxje1aSfzW+hkmXJbknwxifneQpSX5om7XC5lprbm7d3TKEo9uOsszHkjx94vG3JrlxvP/E\nJJ9L8qCJ+Yeb9qwk7zvkeVuSrxjvX5bkTRPzjk1yb5LTDl12fPyaJD97yPN9JMm/zdAQb01SE/P+\nOMlLjvD6NtSWoZl+78TjNyd5zcTj5yf57SM818PGWh+a5Jgk/5Tkqybmv+TgtpJ8V5I/PGT91yb5\nmUXvF25ubm6rfht7wWeS3D2+r78rycPGeZXhQ7gvn1j+G5P89Xj/tUleeZjnPG3sbcdNTPv5JJeN\n938oyR9MbOPjSZ4wPn5HkmdPrPcFSf4uyenj45bkmyfmH7HHTPSnx0zMe+mhfXpi3hnj86+Njy9L\ncsnE/OcnuX7i8b9M8qlNxvaaJOeO9/8gyXMm5n3LwW1luBzjH5N80cT885O8e9H7h9veujmiRq/+\nJskJtfn57KckuWni8U3jtIPubK39wyHrHG7a0Xz84J3W2meS3HXIdiadnuTHx9MeP1VVn8rQAE8Z\nb7e01tohNW/H7RP3//4wj49N/vmUjZeNp3R8Ovd9YnpChk881yZf1yH3T0/y9Ye8hu/NcPQNgMU7\nr7V2XIYPIB+T4b09Gd7fH5zkqon3798dpydDP/rYYZ7vlCR3tdbunph2U4ajccnwweA3jkebnpDk\n8xmOnCVDz/ifE9u7K0OYO3XiubbaYw7Xn2bSJ5Okqr6/7rtU4VNJvjb3jeUp2bxPPiDJgYl1X5vh\nCCHsGhd10qv3Z/i06rwMpyoczq0Z3iw/PD5+5DjtoHa/Ne4/7bMZmlqSpKoOF0ZOm5h/bIbTFm89\nzHLJ8Eb+c621nzt0RlX92ySnVlVNhLVH5vBNc6e+J8m5GT4BvDHDkbRPZmied2Y4XeMRSf5yXP60\niXU/nuQ9rbUnz6AuAHZJa+09VXVZkpdn6JefyBBGvqa1dsthVvl4Dn/N161JHl5Vx02EtUcmuWXc\nzier6vczHA376gxnmhzsYwf73usPfdLJUg+p4bA9ZrzG654MPekvJurYdeN1dJckeVKS97fW7q2q\nazL0ySQ5kKFPHnRon/zHJCe01u6ZRX2QuEaNTrXW/jbDOfWvrqrzqurBVfWAqnpaVf3iuNgbk/xU\nVZ1YVSeMy//aNjf150m+pqrOqqoHJVk/zDJPr6rHj9eY/WySP2mtHfxk7fYkj5pY9pIk/6Wqvr4G\nD6mqf19Vx2UIn/ck+ZHxtXx7huvqZuG4DE3kbzIE0ZcenNFauzfJW5Ksj+P6mCTfP7Hu7yT5yqr6\nvrHOB1TVv66qr55RrQBM71VJnlxVj22tfT5DH3plVX1JklTVqVX1reOyr0vyA1X1pKr6gnHeY8ae\n9sdJfr6qHlTDl2A9Oxt76hsy9IrvHO8f9MtJfrLGL6wav2TjP21S7xF7zGH605mZuIZ6lz0kQ4C8\nc6z7BzIcUTvoN5K8YByjhyV54cEZrbUDSX4/yUVV9S/Gsfzy8QNZ2DWCGt1qrV2U5McyXBR8Z4ZP\nsJ6X4Qs5kuG6qv1JPpjkQxkuFt7WH49urf1lhm+XfGeGb6M63N8te0OGc+fvSvK4JP95Yt56ksvH\nUx+e2Vrbn+EC5P+V4QjWDRmuNUtr7XMZLrZ+1vhc35WhIc3Cr2Y4XeSWJNcl+ZND5j8vw1G22zJ8\nuckbMwS7jJ+mPiXDBd63jsv8QpIvnFGtAEyptXZnhvf8nx4nvTBD7/mT8dT3dyb5qnHZP0vyA0le\nmeRvk7wnw5kpyXCN1RkZ3vffmuG65HdObOqKDF9QdVtr7c8ntv/WDD3iTeP2rs3wJRxHqvdoPeZ5\nGU5PvC3DNWe/so3h2LLW2nVJLsrwIertGa5f+6OJRS7JEMY+mOT/Jnl7hg9b7x3nf3+SB2bosZ/M\ncPbPyYFdVBsvlwEmjaeU3Nxa+6mjLbvMquoXknxpa21Wn1wCwNKqqqcl+eXW2ulHXRh2iSNqsIJq\n+Bs2XzeennlOhlNcNvtTCACwMqrqi6rq6VW1VlWnZjizRp9krgQ1WE3HZTjt8rNJfj3D6R9vW2hF\nANCPyvC31T6Z4dTH63Pf6aUwF059BAAA6IwjagAAAJ2Z699RqxMe3HLGw7a/4lVH+tvCMGePO9Kf\nT9vEtPvvNNvayfb2qnn+zJhwY1r7RB19OZKk6oQ2fOEeAHvfVZ9orZ14tKXm+wevz3hYsv/C7a9X\n67teCkxl//r215l2/51mWzvZ3l41z58ZE/YtuoAlc0aGvzYCwN5XN21lKac+AgAAdEZQAwAA6MyO\nglpVPbWqPlJVN1TVi3arKABYdnokADsxdVCrqmOSvDrJ05KcmeT8qjpztwoDgGWlRwKwUzs5onZO\nkhtaa3/VWvtckjclOXd3ygKApaZHArAjOwlqpyb5+MTjm8dpG1TVhVW1v6r2586/28HmAGBpHLVH\nbuiPuXOuxQHQv5l/mUhr7eLW2r7W2r6c+OBZbw4AlsKG/pij/jkdAFbMToLaLUlOm3j8iHEaAKw6\nPRKAHdlJUPtAkkdX1ZdV1QOTfHeSK3anLABYanokADuyNu2KrbV7qup5SX4vyTFJLm2tfXjXKgOA\nJaVHArBTUwe1JGmtvT3J23epFgDYM/RIAHZi5l8mAgAAwPbs6Ijatl11SlLrc90kLC2/K7vDOAIA\nS8gRNQAAgM4IagAAAJ0R1AAAADojqAEAAHRGUAMAAOiMoAYAANAZQQ0AAKAzghoAAEBnBDUAAIDO\nCGoAAACdEdQAAAA6I6gBAAB0Zm3RBcBSqfVFVwAAwApwRA0AAKAzghoAAEBnBDUAAIDOCGoAAACd\nEdQAAAA6I6gBAAB0RlADAADojKAGAADQGUENAACgM4IaAABAZwQ1AACAzghqAAAAnRHUAAAAOrO2\n6ALoXFvf/jo1xTrzNs3rSpbjtc3TPMfRzwwAWCGOqAEAAHRGUAMAAOiMoAYAANAZQQ0AAKAzghoA\nAEBnBDUAAIDOCGoAAACdEdQAAAA6I6gBAAB0RlADAADojKAGAADQGUENAACgM2uLLoA9qK1Pt15N\nuR4bTTP+0479PH9m89yWfRgAWDBH1AAAADojqAEAAHRGUAMAAOiMoAYAANAZQQ0AAKAzghoAAEBn\nBDUAAIDOCGoAAACdEdQAAAA6I6gBAAB0RlADAADojKAGAADQGUENAACgM2uLLoDO1fr212lTrDNv\n07yuZbFXX9sy7FcAMG/L0B/36v9NZswRNQAAgM4IagAAAJ0R1AAAADqzo2vUqurGJHcnuTfJPa21\nfbtRFAAsOz0SgJ3YjS8T+XettU/swvMAwF6jRwIwFac+AgAAdGanQa0leWdVXVVVFx5ugaq6sKr2\nV9X+5M4dbg4AlsamPVJ/BGAzOz318fGttVuq6kuSXFlVf9Fae+/kAq21i5NcnCRV+9oOtwcAy2LT\nHqk/ArCZHR1Ra63dMv57R5K3JjlnN4oCgGWnRwKwE1MHtap6SFUdd/B+kqckuXa3CgOAZaVHArBT\nOzn18aQkb62qg8/zhtba7+5KVQCw3PRIAHZk6qDWWvurJI/dxVoAYE/QIwHYKV/PDwAA0Jnd+IPX\nW/e4W5P969tfr6ZYZ1ptjtua1jzHYxq918dqmWZ/XIb3AVgG8/5d0n+APcQRNQAAgM4IagAAAJ0R\n1AAAADojqAEAAHRGUAMAAOiMoAYAANAZQQ0AAKAzghoAAEBnBDUAAIDOCGoAAACdEdQAAAA6I6gB\nAAB0Zm2uW7vqlKTW57rJbZumvjbFOrDK5vk74/cTgGWhZzHBETUAAIDOCGoAAACdEdQAAAA6I6gB\nAAB0RlADAADojKAGAADQGUENAACgM4IaAABAZwQ1AACAzghqAAAAnRHUAAAAOiOoAQAAdEZQAwAA\n6MzaogsAjqCtT7deTbke9zGGAHB/+uNcOaIGAADQGUENAACgM4IaAABAZwQ1AACAzghqAAAAnRHU\nAAAAOiOoAQAAdEZQAwAA6IygBgAA0BlBDQAAoDOCGgAAQGcENQAAgM6sLbqAlVbri66Ani3D/tHW\nF11BX6Ydj2X4WQMAc+WIGgAAQGcENQAAgM4IagAAAJ0R1AAAADojqAEAAHRGUAMAAOiMoAYAANAZ\nQQ0AAKAzghoAAEBnBDUAAIDOCGoAAACdEdQAAAA6I6gBAAB0Zm3RBXSnrfe/rZpyPTiSee+L8/w9\nmye/m+xle/X3FqBTjqgBAAB0RlADAADojKAGAADQmaMGtaq6tKruqKprJ6Y9vKqurKqPjv8eP9sy\nAaA/eiQAs7KVI2qXJXnqIdNelORdrbVHJ3nX+BgAVs1l0SMBmIGjBrXW2nuT3HXI5HOTXD7evzzJ\nebtcFwB0T48EYFamvUbtpNbagfH+bUlOOtKCVXVhVe2vqv3JnVNuDgCWxpZ6pP4IwGZ2/GUirbWW\npG0y/+LW2r7W2r7kxJ1uDgCWxmY9Un8EYDPTBrXbq+rkJBn/vWP3SgKApaZHArBj0wa1K5JcMN6/\nIMnbdqccAFh6eiQAO7aVr+d/Y5L3J/mqqrq5qp6d5GVJnlxVH03yLeNjAFgpeiQAs7J2tAVaa+cf\nYdaTdrkWAFgqeiQAs7LjLxMBAABgdx31iBqwAmp9uvXalOsBwF6mP7ILHFEDAADojKAGAADQGUEN\nAACgM4IaAABAZwQ1AACAzghqAAAAnRHUAAAAOiOoAQAAdEZQAwAA6IygBgAA0BlBDQAAoDOCGgAA\nQGfWFl3AlrT17a9TU6wz7XrT1MfqmHb/mHYfnufvy141759Z76YZj3237noZALBKHFEDAADojKAG\nAADQGUENAACgM4IaAABAZwQ1AACAzghqAAAAnRHUAAAAOiOoAQAAdEZQAwAA6IygBgAA0BlBDQAA\noDOCGgAAQGcENQAAgM6szXVrj7s12b++/fVqinWm1abY1rT1TbMtmIW9vC/2/v6RTFfjPLc11Rj+\nzhTrAGzBXu5Z8zTP/shUHFEDAADojKAGAADQGUENAACgM4IaAABAZwQ1AACAzghqAAAAnRHUAAAA\nOiOoAQAAdEZQAwAA6IygBgAA0BlBDQAAoDOCGgAAQGfW5rq1q05Jan2um5yLtr7oCvpiPDaadp83\njrtjmnGc9mc2z/e3vfheCnud93VgGxxRAwAA6IygBgAA0BlBDQAAoDOCGgAAQGcENQAAgM4IagAA\nAJ0R1AAAADojqAEAAHRGUAMAAOiMoAYAANAZQQ0AAKAzghoAAEBnBDUAAIDOrC26AKbQ1hddAdvh\n53V/tT6/bRl/WB1+31lF8+ypzJUjagAAAJ0R1AAAADojqAEAAHTmqEGtqi6tqjuq6tqJaetVdUtV\nXTPenj7bMgGgP3okALOylSNqlyV56mGmv7K1dtZ4e/vulgUAS+Gy6JEAzMBRg1pr7b1J7ppDLQCw\nVPRIAGZlJ9eoPb+qPjie9nH8kRaqqguran9V7U/u3MHmAGBpHLVH6o8AbGbaoPaaJI9KclaSA0ku\nOtKCrbWLW2v7Wmv7khOn3BwALI0t9Uj9EYDNTBXUWmu3t9buba19PsklSc7Z3bIAYDnpkQDshqmC\nWlWdPPHwGUmuPdKyALBK9EgAdsPa0RaoqjcmeWKSE6rq5iQ/k+SJVXVWkpbkxiTPmWGNANAlPRKA\nWTlqUGutnX+Yya+bQS0AsFT0SABmZSff+ggAAMAMHPWI2sqp9e2v06ZYZyeWocZ5mmY8pjXNOM6z\nvmnNe/+Y5zguw/jDMtB7VoP3zI3swyyQI2oAAACdEdQAAAA6I6gBAAB0RlADAADojKAGAADQGUEN\nAACgM4IaAABAZwQ1AACAzghqAAAAnRHUAAAAOiOoAQAAdEZQAwAA6MzaXLf2uFuT/evbX6+mWGee\neq8vmb7GNsV6yzAe05rmtU0zhtNui+Xk94y9yn4KMDVH1AAAADojqAEAAHRGUAMAAOiMoAYAANAZ\nQQ0AAKAzghoAAEBnBDUAAIDOCGoAAACdEdQAAAA6I6gBAAB0RlADAADojKAGAADQGUENAACgM2tz\n3dpVpyS1vv312hTrTLOdZTHP8ViGcex9/1iGMVwG0/yck+UY/2WoEQCYK0fUAAAAOiOoAQAAdEZQ\nAwAA6IygBgAA0BlBDQAAoDOCGgAAQGcENQAAgM4IagAAAJ0R1AAAADojqAEAAHRGUAMAAOiMoAYA\nANAZQQ0AAKAza4suYEtqfdEVbK6tT7fetK+r9/GYN+OxfPbqz2ze7wUAwJ7liBoAAEBnBDUAAIDO\nCGoAAACdEdQAAAA6I6gBAAB0RlADAADojKAGAADQGUENAACgM4IaAABAZwQ1AACAzghqAAAAnRHU\nAAAAOrO26AL2hFpfdAWz09a3v85eHg92h/0KAGBTjqgBAAB0RlADAADojKAGAADQmaMGtao6rare\nXVXXVdWHq+oF4/SHV9WVVfXR8d/jZ18uAPRBfwRglrZyRO2eJD/eWjszyTckeW5VnZnkRUne1Vp7\ndJJ3jY8BYFXojwDMzFGDWmvtQGvt6vH+3UmuT3JqknOTXD4udnmS82ZVJAD0Rn8EYJa2dY1aVZ2R\n5Owkf5rkpNbagXHWbUlOOsI6F1bV/qran9y5g1IBoE/6IwC7bctBraqOTfLmJD/aWvv05LzWWkvS\nDrdea+3i1tq+1tq+5MQdFQsAvdEfAZiFLQW1qnpAhib0+tbaW8bJt1fVyeP8k5PcMZsSAaBP+iMA\ns7KVb32sJK9Lcn1r7RUTs65IcsF4/4Ikb9v98gCgT/ojALO0toVlvinJ9yX5UFVdM057cZKXJfmN\nqnp2kpuSPHM2JQJAl/RHAGbmqEGttfa+JHWE2U/a3XIAYDnojwDM0ra+9REAAIDZ28qpj6yyWl90\nBaurrU+33jL8zJahxmns1dcFAMydI2oAAACdEdQAAAA6I6gBAAB0RlADAADojKAGAADQGUENAACg\nM4IaAABAZwQ1AACAzghqAAAAnRHUAAAAOiOoAQAAdEZQAwAA6MzaogvoTlvf/jo1xTpwNHt5v/J7\nBgCwKUfUAAAAOiOoAQAAdEZQAwAA6IygBgAA0BlBDQAAoDOCGgAAQGcENQAAgM4IagAAAJ0R1AAA\nADojqAEAAHRGUAMAAOiMoAYAANAZQQ0AAKAza4suoDu1vugKYP6m3e/blOv1btrX5f0DANgljqgB\nAAB0RlADAADojKAGAADQGUENAACgM4IaAABAZwQ1AACAzghqAAAAnRHUAAAAOiOoAQAAdEZQAwAA\n6IygBgAA0BlBDQAAoDOCGgAAQGfWFl0ALERbn269mnK93k07HstgL782AGDPckQNAACgM4IaAABA\nZwQ1AACAzghqAAAAnRHUAAAAOiOoAQAAdEZQAwAA6IygBgAA0BlBDQAAoDOCGgAAQGcENQAAgM4I\nagAAAJ1ZW3QBsBC1vugKZqetb3+dvTwee/m1ATBbeggL5IgaAABAZwQ1AACAzghqAAAAnTlqUKuq\n06rq3VV1XVV9uKpeME5fr6pbquqa8fb02ZcLAH3QHwGYpa18mcg9SX68tXZ1VR2X5KqqunKc98rW\n2stnVx4AdEt/BGBmjhrUWmsHkhwY799dVdcnOXXWhQFAz/RHAGZpW9eoVdUZSc5O8qfjpOdX1Qer\n6tKqOv4I61xYVfuran9y546KBYAe6Y8A7LYtB7WqOjbJm5P8aGvt00lek+RRSc7K8IniRYdbr7V2\ncWttX2ttX3LiLpQMAP3QHwGYhS0Ftap6QIYm9PrW2luSpLV2e2vt3tba55NckuSc2ZUJAP3RHwGY\nla1862MleV2S61trr5iYfvLEYs9Icu3ulwcAfdIfAZilrXzr4zcl+b4kH6qqa8ZpL05yflWdlaQl\nuTHJc2ZSIQD0SX8EYGa28q2P70tSh5n19t0vBwCWg/4IwCxt61sfAQAAmL2tnPq4Wtr69tepKdaB\no5lmX0ym2x+n3da09urvzDx/ZgDAnuaIGgAAQGcENQAAgM4IagAAAJ0R1AAAADojqAEAAHRGUAMA\nAOiMoAYAANAZQQ0AAKAzghoAAEBnBDUAAIDOCGoAAACdEdQAAAA6s7boArakrW9/nZpinZ2st1dN\nM/bT2qtjP+0YznM89urYz9u04zjP9zgAYCk4ogYAANAZQQ0AAKAzghoAAEBnBDUAAIDOCGoAAACd\nEdQAAAA6I6gBAAB0RlADAADojKAGAADQGUENAACgM4IaAABAZwQ1AACAzghqAAAAnanW2vw2VnVn\nkpuOMPuEJJ+YWzH9Mx4bGY+NjMd9jMVGvYzH6a21ExddxLLQH7fFeGxkPDYyHhsZj416GY8t9ci5\nBrXNVNX+1tq+RdfRC+OxkfHYyHjcx1hsZDz2Hj/TjYzHRsZjI+OxkfHYaNnGw6mPAAAAnRHUAAAA\nOtNTULt40QV0xnhsZDw2Mh73MRYbGY+9x890I+OxkfHYyHhsZDw2Wqrx6OYaNQAAAAY9HVEDAAAg\nghoAAEB3Fh7UquqpVfWRqrqhql606HoWrapurKoPVdU1VbV/0fXMW1VdWlV3VNW1E9MeXlVXVtVH\nx3+PX2SN83SE8VivqlvGfeSaqnr6Imucp6o6rareXVXXVdWHq+oF4/SV3Ec2GY+V3Uf2Gj1yIz1S\nj5ykR95Hf9xor/THhV6jVlXHJPnLJE9OcnOSDyQ5v7V23cKKWrCqujHJvtZaD3+Mb+6q6glJPpPk\nV1trXztO+8Ukd7XWXjb+R+X41toLF1nnvBxhPNaTfKa19vJF1rYIVXVykpNba1dX1XFJrkpyXpJn\nZQX3kU3G45lZ0X1kL9Ej70+P1CMn6ZH30R832iv9cdFH1M5JckNr7a9aa59L8qYk5y64Jhaotfbe\nJHcdMvncJJeP9y/P8Iu2Eo4wHiurtXagtXb1eP/uJNcnOTUruo9sMh7sDXokG+iRG+mR99EfN9or\n/XHRQe3UJB+feHxzlnAQd1lL8s6quqqqLlx0MZ04qbV2YLx/W5KTFllMJ55fVR8cT/tYidMYDlVV\nZyQ5O8mfxj5y6Hgk9pG9QI+8Pz3y/lb+/e8wVvr9T3/caJn746KDGvf3+NbaWUmeluS542F9Rm04\nV3fV/6bEa5I8KslZSQ4kuWix5cxfVR2b5M1JfrS19unJeau4jxxmPFZ+H2HP0iM3sYrvf4ex0u9/\n+uNGy94fFx3Ubkly2sTjR4zTVlZr7Zbx3zuSvDXDqS+r7vbxXOOD5xzfseB6Fqq1dntr7d7W2ueT\nXJIV20eq6gEZ3nRf31p7yzh5ZfeRw43Hqu8je4geeQg98rBW9v3vcFb5/U9/3Ggv9MdFB7UPJHl0\nVX1ZVT0wyXcnuWLBNS1MVT1kvOAxVfWQJE9Jcu3ma62EK5JcMN6/IMnbFljLwh18wx09Iyu0j1RV\nJXldkutba6+YmLWS+8iRxmOV95E9Ro+coEce0Uq+/x3Jqr7/6Y8b7ZX+uNBvfUyS8WsxX5XkmCSX\nttZ+bqEFLVBVPSrDJ4RJspbkDas2HlX1xiRPTHJCktuT/EyS307yG0kemeSmJM9sra3ExcNHGI8n\nZjhk35LcmOQ5E+ef72lV9fgkf5jkQ0k+P05+cYbzzlduH9lkPM7Piu4je40eeR89Uo88lB55H/1x\no73SHxe1MQf2AAAAOklEQVQe1AAAANho0ac+AgAAcAhBDQAAoDOCGgAAQGcENQAAgM4IagAAAJ0R\n1AAAADojqAEAAHTm/wMuyf4xGG3ezgAAAABJRU5ErkJggg==\n", 233 | "text/plain": [ 234 | "" 235 | ] 236 | }, 237 | "metadata": {}, 238 | "output_type": "display_data" 239 | } 240 | ], 241 | "source": [ 242 | "# (For imaging purposes)\n", 243 | "fig, ax = plt.subplots(nrows = 1, ncols = 2, figsize=(15, 7.5))\n", 244 | "ax[0].imshow(corrupted_image, interpolation='None', cmap='winter')\n", 245 | "ax[0].set_title('Corrupted Image')\n", 246 | "\n", 247 | "# The number of iterations where we randomly update neurons/pixels\n", 248 | "N.iterations = 5000\n", 249 | "\n", 250 | "# This will store the energy of the Hopfield net.\n", 251 | "energy_vec = np.zeros((1,N.iterations))\n", 252 | "\n", 253 | "# Loop through, and recover the image from it's corrupted self. (And vectorize)\n", 254 | "cleaned_image = np.copy(corrupted_image).reshape(1,784)\n", 255 | "for nn in xrange(N.iterations):\n", 256 | "\n", 257 | " # Compute energy:\n", 258 | " energy_vec[0,nn] = energy(cleaned_image, weights)\n", 259 | "\n", 260 | " # Pick a neuron at random to evaluate. \n", 261 | " I.index_to_clean = np.random.randint(N.neurons)\n", 262 | "\n", 263 | " cleaned_image[0,I.index_to_clean] = np.sign(cleaned_image.dot(weights[:,I.index_to_clean]))\n", 264 | "\n", 265 | " # We will show the cleaned_image as it is being cleaned! (Image only every modulo 100 for speed)\n", 266 | " if (nn % 100) == 0:\n", 267 | " ax[1].cla()\n", 268 | " ax[1].imshow(np.reshape(cleaned_image, (28,28)), interpolation='None', cmap='winter')\n", 269 | " ax[1].set_title('Recovered Image')\n" 270 | ] 271 | }, 272 | { 273 | "cell_type": "markdown", 274 | "metadata": {}, 275 | "source": [ 276 | "Our original pattern has been very nicely recovered. What about masking noise _and_ bernouli noise? Let's try that.\n" 277 | ] 278 | }, 279 | { 280 | "cell_type": "code", 281 | "execution_count": 84, 282 | "metadata": { 283 | "collapsed": false 284 | }, 285 | "outputs": [ 286 | { 287 | "data": { 288 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2oAAAGfCAYAAAAu6yGIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFp9JREFUeJzt3U+orHeZJ/DvMx57oy4iZkJMp8cWZEAGJuIhDLQMNv0H\n2010I51Fk4GGuOgRhV60uOnqRYMMrT2bQYgYzIBt06COWcgMaRGchkE8V4JGMz2KRDoxJhEX2qtG\n/c3iltxTyb3nnPvWn/epqs8HLqfOW/Xmfc5Tv1PP/eZ9q26NMQIAAEAf/2ruAgAAAFglqAEAADQj\nqAEAADQjqAEAADQjqAEAADQjqAEAADQjqAEAADQjqAEAADQjqAEAADRzssuDVb1hJG/a5SEBmMUz\nGePHNXcV+2Kn8/HtP5y237U3braOfTelj3q4ylrcP56zDbn24zHGnZc9qsYYu6jm+sHqdCRnOzse\nAHM5zRhngtoV7XQ+jsW0/WrifodqSh/1cJW1uH88ZxtS18YYp5c9yqWPAAAAzawV1KrqXVX1j1X1\nvar68KaKAoB9Z0YCsI7JQa2qXpXkvyX5gyRvTfJgVb11U4UBwL4yIwFY1zpn1O5P8r0xxvfHGP+S\n5G+TPLCZsgBgr5mRAKxlnaB2T5J/Ovf9s8ttAHDszEgA1rL1DxOpqoer6qyqzpKXtn04ANgL5iMA\nF1knqD2X5N5z3//6ctuKMcYjY4zT6x9Beek/FwAAh+DSGWk+AnCRdYLa15O8pap+s6p+LckfJnl8\nM2UBwF4zIwFYy8nUHccYP6+q/5zkfyV5VZJHxxjf3lhlALCnzEgA1jU5qCXJGONLSb60oVoA4GCY\nkQCsY+sfJgIAAMDtEdQAAACaWevSRwBgz9Ri7goOgz4CW+aMGgAAQDOCGgAAQDOCGgAAQDOCGgAA\nQDOCGgAAQDOCGgAAQDOCGgAAQDOCGgAAQDOCGgAAQDOCGgAAQDOCGgAAQDOCGgAAQDMncxcAAOyB\nsbj9fWrCPrzSlN4n/fvfvT5eyXO2U86oAQAANCOoAQAANCOoAQAANCOoAQAANCOoAQAANCOoAQAA\nNCOoAQAANCOoAQAANCOoAQAANCOoAQAANCOoAQAANCOoAQAANCOoAQAANHMydwEAwERjcfv71IR9\n1tlvl3bZD9h3+/D7sg81bpEzagAAAM0IagAAAM0IagAAAM0IagAAAM0IagAAAM0IagAAAM0IagAA\nAM0IagAAAM0IagAAAM0IagAAAM0IagAAAM0IagAAAM3UGGN3B6vTkZzt7HgAzOU0Y5zV3FXsi53O\nx7GYtl9N3A/A687L1LUxxullj3JGDQAAoBlBDQAAoBlBDQAAoBlBDQAAoBlBDQAAoBlBDQAAoBlB\nDQAAoBlBDQAAoBlBDQAAoBlBDQAAoBlBDQAAoBlBDQAAoBlBDQAAoJmTuQsAuJKxmLuCy9Vi7grg\nctYp27DL12hreJX5eLCcUQMAAGhGUAMAAGhmrUsfq+qZJD9L8oskPx9jnG6iKADYd2YkAOvYxHvU\nfnuM8eMN/HcA4NCYkQBM4tJHAACAZtYNaiPJ31fVtap6eBMFAcCBMCMBmGzdSx/fMcZ4rqr+dZIn\nqur/jjG+ev4By+G0HFC/sebhAGBvXDgjzUcALrLWGbUxxnPLry8m+UKS+2/ymEfGGKfX30R95zqH\nA4C9cdmMNB8BuMjkoFZVr6mq1/3qdpLfT/LUpgoDgH1lRgKwrnUufbwryReq6lf/nb8ZY/zPjVQF\nAPvNjARgLZOD2hjj+0n+/QZrAYCDYEYCsC4fzw8AANCMoAYAANDMuh/PT5KMxW6PVzs+HgBMYT7u\nn6nPmd6z76as/S2ve2fUAAAAmhHUAAAAmhHUAAAAmhHUAAAAmhHUAAAAmhHUAAAAmhHUAAAAmhHU\nAAAAmhHUAAAAmhHUAAAAmhHUAAAAmhHUAAAAmjmZuwDgCI3F3BUAXOf1aDP0cf/U4vb3mfo873J9\nTPm51tlvi5xRAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQ\nAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAAaOZk7gIA2qnF3BUAuzL1931M3I/1HfJzNuVnm/pz\n7UM/jpwzagAAAM0IagAAAM0IagAAAM0IagAAAM0IagAAAM0IagAAAM0IagAAAM0IagAAAM0IagAA\nAM0IagAAAM0IagAAAM0IagAAAM2czF0AADDRWNz+PjVhn0M2pYf7YspzvQ/92Icap9rl7/Sh9nHq\nz9XwtdEZNQAAgGYENQAAgGYENQAAgGYENQAAgGYENQAAgGYENQAAgGYENQAAgGYENQAAgGYENQAA\ngGYENQAAgGYENQAAgGYENQAAgGYENQAAgGZO5i6gnbGYuwIAuJpa7O5Yhzofp/ZwH/qxyxqn9HEf\nergP9HHVLl8Xt8wZNQAAgGYENQAAgGYuDWpV9WhVvVhVT53b9vqqeqKqvrv8esd2ywSAfsxIALbl\nKmfUPp3kXS/b9uEkXx5jvCXJl5ffA8Cx+XTMSAC24NKgNsb4apKfvGzzA0keW95+LMl7NlwXALRn\nRgKwLVPfo3bXGOP55e0fJblrQ/UAwL4zIwFY29ofJjLGGEnGre6vqoer6qyqzpKX1j0cAOyNi2ak\n+QjARaYGtReq6u4kWX598VYPHGM8MsY4HWOcJndOPBwA7I0rzUjzEYCLTA1qjyd5aHn7oSRf3Ew5\nALD3zEgA1naVj+f/bJL/k+TfVtWzVfXHST6a5Peq6rtJfnf5PQAcFTMSgG05uewBY4wHb3HX72y4\nFgDYK2YkANuy9oeJAAAAsFmCGgAAQDOXXvoIcEtjMXcFwO2a+ntbE/bzGrF/pjzPied6Tp6zg+WM\nGgAAQDOCGgAAQDOCGgAAQDOCGgAAQDOCGgAAQDOCGgAAQDOCGgAAQDOCGgAAQDOCGgAAQDOCGgAA\nQDOCGgAAQDOCGgAAQDMncxcAAEfv7T9Mzha3v1/taJ8kGRP3A7bL7+aqqf2Y8to4+VhXe5gzagAA\nAM0IagAAAM0IagAAAM0IagAAAM0IagAAAM0IagAAAM0IagAAAM0IagAAAM0IagAAAM0IagAAAM0I\nagAAAM0IagAAAM0IagAAAM2czF0AcAtjMXcFh6EWc1cAl7v2RmuVHnY5e6au+ak1TjmeWbx/dvla\nOvlYf3GlRzmjBgAA0IygBgAA0IygBgAA0IygBgAA0IygBgAA0IygBgAA0IygBgAA0IygBgAA0Iyg\nBgAA0IygBgAA0IygBgAA0IygBgAA0MzJ3AVwgMZi7goA9svbf5icLW5/v5qwD3Sx678v7PJ4U383\n/R2Kc5xRAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAA\naEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAAaOZk7gKYYCzmrgB2rxZzVwDbc+2N1vh5U3thPh6HfVgf\n1uJxmPo819Ue5owaAABAM4IaAABAM5cGtap6tKperKqnzm1bVNVzVfXk8s+7t1smAPRjRgKwLVc5\no/bpJO+6yfa/HmPct/zzpc2WBQB74dMxIwHYgkuD2hjjq0l+soNaAGCvmJEAbMs671H7QFV9c3nZ\nxx0bqwgA9p8ZCcBapga1TyR5c5L7kjyf5GO3emBVPVxVZ1V1lrw08XAAsDeuNCPNRwAuMimojTFe\nGGP8YozxyySfTHL/BY99ZIxxOsY4Te6cWicA7IWrzkjzEYCLTApqVXX3uW/fm+SpWz0WAI6JGQnA\nJpxc9oCq+mySdyZ5Q1U9m+TPk7yzqu5LMpI8k+T9W6wRAFoyIwHYlkuD2hjjwZts/tQWagGAvWJG\nArAt63zqIwAAAFsgqAEAADRTY4zdHaxOR3K2s+PtzFjMXQG3qxZzV9DLPqxhz9meOc0YZzV3FfvC\nfIQdmDJHpq7hqTNrl78zu+zHLu3F3xfq2vVP/L2YM2oAAADNCGoAAADNCGoAAADNCGoAAADNCGoA\nAADNCGoAAADNCGoAAADNCGoAAADNCGoAAADNCGoAAADNCGoAAADNCGoAAADNnMxdwEGoxdwVAEA/\nhzwfx2LuCuhs6vqY8jsz9VjWcHvOqAEAADQjqAEAADQjqAEAADQjqAEAADQjqAEAADQjqAEAADQj\nqAEAADQjqAEAADQjqAEAADQjqAEAADQjqAEAADQjqAEAADQjqAEAADRzMncBAACwdWMxdwWX24ca\nd6kWc1cwK2fUAAAAmhHUAAAAmhHUAAAAmhHUAAAAmhHUAAAAmhHUAAAAmhHUAAAAmhHUAAAAmhHU\nAAAAmhHUAAAAmhHUAAAAmhHUAAAAmhHUAAAAmjmZuwAAYKKxuP19asI+AOycM2oAAADNCGoAAADN\nCGoAAADNCGoAAADNCGoAAADNCGoAAADNCGoAAADNCGoAAADNCGoAAADNCGoAAADNCGoAAADNCGoA\nAADNnMxdAAAwUS3mruBiYzFtv6k/15TjTT1W996zGVPX8C5ZiwfLGTUAAIBmBDUAAIBmLg1qVXVv\nVX2lqr5TVd+uqg8ut7++qp6oqu8uv96x/XIBoAfzEYBtusoZtZ8n+dMxxluT/Ickf1JVb03y4SRf\nHmO8JcmXl98DwLEwHwHYmkuD2hjj+THGN5a3f5bk6ST3JHkgyWPLhz2W5D3bKhIAujEfAdim23qP\nWlW9KcnbknwtyV1jjOeXd/0oyV0brQwA9oT5CMCmXTmoVdVrk3wuyYfGGD89f98YYyQZt9jv4ao6\nq6qz5KW1igWAbsxHALbhSkGtql6d60PoM2OMzy83v1BVdy/vvzvJizfbd4zxyBjjdIxxmty5iZoB\noAXzEYBtucqnPlaSTyV5eozx8XN3PZ7koeXth5J8cfPlAUBP5iMA23Ryhcf8VpI/SvKtqnpyue0j\nST6a5O+q6o+T/CDJ+7ZTIgC0ZD4CsDWXBrUxxj8kqVvc/TubLQcA9oP5CMA23danPgIAALB9ghoA\nAEAzV3mPGgDA7avFYR/vEI3F7o7l+YILOaMGAADQjKAGAADQjKAGAADQjKAGAADQjKAGAADQjKAG\nAADQjKAGAADQjKAGAADQjKAGAADQjKAGAADQjKAGAADQjKAGAADQzMncBQAAe2Asbn+fmrDPIZvS\nw2S3ffScse/24ffsipxRAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQ\nAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAAaEZQAwAAaOZk7gIAgD1Qi90da0w81i5rnGJqfYfa\nj0O26+d6V8fahzW1DzVekTNqAAAAzQhqAAAAzQhqAAAAzQhqAAAAzQhqAAAAzQhqAAAAzQhqAAAA\nzQhqAAAAzQhqAAAAzQhqAAAAzQhqAAAAzQhqAAAAzQhqAAAAzZzMXQAAwN4Zi9vfpybsw36asj52\n7VDX49TeN+yHM2oAAADNCGoAAADNCGoAAADNCGoAAADNCGoAAADNCGoAAADNCGoAAADNCGoAAADN\nCGoAAADNCGoAAADNCGoAAADNCGoAAADNnMxdANBALeauAOCGXb4mjYnH2mWN+rF/9GM+U3s/Ze1v\n+Xl2Rg0AAKAZQQ0AAKCZS4NaVd1bVV+pqu9U1ber6oPL7Yuqeq6qnlz+eff2ywWAHsxHALbpKu9R\n+3mSPx1jfKOqXpfkWlU9sbzvr8cYf7W98gCgLfMRgK25NKiNMZ5P8vzy9s+q6ukk92y7MADozHwE\nYJtu6z1qVfWmJG9L8rXlpg9U1Ter6tGqumPDtQHAXjAfAdi0Kwe1qnptks8l+dAY46dJPpHkzUnu\ny/X/o/ixW+z3cFWdVdVZ8tIGSgaAPsxHALbhSkGtql6d60PoM2OMzyfJGOOFMcYvxhi/TPLJJPff\nbN8xxiNjjNMxxmly56bqBoDZmY8AbMtVPvWxknwqydNjjI+f2373uYe9N8lTmy8PAHoyHwHYpqt8\n6uNvJfmjJN+qqieX2z6S5MGqui/JSPJMkvdvpUIA6Ml8BGBrrvKpj/+QpG5y15c2Xw4A7AfzEYBt\nuq1PfQQAAGD7BDUAAIBmaoyxu4PV6UjOdnY8AOZymjHObnZZIDdhPnKhsZi2X03cD9iyunb9E38v\n5owaAABAM4IaAABAM4IaAABAM4IaAABAM4IaAABAM4IaAABAM4IaAABAM4IaAABAM4IaAABAM4Ia\nAABAM4IaAABAM4IaAABAMydzFwAAwAVqMXcFwAycUQMAAGhGUAMAAGhGUAMAAGhGUAMAAGhGUAMA\nAGhGUAMAAGhGUAMAAGhGUAMAAGhGUAMAAGhGUAMAAGhGUAMAAGhGUAMAAGhGUAMAAGimxhi7O1jV\nS0l+cIu735Dkxzsrpj/9WKUfq/TjBr1Y1aUf/2aMcefcRewL8/G26Mcq/VilH6v0Y1WXflxpRu40\nqF2kqs7GGKdz19GFfqzSj1X6cYNerNKPw+M5XaUfq/RjlX6s0o9V+9YPlz4CAAA0I6gBAAA00ymo\nPTJ3Ac3oxyr9WKUfN+jFKv04PJ7TVfqxSj9W6ccq/Vi1V/1o8x41AAAArut0Rg0AAIA0CGpV9a6q\n+seq+l5VfXjueuZWVc9U1beq6smqOpu7nl2rqker6sWqeurcttdX1RNV9d3l1zvmrHGXbtGPRVU9\nt1wjT1bVu+escZeq6t6q+kpVfaeqvl1VH1xuP8o1ckE/jnaNHBozcpUZaUaeZ0beYD6uOpT5OOul\nj1X1qiT/L8nvJXk2ydeTPDjG+M5sRc2sqp5JcjrG6PBvPOxcVf3HJP+c5L+PMf7dctt/SfKTMcZH\nl39RuWOM8Wdz1rkrt+jHIsk/jzH+as7a5lBVdye5e4zxjap6XZJrSd6T5D/lCNfIBf14X450jRwS\nM/KVzEgz8jwz8gbzcdWhzMe5z6jdn+R7Y4zvjzH+JcnfJnlg5pqY0Rjjq0l+8rLNDyR5bHn7sVz/\nRTsKt+jH0RpjPD/G+Mby9s+SPJ3knhzpGrmgHxwGM5IVZuQqM/IG83HVoczHuYPaPUn+6dz3z2YP\nm7hhI8nfV9W1qnp47mKauGuM8fzy9o+S3DVnMU18oKq+ubzs4yguY3i5qnpTkrcl+VqskZf3I7FG\nDoEZ+Upm5Csd/evfTRz165/5uGqf5+PcQY1XescY474kf5DkT5an9Vka16/VPfaPKv1EkjcnuS/J\n80k+Nm85u1dVr03yuSQfGmP89Px9x7hGbtKPo18jHCwz8gLH+Pp3E0f9+mc+rtr3+Th3UHsuyb3n\nvv/15bajNcZ4bvn1xSRfyPVLX47dC8trjX91zfGLM9czqzHGC2OMX4wxfpnkkzmyNVJVr871F93P\njDE+v9x8tGvkZv049jVyQMzIlzEjb+poX/9u5phf/8zHVYcwH+cOal9P8paq+s2q+rUkf5jk8Zlr\nmk1VvWb5hsdU1WuS/H6Spy7e6yg8nuSh5e2Hknxxxlpm96sX3KX35ojWSFVVkk8leXqM8fFzdx3l\nGrlVP455jRwYM/IcM/KWjvL171aO9fXPfFx1KPNx9n/wevmxmP81yauSPDrG+MtZC5pRVb051/8P\nYZKcJPmbY+tHVX02yTuTvCHJC0n+PMn/SPJ3SX4jyQ+SvG+McRRvHr5FP96Z66fsR5Jnkrz/3PXn\nB62q3pHkfyf5VpJfLjd/JNevOz+6NXJBPx7Mka6RQ2NG3mBGmpEvZ0beYD6uOpT5OHtQAwAAYNXc\nlz4CAADwMoIaAABAM4IaAABAM4IaAABAM4IaAABAM4IaAABAM4IaAABAM4IaAABAM/8fX5XdEejb\nZuYAAAAASUVORK5CYII=\n", 289 | "text/plain": [ 290 | "" 291 | ] 292 | }, 293 | "metadata": {}, 294 | "output_type": "display_data" 295 | } 296 | ], 297 | "source": [ 298 | "# Masking and Bernouli noise\n", 299 | "mask_matrix = np.zeros((2,2)).astype(np.int32)\n", 300 | "mask_matrix[0,0] = 7\n", 301 | "mask_matrix[0,1] = 20\n", 302 | "mask_matrix[1,0] = 8\n", 303 | "mask_matrix[1,1] = 25\n", 304 | "corrupted_image = corrupt_input(image, method='mask', mask_value=1, mask_matrix=mask_matrix)\n", 305 | "corrupted_image = corrupt_input(corrupted_image, method='bernouli', bernouli_noise_probability=0.1)\n", 306 | "fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(15,7.5))\n", 307 | "ax[0].imshow(image, interpolation='None', cmap='winter')\n", 308 | "ax[1].imshow(corrupted_image, interpolation='None', cmap='winter')\n", 309 | "plt.show()\n" 310 | ] 311 | }, 312 | { 313 | "cell_type": "markdown", 314 | "metadata": {}, 315 | "source": [ 316 | "Now that's one ugly looking '4'. Can we recover the hidden pattern?" 317 | ] 318 | }, 319 | { 320 | "cell_type": "code", 321 | "execution_count": 86, 322 | "metadata": { 323 | "collapsed": false 324 | }, 325 | "outputs": [ 326 | { 327 | "data": { 328 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2oAAAGrCAYAAACxAGQzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X2QbHdZJ/DvYyaIkiwEE2MSQhBBMbqSLHejliyyIgjs\nugnqotHVoGjYKkAsLRdkLZ1dEdEiwFYtIkkRE1ZefAEka4EakAJRVG6yEUIiEjQsSW5eMCCBVTHh\nt3+cc0nPzdx56Znu/vXM51PVdbvPS59nfn2mn/vtc05PtdYCAABAP75o0QUAAACwlqAGAADQGUEN\nAACgM4IaAABAZwQ1AACAzghqAAAAnRHUYIGq6rKqetGi6wCAHlXVM6rqvYuuAxZBUKNrVfX9VXWw\nqj5TVYeq6u1V9dhF13VYVbWqesSMnltzAmCNqrqxqv5h7Iu3jh/4Hbfouhahqh429uGVRdcCsyCo\n0a2q+skkr0jy4iQnJ3loklcm+Q9TPNd93sS9sQOwpL6ztXZckrOSnJ3kZxZcz5bou7A9ghpdqqoH\nJvnvSZ7dWntza+2zrbV/bq39Xmvtv4zLfHFVvaKqbhlvr6iqLx7nPb6qbqqq51fVrUl+/SjT7nPU\navIo2fhJ5a9V1ZVVdVdVvbuqzhjnvWdc5S/HTza/d5z+76vqmqr6VFX9aVV9w8Rzn11VV4/P9ZtJ\n7r+NMbmxqn66qj5QVZ+tqtdU1cnjUca7quodVXXCxPK/PX7a+vdV9Z6q+rqJeV9WVf+7qj5dVe+v\nqhdNjkNVPWr8me+sqg9X1dO3WicA89FauzXJH2QIbEm+0BtfWlX/t6puG3vYl0zMP3fsUZ+uqo9W\n1ZPH6adW1RXj+/4NVfVjE9P/oaoePPEcZ1fVJ6rq2PHxj1TV9VX1yar6g8N9cpzXqurZVfWRJB8Z\npx21x4z96Yqxvr9I8lVbHY+xZ//q2Bc/U1V/UlVfMf7/4JNV9VdVdfbE8i8Yx+Cuqrquqp42Me+Y\nqrpo/Dn/tqqeUxNH76rqgWMfPlRVN4999Jit1gpbIajRq2/OEGLessEy/zXJN2VoUI9Ock6Sn52Y\n/xVJHpzkjCQXbjBtMz+Q5BeSnJjkmiSvS5LW2uPG+Y9urR3XWvvNsQFcmuRZSb4syauTXDE2zvsl\n+d0k/2us4beTfPcWazjsu5M8MclXJ/nOJG9P8sIkJ2X4ff7xiWXfnuSRSb48ydWH6x69MslnM4zH\nBeMtSVJVD0hyZZLXj+t+X5Jfraozt1krADNUVQ9J8pQkN0xMfkmGHnFWkkckOS3Jz43Ln5PktUl+\nOsmDkjwuyY3jem9MclOSU5N8T5IXV9W3tdZuSfK+rO1X35/kd1pr/1xV52boQ9+VoRf9cZI3HFHq\neUm+McmZW+gxr0zyj0lOSfIj4207np7h/wInJvmnsfarx8e/k+RlE8t+NMm/SfLAJP8tyW9U1Snj\nvB/LMLZnJflX488w6bIkd2cY47OTPCnJj26zVthYa83NrbtbhnB06ybLfDTJUycef0eSG8f7j0/y\nuST3n5i/3rRnJHnvEc/bkjxivH9ZkjdOzDsuyT1JTj9y2fHxq5L8whHP9+Ek35qhId6SpCbm/WmS\nFx3l51tTW4Zm+gMTj9+U5FUTj5+b5HeP8lwPGmt9YJJjkvxzkq+ZmP+iw9tK8r1J/viI9V+d5OcX\nvV+4ubm57ffb2As+k+Su8X39nUkeNM6rDB/CfdXE8t+c5G/H+69O8vJ1nvP0sbcdPzHtl5JcNt7/\n0SR/NLGNjyd53Pj47UmeObHeFyX5f0nOGB+3JN82Mf+oPWaiPz1qYt6Lj+zTE/MeNj7/yvj4siSX\nTMx/bpLrJx7/yySf2mBsr0ly7nj/j5I8a2Letx/eVobLMf4pyZdMzD8/ybsWvX+47a2bI2r06u+S\nnFgbn89+apKPTTz+2DjtsDtaa/94xDrrTdvMxw/faa19JsmdR2xn0hlJfmo87fFTVfWpDA3w1PF2\nc2utHVHzdtw2cf8f1nl8XPKFUzZeMp7S8enc+4npiRk+8VyZ/LmOuH9Gkm884mf4gQxH3wBYvPNa\na8dn+ADyURne25Ph/f1Lk1w18f79++P0ZOhHH13n+U5Ncmdr7a6JaR/LcDQuGT4Y/ObxaNPjknw+\nw5GzZOgZ/2Nie3dmCHOnTTzXVnvMev1pJn0ySarqh+reSxU+leTrc+9YnpqN++SxSQ5NrPvqDEcI\nYde4qJNevS/Dp1XnZThVYT23ZHiz/ND4+KHjtMPafda477TPZmhqSZKqWi+MnD4x/7gMpy3ess5y\nyfBG/outtV88ckZVfWuS06qqJsLaQ7N+09yp709yboZPAG/McCTtkxma5x0ZTtd4SJK/Hpc/fWLd\njyd5d2vtiTOoC4Bd0lp7d1VdluSlGfrlJzKEka9rrd28ziofz/rXfN2S5MFVdfxEWHtokpvH7Xyy\nqv4ww9Gwr81wpsnhPna4773uyCedLPWIGtbtMeM1Xndn6El/NVHHrhuvo7skyROSvK+1dk9VXZOh\nTybJoQx98rAj++Q/JTmxtXb3LOqDxDVqdKq19vcZzql/ZVWdV1VfWlXHVtVTqupXxsXekORnq+qk\nqjpxXP43trmpv0zydVV1VlXdP8nqOss8taoeO15j9gtJ/qy1dviTtduSPHxi2UuS/Oeq+sYaPKCq\n/l1VHZ8hfN6d5MfHn+W7MlxXNwvHZ2gif5chiL748IzW2j1J3pxkdRzXRyX5oYl1fy/JV1fVD451\nHltV/7qqvnZGtQIwvVckeWJVPbq19vkMfejlVfXlSVJVp1XVd4zLvibJD1fVE6rqi8Z5jxp72p8m\n+aWqun8NX4L1zKztqa/P0Cu+Z7x/2K8l+Zkav7Bq/JKN/7hBvUftMev0pzMzcQ31LntAhgB5x1j3\nD2c4onbYbyV53jhGD0ry/MMzWmuHkvxhkouq6l+MY/lV4weysGsENbrVWrsoyU9muCj4jgyfYD0n\nwxdyJMN1VQeTfCDJBzNcLLytPx7dWvvrDN8u+Y4M30a13t8te32Gc+fvTPKYJP9pYt5qksvHUx+e\n3lo7mOEC5P+Z4QjWDRmuNUtr7XMZLrZ+xvhc35uhIc3CazOcLnJzkuuS/NkR85+T4SjbrRm+3OQN\nGYJdxk9Tn5ThAu9bxmV+OckXz6hWAKbUWrsjw3v+z42Tnp+h9/zZeOr7O5J8zbjsXyT54SQvT/L3\nSd6d4cyUZLjG6mEZ3vffkuG65HdMbOqKDF9QdWtr7S8ntv+WDD3ijeP2rs3wJRxHq3ezHvOcDKcn\n3prhmrNf38ZwbFlr7bokF2X4EPW2DNev/cnEIpdkCGMfSPJ/krwtw4et94zzfyjJ/TL02E9mOPvn\nlMAuqrWXywCTxlNKbmqt/exmyy6zqvrlJF/RWpvVJ5cAsLSq6ilJfq21dsamC8MucUQN9qEa/obN\nN4ynZ56T4RSXjf4UAgDsG1X1JVX11KpaqarTMpxZo08yV4Ia7E/HZzjt8rNJfjPD6R9vXWhFANCP\nyvC31T6Z4dTH63Pv6aUwF059BAAA6IwjagAAAJ2Z699RqzqxDV8oNAePOdqfudrEVUf7O8b71DTj\naAzXsi8uH6/ZLrgxrX2iNl+OZM79EYAFu+oTrbWTNltqrqc+Vh1ow7epz0FbnW69mnK9vWqacTSG\na9kXl4/XbBccSGsHBbUtmmt/BGDB6qrW2oHNlnLqIwAAQGcENQAAgM7sKKhV1ZOr6sNVdUNVvWC3\nigKAZadHArATUwe1qjomySuTPCXJmUnOr6ozd6swAFhWeiQAO7WTI2rnJLmhtfY3rbXPJXljknN3\npywAWGp6JAA7spOgdlqSj088vmmctkZVXVhVB6vqYHLHDjYHAEtj0x6pPwKwkZl/mUhr7eLW2oHh\nKyg3/XMBALAv6I8AbGQnQe3mJKdPPH7IOA0A9js9EoAd2UlQe3+SR1bVV1bV/ZJ8X5IrdqcsAFhq\neiQAO7Iy7Yqttbur6jlJ/iDJMUkuba19aNcqA4AlpUcCsFNTB7Ukaa29LcnbdqkWANgz9EgAdmLm\nXyYCAADA9uzoiFrXanXRFewNxhEAAObOETUAAIDOCGoAAACdEdQAAAA6I6gBAAB0RlADAADojKAG\nAADQGUENAACgM4IaAABAZwQ1AACAzghqAAAAnRHUAAAAOiOoAQAAdGZl0QV0p61uf52aYh3ua5qx\nT/of/97r4768ZgDAgjmiBgAA0BlBDQAAoDOCGgAAQGcENQAAgM4IagAAAJ0R1AAAADojqAEAAHRG\nUAMAAOiMoAYAANAZQQ0AAKAzghoAAEBnBDUAAIDOCGoAAACdWVl0AVvSVre/Tk2xzk7Wm6d5jgcs\nu2X4fVmGGgGAuXJEDQAAoDOCGgAAQGcENQAAgM4IagAAAJ0R1AAAADojqAEAAHRGUAMAAOiMoAYA\nANAZQQ0AAKAzghoAAEBnBDUAAIDOCGoAAACdqdba/DZWB1pycD4ba6vTrVdTrgfgfWfCgbR2sBZd\nxbKYa38EYMHqqtbagc2WckQNAACgM4IaAABAZwQ1AACAzghqAAAAnRHUAAAAOiOoAQAAdEZQAwAA\n6IygBgAA0BlBDQAAoDOCGgAAQGcENQAAgM4IagAAAJ0R1AAAADqzsugCZqZWF10Be1Fbnd+27MNr\nzXPsp+U1A2De9Mc9yxE1AACAzghqAAAAnRHUAAAAOrOja9Sq6sYkdyW5J8ndrbUDu1EUACw7PRKA\nndiNLxP5t621T+zC8wDAXqNHAjAVpz4CAAB0ZqdBrSV5R1VdVVUXrrdAVV1YVQer6mByxw43BwBL\nY8MeqT8CsJGdnvr42NbazVX15UmurKq/aq29Z3KB1trFSS5OkqoDbYfbA4BlsWGP1B8B2MiOjqi1\n1m4e/709yVuSnLMbRQHAstMjAdiJqYNaVT2gqo4/fD/Jk5Jcu1uFAcCy0iMB2KmdnPp4cpK3VNXh\n53l9a+33d6UqAFhueiQAOzJ1UGut/U2SR+9iLQCwJ+iRAOyUr+cHAADozG78wWva6ny3V3Pe3l40\n7Wtm7Fl20+z79numpT8CTM0RNQAAgM4IagAAAJ0R1AAAADojqAEAAHRGUAMAAOiMoAYAANAZQQ0A\nAKAzghoAAEBnBDUAAIDOCGoAAACdEdQAAAA6I6gBAAB0ZmXRBTAnbXXRFewNxnH51Or215n2dZ7n\n/jHNz7WT9QCYPf/PYIIjagAAAJ0R1AAAADojqAEAAHRGUAMAAOiMoAYAANAZQQ0AAKAzghoAAEBn\nBDUAAIDOCGoAAACdEdQAAAA6I6gBAAB0RlADAADojKAGAADQmZVFF8Cc1Op067Up12Pn9vJrNs3P\nNu3PtQzjAQDLYNr/mzAVR9QAAAA6I6gBAAB0RlADAADojKAGAADQGUENAACgM4IaAABAZwQ1AACA\nzghqAAAAnRHUAAAAOiOoAQAAdEZQAwAA6IygBgAA0JmVRRewJW11++vUFOvsZdOM4bKY5rVehvFY\nhhqnNc/f6b06jtP+XN4bAWApOKIGAADQGUENAACgM4IaAABAZwQ1AACAzghqAAAAnRHUAAAAOiOo\nAQAAdEZQAwAA6IygBgAA0BlBDQAAoDOCGgAAQGcENQAAgM4IagAAAJ1ZWXQBW1Kr89tWm+O25mna\nMVyG8ZhnjdOM4zKM4TIwjmvN830REr+DAHPmiBoAAEBnBDUAAIDOCGoAAACd2TSoVdWlVXV7VV07\nMe3BVXVlVX1k/PeE2ZYJAP3RIwGYla0cUbssyZOPmPaCJO9srT0yyTvHxwCw31wWPRKAGdg0qLXW\n3pPkziMmn5vk8vH+5UnO2+W6AKB7eiQAszLtNWont9YOjfdvTXLy0Rasqgur6mBVHUzumHJzALA0\nttQj9UcANrLjLxNprbUkbYP5F7fWDrTWDiQn7XRzALA0NuqR+iMAG5k2qN1WVackyfjv7btXEgAs\nNT0SgB2bNqhdkeSC8f4FSd66O+UAwNLTIwHYsa18Pf8bkrwvyddU1U1V9cwkL0nyxKr6SJJvHx8D\nwL6iRwIwKyubLdBaO/8os56wy7UAwFLRIwGYlR1/mQgAAAC7a9MjakurrU63Xk2x3rTbYnGmeZ0T\nr/Uiec0AWBZ6D7vAETUAAIDOCGoAAACdEdQAAAA6I6gBAAB0RlADAADojKAGAADQGUENAACgM4Ia\nAABAZwQ1AACAzghqAAAAnRHUAAAAOiOoAQAAdGZlrlt7zC3JwdXtr1dzWidJ2pTrAbPld3Otacdj\nmvfGabZ14JbtrwMAfIEjagAAAJ0R1AAAADojqAEAAHRGUAMAAOiMoAYAANAZQQ0AAKAzghoAAEBn\nBDUAAIDOCGoAAACdEdQAAAA6I6gBAAB0RlADAADojKAGAADQmZW5bu2qU5NanesmYV1tdX7bmnaf\nn7bGabY3z/Fgd8zzvXSqbf3eblcBMNCzdof/k3fPETUAAIDOCGoAAACdEdQAAAA6I6gBAAB0RlAD\nAADojKAGAADQGUENAACgM4IaAABAZwQ1AACAzghqAAAAnRHUAAAAOiOoAQAAdGZlrlt7zC3JwdXt\nr1dTrAO9aKt7d3vT/m7Oe0wAeuC9D9gGR9QAAAA6I6gBAAB0RlADAADojKAGAADQGUENAACgM4Ia\nAABAZwQ1AACAzghqAAAAnRHUAAAAOiOoAQAAdEZQAwAA6IygBgAA0BlBDQAAoDMrc93aVacmtTrX\nTXZt2rFoU67HclmG/cO+uD9M8zofuGXXy4BNeU9iP/J/6z3LETUAAIDOCGoAAACdEdQAAAA6s2lQ\nq6pLq+r2qrp2YtpqVd1cVdeMt6fOtkwA6I8eCcCsbOWI2mVJnrzO9Je31s4ab2/b3bIAYClcFj0S\ngBnYNKi11t6T5M451AIAS0WPBGBWdnKN2nOr6gPjaR8nHG2hqrqwqg5W1cHkjh1sDgCWxqY9Un8E\nYCPTBrVXJXl4krOSHEpy0dEWbK1d3Fo70Fo7kJw05eYAYGlsqUfqjwBsZKqg1lq7rbV2T2vt80ku\nSXLO7pYFAMtJjwRgN0wV1KrqlImHT0ty7dGWBYD9RI8EYDesbLZAVb0hyeOTnFhVNyX5+SSPr6qz\nkrQkNyZ51gxrBIAu6ZEAzMqmQa21dv46k18zg1oAYKnokQDMyk6+9REAAIAZqNba/DZWB1pycG7b\nm5u2uugK4F61uv11pt2Hp9nWTrY3jXmOxzxNO/ZzcyCtHaxFV7Es9Ee60f17y5wtwz7sNVtCddXw\njb8bc0QNAACgM4IaAABAZwQ1AACAzghqAAAAnRHUAAAAOiOoAQAAdEZQAwAA6IygBgAA0BlBDQAA\noDOCGgAAQGcENQAAgM4IagAAAJ1ZWXQBe0KtLrqC2Wmri66Ank27f0zzOzPttuzDsDh7uT8CzJgj\nagAAAJ0R1AAAADojqAEAAHRGUAMAAOiMoAYAANAZQQ0AAKAzghoAAEBnBDUAAIDOCGoAAACdEdQA\nAAA6I6gBAAB0RlADAADojKAGAADQmZVFFwDssra66Ao2tww1zlOtLroCAKAzjqgBAAB0RlADAADo\njKAGAADQGUENAACgM4IaAABAZwQ1AACAzghqAAAAnRHUAAAAOiOoAQAAdEZQAwAA6IygBgAA0BlB\nDQAAoDOCGgAAQGdWFl3AlrTV7a9TU6wDAADQAUfUAAAAOiOoAQAAdEZQAwAA6IygBgAA0BlBDQAA\noDOCGgAAQGcENQAAgM4IagAAAJ0R1AAAADojqAEAAHRGUAMAAOiMoAYAANCZlUUXsCW1uugKNtZW\np1tv2p9rmu1Nu63ex57dMe0+PE/2RQBgH3FEDQAAoDOCGgAAQGcENQAAgM5sGtSq6vSqeldVXVdV\nH6qq543TH1xVV1bVR8Z/T5h9uQDQB/0RgFnayhG1u5P8VGvtzCTflOTZVXVmkhckeWdr7ZFJ3jk+\nBoD9Qn8EYGY2DWqttUOttavH+3cluT7JaUnOTXL5uNjlSc6bVZEA0Bv9EYBZ2tY1alX1sCRnJ/nz\nJCe31g6Ns25NcvJR1rmwqg5W1cHkjh2UCgB90h8B2G1bDmpVdVySNyX5idbapyfntdZakrbeeq21\ni1trB1prB5KTdlQsAPRGfwRgFrYU1Krq2AxN6HWttTePk2+rqlPG+ackuX02JQJAn/RHAGZlK9/6\nWElek+T61trLJmZdkeSC8f4FSd66++UBQJ/0RwBmaWULy3xLkh9M8sGqumac9sIkL0nyW1X1zCQf\nS/L02ZQIAF3SHwGYmU2DWmvtvUnqKLOfsLvlAMBy0B8BmKVtfesjAAAAs7eVUx/ZTK3u7e3tRW11\nftvyegEAsE2OqAEAAHRGUAMAAOiMoAYAANAZQQ0AAKAzghoAAEBnBDUAAIDOCGoAAACdEdQAAAA6\nI6gBAAB0RlADAADojKAGAADQGUENAACgMyuLLqA7bXX769QU6+xl04xhMt9x9Jqx7Jbh9wwAmJoj\nagAAAJ0R1AAAADojqAEAAHRGUAMAAOiMoAYAANAZQQ0AAKAzghoAAEBnBDUAAIDOCGoAAACdEdQA\nAAA6I6gBAAB0RlADAADojKAGAADQmZVFF9CdWp3fttqU25pnjdOYtr69Oh572bxf63ltaxn2qWWo\nEQCYmiNqAAAAnRHUAAAAOiOoAQAAdEZQAwAA6IygBgAA0BlBDQAAoDOCGgAAQGcENQAAgM4IagAA\nAJ0R1AAAADojqAEAAHRGUAMAAOiMoAYAANCZlUUXQOfa6vbXqSnWYTlNs3/M217dH6cd+706HgCw\nxziiBgAA0BlBDQAAoDOCGgAAQGcENQAAgM4IagAAAJ0R1AAAADojqAEAAHRGUAMAAOiMoAYAANAZ\nQQ0AAKAzghoAAEBnBDUAAIDOrCy6gH2tVue3rTbltuZZo/FYPsZjcaYd+2n2fa8zsF95/2OBHFED\nAADojKAGAADQGUENAACgM5sGtao6vareVVXXVdWHqup54/TVqrq5qq4Zb0+dfbkA0Af9EYBZ2sqX\nidyd5Kdaa1dX1fFJrqqqK8d5L2+tvXR25QFAt/RHAGZm06DWWjuU5NB4/66quj7JabMuDAB6pj8C\nMEvbukatqh6W5Owkfz5Oem5VfaCqLq2qE46yzoVVdbCqDiZ37KhYAOiR/gjAbttyUKuq45K8KclP\ntNY+neRVSR6e5KwMnyhetN56rbWLW2sHWmsHkpN2oWQA6If+CMAsbCmoVdWxGZrQ61prb06S1tpt\nrbV7WmufT3JJknNmVyYA9Ed/BGBWtvKtj5XkNUmub629bGL6KROLPS3JtbtfHgD0SX8EYJa28q2P\n35LkB5N8sKquGae9MMn5VXVWkpbkxiTPmkmFANAn/RGAmdnKtz6+N0mtM+ttu18OACwH/RGAWdrW\ntz4CAAAwe9Vam9/G6kBLDs5teyyZtjrdejXlesAMHUhrB9c72sQ69EeA/aSuGr7xd2OOqAEAAHRG\nUAMAAOiMoAYAANAZQQ0AAKAzghoAAEBnBDUAAIDOCGoAAACdEdQAAAA6I6gBAAB0RlADAADojKAG\nAADQGUENAACgMyuLLgC+oFYXXQEAAHTBETUAAIDOCGoAAACdEdQAAAA6I6gBAAB0RlADAADojKAG\nAADQGUENAACgM4IaAABAZwQ1AACAzghqAAAAnRHUAAAAOiOoAQAAdEZQAwAA6Ey11ua3sao7knzs\nKLNPTPKJuRXTP+OxlvFYy3jcy1is1ct4nNFaO2nRRSwL/XFbjMdaxmMt47GW8Virl/HYUo+ca1Db\nSFUdbK0dWHQdvTAeaxmPtYzHvYzFWsZj7/GarmU81jIeaxmPtYzHWss2Hk59BAAA6IygBgAA0Jme\ngtrFiy6gM8ZjLeOxlvG4l7FYy3jsPV7TtYzHWsZjLeOxlvFYa6nGo5tr1AAAABj0dEQNAACACGoA\nAADdWXhQq6onV9WHq+qGqnrBoutZtKq6sao+WFXXVNXBRdczb1V1aVXdXlXXTkx7cFVdWVUfGf89\nYZE1ztNRxmO1qm4e95Frquqpi6xxnqrq9Kp6V1VdV1UfqqrnjdP35T6ywXjs231kr9Ej19Ij9chJ\neuS99Me19kp/XOg1alV1TJK/TvLEJDcleX+S81tr1y2sqAWrqhuTHGit9fDH+Oauqh6X5DNJXtta\n+/px2q8kubO19pLxPyontNaev8g65+Uo47Ga5DOttZcusrZFqKpTkpzSWru6qo5PclWS85I8I/tw\nH9lgPJ6efbqP7CV65H3pkXrkJD3yXvrjWnulPy76iNo5SW5orf1Na+1zSd6Y5NwF18QCtdbek+TO\nIyafm+Ty8f7lGX7R9oWjjMe+1Vo71Fq7erx/V5Lrk5yWfbqPbDAe7A16JGvokWvpkffSH9faK/1x\n0UHttCQfn3h8U5ZwEHdZS/KOqrqqqi5cdDGdOLm1dmi8f2uSkxdZTCeeW1UfGE/72BenMRypqh6W\n5Owkfx77yJHjkdhH9gI98r70yPva9+9/69jX73/641rL3B8XHdS4r8e21s5K8pQkzx4P6zNqw7m6\n+/1vSrwqycOTnJXkUJKLFlvO/FXVcUnelOQnWmufnpy3H/eRdcZj3+8j7Fl65Ab24/vfOvb1+5/+\nuNay98dFB7Wbk5w+8fgh47R9q7V28/jv7UnekuHUl/3utvFc48PnHN++4HoWqrV2W2vtntba55Nc\nkn22j1TVsRnedF/XWnvzOHnf7iPrjcd+30f2ED3yCHrkuvbt+9969vP7n/641l7oj4sOau9P8siq\n+sqqul+S70tyxYJrWpiqesB4wWOq6gFJnpTk2o3X2heuSHLBeP+CJG9dYC0Ld/gNd/S07KN9pKoq\nyWuSXN9ae9nErH25jxxtPPbzPrLH6JET9Mij2pfvf0ezX9//9Me19kp/XOi3PibJ+LWYr0hyTJJL\nW2u/uNCCFqiqHp7hE8IkWUny+v02HlX1hiSPT3JiktuS/HyS303yW0kemuRjSZ7eWtsXFw8fZTwe\nn+GQfUtyY5JnTZx/vqdV1WOT/HGSDyb5/Dj5hRnOO993+8gG43F+9uk+stfokffSI/XII+mR99If\n19or/XGkLKPwAAAAOklEQVThQQ0AAIC1Fn3qIwAAAEcQ1AAAADojqAEAAHRGUAMAAOiMoAYAANAZ\nQQ0AAKAzghoAAEBn/j+QHzSteERxagAAAABJRU5ErkJggg==\n", 329 | "text/plain": [ 330 | "" 331 | ] 332 | }, 333 | "metadata": {}, 334 | "output_type": "display_data" 335 | } 336 | ], 337 | "source": [ 338 | "# (For imaging purposes)\n", 339 | "fig, ax = plt.subplots(nrows = 1, ncols = 2, figsize=(15, 7.5))\n", 340 | "ax[0].imshow(corrupted_image, interpolation='None', cmap='winter')\n", 341 | "ax[0].set_title('Corrupted Image')\n", 342 | "\n", 343 | "# The number of iterations where we randomly update neurons/pixels\n", 344 | "N.iterations = 5000\n", 345 | "\n", 346 | "# This will store the energy of the Hopfield net.\n", 347 | "energy_vec = np.zeros((1,N.iterations))\n", 348 | "\n", 349 | "# Loop through, and recover the image from it's corrupted self. (And vectorize)\n", 350 | "cleaned_image = np.copy(corrupted_image).reshape(1,784)\n", 351 | "for nn in xrange(N.iterations):\n", 352 | "\n", 353 | " # Compute energy:\n", 354 | " energy_vec[0,nn] = energy(cleaned_image, weights)\n", 355 | "\n", 356 | " # Pick a neuron at random to evaluate. \n", 357 | " I.index_to_clean = np.random.randint(N.neurons)\n", 358 | "\n", 359 | " # Update the neuron in question, based on it's activation.\n", 360 | " cleaned_image[0,I.index_to_clean] = np.sign(cleaned_image.dot(weights[:,I.index_to_clean]))\n", 361 | "\n", 362 | " # We will show the cleaned_image as it is being cleaned! (Image only every modulo 100 for speed)\n", 363 | " if (nn % 100) == 0:\n", 364 | " ax[1].cla()\n", 365 | " ax[1].imshow(np.reshape(cleaned_image, (28,28)), interpolation='None', cmap='winter')\n", 366 | " ax[1].set_title('Recovered Image')\n" 367 | ] 368 | }, 369 | { 370 | "cell_type": "markdown", 371 | "metadata": {}, 372 | "source": [ 373 | "Marvelously recovered. The Hopfield energy plotted as the recovery iteration index is shown below. We can see that as the recovery process was being run, the total net energy was being decreased." 374 | ] 375 | }, 376 | { 377 | "cell_type": "code", 378 | "execution_count": 93, 379 | "metadata": { 380 | "collapsed": false 381 | }, 382 | "outputs": [ 383 | { 384 | "data": { 385 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZYAAAD8CAYAAABU4IIeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt8VdWd9/HPLxcS5BJIuF+UcFMRFQUj1ltUCoydGWjr\nhdZW+oyVtjqt1nlatbbaaulTO9Pap9OndhxlKrZTZLQWxooI1UxrC3IRBEGRcJNLlEtCIEAggd/z\nx1lpjzEQyDnJPjnn+3699oudtfda+f36ovzce+29l7k7IiIiyZIVdQAiIpJeVFhERCSpVFhERCSp\nVFhERCSpVFhERCSpVFhERCSpVFhERCSpVFhERCSpVFhERCSpcqIOIAo9evTwQYMGtajvgQMH6NSp\nU3IDSnHKOTMo5/SXaL7Lly/f7e49mzsvIwvLoEGDWLZsWYv6lpWVUVpamtyAUpxyzgzKOf0lmq+Z\nbTmZ83QrTEREkkqFRUREkkqFRUREkkqFRUREkkqFRUREkkqFRUREkkqFRUREkkqF5RQ8t2Ibb+yq\njzoMEZGUpsJyCh7877U8svww7h51KCIiKUuF5RRMKTkdgNXbqyOOREQkdamwnIKPX9AfgC/98vWI\nIxERSV0qLKdgeO8uXNArm+17D3HoyNGowxERSUkqLKfogl7ZAFz68MuaaxERaYIKyym6qE8OxT06\nUXngCDWH9YSYiEhjKiynqGOOcVvpEAAqDxyJOBoRkdSjwtICvbrmAzBz0UktTSAiklFUWFrgksFF\nADzx6iZ27quNOBoRkdSiwtICHXKy+PrEMwFYv7Mm4mhERFKLCksLjR/RG4Bnl2+LOBIRkdSiwtJC\nZxR1AuA3K7ZzuF7vtIiINFBhaaHc7Cxuujj2iZd5q9+LOBoRkdShwpKAO8cNB+Drz67i6DG9LCki\nAiosCenZJY/zBxRwpP4YK96tijocEZGUoMKSoAcnjQTg1fLdEUciIpIaVFgSdE6/rgD8eOF6jul2\nmIiICkuicrKz+Pvz+wGwtmJfxNGIiERPhSUJbr7kDAD26NthIiIqLMlQ2KkDANurDkUciYhI9BIq\nLGb2kJmtMrOVZvaSmfWLO3avmZWb2TozmxDXPtrMVodjPzEzC+15ZvZ0aH/NzAbF9ZlqZuvDNjWu\nvTicWx76dkgkn5bq2SUPQE+GiYiQ+BXLP7v7ee4+CngeuB/AzEYAU4BzgInAz8wsO/R5FLgVGBa2\niaH9FqDK3YcCjwAPh7EKgQeAi4ES4AEz6x76PAw8EvpUhTHaXJf8XPoW5LO16mAUv15EJKUkVFjc\nPX62uhPQ8FjUJGCWux92901AOVBiZn2Bru6+2GPLL84EJsf1eTLsPwNcE65mJgAL3L3S3auABcDE\ncOzqcC6hb8NYbe4jQ3rwVsV+vSgpIhkv4TkWM5tuZluBmwhXLEB/YGvcadtCW/+w37j9A33cvR6o\nBopOMFYRsDec23isNnfegAKqD9Xxh3d2RRWCiEhKyGnuBDNbCPRp4tB97j7H3e8D7jOze4F/JHbb\nKuWY2TRgGkDv3r0pKytr0Tg1NTVN9s0/cAyA5/64EnsvkqmeVnO8nNOZcs4MmZZzW+XbbGFx93En\nOdavgBeIFZbtwMC4YwNC2/aw37iduD7bzCwHKAD2hPbSRn3KwrFuZpYTrlrix2oqj8eAxwDGjBnj\npaWlxzv1hMrKyjhe3+8vf4mtdZ0oLb20RWOnqhPlnK6Uc2bItJzbKt9EnwobFvfjJODtsD8XmBKe\n9ComNkm/xN0rgH1mNjbMkdwMzInr0/DE13XAy2EeZj4w3sy6h0n78cD8cOyVcC6hb8NYkejZJY8V\n7+7lSP2xKMMQEYlUonMs3zezN81sFbF/8O8AcPc1wGxgLfAicLu7NyxachvwOLEJ/Q3AvND+BFBk\nZuXAXcA9YaxK4CFgadgeDG0AdwN3hT5FYYzI3HzJIACeX7UjyjBERCLV7K2wE3H3T57g2HRgehPt\ny4CRTbTXAtcfZ6wZwIwm2jcSewQ5Jfzd+f345m/f5McL1/OJCwc030FEJA3pzfskKuiYy9BenXm3\n8iDv7tE7LSKSmVRYkuzrE84E4KuzV0YciYhINFRYkuzKM3vSo3Mey7dUMXvZ1uY7iIikGRWWJMvL\nyebVu68CYMOumoijERFpeyosrSA/N5v+3TqydFNl8yeLiKQZFZZWkpNtvP7uXmrrjjZ/sohIGlFh\naSUTz4l9BefZ17c1c6aISHpRYWklX594FgAVe2sjjkREpG2psLSS7Cyja34OMxdtjjoUEZE2pcLS\niob37sK+2nreqtjX/MkiImlChaUVffma2Dc6pz21TJP4IpIxVFha0eVDezC4Rye2Vh5i6WY9eiwi\nmUGFpRVlZRmP3TwGgIpqTeKLSGZQYWllPbvkAbBow56IIxERaRsqLK2soGMuBR1zeW7FdnbXHI46\nHBGRVqfC0ga+cOVgAH7xp83RBiIi0gZUWNpAw8qSz76+TcsWi0jaU2FpA53zcvjYeX2pqK7Vp/RF\nJO2psLSR7338XAAqqg9FHImISOtSYWkjBR1zKezUgf/3ygbcPepwRERajQpLGxraqzMAVQfrIo5E\nRKT1qLC0oesuHADAMr2FLyJpTIWlDV0ypAiAOW/soO6ong4TkfSUE3UAmWRg4Wn0Lcjnd6sqmLe6\ngvMGdOPJfyihoGNu1KGJiCSNrlja2H/eOpa7J57F+QO7sXLrXl5YXRF1SCIiSaXC0saKe3TiS6VD\nmDVtLAB79JkXEUkzSSksZvZPZuZm1iOu7V4zKzezdWY2Ia59tJmtDsd+YmYW2vPM7OnQ/pqZDYrr\nM9XM1odtalx7cTi3PPTtkIx82kJeTjZ5OVm8/u7eqEMREUmqhAuLmQ0ExgPvxrWNAKYA5wATgZ+Z\nWXY4/ChwKzAsbBND+y1AlbsPBR4BHg5jFQIPABcDJcADZtY99HkYeCT0qQpjtBt5OVls3FUTdRgi\nIkmVjCuWR4CvA/Fv/U0CZrn7YXffBJQDJWbWF+jq7os99pbgTGByXJ8nw/4zwDXhamYCsMDdK929\nClgATAzHrg7nEvo2jNUunD+wGzv2ap0WEUkvCRUWM5sEbHf3Nxod6g/EfxRrW2jrH/Ybt3+gj7vX\nA9VA0QnGKgL2hnMbj9UujOjXlSNHj+mqRUTSSrOPG5vZQqBPE4fuA75B7DZYyjOzacA0gN69e1NW\nVtaicWpqalrct7Huh44C8KPn/sx1w1N3eiiZObcXyjkzZFrObZVvs4XF3cc11W5m5wLFwBth/n0A\n8LqZlQDbgYFxpw8IbdvDfuN24vpsM7McoADYE9pLG/UpC8e6mVlOuGqJH6upPB4DHgMYM2aMl5aW\nHu/UEyorK6OlfRsbW3eU7y95kdq8QkpLxyRlzNaQzJzbC+WcGTIt57bKt8W3wtx9tbv3cvdB7j6I\n2K2oC939PWAuMCU86VVMbJJ+ibtXAPvMbGyYI7kZmBOGnAs0PPF1HfBymIeZD4w3s+5h0n48MD8c\neyWcS+jbMFa7kJ+bzUeGFLH34JGoQxERSZpWeY/F3dcAs4G1wIvA7e5+NBy+DXic2IT+BmBeaH8C\nKDKzcuAu4J4wViXwELA0bA+GNoC7gbtCn6IwRrtS0DGXZVuqKN+5P+pQRESSImmfdAlXLfE/Twem\nN3HeMmBkE+21wPXHGXsGMKOJ9o3EHkFutz46ojfz3nyPtyr2M7RXl6jDERFJmN68j9hlw2LvlC7a\nuCfiSEREkkOFJWLdT4s9DbZogwqLiKQHFZaI5WZnMfGcPmzafSDqUEREkkKFJQUM6tEJgIrqQxFH\nIiKSOBWWFDC8d2zJ4i17DkYciYhI4lRYUsCZfWJPg23W7TARSQMqLCmgb0FHABbryTARSQMqLCmg\nsFMHBvfoxG9X7uB3q7SipIi0byosKeKLpUMAeHLR5kjjEBFJlApLirhhzEAuOL0bb2zVipIi0r6p\nsKSQc/p15XD9MaoP1UUdiohIi6mwpJCR/QoAmPnnzWyt1KPHItI+qbCkkMuG9cAMfrjgHb41582o\nwxERaREVlhQyoPtpvHbvNVxcXMieGq3RIiLtkwpLiunVNZ9eXfOpOVwfdSgiIi2iwpKCuuTnsL9W\nE/gi0j6psKSgLnk57DmgW2Ei0j6psKSg0zrk4A5/3rA76lBERE6ZCksK+ptz+wDwh3dUWESk/VFh\nSUFDe8Y+o79mRzXLt1RGHI2IyKlRYUlBWVnGhad344/rd3PLk8uiDkdE5JSosKSoX08by62XF7P3\nYJ0+8SIi7YoKS4rKy8mmuEfslticldsjjkZE5OSpsKSwq87qCcAf3tkVcSQiIidPhSWF9S3oyOgz\nurPwrZ2UrdsZdTgiIidFhSXFfePaswD43H8sZZ/exheRdiChwmJm3zaz7Wa2MmzXxh2718zKzWyd\nmU2Iax9tZqvDsZ+YmYX2PDN7OrS/ZmaD4vpMNbP1YZsa114czi0PfTskkk8quvD07lxwejcAZi/d\nGnE0IiLNS8YVyyPuPipsLwCY2QhgCnAOMBH4mZllh/MfBW4FhoVtYmi/Bahy96HAI8DDYaxC4AHg\nYqAEeMDMuoc+D4ffPxSoCmOkFTPj2S9+BIDv/u4trdMiIimvtW6FTQJmufthd98ElAMlZtYX6Oru\ni93dgZnA5Lg+T4b9Z4BrwtXMBGCBu1e6exWwAJgYjl0dziX0bRgrrWRlGbdfNQSATz++mN01hyOO\nSETk+JJRWL5sZqvMbEbclUR/IP6+zbbQ1j/sN27/QB93rweqgaITjFUE7A3nNh4r7XzxyiH0K8hn\na+Uh/rxhT9ThiIgcV05zJ5jZQqBPE4fuI3Zb6yHAw58/BP4hmQEmi5lNA6YB9O7dm7KyshaNU1NT\n0+K+ibrnwiy+8gosWbmGrlXvtNnvjTLnqCjnzJBpObdVvs0WFncfdzIDmdm/A8+HH7cDA+MODwht\n28N+4/b4PtvMLAcoAPaE9tJGfcrCsW5mlhOuWuLHaiqPx4DHAMaMGeOlpaXHO/WEysrKaGnfRNUf\nPcZXXpnHtmMFlJaWtNnvjTLnqCjnzJBpObdVvok+FdY37sePAw0Ltc8FpoQnvYqJTdIvcfcKYJ+Z\njQ1zJDcDc+L6NDzxdR3wcpiHmQ+MN7Pu4VbbeGB+OPZKOJfQt2GstJSTnUVhpw6UrdtF3dFjUYcj\nItKkROdYfhAeHV4FXAV8FcDd1wCzgbXAi8Dt7n409LkNeJzYhP4GYF5ofwIoMrNy4C7gnjBWJbHb\nbEvD9mBoA7gbuCv0KQpjpLUbL4pdCH7jN6uJ1VYRkdTS7K2wE3H3z57g2HRgehPty4CRTbTXAtcf\nZ6wZwIwm2jcSewQ5Y0wa1Y9HyzbwX8u3cce4YQzoflrUIYmIfIDevG9nzurTlSemjgHgd6sqIo5G\nROTDVFjaodIzewGw8K33I45EROTDVFjaoewso/tpuew7VM+u/XpZUkRSiwpLO/W35/Vj3fv7uWj6\nQhas1ZWLiKQOFZZ26stXD2X6x2PPQPzry+sjjkZE5K9UWNqpXl3z+XTJ6eRkGau2VUcdjojIX6iw\ntGNmxmfGngHAzn21EUcjIhKjwtLOjR1cCMCPFrTdt8NERE5EhaWd+8jQHgD8ecMefvGnTTy99F2O\n1OtzLyISnYTevJfodc3P5fOXFfP4q5v49n+vBWLzL1eFd11ERNqarljSwH0fO5sV3/oo8+64HID/\n9R9LKZm+kJrD9c30FBFJPhWWNGBmdO/UgbP6dOGhySP5u/P7sXP/YX6vN/NFJAIqLGnEzPjs2DP4\n0pWxZYzvmLWSc789n/vnvKnP7ItIm1FhSUMj+nVl5j+UcNPFp7O/tp6Zi7bw3OvHXQNNRCSpVFjS\n1BXDezL94+ey4lsfBWDxxj0RRyQimUKFJc1179SBXl3y+M2K7bym4iIibUCFJQN8d3Lsm2JbKg9G\nHImIZAIVlgxwyZAiAPYePBJxJCKSCVRYMkDnvNh7sN974W1q645GHI2IpDsVlgxgZlwxvCcA7+p2\nmIi0MhWWDPHFKwcD8OZ2fWJfRFqXCkuGGNKzMwCPLHyHY8c84mhEJJ2psGSIHp3z6JCdxdbKQ5Tv\nqok6HBFJYyosGSI7y5jxuYsAWLq5MuJoRCSdqbBkkHMHFABw33NvskzFRURaiQpLBuman0NJcWzF\nyet+vojH/7gx4ohEJB2psGQQM2P2Fy5h1rSxADz7+natOCkiSZdwYTGzL5vZ22a2xsx+ENd+r5mV\nm9k6M5sQ1z7azFaHYz8xMwvteWb2dGh/zcwGxfWZambrwzY1rr04nFse+nZINJ9MMHZwEZNH9eOt\nin3c/exq/rxhd9QhiUgaSaiwmNlVwCTgfHc/B/iX0D4CmAKcA0wEfmZm2aHbo8CtwLCwTQzttwBV\n7j4UeAR4OIxVCDwAXAyUAA+YWffQ52HgkdCnKowhJ+FHN4zid1+5DID5a7QgmIgkT6JXLF8Cvu/u\nhwHcfWdonwTMcvfD7r4JKAdKzKwv0NXdF7u7AzOByXF9ngz7zwDXhKuZCcACd6909ypgATAxHLs6\nnEvo2zCWNCMryxjRtysAa3dU89TiLRw8oqWMRSRxOQn2Hw5cbmbTgVrgf7v7UqA/sDjuvG2hrS7s\nN24n/LkVwN3rzawaKIpvb9SnCNjr7vVNjPUhZjYNmAbQu3dvysrKTjVXAGpqalrcNxVd0CubFduq\neWNbNds2rueSfh/+K5FuOZ8M5ZwZMi3ntsq32cJiZguBPk0cui/0LwTGAhcBs81scFIjTBJ3fwx4\nDGDMmDFeWlraonHKyspoad9UdMUVznv7avnI91/m2Y3OvZ8u/dA56ZbzyVDOmSHTcm6rfJu9Febu\n49x9ZBPbHGJXCb/xmCXAMaAHsB0YGDfMgNC2Pew3bie+j5nlAAXAnhOMtQfoFs5tPJacpKwso1+3\njvQryGffoXpmLXmXo/rki4gkINE5lt8CVwGY2XCgA7AbmAtMCU96FRObpF/i7hXAPjMbG+ZIbgbm\nhLHmAg1PfF0HvBzmYeYD482se5i0Hw/MD8deCecS+jaMJafozo8O58jRY9zzm9UM+cYLTJu5jC//\negUV1YeiDk1E2plE51hmADPM7E3gCDA1/IO/xsxmA2uBeuB2d29YCOQ24BdAR2Be2ACeAJ4ys3Kg\nkthTZbh7pZk9BCwN5z3o7g2vjd8NzDKz7wIrwhjSAjeMGcj4Eb25f84a3nl/Pxt3H6B8Zw0lxYUf\nuFwUEWlOQoXF3Y8AnznOsenA9CbalwEjm2ivBa4/zlgziBWxxu0biT2CLEnQ7bQO/ORTFwBw8Eg9\nI+6fz7d++ya3jcrjSnfCK0ciIiekN++lSad1yGH8iN4A/GzlYcp36ovIInJyEr0VJmns3z47mpfW\nvs8XnlrOp/59MR2ys+jeqQM/umEUZ/bpEnV4IpKidMUix2VmXDm8Jx89I4erzuxFdraxZsc+Jvz4\nD1qJUkSOS4VFTig/N5ubzs7jn68/nz9+/WoenHQOALOWvhtxZCKSqlRY5JTcfMkguuTn8KvXVFhE\npGkqLHLKLhvaA3e4a/ZK7pq9kmeWb2u+k4hkDE3eyym76eIzeKtiH0s2VVJ14Ah/Kt/NdaMHNN9R\nRDKCrljklF02rAdlX7uKV+++mk9cOID39x1md83hqMMSkRShwiIJGdk/9un9uSt3RByJiKQKFRZJ\nyKVDewDwyrqdxL7mIyKZTnMskpAB3U/jqjN78sq6XRTf+wLn9i/g3AEFHzpv0vn9uHhwUQQRikhb\nU2GRhP3T+DM5f2A3nl66lYrqWiqqaz9wfO/BI7xfXavCIpIhVFgkYSP7FzCyfwF3jhve5PEb/20R\nv397Z5PHRCT9aI5FWl1R5w4A7NqvJ8dEMoEKi7S68SNiK1vPXrY14khEpC2osEirm3xBf/Jzs/jX\nl9cz+qEFjH5oAb/406aowxKRVqLCIm3iO39/DteNHsDEkX3Yc+AIL6x+L+qQRKSVaPJe2sSNF53O\njRfF9vcerGPemxWMevAlAEb2K+ATF/bnExfqszAi6UCFRdrcrVcMpkeY0J+1dCuvlu/m1fLdXHtu\nX/JzsyOOTkQSpcIibW7UwG6MGtgNgO9MGskvF2/hm799k1XbqikpLow4OhFJlAqLRO6ssMzx86t2\nUFF9iJH9CxjSs3PEUYlIS2nyXiJ3dt+u5OdmMXPRFu6YtZKvPr0y6pBEJAG6YpHIdcrL4U93X83e\nQ3X88KV1LHxrJ9f//M9cNrQnN409nR6d86IOUUROgQqLpISiznkUdc5jykWnU3WgjkUb97B0cxWP\nLHyH/z1+OP949bCoQxSRk6RbYZJSrhjek19PG8vyb47jmx87G4BfLn434qhE5FSosEhKKuqcx+cv\nH8zYwYW8t6+W6kN1UYckIicpoVthZvY0cGb4sRuw191HhWP3ArcAR4GvuPv80D4a+AXQEXgBuMPd\n3czygJnAaGAPcKO7bw59pgLfDL/nu+7+ZGgvBmYBRcBy4LPufiSRnCS1TB7Vn8UbK5k2cxkdO2Qz\n9ZJBZGfZB84Z3LMTA7qfFlGEItJYQoXF3W9s2DezHwLVYX8EMAU4B+gHLDSz4e5+FHgUuBV4jVhh\nmQjMI1aEqtx9qJlNAR4GbjSzQuABYAzgwHIzm+vuVeGcR9x9lpn9PIzxaCI5SWq5dGgPSooLefu9\n/VQfqqNs3a4PnTOsV2cW3HVlBNGJSFOSMnlvZgbcAFwdmiYBs9z9MLDJzMqBEjPbDHR198Wh30xg\nMrHCMgn4duj/DPDTMO4EYIG7V4Y+C4CJZjYr/L5Phz5Phv4qLGlkYOFpzP7CJbg7a3bs43D90Q8c\n/48/beb5VRXMWvIuU0pOjyhKEYmXrKfCLgfed/f14ef+wOK449tCW13Yb9ze0GcrgLvXm1k1sVtc\nf2lv1KeI2K23+ibGkjRjZozs/+Elj/fV1vP8qgrun7tGhUUkRTRbWMxsIdCniUP3ufucsP8p4NfJ\nDCzZzGwaMA2gd+/elJWVtWicmpqaFvdtr1I5ZwP+pjiXeZvquPXR+Vw3rAN5OdZsv+akcs6tRTmn\nv7bKt9nC4u7jTnTczHKATxCbdG+wHRgY9/OA0LY97Dduj++zLYxZQGwSfztQ2qhPWTjWzcxywlVL\n/FhN5fEY8BjAmDFjvLS09HinnlBZWRkt7dtepXrOp51RyYpfr2DBllqmjhvNZcN6JDxmqufcGpRz\n+murfJPxuPE44G13j7/FNReYYmZ54cmtYcASd68A9pnZ2DB/cjMwJ67P1LB/HfCyuzswHxhvZt3N\nrDswHpgfjr0SziX0bRhLMkhJcSG//HwJAJ954jW+98JbEUckktmSUVim0Og2mLuvAWYDa4EXgdvD\nE2EAtwGPA+XABmIT9wBPAEVhov8u4J4wViXwELA0bA82TOQDdwN3hT5FYQzJQEN6dubbfzeCgYUd\nWbj2/ajDEcloCU/eu/vnjtM+HZjeRPsyYGQT7bXA9ccZawYwo4n2jUDJqUUs6cjM+NylxTy/qoIN\nu2qiDkcko+nNe0kr5w/sRtXBOo7UH4s6FJGMpcIiaaWgYy4Aq7fvjTgSkcylwiJp5aJBsRUoP/no\nIv5UvjviaEQykwqLpJWS4kIe/uS5AMxf8x6H649Sf1S3xUTakgqLpJXsLOMTF8ZelZq5aAtnfvNF\nzv32S2zfeyjiyEQyhwqLpJ3c7Cx+e/ulfG3CmXz64tM5VHeUWUvepbbuaPOdRSRhWkFS0tKogd0Y\nNbAbFdWHmL10K//6cjlDenZm8gX6nJxIa9MVi6S1vgUdKftaKQC7aw5HG4xIhtAVi6S9fgUdyTL4\n+f9s4Jnl28jLyeKHN4xiaK/OUYcmkpZ0xSJpLyvL+Mo1wxh9Rnf6FOTzxrZqXlr7XtRhiaQtXbFI\nRrhz3HAADh05ytn3v8gPXlzH4o2V5GY1/Yn93XtqeWrzUgA+MrQHt1xW3GaxirR3KiySUTp2yObh\nT57LzEVbqDxw/DmX/Yedo/tr2V51iDU79nHTxbFFxPJzs9sqVJF2S4VFMs6NF53OjRedeLXJ2LoV\nl/N/XniLf/vDRs761osADOnZiW9cezbnDehGUacOZB3nikckk6mwiJzA5y4dRGGnDhxzeGrRZjbs\nOsAtTy4DoFOHbF75Wim9uuRHG6RIilFhETmBvgUd+cKVQwD4UukQ3n5vH0s3V/HLRVtY9/5+Lv3+\ny6yffm3EUYqkFj0VJnIKzurTlc+OPYP5X72CkuJC6o46B4/URx2WSErRFYtIC33ywv4s2VTJiPvn\n06NzHp+//MRPjmWbMemCfrp1JmlPhUWkhSZf0J+K6lp+t6qC9Ttr+P68t5vtc6juKF+5ZlgbRCcS\nHRUWkRbKy8nmznHDuXPccGrrjuJ+4vNLvreQpxZv4fdv7/xL2/Benfnn689v5UhF2pbmWESSID83\nm44dTrxNu3wwI/p2pVvHXLp1zGV/bR3/tXwbh+v11WVJL7piEWkjX250C2zmos3cP2cNZet2MeGc\nPtEEJdIKVFhEIlI6vBewhv9+YwcVcQuRXXN2bwYWnhZdYCIJUmERiUi/bvn06JzH86sqeH5VxV/a\n1+zYp3kXaddUWEQikpOdxat3X8WhI3+dY/nUvy/m5bd3ctPjixMae/Ko/lw/ZmCiIYq0iAqLSITy\nc7M/8GHLKRcN5PlVFRyuO9biMde9v5/aumP87Xn9mj3XTB/WlORTYRFJIZ+7tJjPXZrYJ/pv/9Xr\n/G51BWff/+JJnf+D687jBl3dSBIlVFjMbBTwcyAfqAduc/cl4di9wC3AUeAr7j4/tI8GfgF0BF4A\n7nB3N7M8YCYwGtgD3Ojum0OfqcA3w6/9rrs/GdqLgVlAEbAc+Ky7H0kkJ5H27qsfHca5AwpO6twf\nvfQOT/xxE9ePHtDKUUkmSfSK5QfAd9x9npldG34uNbMRwBTgHKAfsNDMhrv7UeBR4FbgNWKFZSIw\nj1gRqnL3oWY2BXgYuNHMCoEHgDGAA8vNbK67V4VzHnH3WWb28zDGownmJNKuDe3VhaG9upzUuU+F\nj2meff9MzNgyAAAF9ElEQVSLFOY5nV7/n1aOLrUcOHgwo3I+cPAgs849yOlFrfvUYaKFxYGuYb8A\n2BH2JwGz3P0wsMnMyoESM9sMdHX3xQBmNhOYTKywTAK+Hfo/A/zUzAyYACxw98rQZwEw0cxmAVcD\nnw59ngz9VVhETtLcf7yUB+au4Zg7O3fuolevzlGH1KZ27jyUUTnv3HmIDjmt/158ooXlTmC+mf0L\nsbf4PxLa+wPxj7VsC211Yb9xe0OfrQDuXm9m1cRucf2lvVGfImCvu9c3MZaInISiznn89NMXAg2L\nm42OOKK2lWk5l5WV0aeg9T+C2mxhMbOFQFOvBd8HXAN81d2fNbMbgCeAcckNMTnMbBowDaB3796U\nlZW1aJyampoW922vlHNmUM7pr63ybbawuPtxC0W4lXVH+PG/gMfD/nYg/jGTAaFte9hv3B7fZ5uZ\n5RC7tbYntJc26lMWjnUzs5xw1RI/VlN5PAY8BjBmzBgvLS093qknFPsvnJb1ba+Uc2ZQzumvrfJN\n9GbbDuDKsH81sD7szwWmmFleeHJrGLDE3SuAfWY2Nsyf3AzMieszNexfB7zs7g7MB8abWXcz6w6M\nB+aHY6+Ecwl9G8YSEZGIJDrHcivwf8MVRi3hVpO7rzGz2cBaYo8h3x6eCAO4jb8+bjwvbBC7jfZU\nmOivJPZUGe5eaWYPAUvDeQ82TOQDdwOzzOy7wIowhoiIRCihwuLurxJ776SpY9OB6U20LwNGNtFe\nC1x/nLFmADOaaN8IlJxa1CIi0pq0HouIiCSVCouIiCSVCouIiCSVeXMLdachM9sFbGlh9x7A7iSG\n0x4o58ygnNNfovme4e49mzspIwtLIsxsmbuPiTqOtqScM4NyTn9tla9uhYmISFKpsIiISFKpsJy6\nx6IOIALKOTMo5/TXJvlqjkVERJJKVywiIpJUKiwnycwmmtk6Mys3s3uijicRZjbDzHaa2ZtxbYVm\ntsDM1oc/u8cduzfkvc7MJsS1jzaz1eHYT8KHRVOSmQ00s1fMbK2ZrTGzO0J72uZtZvlmtsTM3gg5\nfye0p23OAGaWbWYrzOz58HNa5wtgZptDvCvNbFloiy5vd9fWzAZkAxuAwUAH4A1gRNRxJZDPFcCF\nwJtxbT8A7gn79wAPh/0RId88oDj875Adji0BxgJG7GOifxN1bifIuS9wYdjvArwTckvbvEN8ncN+\nLrHlwMemc84h1ruA/wSez4S/2yHezUCPRm2R5a0rlpNTApS7+0Z3PwLMIraUcrvk7n8g9gXpeJOI\nLe9M+HNyXPssdz/s7puAhmWm+xKWmfbY38iZcX1SjrtXuPvrYX8/8BaxFUfTNm+PqQk/5obNSeOc\nzWwA8DH+ujYUpHG+zYgsbxWWk3O85ZHTSW+PrZcD8B7QO+wfL/f+HH+Z6ZRmZoOAC4j9F3xa5x1u\nC60EdgIL3D3dc/4x8HXgWFxbOufbwIGFZrbcYqvlQoR5J7oei6Qhd3czS8vHBc2sM/AscKe774u/\nhZyOeXtsHaRRZtYNeM7MRjY6njY5m9nfAjvdfbmZlTZ1Tjrl28hl7r7dzHoBC8zs7fiDbZ23rlhO\nzvGWWk4n74dLYcKfO0N7S5aZTklmlkusqPzK3X8TmtM+bwB330tsxdWJpG/OlwJ/b2abid2uvtrM\nfkn65vsX7r49/LkTeI7Y7fvI8lZhOTlLgWFmVmxmHYitbjk34piSLX5p6PhlnluyzHTKCTE+Abzl\n7j+KO5S2eZtZz3Clgpl1BD4KvE2a5uzu97r7AHcfROz/oy+7+2dI03wbmFknM+vSsE9s+fY3iTLv\nqJ9maC8bcC2xJ4k2APdFHU+CufwaqADqiN1HvQUoAn4PrAcWAoVx598X8l5H3FMiwJjwF3gD8FPC\nC7epuAGXEbsPvQpYGbZr0zlv4DxiS3avCvHeH9rTNue4eEv561NhaZ0vsadV3wjbmoZ/n6LMW2/e\ni4hIUulWmIiIJJUKi4iIJJUKi4iIJJUKi4iIJJUKi4iIJJUKi4iIJJUKi4iIJJUKi4iIJNX/B9Gi\n+zbUsLSQAAAAAElFTkSuQmCC\n", 386 | "text/plain": [ 387 | "" 388 | ] 389 | }, 390 | "metadata": {}, 391 | "output_type": "display_data" 392 | } 393 | ], 394 | "source": [ 395 | "# Plot the Hopfield energy during recovery\n", 396 | "plt.plot(energy_vec.T); plt.grid('on'); plt.show()" 397 | ] 398 | }, 399 | { 400 | "cell_type": "markdown", 401 | "metadata": { 402 | "collapsed": false 403 | }, 404 | "source": [ 405 | "\n", 406 | "Now this is exactly what it means to be 'content addressable' - the fact that we are able to 'snap back' to a previous pattern among many patterns, starting from a corrupted variant was perhaps one of the first success stories of the associationist movement - Hopefield nets - totally connected RNNs, where the relationships between neurons/pixels are impressed into the weights as minimum energy states. In doing so, the iterative changing of neural activations becomes exactly like finding a configuration of least energy - something that is found in nature all the time.\n" 407 | ] 408 | } 409 | ], 410 | "metadata": { 411 | "kernelspec": { 412 | "display_name": "Python 2", 413 | "language": "python", 414 | "name": "python2" 415 | }, 416 | "language_info": { 417 | "codemirror_mode": { 418 | "name": "ipython", 419 | "version": 2 420 | }, 421 | "file_extension": ".py", 422 | "mimetype": "text/x-python", 423 | "name": "python", 424 | "nbconvert_exporter": "python", 425 | "pygments_lexer": "ipython2", 426 | "version": "2.7.13" 427 | } 428 | }, 429 | "nbformat": 4, 430 | "nbformat_minor": 2 431 | } 432 | --------------------------------------------------------------------------------