├── .gitignore ├── LICENSE ├── README.md └── notebooks ├── 10_MNIST_Reccurent_Neural_Network.ipynb ├── 13_Generative_Adversarial_Network.ipynb ├── 14_DCGAN_with_MNIST.ipynb ├── 1_MNIST_Softmax_Regression.ipynb ├── 2_MNIST_Convolutional_Network.ipynb ├── 3_MNIST_CNN_Layer_Visualization.ipynb ├── 4_Save_Restore_Model.ipynb ├── 5_Autoencoder.ipynb ├── 6_Sparse_Autoencoder.ipynb ├── 7_Convolutional_Autoencoder.ipynb ├── 8_Denoising_Autoencoder.ipynb ├── 9_Variational_Autoencoder.ipynb ├── Logistic_Regression.ipynb ├── images ├── 10_Reccurent_Neural_Network │ ├── gradient.jpg │ ├── rnn_cell.jpg │ └── rnn_expand.jpg ├── 11_Char_RNN │ ├── lstm_inner1.jpg │ ├── lstm_inner2.jpg │ ├── lstm_inner3.jpg │ └── lstm_inner4.jpg ├── 12_word2vec │ ├── skip_gram_example.jpg │ └── word2vec_example.gif ├── 13_Generative_Adversarial_Network │ ├── GAN_algorithm.jpg │ └── GAN_result.gif └── 14_DCGAN │ ├── dcgan.gif │ └── dcgan.jpg └── libs ├── __init__.py └── utils.py /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | *.pyc 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | License 2 | 3 | The MIT License (MIT) 4 | 5 | Copyright (c) 2016 c1mone 6 | 7 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 8 | 9 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 10 | 11 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 12 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Tensorflow-101 2 | 3 | 嗨！在這裡我把[Tensorflow 官網教學](http://tensorflow.org)翻譯成中文以及我自己在 ipython 的實作程式碼記錄在 Jupyter Notebook 裡，歡迎大家取用． 4 | 5 | 0. [Logistic Regression](https://github.com/c1mone/Tensorflow-101/blob/master/notebooks/Logistic_Regression.ipynb) 6 | 7 | 1. [Softmax Regressions with MNIST](https://github.com/c1mone/Tensorflow-101/blob/master/notebooks/1_MNIST_Softmax_Regression.ipynb) 8 | 2. [Convolutional Network with MNIST](https://github.com/c1mone/Tensorflow-101/blob/master/notebooks/2_MNIST_Convolutional_Network.ipynb) 9 | 3. [CNN layer visualization](https://github.com/c1mone/Tensorflow-101/blob/master/notebooks/3_MNIST_CNN_Layer_Visualization.ipynb) 10 | 4. [Save and Restore Model](https://github.com/c1mone/Tensorflow-101/blob/master/notebooks/4_Save_Restore_Model.ipynb) 11 | 5. [Autoencoder](https://github.com/c1mone/Tensorflow-101/blob/master/notebooks/5_Autoencoder.ipynb) 12 | 6. [Sparse Autoencoder](https://github.com/c1mone/Tensorflow-101/blob/master/notebooks/6_Sparse_Autoencoder.ipynb) 13 | 7. [Convolutional Autoencoder](https://github.com/c1mone/Tensorflow-101/blob/master/notebooks/7_Convolutional_Autoencoder.ipynb) 14 | 8. [Denoising Autoencoder](https://github.com/c1mone/Tensorflow-101/blob/master/notebooks/8_Denoising_Autoencoder.ipynb) 15 | 9. [Variational Autoencoder](https://github.com/c1mone/Tensorflow-101/blob/master/notebooks/9_Variational_Autoencoder.ipynb) 16 | 10. [Reccurent Neural Network with MNIST](https://github.com/c1mone/Tensorflow-101/blob/master/notebooks/10_MNIST_Reccurent_Neural_Network.ipynb) 17 | 11. Char RNN 18 | 12. word2vec 19 | 13. [Generative Adversarial Network with MNIST](https://github.com/c1mone/Tensorflow-101/blob/master/notebooks/13_Generative_Adversarial_Network.ipynb) 20 | 14. [DCGAN with MNIST](https://github.com/c1mone/Tensorflow-101/blob/master/notebooks/14_DCGAN_with_MNIST.ipynb) 21 | 22 | 23 | 24 | # License 25 | 26 | The MIT License (MIT) 27 | 28 | Copyright (c) 2016 c1mone 29 | 30 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 31 | 32 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 33 | 34 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------------------- /notebooks/10_MNIST_Reccurent_Neural_Network.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Reccurent Neural Network with MNIST\n", 8 | "\n", 9 | "## Introduction\n", 10 | "\n", 11 | "Reccurent Neural Network 簡稱 RNN．跟之前提到的 CNN (找出特徵)，Autoencoder (降維 重建) 不同．它關注的是 **時間序列** 有關的問題，舉個例子，一篇文章中的文字會是跟前後文有前因後果的，而如果想要製作一個文章產生器，就會需要用到 RNN．\n", 12 | "\n", 13 | "那 RNN 是如何解決這個問題呢？．觀察下面這個 RNN 基本的結構圖．其中 `Xt` 以及 `Ht` 分別是 `t` 時刻的輸入以及輸出，可以看到 `Ht` 會跟 `Ht-1` 以及 `Xt` 有關，可以簡單地把它想像成多一個輸入的神經網路．\n", 14 | "\n", 15 | "![](https://github.com/c1mone/Tensorflow-101/blob/master/notebooks/images/10_Reccurent_Neural_Network/rnn_cell.jpg?raw=true)\n", 16 | "\n", 17 | "那如果我們依照時間序列展開 RNN 就會變成以下的樣子：\n", 18 | "\n", 19 | "![](https://github.com/c1mone/Tensorflow-101/blob/master/notebooks/images/10_Reccurent_Neural_Network/rnn_expand.jpg?raw=true)\n", 20 | "\n", 21 | "每一個時間點的輸出，除了跟當前的輸入有關，也會跟前一刻，前前一刻．．．等時間點的輸出有關係．如此就相當於把各個時間序列點的輸出連結了起來．\n", 22 | "\n", 23 | "理論上來說這是一個很完美結構，可以處理許多跟時間序列有關的問題，但是實際上會遇到許多的問題，什麼問題呢？想像一下在訓練模型的時候會利用 Backpropagation 來更新權重，而 RNN 的輸出會跟前一刻有關，所以也會傳遞到前一刻的模型更新權重，依此類推．當這些更新同時發生的時候就可能會產生兩個結果，一個是梯度爆炸 (Gradient Exploding)另一個是梯度消失 (Gradient Vanishing)．" 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "metadata": {}, 29 | "source": [ 30 | "\n", 31 | "### Gradient Exploding\n", 32 | "\n", 33 | "梯度爆炸，也就是說隨著序列增加，後續權重的更新大到無法處理的地步，如下圖．而解決這個問題的方式比較簡單的方法就硬性規定一個更新上限，更新值大於上限就用上限值取代．\n" 34 | ] 35 | }, 36 | { 37 | "cell_type": "markdown", 38 | "metadata": {}, 39 | "source": [ 40 | "### Gradient Vanishing\n", 41 | "\n", 42 | "梯度消失，也就是說隨著序列增加，後續權重的更新小到趨近於 0．使得 RNN 只**記得**最近發生的事情，而沒有辦法記起時間較久前的結果．而解決這個問題的方式就是 **LSTMs**，在[不可思議的 RNN](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)這篇文章中，提到了許多 RNN 非常有效的使用場景，就是因為使用了 LSTMs．\n", 43 | "\n", 44 | "使用 LSTM 的方法非常簡單就是把 RNN cell 替換成 LSTM cell，關於 LSTM 的內容這裡不會細講，可以看這一篇非常棒的[文章](http://colah.github.io/posts/2015-08-Understanding-LSTMs/)，目前請先想像成它可以有效的解決梯度消失的問題．" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": {}, 50 | "source": [ 51 | "![](https://github.com/c1mone/Tensorflow-101/blob/master/notebooks/images/10_Reccurent_Neural_Network/gradient.jpg?raw=true)\n", 52 | "\n", 53 | "圖片來自 Udacity [course](https://www.youtube.com/watch?time_continue=4&v=VuamhbEWEWA)" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "## MNIST Test with RNN\n", 61 | "\n", 62 | "接下來我們會用 Tensorflow 中的 RNN 來處理 MNIST 手寫數字辨識的問題．但是一個圖片要怎麼跟時間序列扯上關係呢？簡單的想法就是我們把 MNIST 中 28 x 28 維的資料，想像成 28 個時間點，而每一時間點就給 RNN 28 維的向量．換句話說就是讓 RNN **'一列一列地看'**手寫數字，當看完整個圖片的以後，把他的輸出 `H28` 先丟給一個全連結層之後再丟給分類器來決定說這個看到的數字屬於哪一類．\n", 63 | "\n", 64 | "### Imports" 65 | ] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": 1, 70 | "metadata": { 71 | "collapsed": false 72 | }, 73 | "outputs": [ 74 | { 75 | "name": "stdout", 76 | "output_type": "stream", 77 | "text": [ 78 | "Extracting /tmp/data/train-images-idx3-ubyte.gz\n", 79 | "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n", 80 | "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n", 81 | "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n", 82 | "Package loaded\n" 83 | ] 84 | } 85 | ], 86 | "source": [ 87 | "import tensorflow as tf\n", 88 | "from tensorflow.python.ops import rnn, rnn_cell\n", 89 | "from libs.utils import weight_variable, bias_variable\n", 90 | "import numpy as np\n", 91 | "\n", 92 | "from tensorflow.examples.tutorials.mnist import input_data\n", 93 | "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n", 94 | "\n", 95 | "print(\"Package loaded\")" 96 | ] 97 | }, 98 | { 99 | "cell_type": "markdown", 100 | "metadata": {}, 101 | "source": [ 102 | "### Configuraions\n", 103 | "\n", 104 | "設定相關的參數，其中要讓 RNN 看 MNIST 圖片的話就是，一次給它看一列 (n_input = 28維)，由上看到下看 28 個輸入 (n_steps = 28 steps)．" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": 2, 110 | "metadata": { 111 | "collapsed": false 112 | }, 113 | "outputs": [], 114 | "source": [ 115 | "n_input = 28 # MNIST data input (image shape: 28*28)\n", 116 | "n_steps = 28 # steps\n", 117 | "n_hidden = 128 # number of neurons in fully connected layer \n", 118 | "n_classes = 10 # (0-9 digits)\n", 119 | "\n", 120 | "x = tf.placeholder(\"float\", [None, n_steps, n_input])\n", 121 | "y = tf.placeholder(\"float\", [None, n_classes])\n", 122 | "\n", 123 | "weights = {\n", 124 | " \"w_fc\" : weight_variable([n_hidden, n_classes], \"w_fc\")\n", 125 | "}\n", 126 | "biases = {\n", 127 | " \"b_fc\" : bias_variable([n_classes], \"b_fc\") \n", 128 | "}" 129 | ] 130 | }, 131 | { 132 | "cell_type": "markdown", 133 | "metadata": {}, 134 | "source": [ 135 | "### Adjust input x to RNN\n", 136 | "\n", 137 | "因應 tensorflow 中 RNN 輸入的要求，現在要來變化輸入向量的形式\n", 138 | "\n", 139 | "- 先把 x 交換維度成 `n_step, None, n_input`" 140 | ] 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": 3, 145 | "metadata": { 146 | "collapsed": false 147 | }, 148 | "outputs": [ 149 | { 150 | "name": "stdout", 151 | "output_type": "stream", 152 | "text": [ 153 | "x_transpose shape: (28, ?, 28)\n" 154 | ] 155 | } 156 | ], 157 | "source": [ 158 | "x_transpose = tf.transpose(x, [1, 0, 2])\n", 159 | "print(\"x_transpose shape: %s\" % x_transpose.get_shape())" 160 | ] 161 | }, 162 | { 163 | "cell_type": "markdown", 164 | "metadata": {}, 165 | "source": [ 166 | "- 再來變成 `n_step * None, n_input`" 167 | ] 168 | }, 169 | { 170 | "cell_type": "code", 171 | "execution_count": 4, 172 | "metadata": { 173 | "collapsed": false 174 | }, 175 | "outputs": [ 176 | { 177 | "name": "stdout", 178 | "output_type": "stream", 179 | "text": [ 180 | "x_reshape shape: (?, 28)\n" 181 | ] 182 | } 183 | ], 184 | "source": [ 185 | "x_reshape = tf.reshape(x_transpose, [-1, n_input])\n", 186 | "print(\"x_reshape shape: %s\" % x_reshape.get_shape())" 187 | ] 188 | }, 189 | { 190 | "cell_type": "markdown", 191 | "metadata": {}, 192 | "source": [ 193 | "- 最後會把它切成長度為 n_steps 的 list，其中第 i 個元素就是對應第 i 個 step．每個元素會有 None x n_inputs" 194 | ] 195 | }, 196 | { 197 | "cell_type": "code", 198 | "execution_count": 5, 199 | "metadata": { 200 | "collapsed": false 201 | }, 202 | "outputs": [ 203 | { 204 | "name": "stdout", 205 | "output_type": "stream", 206 | "text": [ 207 | "type of x_split: \n", 208 | "length of x_split: 28\n", 209 | "shape of x_split[0]: (?, 28)\n" 210 | ] 211 | } 212 | ], 213 | "source": [ 214 | "x_split = tf.split(0, n_steps, x_reshape)\n", 215 | "print(\"type of x_split: %s\" % type(x_split))\n", 216 | "print(\"length of x_split: %d\" % len(x_split))\n", 217 | "print(\"shape of x_split[0]: %s\" % x_split[0].get_shape())" 218 | ] 219 | }, 220 | { 221 | "cell_type": "markdown", 222 | "metadata": {}, 223 | "source": [ 224 | "接下來就是要建立模型，這裡直接使用 RNN cell，其中 n_hidden 是輸出的維度為 128．\n", 225 | "觀察它輸出的 h 是一個長度 28 的 list，第 i 個元素代表著第 i 個 step 的輸出，每個元素會有 None x n_hidden" 226 | ] 227 | }, 228 | { 229 | "cell_type": "code", 230 | "execution_count": 6, 231 | "metadata": { 232 | "collapsed": false, 233 | "scrolled": true 234 | }, 235 | "outputs": [ 236 | { 237 | "name": "stdout", 238 | "output_type": "stream", 239 | "text": [ 240 | "type of outputs: \n", 241 | "length of outputs: 28\n", 242 | "shape of h[0]: (?, 128)\n", 243 | "type of states: \n" 244 | ] 245 | } 246 | ], 247 | "source": [ 248 | "basic_rnn_cell = rnn_cell.BasicRNNCell(n_hidden)\n", 249 | "h, states = rnn.rnn(basic_rnn_cell, x_split, dtype=tf.float32)\n", 250 | "print(\"type of outputs: %s\" % type(h))\n", 251 | "print(\"length of outputs: %d\" % len(h))\n", 252 | "print(\"shape of h[0]: %s\" % h[0].get_shape())\n", 253 | "print(\"type of states: %s\" % type(states))" 254 | ] 255 | }, 256 | { 257 | "cell_type": "markdown", 258 | "metadata": {}, 259 | "source": [ 260 | "### fully connnected layer\n", 261 | "\n", 262 | "接一個 128 -> 10 的全連結層" 263 | ] 264 | }, 265 | { 266 | "cell_type": "code", 267 | "execution_count": 7, 268 | "metadata": { 269 | "collapsed": false 270 | }, 271 | "outputs": [], 272 | "source": [ 273 | "h_fc = tf.matmul(h[-1], weights['w_fc']) + biases['b_fc']\n", 274 | "y_ = h_fc" 275 | ] 276 | }, 277 | { 278 | "cell_type": "markdown", 279 | "metadata": {}, 280 | "source": [ 281 | "### cost function\n", 282 | "\n", 283 | "用 softmax 作分類" 284 | ] 285 | }, 286 | { 287 | "cell_type": "code", 288 | "execution_count": 8, 289 | "metadata": { 290 | "collapsed": true 291 | }, 292 | "outputs": [], 293 | "source": [ 294 | "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(h_fc, y))\n", 295 | "optimizer = tf.train.AdamOptimizer(0.01).minimize(cost)" 296 | ] 297 | }, 298 | { 299 | "cell_type": "markdown", 300 | "metadata": {}, 301 | "source": [ 302 | "### accuracy function" 303 | ] 304 | }, 305 | { 306 | "cell_type": "code", 307 | "execution_count": 9, 308 | "metadata": { 309 | "collapsed": false 310 | }, 311 | "outputs": [], 312 | "source": [ 313 | "correct_prediction = tf.equal(tf.argmax(y_, 1), tf.argmax(y, 1))\n", 314 | "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n", 315 | "\n" 316 | ] 317 | }, 318 | { 319 | "cell_type": "markdown", 320 | "metadata": {}, 321 | "source": [ 322 | "### Training" 323 | ] 324 | }, 325 | { 326 | "cell_type": "code", 327 | "execution_count": 11, 328 | "metadata": { 329 | "collapsed": false 330 | }, 331 | "outputs": [ 332 | { 333 | "name": "stdout", 334 | "output_type": "stream", 335 | "text": [ 336 | "step 0, loss 2.32416, accuracy 0.090, mean of rnn weight -0.00024, mean of rnn out -0.00735\n", 337 | "step 100, loss 1.64264, accuracy 0.380, mean of rnn weight -0.00052, mean of rnn out -0.09297\n", 338 | "step 200, loss 1.13360, accuracy 0.600, mean of rnn weight 0.00075, mean of rnn out 0.00324\n", 339 | "step 300, loss 1.03078, accuracy 0.670, mean of rnn weight 0.00082, mean of rnn out -0.00883\n", 340 | "step 400, loss 1.29169, accuracy 0.510, mean of rnn weight 0.00108, mean of rnn out 0.00112\n", 341 | "step 500, loss 1.48408, accuracy 0.420, mean of rnn weight 0.00160, mean of rnn out -0.01736\n", 342 | "step 600, loss 1.43396, accuracy 0.570, mean of rnn weight 0.00256, mean of rnn out -0.05415\n", 343 | "step 700, loss 2.06715, accuracy 0.350, mean of rnn weight 0.00297, mean of rnn out -0.04546\n", 344 | "step 800, loss 1.53593, accuracy 0.390, mean of rnn weight 0.00282, mean of rnn out 0.00934\n", 345 | "step 900, loss 1.58583, accuracy 0.370, mean of rnn weight 0.00266, mean of rnn out 0.01959\n", 346 | "step 1000, loss 1.36978, accuracy 0.470, mean of rnn weight 0.00299, mean of rnn out 0.04775\n", 347 | "step 1100, loss 2.12206, accuracy 0.360, mean of rnn weight 0.00161, mean of rnn out -0.00393\n", 348 | "step 1200, loss 1.50930, accuracy 0.470, mean of rnn weight 0.00138, mean of rnn out -0.01369\n", 349 | "step 1300, loss 1.39899, accuracy 0.520, mean of rnn weight 0.00152, mean of rnn out 0.00569\n", 350 | "step 1400, loss 1.44504, accuracy 0.430, mean of rnn weight 0.00158, mean of rnn out -0.00496\n", 351 | "step 2000, loss 2.32795, accuracy 0.170, mean of rnn weight 0.00122, mean of rnn out 0.09313\n", 352 | "step 3000, loss 2.43317, accuracy 0.100, mean of rnn weight 0.00119, mean of rnn out 0.07819\n", 353 | "step 4000, loss 2.42197, accuracy 0.110, mean of rnn weight 0.00111, mean of rnn out 0.07806\n" 354 | ] 355 | } 356 | ], 357 | "source": [ 358 | "batch_size = 100\n", 359 | "init_op = tf.global_variables_initializer()\n", 360 | "sess = tf.InteractiveSession()\n", 361 | "sess.run(init_op)\n", 362 | "\n", 363 | "variables_names =[v.name for v in tf.trainable_variables()]\n", 364 | "\n", 365 | "for step in range(5000):\n", 366 | " batch_x, batch_y = mnist.train.next_batch(batch_size)\n", 367 | " batch_x = np.reshape(batch_x, (batch_size, n_steps, n_input))\n", 368 | " cost_train, accuracy_train, states_train, rnn_out = sess.run([cost, accuracy, states, h[-1]], feed_dict = {x: batch_x, y: batch_y})\n", 369 | " values = sess.run(variables_names)\n", 370 | " rnn_out_mean = np.mean(rnn_out)\n", 371 | " for k,v in zip(variables_names, values):\n", 372 | " if k == 'RNN/BasicRNNCell/Linear/Matrix:0':\n", 373 | " w_rnn_mean = np.mean(v)\n", 374 | "\n", 375 | " if step < 1500:\n", 376 | " if step % 100 == 0:\n", 377 | " print(\"step %d, loss %.5f, accuracy %.3f, mean of rnn weight %.5f, mean of rnn out %.5f\" % (step, cost_train, accuracy_train, w_rnn_mean, rnn_out_mean))\n", 378 | " else:\n", 379 | " if step%1000 == 0: \n", 380 | " print(\"step %d, loss %.5f, accuracy %.3f, mean of rnn weight %.5f, mean of rnn out %.5f\" % (step, cost_train, accuracy_train, w_rnn_mean, rnn_out_mean))\n", 381 | " optimizer.run(feed_dict={x: batch_x, y: batch_y})\n" 382 | ] 383 | }, 384 | { 385 | "cell_type": "code", 386 | "execution_count": 12, 387 | "metadata": { 388 | "collapsed": false 389 | }, 390 | "outputs": [ 391 | { 392 | "name": "stdout", 393 | "output_type": "stream", 394 | "text": [ 395 | "final loss 2.41618, accuracy 0.10320\n" 396 | ] 397 | } 398 | ], 399 | "source": [ 400 | "cost_test, accuracy_test = sess.run([cost, accuracy], feed_dict={x: np.reshape(mnist.test.images, [-1, 28, 28]), y: mnist.test.labels})\n", 401 | "print(\"final loss %.5f, accuracy %.5f\" % (cost_test, accuracy_test) )" 402 | ] 403 | }, 404 | { 405 | "cell_type": "markdown", 406 | "metadata": {}, 407 | "source": [ 408 | "### Result\n", 409 | "\n", 410 | "可以看到預測結果非常不好，測試的準確率只有 10%，rnn 的權重平均值滿低的．下面印出最後輸出 128 維的矩陣，可以看到每個值都接近 1 或是 -1，然後搜尋一下以後知道 RNN 裡面輸出之前會經過一個 `tanh`，而當 `tanh` 在 1 或 -1 的時候做微分是 0．我想這樣的情形就是**Gradient Vanishing**了．" 411 | ] 412 | }, 413 | { 414 | "cell_type": "code", 415 | "execution_count": 13, 416 | "metadata": { 417 | "collapsed": false 418 | }, 419 | "outputs": [ 420 | { 421 | "name": "stdout", 422 | "output_type": "stream", 423 | "text": [ 424 | "[ 0.99999559 1. 1. -1. -0.99998862 1. 1.\n", 425 | " -1. 1. 1. -0.99999774 1. -0.99999982\n", 426 | " 1. 1. 1. 1. 0.9999997 0.99999994\n", 427 | " 0.9999994 -0.99999225 -1. -1. 1. -1. 1.\n", 428 | " -0.99999952 1. 0.99999928 -1. 0.99674076 1. -1.\n", 429 | " -0.99999458 -0.99956894 1. 0.99983639 0.99999982 0.99956954\n", 430 | " -0.99999893 1. -0.99999994 1. -0.99997771 1. 1.\n", 431 | " 1. -1.00000012 -1. 1. -0.99970055 0.99998623\n", 432 | " -0.99999619 -1. -0.99960238 0.99785262 -1. 0.99962986\n", 433 | " -1. 1. -1. -1. 1. -1. 1.\n", 434 | " 0.99979544 1. 1. 1. 1. 1. 1.\n", 435 | " 1. 1. 1. 1. 0.99939382 -1. -1.\n", 436 | " -0.99976331 -0.99999881 -1. -0.99999976 -1. -0.99999964\n", 437 | " 1. -1. -0.99999934 0.99999392 0.99910891 -0.99995011\n", 438 | " -1. -1. -1. -0.99998069 0.99999958 -0.99999964\n", 439 | " 1. -1. 0.99999958 1. -1. 1.\n", 440 | " -0.99998337 0.99999732 1. 1. 1. -0.99997371\n", 441 | " -1. -0.999376 0.99992633 0.9999997 1. -1.\n", 442 | " 0.99999499 1. -1. -1. 1. -1.\n", 443 | " -0.99995339 0.99957949 -1. -0.999933 -0.99999905 -0.99999183\n", 444 | " 1. ]\n" 445 | ] 446 | } 447 | ], 448 | "source": [ 449 | "print h[-1].eval(feed_dict={x: np.reshape(mnist.test.images, [-1, 28, 28]), y: mnist.test.labels})[0,:]" 450 | ] 451 | }, 452 | { 453 | "cell_type": "markdown", 454 | "metadata": {}, 455 | "source": [ 456 | "### LSTM\n", 457 | "\n", 458 | "因為上面 RNN 的表現非常不好，讓我們來用一下 tensorflow 中的 LSTMs cell 看看成果如何．而更改 cell 的設定非常簡單，只要把 BasicRNNCell 改成 BasicLSTMCell 就可以了．" 459 | ] 460 | }, 461 | { 462 | "cell_type": "code", 463 | "execution_count": 16, 464 | "metadata": { 465 | "collapsed": false 466 | }, 467 | "outputs": [ 468 | { 469 | "name": "stdout", 470 | "output_type": "stream", 471 | "text": [ 472 | "step 0, loss 2.30415, accuracy 0.110, mean of lstm weight 0.00013, mean of lstm out 0.01331\n", 473 | "step 100, loss 0.31279, accuracy 0.880, mean of lstm weight -0.00529, mean of lstm out -0.00088\n", 474 | "step 200, loss 0.17318, accuracy 0.940, mean of lstm weight -0.00648, mean of lstm out 0.00784\n", 475 | "step 300, loss 0.15617, accuracy 0.950, mean of lstm weight -0.00778, mean of lstm out -0.00153\n", 476 | "step 400, loss 0.08717, accuracy 0.980, mean of lstm weight -0.00872, mean of lstm out 0.00838\n", 477 | "step 500, loss 0.13275, accuracy 0.960, mean of lstm weight -0.00991, mean of lstm out 0.00275\n", 478 | "step 600, loss 0.11011, accuracy 0.970, mean of lstm weight -0.01076, mean of lstm out 0.00076\n", 479 | "step 700, loss 0.12507, accuracy 0.960, mean of lstm weight -0.01037, mean of lstm out 0.00274\n", 480 | "step 800, loss 0.09086, accuracy 0.970, mean of lstm weight -0.01050, mean of lstm out 0.00409\n", 481 | "step 900, loss 0.05551, accuracy 0.990, mean of lstm weight -0.01066, mean of lstm out -0.00078\n", 482 | "step 1000, loss 0.03132, accuracy 0.990, mean of lstm weight -0.01064, mean of lstm out -0.00035\n", 483 | "step 1100, loss 0.06873, accuracy 0.980, mean of lstm weight -0.01098, mean of lstm out -0.00248\n", 484 | "step 1200, loss 0.08930, accuracy 0.980, mean of lstm weight -0.01073, mean of lstm out -0.00918\n", 485 | "step 1300, loss 0.10252, accuracy 0.980, mean of lstm weight -0.01027, mean of lstm out -0.00038\n", 486 | "step 1400, loss 0.00594, accuracy 1.000, mean of lstm weight -0.01041, mean of lstm out 0.00762\n", 487 | "step 2000, loss 0.04595, accuracy 0.990, mean of lstm weight -0.01243, mean of lstm out 0.00263\n", 488 | "step 3000, loss 0.12044, accuracy 0.960, mean of lstm weight -0.01405, mean of lstm out -0.00691\n", 489 | "step 4000, loss 0.03068, accuracy 0.990, mean of lstm weight -0.01488, mean of lstm out -0.00371\n" 490 | ] 491 | } 492 | ], 493 | "source": [ 494 | "tf.reset_default_graph()\n", 495 | "x = tf.placeholder(\"float\", [None, n_steps, n_input])\n", 496 | "y = tf.placeholder(\"float\", [None, n_classes])\n", 497 | "\n", 498 | "weights = {\n", 499 | " \"w_fc\" : weight_variable([n_hidden, n_classes], \"w_fc\")\n", 500 | "}\n", 501 | "biases = {\n", 502 | " \"b_fc\" : bias_variable([n_classes], \"b_fc\") \n", 503 | "}\n", 504 | "x_transpose = tf.transpose(x, [1, 0, 2])\n", 505 | "x_reshape = tf.reshape(x_transpose, [-1, n_input])\n", 506 | "x_split = tf.split(0, n_steps, x_reshape)\n", 507 | "lstm_cell = rnn_cell.BasicLSTMCell(n_hidden, forget_bias=1.0)\n", 508 | "h, states = rnn.rnn(lstm_cell, x_split, dtype=tf.float32)\n", 509 | "\n", 510 | "h_fc = tf.matmul(h[-1], weights['w_fc']) + biases['b_fc']\n", 511 | "y_ = h_fc\n", 512 | "\n", 513 | "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(h_fc, y))\n", 514 | "optimizer = tf.train.AdamOptimizer(0.01).minimize(cost)\n", 515 | "correct_prediction = tf.equal(tf.argmax(y_, 1), tf.argmax(y, 1))\n", 516 | "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n", 517 | "\n", 518 | "batch_size = 100\n", 519 | "init_op = tf.global_variables_initializer()\n", 520 | "sess = tf.InteractiveSession()\n", 521 | "sess.run(init_op)\n", 522 | "\n", 523 | "variables_names =[v.name for v in tf.trainable_variables()]\n", 524 | "\n", 525 | "for step in range(5000):\n", 526 | " batch_x, batch_y = mnist.train.next_batch(batch_size)\n", 527 | " batch_x = np.reshape(batch_x, (batch_size, n_steps, n_input))\n", 528 | " cost_train, accuracy_train, states_train, rnn_out = sess.run([cost, accuracy, states, h[-1]], feed_dict = {x: batch_x, y: batch_y})\n", 529 | " values = sess.run(variables_names)\n", 530 | " rnn_out_mean = np.mean(rnn_out)\n", 531 | " for k,v in zip(variables_names, values):\n", 532 | " if k == 'RNN/BasicLSTMCell/Linear/Matrix:0':\n", 533 | " w_rnn_mean = np.mean(v)\n", 534 | "\n", 535 | " if step < 1500:\n", 536 | " if step % 100 == 0:\n", 537 | " print(\"step %d, loss %.5f, accuracy %.3f, mean of lstm weight %.5f, mean of lstm out %.5f\" % (step, cost_train, accuracy_train, w_rnn_mean, rnn_out_mean))\n", 538 | " else:\n", 539 | " if step%1000 == 0: \n", 540 | " print(\"step %d, loss %.5f, accuracy %.3f, mean of lstm weight %.5f, mean of lstm out %.5f\" % (step, cost_train, accuracy_train, w_rnn_mean, rnn_out_mean))\n", 541 | " optimizer.run(feed_dict={x: batch_x, y: batch_y})\n", 542 | "\n" 543 | ] 544 | }, 545 | { 546 | "cell_type": "code", 547 | "execution_count": 17, 548 | "metadata": { 549 | "collapsed": false 550 | }, 551 | "outputs": [ 552 | { 553 | "name": "stdout", 554 | "output_type": "stream", 555 | "text": [ 556 | "final loss 0.06450, accuracy 0.98390\n" 557 | ] 558 | } 559 | ], 560 | "source": [ 561 | "cost_test, accuracy_test = sess.run([cost, accuracy], feed_dict={x: np.reshape(mnist.test.images, [-1, 28, 28]), y: mnist.test.labels})\n", 562 | "print(\"final loss %.5f, accuracy %.5f\" % (cost_test, accuracy_test) )" 563 | ] 564 | }, 565 | { 566 | "cell_type": "markdown", 567 | "metadata": {}, 568 | "source": [ 569 | "Bingo!可以看到準確率提升到非常高，看來 LSTMs 真的解決了 RNN 的缺點．" 570 | ] 571 | }, 572 | { 573 | "cell_type": "markdown", 574 | "metadata": {}, 575 | "source": [ 576 | "## 小結\n", 577 | "\n", 578 | "了解了 RNN 的架構，以及會遇到的問題 `Gradient Exploding` 以及 `Gradient Vanishing`，並且使用 MNIST 手寫數字資料集來練習 RNN．\n", 579 | "\n", 580 | "在 MNIST 中純粹的 RNN 會遇到梯度消失問題，而改用 LSTMs 之後就成功了提高極多的準確度．\n", 581 | "\n", 582 | "### 問題\n", 583 | "\n", 584 | "- 找找看 tensorflow 有沒有顯示各階段 gradient 的函數\n", 585 | "- 看看 RNN 的 backpropagation" 586 | ] 587 | }, 588 | { 589 | "cell_type": "markdown", 590 | "metadata": {}, 591 | "source": [ 592 | "## 學習資源連結\n", 593 | "\n", 594 | "- [Colah Blog : Understanding LSTMs](http://colah.github.io/posts/2015-08-Understanding-LSTMs/)\n", 595 | "- [The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)" 596 | ] 597 | } 598 | ], 599 | "metadata": { 600 | "kernelspec": { 601 | "display_name": "Python 2", 602 | "language": "python", 603 | "name": "python2" 604 | }, 605 | "language_info": { 606 | "codemirror_mode": { 607 | "name": "ipython", 608 | "version": 2 609 | }, 610 | "file_extension": ".py", 611 | "mimetype": "text/x-python", 612 | "name": "python", 613 | "nbconvert_exporter": "python", 614 | "pygments_lexer": "ipython2", 615 | "version": "2.7.6" 616 | } 617 | }, 618 | "nbformat": 4, 619 | "nbformat_minor": 2 620 | } 621 | -------------------------------------------------------------------------------- /notebooks/1_MNIST_Softmax_Regression.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "# MNIST 手寫數字辨識 - Softmax Regression" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": { 16 | "deletable": true, 17 | "editable": true 18 | }, 19 | "source": [ 20 | "## 目標\n", 21 | "* 下載並熟悉 MNIST 資料集\n", 22 | "* 建立 tensorflow softmax regression model\n", 23 | "* 訓練 model 並計算出準確度" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": 6, 29 | "metadata": { 30 | "collapsed": false, 31 | "deletable": true, 32 | "editable": true 33 | }, 34 | "outputs": [], 35 | "source": [ 36 | "import numpy as np\n", 37 | "import tensorflow as tf\n", 38 | "import matplotlib.pyplot as plt\n", 39 | "%matplotlib inline" 40 | ] 41 | }, 42 | { 43 | "cell_type": "markdown", 44 | "metadata": { 45 | "deletable": true, 46 | "editable": true 47 | }, 48 | "source": [ 49 | "當我們在學習一個新的程式語言的時候，譬如 `java`, `python` 之類的語言的時候，第一堂課都是學習印出 \"Hello World!\"．而在機器學習的領域裡面，MNIST 就有著相當於 Hello World 的地位．\n", 50 | "\n", 51 | "而什麼是 MNIST 呢？就是一個手寫數字的資料集．它包含了一連串的手寫數字圖片，並且有相對應的數字，如下圖，看得出來就是 5, 0, 4, 1 這四個數字．\n", 52 | "\n", 53 | "![](https://www.tensorflow.org/images/MNIST.png)\n", 54 | "\n", 55 | "在這裡會嘗試用一個簡單的 Softmax Regression 的模型來解決這個問題，當然沒辦法達到頂尖的辨識水準，不過會是一個好的開始，接下來也會學到比較複雜但更為準確的模型，讓我們先來看看 MNIST 資料集吧．" 56 | ] 57 | }, 58 | { 59 | "cell_type": "markdown", 60 | "metadata": { 61 | "deletable": true, 62 | "editable": true 63 | }, 64 | "source": [ 65 | "## 下載並熟悉 MNIST data" 66 | ] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "metadata": { 71 | "deletable": true, 72 | "editable": true 73 | }, 74 | "source": [ 75 | "MNIST 資料集的官網是在[Yann LeCun's website](http://yann.lecun.com/exdb/mnist/)．這裡我們只要在 python 內把以下的兩行程式碼貼上，就可以下載 MNIST 的資料集．" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": 7, 81 | "metadata": { 82 | "collapsed": false, 83 | "deletable": true, 84 | "editable": true 85 | }, 86 | "outputs": [ 87 | { 88 | "name": "stdout", 89 | "output_type": "stream", 90 | "text": [ 91 | "Extracting MNIST_data/train-images-idx3-ubyte.gz\n", 92 | "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n", 93 | "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n", 94 | "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n" 95 | ] 96 | } 97 | ], 98 | "source": [ 99 | "from tensorflow.examples.tutorials.mnist import input_data\n", 100 | "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)" 101 | ] 102 | }, 103 | { 104 | "cell_type": "markdown", 105 | "metadata": { 106 | "deletable": true, 107 | "editable": true 108 | }, 109 | "source": [ 110 | "MNIST 資料集分成三個部分\n", 111 | "\n", 112 | "1. 55,000 筆的 training data (`mnist.train`)\n", 113 | "2. 10,000 筆的 test data (`mnist.test`)\n", 114 | "3. 5,000 筆的 validation data (`mnist.validation`)\n", 115 | "\n", 116 | "把資料集分成這三個部分在機器學習裡面是非常重要的，因為我們必須從學習資料以及驗證資料來看我們的學習到底有沒有效果 (generalizes)\n", 117 | "\n", 118 | "如同上面講到的，每筆 MNIST 資料都有兩個部分，第一個是手寫的數字 image，另一個是對應的 label．在這裡我們把 image 稱作 `x`，而 label 稱作 `y`． training set 還有 test set 都有 image 以及 label．例如 training images 被稱作 `mnist.train.images`， training labels 稱作 `mnist.train.labels`．" 119 | ] 120 | }, 121 | { 122 | "cell_type": "code", 123 | "execution_count": 8, 124 | "metadata": { 125 | "collapsed": false, 126 | "deletable": true, 127 | "editable": true 128 | }, 129 | "outputs": [ 130 | { 131 | "name": "stdout", 132 | "output_type": "stream", 133 | "text": [ 134 | "\n", 135 | "55000\n", 136 | "5000\n", 137 | "10000\n" 138 | ] 139 | } 140 | ], 141 | "source": [ 142 | "# 來看看 mnist 的型態\n", 143 | "print type(mnist)\n", 144 | "print mnist.train.num_examples\n", 145 | "print mnist.validation.num_examples\n", 146 | "print mnist.test.num_examples" 147 | ] 148 | }, 149 | { 150 | "cell_type": "code", 151 | "execution_count": 9, 152 | "metadata": { 153 | "collapsed": false, 154 | "deletable": true, 155 | "editable": true 156 | }, 157 | "outputs": [ 158 | { 159 | "name": "stdout", 160 | "output_type": "stream", 161 | "text": [ 162 | "讓我們看一下 MNIST 訓練還有測試的資料集長得如何\n", 163 | "\n", 164 | " train_img 的 type : \n", 165 | " train_img 的 dimension : (55000, 784)\n", 166 | " train_label 的 type : \n", 167 | " train_label 的 dimension : (55000, 10)\n", 168 | " test_img 的 type : \n", 169 | " test_img 的 dimension : (10000, 784)\n", 170 | " test_label 的 type : \n", 171 | " test_label 的 dimension : (10000, 10)\n" 172 | ] 173 | } 174 | ], 175 | "source": [ 176 | "print(\"讓我們看一下 MNIST 訓練還有測試的資料集長得如何\")\n", 177 | "train_img = mnist.train.images\n", 178 | "train_label = mnist.train.labels\n", 179 | "test_img = mnist.test.images\n", 180 | "test_label = mnist.test.labels\n", 181 | "print\n", 182 | "print(\" train_img 的 type : %s\" % (type(train_img)))\n", 183 | "print(\" train_img 的 dimension : %s\" % (train_img.shape,))\n", 184 | "print(\" train_label 的 type : %s\" % (type(train_label)))\n", 185 | "print(\" train_label 的 dimension : %s\" % (train_label.shape,))\n", 186 | "print(\" test_img 的 type : %s\" % (type(test_img)))\n", 187 | "print(\" test_img 的 dimension : %s\" % (test_img.shape,))\n", 188 | "print(\" test_label 的 type : %s\" % (type(test_label)))\n", 189 | "print(\" test_label 的 dimension : %s\" % (test_label.shape,))" 190 | ] 191 | }, 192 | { 193 | "cell_type": "markdown", 194 | "metadata": { 195 | "deletable": true, 196 | "editable": true 197 | }, 198 | "source": [ 199 | "從上面可以看到每個 image 有 784 個數字，因為每張圖片其實是 28 pixels X 28 pixels，我們可以把它看成一個很大的 array 如下圖\n", 200 | "\n", 201 | "![](https://www.tensorflow.org/images/MNIST-Matrix.png)\n", 202 | "\n", 203 | "把這個 array `拉平` 成一個 28x28 = 784 的向量，是用什麼方法拉平的並不重要，只要確保對於每張圖片都是用同樣的方法拉平就可以了．從這個角度來看， MNIST 就是一個 784 維向量空間裡面的點，只是有著比較複雜的[結構](http://colah.github.io/posts/2014-10-Visualizing-MNIST/)（注意，這個連結有很複雜的視覺化計算）\n", 204 | "\n", 205 | "或許你會想問說把這個二維圖片變成一個向量，會不會造成什麼不好的效果．當然有一些更強大的電腦視覺技巧是可以學習這一些原始的數字結構的，之後我們還會看到．但在這裡我們使用的方法是 Softmax Regression ，他並不需要保留原始的數字結構，直接把它變成一個向量即可．\n", 206 | "\n", 207 | "我們把 `mnist.train.images` 稱作一個 `tensor`（喔喔！tensorflow 的 tensor 出現了，他其實就是一個 n-dimensional array．而 tensor + flow = tensorflow，也就是指這個 n-dimensional array 變化流動的意思）這個 `tensor` 形狀為 [55000, 784]．第一個維度指的是圖片的 index，第二個則是每個圖片的 pixel 點，這個 pixel 點是一個介於 0 到 1 的值，來表示 pixel 點的強度．\n", 208 | "\n", 209 | "![](https://www.tensorflow.org/images/mnist-train-xs.png)\n", 210 | "\n", 211 | "而每個 MNIST 中的圖片都有一個對應的 label 也就是從 0 到 9 的數值．在這裡每個 label 都是一個 **one-hot vectors**． one-hot vector 是指說只有一個維度是 1 其他都是 0．在這裡數字 n 表示一個只有在第 n 維度（從 0 開始）數字為 1 的 10 維向量．例如 label 0 的表示法就是（[1, 0, 0, 0, 0, 0, 0, 0, 0, 0]．因此，`mnist.train.labels` 是一個 [60000, 10] 的矩陣．\n", 212 | "\n", 213 | "![](https://www.tensorflow.org/images/mnist-train-ys.png)\n", 214 | "\n", 215 | "以下我們實際印出了 MNIST 的資料集來看看他長得怎麼樣" 216 | ] 217 | }, 218 | { 219 | "cell_type": "code", 220 | "execution_count": 10, 221 | "metadata": { 222 | "collapsed": false, 223 | "deletable": true, 224 | "editable": true 225 | }, 226 | "outputs": [ 227 | { 228 | "data": { 229 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQQAAAEMCAYAAAAiW8hnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAE0xJREFUeJzt3X2QVfV9x/H3J4jRCiJWg/hIUaNo0xAljE5MSkxMxalR\nW6shtRLTzCaOzoDRPxzHNKRjkiYjajJ2dEikaqJktBpEJ0kFSgYpMQYyhAc1mmYgCriICixqsOK3\nf5yzP25W9tx79z6cu8vnNXNnd+/3PHzvYfnc8/DbcxURmJkBvKfsBsysczgQzCxxIJhZ4kAws8SB\nYGaJA8HMEgdCH5JmSfphSeseL2lns6fdl0j6oaRZzZ5X0nRJP22kt8FgyAeCpKslrZC0S9LdfWpT\nJL04wOUeK2lnxSMkvV7x80frXWZE/D4iRjR72nrl/zHektSTP9ZI+rqkg+tYxouSpjTQwzJJnxvo\n/M0WEfdExNR655P02z6/J29L+nEremyGIR8IwCbgJmBuMxcaEX+IiBG9j/zpD1Y890TfeSQNa2YP\nLfaNiBgJHA78M/BR4AlJB5bb1uASESdV/I4cTPb7+GDJbfVryAdCRDwcEfOBVyqfl3QQ8FPgyIr0\nPjIv7y/p3vzdcZ2kSQNZd/5O+++SfibpdeCjkj4taZWkHZL+IOkrFdOfICkqfl4m6WuSlue9/EzS\nofVOm9evyNe3VdINtb6DR8QfI+Ip4HzgCGB6vrwTJS2R9Gq+zB9IGpXX5gFHAj/Nt+uXJb1H0n9K\neknSNkk/lzRhANu0luUcLmlxvh2WSDqmYv5TJC3K+35W0t/XuN4vSPp5RQ/flbRF0nZJqyWdUsNi\nPg6MAryH0Gki4nVgKrCp4l19U17+NPAj4BBgAXB7A6v6LPA1YCTwC2An8I/5ss8HZkj62yrzTwfG\nAAcBX653WkkfAL4LfAY4iuxd/4h6XkREbAcWk+0pAIhsz+sI4BRgPPCVfNppZO+EU/Pteks+z2PA\nifk8a4Ef1NNDhWrLuQz4F+Aw4OneuqQRwELgXuB9ZP8OcySdVOf6pwJn5D2MJtuur9Yw33TgwYh4\ns871tc0+GwhVLIuIn0TEbrJfpg82sKwfR8QvIuKdiNgVEf8dEevyn39DFjx/XTD/XRHxfES8Qbar\nOXEA0/4DMD8ilkfELuDGAb6WTcChABHxXEQsjoi3ImILcGvR68hf790R0RMRfwRmAafne2o1q3E5\nj0bE/+Sv9QbgY5LGAhcAz0XEvRHxdkSsBOYDF9fTA/B/ZLv/J+c9PR0RLxXNkIfR3wF317mutnIg\n7F3lP+4bwAGS9hvgsl6o/EHSmflu7suStgNfIHsnq7WXohOJ/U17ZGUf+d7RazX03tdR5O+Eko6Q\n9ICkjZJ2kP2i9/s6JA2T9G1Jv8+n/11eKnrtA11O5WvdDmwn2wbHAR/JDzW2SdoGXAqMraeHiHgc\nuBO4A+iWdKekkVVmuxh4KSKW1bOudtvXA6Edf+rZdx0/Ah4CjomIUcD3yXa/W2kzcHTvD/m76eh6\nFpBfYTgb6D1Z+i1gF/CBiDgY+Bx/+jr6vu7LgfPyZYwCTuhddD191LicynMGo/LpNpEFxeKIOKTi\nMSIirq6zByLitog4DfhLskOmokM5yA4X7q13Pe025ANB0n6SDgCGAcMkVb7bdwN/3nsyrE1GAq9G\nxB8lnUF2/NlqDwIXSjpD0v7Av9Y6o6T35idVHwFeZs8v9UjgdWB7ftLuuj6zdpOdV6Bi+l1kJ3f/\nDPh6Dasfnv979T6G17ic8/M9sfeSned4IiI2k50POlXSZyUNzx+T6z2HkM8zOf89eh14C3inYPrj\nyM69OBA6wI3Am8D1ZCeb3syfIyKeBeYBv893IY/sdynNcyXwTUk9ZMe3D7R6hRGxGriGLBg2kf1n\neoXsP1Z/bsh7fAW4B3gS+Eh+fgLgq8Bkst3xBWR7PZW+AXwt364zgf/I170JWAcsr6H1OWT/Xr2P\n79W4nB+SBcFW4K/I9ip6Dx/+huz3YDPZIdY3gffW0EulQ4C7gG3A+nxZtxRMfzlZKK2vcz3tFxFt\nfwDnAr8lO/67voweqvS3HlgDrAJWdEA/c4EtwNqK5w4lO2P+fP51dB3LO5jsHe2YFvY3C9iYb8NV\nwHklbr9jgCVkVxzWATMa3YZt6q/t27CMFz8M+F+y3cn9gd8Ap5T1y9JPj+uBw8ruo6KfjwGn9fkP\n9+3eMCXb+/lWlWV8mmwXewTZO23Tgq6f/mYB15W97fJexgKn5d+PBJ4jO+6vaxuW0F/bt2EZhwyT\ngd9FNvT2LbKTbBeU0MegERFLefd17gvIduXJv15YZTEXke1mvwiMA6a1uL+OERGbI+LX+fc9wDNk\nV0zq3Ybt7q/tygiEo/jTS3EvUtKLLxDA45JWSuoqu5l+jInsRBlkx8JjiiaOiCtiz5n1cyLi+da3\nyNX5KL65kuq6qtEqksYBHwJ+SZ3bsB369Adt3ob7wknFgTgrsktKU4GrJH2s7IaKRLav2Wl3y70D\nOJ5scNRmYHa57aTBQQ8BMyNiR2WtE7bhXvpr+zYsIxA2UnGdmOz6+MYS+uhXRGzMv24hG3c+udyO\n9qo7H31H/nVLyf38iYjojojdEfEO2TmLUrdhfsnyIeC+iHg4f7pjtuHe+itjG5YRCL8CTpT0F/k1\n8c+QXbbqCJIO6h11lg/g+RTZePlOs4D8D43yr4+U2Mu79P5Hy11EidtQksguEz4Te/6uAjpkG/bX\nXxnbUPmZzbaSdB5wG9kVh7kRUcsglbaQNJ49f422H3B/2f0p++vBKWTDc7vJxgDMJxvDcCywAbgk\nIko5sddPf1PIdnWD7KrNFyuO19vd31lkIyzXsGcA0Q1kx+mlb8OC/qbR5m1YSiCYWWfySUUzSxwI\nZpY4EMwscSCYWeJAMLOk1EDo4GHBgPtrVCf318m9QXn9lb2H0NH/KLi/RnVyf53cG5TUX9mBYGYd\npKGBSZLOBb5DNuLw+xHxb1Wm9ygos5JERNX7Vw44EJR9CtFzwDlkf8L8K2BaRDxdMI8DwawktQRC\nI4cMvtGJ2RDTSCAMhhudmFkdBvrhIzXLL590+hldM6OxQKjpRicRMYfsdto+h2DW4Ro5ZOjoG52Y\nWf0GvIcQEW9Luhr4L/bc6GRd0zozs7Zr6w1SfMhgVp5WX3Y0syHGgWBmiQPBzBIHgpklDgQzSxwI\nZpY4EMwscSCYWeJAMLPEgWBmiQPBzBIHgpklDgQzSxwIZpY4EMwscSCYWeJAMLPEgWBmiQPBzBIH\ngpklDgQzSxwIZpY4EMwscSCYWeJAMLPEgWBmiQPBzBIHgpklDgQzSxwIZpY4EMws2a+RmSWtB3qA\n3cDbETGpGU2ZWTkaCoTcxyNiaxOWY2Yl8yGDmSWNBkIAj0taKamrGQ2ZWXkaPWQ4KyI2SnofsFDS\nsxGxtHKCPCgcFmaDgCKiOQuSZgE7I+LmgmmaszIzq1tEqNo0Az5kkHSQpJG93wOfAtYOdHlmVr5G\nDhnGAD+W1Luc+yPiZ03pysxK0bRDhppW5kMGs9K09JDBzIYeB4KZJQ4EM0scCGaWOBDMLHEgmFnS\njL92tA5xxRVXFNarXWJ+5ZVXCusTJkworC9fvrywvmzZssK6lc97CGaWOBDMLHEgmFniQDCzxIFg\nZokDwcwSB4KZJUNqHMK0adMK66eddlphvdp1/E53yCGHNDT/7t27C+v7779/Yf3NN98srL/xxhuF\n9TVr1hTWL7nkksL6yy+/XFi36ryHYGaJA8HMEgeCmSUOBDNLHAhmljgQzCxxIJhZMqhuwz579uzC\n+owZMwrrw4YNa2T1VrIlS5YU1quNQ+nu7m5mO4OOb8NuZnVxIJhZ4kAws8SBYGaJA8HMEgeCmSUO\nBDNLBtU4hBdeeKGwfvTRRxfWV69eXViv9vf8rVbtcwvmz5/fpk4G5pxzzimsX3755YX1cePGNbT+\nauMULr300sL6UL+fQlPGIUiaK2mLpLUVzx0qaaGk5/Ovoxtt1szKV8shw93AuX2eux5YHBEnAovz\nn81skKsaCBGxFHi1z9MXAPfk398DXNjkvsysBAM9qTgmIjbn378EjGlSP2ZWooZvshoRUXSyUFIX\n0NXoesys9Qa6h9AtaSxA/nVLfxNGxJyImBQRkwa4LjNrk4EGwgJgev79dOCR5rRjZmWqOg5B0jxg\nCnAY0A18FZgPPAAcC2wALomIvice97ashsYhvP/97y+sn3rqqYX1RYsWFdZ7enrq7slqN378+ML6\nY489VlifMGFCQ+u/7rrrCuvV7rcx2NUyDqHqOYSI6O+uE5+ouyMz62geumxmiQPBzBIHgpklDgQz\nSxwIZpY4EMwsGVT3Q7Ch7eKLLy6sP/jggw0tf+vWrYX1ww8/vKHldzp/LoOZ1cWBYGaJA8HMEgeC\nmSUOBDNLHAhmljgQzCxxIJhZ4kAws8SBYGaJA8HMEgeCmSUOBDNLHAhmljgQzCxp+KPczGp15ZVX\nFtY//OEPt3T9BxxwQGH99NNPL6yvXLmyme10JO8hmFniQDCzxIFgZokDwcwSB4KZJQ4EM0scCGaW\n+HMZhpCxY8cW1i+77LLC+syZM5vZzrtU60+q+rEBLbVjx47C+qhRo9rUSWs05XMZJM2VtEXS2orn\nZknaKGlV/jiv0WbNrHy1HDLcDZy7l+dvjYiJ+eMnzW3LzMpQNRAiYinwaht6MbOSNXJS8WpJq/ND\nitFN68jMSjPQQLgDOB6YCGwGZvc3oaQuSSskrRjgusysTQYUCBHRHRG7I+Id4HvA5IJp50TEpIiY\nNNAmzaw9BhQIkiqvH10ErO1vWjMbPKreD0HSPGAKcJikF4GvAlMkTQQCWA98sYU97jM++clPFtar\n/b1+V1dXYX38+PF197QvmTt3btktlK5qIETEtL08fVcLejGzknnospklDgQzSxwIZpY4EMwscSCY\nWeJAMLPEn8vQRCeccEJh/c477yysn3322YX1Vt8vYMOGDYX11157raHl33jjjYX1Xbt2FdZvv/32\nwvpJJ51Ud0+VNm3a1ND8Q4H3EMwscSCYWeJAMLPEgWBmiQPBzBIHgpklDgQzSzwOoQ7XXHNNYf2q\nq64qrB9//PGF9Z07dxbWt23bVli/7bbbCuvVrrMvX768sF5tnEKrbd++vaH5e3p6CuuPPvpoQ8sf\nCryHYGaJA8HMEgeCmSUOBDNLHAhmljgQzCxxIJhZ4nEIdTjzzDML69XGGSxYsKCwPnt2v5+IB8DS\npUsL64PdxIkTC+vHHXdcQ8uvdr+FZ599tqHlDwXeQzCzxIFgZokDwcwSB4KZJQ4EM0scCGaWOBDM\nLPE4hDp86UtfKqyvXr26sH7TTTc1s50hp9rnWowZM6ah5S9atKih+fcFVfcQJB0jaYmkpyWtkzQj\nf/5QSQslPZ9/Hd36ds2slWo5ZHgbuDYiTgHOAK6SdApwPbA4Ik4EFuc/m9kgVjUQImJzRPw6/74H\neAY4CrgAuCef7B7gwlY1aWbtUddJRUnjgA8BvwTGRMTmvPQS0NgBnpmVruaTipJGAA8BMyNiR+UH\nj0ZESIp+5usCuhpt1Mxar6Y9BEnDycLgvoh4OH+6W9LYvD4W2LK3eSNiTkRMiohJzWjYzFqnlqsM\nAu4CnomIWypKC4Dp+ffTgUea356ZtZMi9rqnv2cC6SzgCWAN8E7+9A1k5xEeAI4FNgCXRMSrVZZV\nvDLbp918882F9WuvvbawXu1zK6ZOnVpYf/LJJwvrg11EqNo0Vc8hRMQyoL8FfaLepsysc3nospkl\nDgQzSxwIZpY4EMwscSCYWeJAMLPE90OwtlmzZk1h/eSTT25o+Y8//nhhfaiPM2gG7yGYWeJAMLPE\ngWBmiQPBzBIHgpklDgQzSxwIZpZ4HIK1zbhx4wrr++1X/Ou4ffv2wvqtt95ab0vWh/cQzCxxIJhZ\n4kAws8SBYGaJA8HMEgeCmSUOBDNLPA7BmmbatGmF9QMPPLCw3tPTU1jv6ir+REDf76Bx3kMws8SB\nYGaJA8HMEgeCmSUOBDNLHAhmljgQzCxRRLRvZVL7VmZNN3z48ML6U089VViv9rkL8+bNK6x//vOf\nL6xbsYhQtWmq7iFIOkbSEklPS1onaUb+/CxJGyWtyh/nNaNpMytPLSMV3waujYhfSxoJrJS0MK/d\nGhE3t649M2unqoEQEZuBzfn3PZKeAY5qdWNm1n51nVSUNA74EPDL/KmrJa2WNFfS6H7m6ZK0QtKK\nhjo1s5arORAkjQAeAmZGxA7gDuB4YCLZHsTsvc0XEXMiYlJETGpCv2bWQjUFgqThZGFwX0Q8DBAR\n3RGxOyLeAb4HTG5dm2bWDrVcZRBwF/BMRNxS8fzYiskuAtY2vz0za6darjJ8BPgnYI2kVflzNwDT\nJE0EAlgPfLElHVrHqDZm5f777y+sr1q1qrC+cOHCwrq1Xi1XGZYBexvQ8JPmt2NmZfLQZTNLHAhm\nljgQzCxxIJhZ4kAws8SBYGaJ74dgto9oyv0QzGzf4UAws8SBYGaJA8HMEgeCmSUOBDNLHAhmltRy\nP4Rm2gpsqPj5sPy5TuX+GtPJ/XVyb9D8/o6rZaK2Dkx618qlFZ18r0X315hO7q+Te4Py+vMhg5kl\nDgQzS8oOhDklr78a99eYTu6vk3uDkvor9RyCmXWWsvcQzKyDOBDMLHEgmFniQDCzxIFgZsn/A7Ef\nH9ZUofg/AAAAAElFTkSuQmCC\n", 230 | "text/plain": [ 231 | "" 232 | ] 233 | }, 234 | "metadata": {}, 235 | "output_type": "display_data" 236 | }, 237 | { 238 | "data": { 239 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQQAAAEMCAYAAAAiW8hnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFDlJREFUeJzt3X2wXHV9x/H3RxO1TXgIJIZAISmKIGhFDAyOIUIdgTAF\nkrGlBmqjFcIwMgOk/oGAEAsJ2ikImXaCUSgogoNAhDKg0hRDqJZCACHJDWBpeAghD1AggIKQb/84\nJz+WmPs7u3f37tncfF4zO/fe/Z6H757kfvY8/O5ZRQRmZgDvqrsBM+sdDgQzSxwIZpY4EMwscSCY\nWeJAMLPEgbAFSbMlXVvTuveW9Eqnp92eSLpW0uxOzytphqQ72ultWzCkA0HSeyVdKelJSRslPSRp\nSkP9cEnPDHDZe0l6peERkl5t+PmwVpcZEU9ExMhOT9uq8hfjjXKbbZT0iKQ5knZsYRnPSDq8jR7u\nkfTFgc7faRFxTURMqZ7ynSRdUm6LlyWtknT2YPTXKUM6EIBhwNPAp4GdgPOAGyRNaHfBEfFURIzc\n/Cif/ljDc0u2nEfSu9tdbxfNjYgdgDHAl4HDgCWS/qjetrY5C4APRcSOFNvwi5KOq7mnfg3pQIiI\nVyNidkSsiohNEXEb8L/AJySNAO4Adm94V9+9nPU9kr5fvjsulzRxIOsv32n/RdJPJb0KHCbpuHJP\n5WVJT0n6esP0H5QUDT/fI+kbkn5Z9vJTSbu0Om1Z/1K5vg2Szmn2HTwifhcR/w0cC+wGzCiXt4+k\nuyS9UC7zB5J2KmvXA7sDd5TbdZakd0m6UdJzkl6U9AtJHx7ANm1mOWMkLSq3w12S9myYf39J/172\nvVLS55pc78mSftHQwzxJ6yS9JOlhSftvbb6IeDQiXtv8I7AJ+GCrr7tbhnQgbEnSWOBDwPKIeBWY\nAjzb8K7+bDnpccCPgJ2BW4F/bmO1JwLfAHYAfgW8ApxULvtY4AxJf1Ex/wxgLDACmNXqtJI+CswD\nPg/sQfGuv1srLyIiXgIWUbzLAQi4qFzO/sDewNfLaacDzwJTyu16aTnPbcA+5TzLgB+00kODquX8\nDXA+MBpYsbkuaSRwJ/B94P0U/w4LJO3b4vqnAIeWPYyi2K4v9DexpHPLN4SngfcC17e4vq7ZbgJB\n0nDgh8A1EbGyYvJ7IuL2iHiL4j/Tx9pY9cKI+FW5h/J6RPxHRCwvf/41RfB8OjP/lRHxePku82Pg\nwAFM+1fATyLilxHxOsWh00A8C+wCEBGPRcSiiHgjItYB3869jvL1Xh0RGyPid8Bs3t5Ta1qTy/m3\niPjP8rWeA0yWNA44HngsIr4fEW9GxFLgJ8BfttID8HtgR2C/sqcVEfFcpuc5wEjgE8C1wMstrq9r\ntotAkPQuil/sN4DTm5il8R/3NeB9koYNcPVPb9HLJ8vd3PWSXgJOpngna7aX3InE/qbdvbGPcu/o\n/5rofUt7UL4TStpN0g2SVkt6GbiazOuQ9G5J/yjpiXL635Sl3Gsf6HIaX+tLwEsU22A88KnyUONF\nSS8Cfw2Ma6WHiPg5cAUwH1gr6QpJO1TMExHxAEWYXNDK+rppyAeCJAFXUuxGfy4ift9Q7safem65\njh8BNwF7RsROwPcodr8H0xrgTzb/UL6bjmplASquMPw5sPlk6beA14GPlifMvsg7X8eWr/tvgWPK\nZezE28fRrb72ZpbTeM5gp3K6ZymCYlFE7NzwGBkRzbxJvENEXBYRBwEfoThkyh3KNRoGfKDV9XXL\nkA8EihT/MHBsRPx2i9paYNfNJ8O6ZAfghYj4naRDKY4/B9uPgamSDpX0HuAfmp1RxaXbicAtwHqK\n428oXserwEvlSbuvbjHrWorzCjRM/zrwPPDHwJwmVj9c0vsaHsObXM6x5Z7YeynOcyyJiDUU54MO\nkHSipOHl45BWzyGU8xxS7jW+SrHnuWkr0w2XdIqkncsTkZ8ETqM4F9OThnQgSBoPnEpxLP2c3r6a\ncBJAeS7heuCJchdy98ziOuU04GJJGymOb28Y7BVGxMPAWRTB8CzFL9PzFL9Y/Tmn7PF54Brgv4BP\nNZwxvwA4hGJ3/FaKvZ5Gc4FvlNv1TOBfy3U/CywHftlE6wuA3zY8vtvkcq6lCIINwJ9R7FVsPnw4\niuKk4xqKQ6yLKU70tWJnir3OF4FV5bIu3cp0QXF+4gmK8wZXl9PNb3F93RMRXX8ARwOPUhz/nV1H\nDxX9rQIeAR4C7u+Bfq4C1gHLGp7bheKM+ePl11EtLG9Hine0PQexv9nA6nIbPgQcU+P22xO4i+KK\nw3LgjHa3YZf66/o2rOPFvxv4H4rdyfcAvwb2r+s/Sz89rgJG191HQz+TgYO2+IX7x81hCpwNfKti\nGcdR7GKPpHin7VjQ9dPfbOCrdW+7spdxwEHl9zsAj1Ec97e0DWvor+vbsI5DhkOA30Qx9PYNipNs\nx9fQxzYjIu7mD69zH0+xK0/5dWrFYqZR7GY/A0wApg9yfz0jItZEcYafiNgI9FFcMWl1G3a7v66r\nIxD24J2X4p6hphefEcDPJS2VNLPuZvoxNooTZVAcC4/NTRwRX4q3z6x/NiIeH/wWOb0cxXeVpJau\nagwWFcPWPw7cS4vbsBu26A+6vA2H9EnFNkyK4pLSFOArkibX3VBOFPuavXa33PkUl9cOpDjpdkm9\n7aSRijcBZ0bEOwYH9cI23Ep/Xd+GdQTCahquE1NcH19dQx/9iojV5dd1wEKKw5xes7YcfUf5dV3N\n/bxDRKyNiLciYhPFOYtat2F5yfIm4IcRcXP5dM9sw631V8c2rCMQ7gP2kfSn5TXxz1NctuoJkkZs\nHnVWDuA5kmK8fK+5lfIPjcqvt9TYyx/Y/ItWmkaN27BhcFpfvP13FdAj27C//urYhirPbHaVpGOA\nyyiuOFwVxVjvniBpb4q9AihGlV1Xd38q/nrwcIrhuWspxgD8hGIMw17Ak8AJEVHLib1++jucYlc3\nKK7anNpwvN7t/iZRjLB8hLcHEJ1DcZxe+zbM9DedLm/DWgLBzHqTTyqaWeJAMLPEgWBmiQPBzBIH\ngpkltQZCDw8LBtxfu3q5v17uDerrr+49hJ7+R8H9tauX++vl3qCm/uoOBDPrIW0NTJJ0NHA5xYjD\n70XENyum9ygos5pEROX9KwccCCo+hegx4LMUf8J8HzA9IlZk5nEgmNWkmUBo55DBNzoxG2LaCYRt\n4UYnZtaCgX74SNPKyye9fkbXzGgvEJq60UlELKC4nbbPIZj1uHYOGXr6Ridm1roB7yFExJuSTgd+\nxts3Olnesc7MrOu6eoMUHzKY1WewLzua2RDjQDCzxIFgZokDwcwSB4KZJQ4EM0scCGaWOBDMLHEg\nmFniQDCzxIFgZokDwcwSB4KZJQ4EM0scCGaWOBDMLHEgmFniQDCzxIFgZokDwcwSB4KZJQ4EM0sc\nCGaWOBDMLHEgmFniQDCzxIFgZokDwcwSB4KZJQ4EM0uG1d2A9Y7x48dn6yeffHK2fu6552brEZGt\nS/lPK+/r68vWzzvvvGx94cKF2bq1GQiSVgEbgbeANyNiYieaMrN6dGIP4YiI2NCB5ZhZzXwOwcyS\ndgMhgJ9LWippZicaMrP6tHvIMCkiVkt6P3CnpJURcXfjBGVQOCzMtgFt7SFExOry6zpgIXDIVqZZ\nEBETfcLRrPcNOBAkjZC0w+bvgSOBZZ1qzMy6T1XXhvudUdqbYq8AikOP6yJiTsU8A1uZNWXMmDHZ\n+te+9rVs/aSTTsrWd91112y9ahxBu+MQquZ/+umns/WDDz44W9+wYWhfLIuI/AamjXMIEfEE8LGB\nzm9mvceXHc0scSCYWeJAMLPEgWBmiQPBzBIHgpklAx6HMKCVeRxCW6ruN3DhhRdm63WPA1i/fn22\nXmX06NHZ+oQJE7L1FStWZOsHHHBAqy1tU5oZh+A9BDNLHAhmljgQzCxxIJhZ4kAws8SBYGaJA8HM\nEo9D2Ibcd9992fpBBx2Urbc7DqHqOv4RRxyRrbd7v4FJkyZl64sXL87Wq17/sGFD+2NKPA7BzFri\nQDCzxIFgZokDwcwSB4KZJQ4EM0scCGaWeBxCD9lvv/2y9apxCM8//3y2XnU/gqpxAmeddVa2fuaZ\nZ2brc+fOzdafeuqpbL1K1f/lTZs2ZeunnXZatr5gwYKWe+olHodgZi1xIJhZ4kAws8SBYGaJA8HM\nEgeCmSUOBDNLPA5hG1I1TqFqHEG79yOYOXNmtj5//vxs/eCDD87WH3jggWx92rRp2fqNN96YrVf9\nX99tt92y9Xa3X906Mg5B0lWS1kla1vDcLpLulPR4+XVUu82aWf2aOWS4Gjh6i+fOBhZFxD7AovJn\nM9vGVQZCRNwNvLDF08cD15TfXwNM7XBfZlaDgZ5UHBsRa8rvnwPGdqgfM6tR23eVjIjInSyUNBPI\nn40ys54w0D2EtZLGAZRf1/U3YUQsiIiJETFxgOsysy4ZaCDcCswov58B3NKZdsysTpWHDJKuBw4H\nRkt6BrgA+CZwg6QvA08CJwxmk1ZYuXJlreuvup/Co48+mq1X3a+h6n4LZ5+dv5hV9bkSgz1OYyio\nDISImN5P6TMd7sXMauahy2aWOBDMLHEgmFniQDCzxIFgZokDwcyStocuW++YPHlytl51P4WqcQZ9\nfX3Z+r777put33vvvdn6mDFjsvWq+xlU9T9lypRs3byHYGYNHAhmljgQzCxxIJhZ4kAws8SBYGaJ\nA8HMEo9DGEJOPPHEbP2UU07J1qvuJ1A1DqBq/qpxBu3ez2DevHnZetXnPpj3EMysgQPBzBIHgpkl\nDgQzSxwIZpY4EMwscSCYWeJxCNuRqnEEdc+/ZMmSbH3WrFnZuscZtM97CGaWOBDMLHEgmFniQDCz\nxIFgZokDwcwSB4KZJR6HMIRcd9112fr48eOz9dGjR2frVZ/rMGLEiGy9yvnnn5+te5zB4KvcQ5B0\nlaR1kpY1PDdb0mpJD5WPYwa3TTPrhmYOGa4Gjt7K89+OiAPLx+2dbcvM6lAZCBFxN/BCF3oxs5q1\nc1LxdEkPl4cUozrWkZnVZqCBMB/4AHAgsAa4pL8JJc2UdL+k+we4LjPrkgEFQkSsjYi3ImIT8F3g\nkMy0CyJiYkRMHGiTZtYdAwoESeMafpwGLOtvWjPbdqiJe+1fDxwOjAbWAheUPx8IBLAKODUi1lSu\nTGrvD+qtVlXjEC666KJsferUqdn6gw8+mK1PmTIlW6/63IbtXUTkP/iCJgYmRcT0rTx95YA6MrOe\n5qHLZpY4EMwscSCYWeJAMLPEgWBmiQPBzJLKcQgdXdk2Pg5hzJgx2fr69eu71Mm26Y477sjWjzrq\nqGy96nMZLrvsspZ72p40Mw7BewhmljgQzCxxIJhZ4kAws8SBYGaJA8HMEgeCmSX+XIYGkydPztYv\nuaTfO8UBsHLlymz9C1/4Qss9DSVz5szJ1o888shsfd999+1kO7YV3kMws8SBYGaJA8HMEgeCmSUO\nBDNLHAhmljgQzCzZrsYhVN3P4IorrsjW161bl61v7+MMRowYka1/5zvfydalyj/Xt0HmPQQzSxwI\nZpY4EMwscSCYWeJAMLPEgWBmiQPBzJLtahzCtGnTsvWqv7dfvHhxJ9vZ5uy3337Z+k033ZStV23f\nqs8IqbrfhLWvcg9B0p6S7pK0QtJySWeUz+8i6U5Jj5dfRw1+u2Y2mJo5ZHgT+PuI2B84FPiKpP2B\ns4FFEbEPsKj82cy2YZWBEBFrIuKB8vuNQB+wB3A8cE052TXA1MFq0sy6o6WTipImAB8H7gXGRsSa\nsvQcMLajnZlZ1zV9UlHSSOAm4MyIeLnxD1EiIvr7IFdJM4GZ7TZqZoOvqT0EScMpwuCHEXFz+fRa\nSePK+jhgq38KGBELImJiREzsRMNmNniaucog4EqgLyIubSjdCswov58B3NL59sysm1R17VfSJGAJ\n8AiwqXz6HIrzCDcAewFPAidExAsVy8qvbJBVXUfv6+vL1lesWJGtX3zxxW0tf+nSpdl6lfHjx2fr\nhx12WLZeNU5j6tT8eeOq+xlU/V+7/PLLs/VZs2Zl65YXEZU3nKg8hxAR9wD9LegzrTZlZr3LQ5fN\nLHEgmFniQDCzxIFgZokDwcwSB4KZJZXjEDq6sprHIVS58cYbs/XBvg7/4IMPZutV9tprr2x91113\nzdbb7b9q/jlz5mTr8+bNy9Y3bNiQrVteM+MQvIdgZokDwcwSB4KZJQ4EM0scCGaWOBDMLHEgmFni\ncQgNxowZk63ffvvt2frEifmbQm3atClbH+xxAFXzv/baa9l61ecizJ07N1tfuHBhtm6Dy+MQzKwl\nDgQzSxwIZpY4EMwscSCYWeJAMLPEgWBmicchtGD06NHZ+oUXXtjW8mfOzH/i3c0335ytt3u/gKrP\nRagah2C9zeMQzKwlDgQzSxwIZpY4EMwscSCYWeJAMLPEgWBmicchmG0nOjIOQdKeku6StELSckln\nlM/PlrRa0kPl45hONG1m9ancQ5A0DhgXEQ9I2gFYCkwFTgBeiYh/anpl3kMwq00zewjDmljIGmBN\n+f1GSX3AHu23Z2a9pqWTipImAB8H7i2fOl3Sw5KukjSqn3lmSrpf0v1tdWpmg67pk4qSRgKLgTkR\ncbOkscAGIIALKQ4r/q5iGT5kMKtJM4cMTQWCpOHAbcDPIuLSrdQnALdFxEcqluNAMKtJp64yCLgS\n6GsMg/Jk42bTgGUDadLMekczVxkmAUuAR4DNHyxwDjAdOJDikGEVcGp5AjK3LO8hmNWkY4cMneJA\nMKuPb5BiZi1xIJhZ4kAws8SBYGaJA8HMEgeCmSUOBDNLHAhmljgQzCxxIJhZ4kAws8SBYGaJA8HM\nEgeCmSWVN1ntsA3Akw0/jy6f61Xurz293F8v9wad7298MxN19X4If7By6f6ImFhbAxXcX3t6ub9e\n7g3q68+HDGaWOBDMLKk7EBbUvP4q7q89vdxfL/cGNfVX6zkEM+stde8hmFkPcSCYWeJAMLPEgWBm\niQPBzJL/By402Tk7k2myAAAAAElFTkSuQmCC\n", 240 | "text/plain": [ 241 | "" 242 | ] 243 | }, 244 | "metadata": {}, 245 | "output_type": "display_data" 246 | }, 247 | { 248 | "data": { 249 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQQAAAEMCAYAAAAiW8hnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEzdJREFUeJzt3XuwXWV9xvHvwyWRkgtJoyGkiWjEtlAxSialBUKoN2Aq\nl2lrRToFRieMFQcQnUEGMCJiSgHTtBQnmBS8wSB3O0hNAwwXLTU4MRBIiWWCuXEiSbnFQm6//rHe\n87KNOWvvffZl7XPyfGb2nL33u9a7fnudc579rrXes48iAjMzgH2qLsDMeocDwcwyB4KZZQ4EM8sc\nCGaWORDMLHMg7EbSXEnfqWjb75T0WruX3ZtI+o6kue1eV9JZkn7YSm1DwbAPhPRN3ijpFUnPSvpU\nTdtsSesG2e9USa/V3ELS1prHxzXbZ0Q8FxGj2r1ss9I+2ybp1XR7UtJXJY1poo91kma3UMOjks4e\n7PrtFhE3R8RJg11f0gRJmyU91May2m7YBwLwNeDQiBgDnAJcKemoVjuNiF9GxKj+W3r6vTXPPbL7\nOpL2bXW7XXRVRIwG3gp8EjgOeETSAdWWNWT9A7Cy6iLqGfaBEBErI+KN/ofpNk3SgcAPgUNq3tUP\nScuNkPSt9O64UtKMwWw7vdNeL+l+SVuB4ySdIml5GrH8UtJlNcu/S1LUPH5U0pcl/TjVcr+k8c0u\nm9rPSdt7UdIljb6DR8TrEfFfwEeBg4GzUn+HSXpQ0pbU57cljU1ttwCHAD9M+/VzkvaRdLukFyS9\nJOkhSX84iH3aSD9vlbQ07YcHJU2pWf9wSf+R6l4l6S8a3O6n+t/dUw0LJG2S9LKkFZIOL1n3OOAw\n4NvNvt5uG/aBACDpXyT9GlgFbATui4itwEnAhpp39Q1plVOAW4GDgHuBf25h858AvgyMBn4CvAac\nmfr+KHC+pD+vs/5ZwETgQOBzzS4r6T3AAuDjwGSKd/2Dm3kREfEysJRipAAg4MrUz+HAO4HL0rJn\nABuAk9J+vS6t828UvxgHA08x+F+Qev38DXA5MAF4ur9d0ihgCfAt4G0U34eFkn6/ye2fBBydahhH\nsV+37GlBSfsB/wScR/Fm1NP2ikCIiL+j+IU8DrgTeKN8DR6NiPsiYifFD9N7W9j8XRHxk4jYFRFv\nRMQDadSyKyJ+ThE8x5esvygiVkfEr4HvA9MHsexfAXdHxI/TaOnSQb6WDcB4gIh4NiKWRsS2iNgE\nfL3sdaTXe1NEvBoRrwNzgaPSSK1hDfbzg4h4LL3WS4BZkiYBpwLPRsS3ImJHRDwB3A38ZTM1ANuB\nMcAfpJqejogXBlj2QuCRiFje5DYqsVcEAkBE7IyIR4HfAz5dZ/Hab+6vgbekpB+MtbUPJP1JGub+\nStLLwKco3skaraXsROJAyx5SW0caHf1vA7XvbjLpnVDSwZJuk7Re0ivATZS8Dkn7Srpa0nNp+V+k\nprLXPth+al/ry8DLFPvg7cAx6VDjJUkvAX8NTGqmhoj4EfAN4AagT9I3JI3eQ61TKH7WLtu9rVft\nNYFQYz9gWrrfjSHc7tu4FbgDmBIRY4FvUgy/O2kjRRACkN5NxzXTQbrC8GdA/8nSv6cYab0nnbA9\nm998Hbu/7r8FTk59jAXe1d91M3U02E/tOYOxabkNFEGxNCIOqrmNiojzmqyBiJgfEe8H/ojikGlP\nh3J/TBE2qyS9AFwL/Gm635OGdSBIepukj0sald5ZPgKcQXEsDNAH/G7/ybAuGQ1siYjXJR1NcfzZ\nad8HTpN0tKQRwBWNrihpZDqpeg/wK4rjbyhex1bg5fRO+PndVu2jOK9AzfJvAJuB3wG+2sDm95f0\nlprb/g3289E0EhtJcZ7jkYjYSHE+6AhJn5C0f7rNbPYcQlpnZho1bgW2Abv2sOgPgHdQHLpNpziX\ntIzyw75KDetAoHiX+jSwjmKIfA1wQUTcCxARq4BbgOfSEPKQAXtqn08DX5P0KsXx7W2d3mBErKA4\nlv0+xTvl5nQrO5dySapxM3Az8J/AMen8BMCXgJkUw/F7KUY9ta4Cvpz26wXAv6Ztb6C4/PbjBkpf\nCPxfze3GBvv5DkUQvAgcSTGq6D98+AjFSceNFIdYXwNGNlBLrYOARcBLwJrU13W7L5TOGb3QfwNe\nAbaVnG+oXkR0/QacCPw3xfHfxVXUUKe+NcCTwHJgWQ/UsxjYBDxV89x4ijPmq9PXcU30N4biHW1K\nB+ubC6xP+3A5cHKF+28K8CDFFYeVwPmt7sMu1df1fVjFi98X+B+K4eQI4OfA4VX9sAxQ4xpgQtV1\n1NQzC3j/br9wV/eHKXAx8Pd1+jiFYog9iuKdtm1BN0B9c4HPV73vUi2TgPen+6OBZymO+5vahxXU\n1/V9WMUhw0zgF1FMvd1GcZLt1ArqGDIi4mF++zr3qRRDedLX0+p0czrFMHsdcCjFuZRO1tczImJj\nRPws3X8VeIbiikmz+7Db9XVdFYEwmd+8FLeOil58iQB+JOkJSXOqLmYAE6M4UQbFsfDEsoUj4px4\n88z6hyJidedL5Lw0i2+xpKauanSKpEOB9wGP0+Q+7Ibd6oMu78PhflJxsI6N4pLSScBnJM2quqAy\nUYw1e20W3A0Ul3enU5x0u7bacvJMxTsoTiy/UtvWC/twD/V1fR9WEQjrqblOTHF9fH0FdQwoItan\nr5uAuygOc3pNX5p9R/q6qeJ6fkNE9EUxGWwXxTmLSvdhumR5B/DdiLgzPd0z+3BP9VWxD6sIhJ8C\nh0l6R7om/nGKy1Y9QdKB/bPO0gSeD1PMl+8195L+0Ch9vafCWn5L/y9acjoV7kNJorhM+Ey8+XcV\n0CP7cKD6qtiHSmc2u0rSycB8iisOiyOikUkqXSHpnRSjAihmNX6v6vpU/PXgbIrpuX0UcwDuppjD\nMBV4HvhYRFRyYm+A+mZTDHWD4qrNuTXH692u71iKGZZP8uYEoksojtMr34cl9Z1Bl/dhJYFgZr3J\nJxXNLHMgmFnmQDCzzIFgZpkDwcyySgOhh6cFA66vVb1cXy/XBtXVV/UIoae/Kbi+VvVyfb1cG1RU\nX9WBYGY9pKWJSZJOBP6RYsbhNyNiXp3lPQvKrCIRUffzKwcdCCr+C9GzwIco/oT5p8AZEfF0yToO\nBLOKNBIIrRwy+INOzIaZVgJhKHzQiZk1YbD/fKRh6fJJr5/RNTNaC4SGPugkIhZSfJy2zyGY9bhW\nDhl6+oNOzKx5gx4hRMQOSecB/86bH3Sysm2VmVnXdfUDUnzIYFadTl92NLNhxoFgZpkDwcwyB4KZ\nZQ4EM8scCGaWORDMLHMgmFnmQDCzzIFgZpkDwcwyB4KZZQ4EM8scCGaWORDMLHMgmFnmQDCzzIFg\nZpkDwcwyB4KZZQ4EM8scCGaWORDMLHMgmFnmQDCzzIFgZpkDwcwyB4KZZQ4EM8scCGaWORDMLNuv\nlZUlrQFeBXYCOyJiRjuKMrNqtBQIyQkR8WIb+jGzivmQwcyyVgMhgB9JekLSnHYUZGbVafWQ4diI\nWC/pbcASSasi4uHaBVJQOCzMhgBFRHs6kuYCr0XENSXLtGdjZta0iFC9ZQZ9yCDpQEmj++8DHwae\nGmx/Zla9Vg4ZJgJ3Serv53sRcX9bqrI9GjFiRGn70qVLS9uPOeaY0vb0vRzQSy+9VNp+5JFHlrav\nXbu2tN2qN+hAiIjngPe2sRYzq5gvO5pZ5kAws8yBYGaZA8HMMgeCmWUOBDPL2vHXjtYm9eYZLFq0\nqLS93jyDeu6+++7S9nnz5pW2b9iwoaXtd9rEiRNL2/v6+rpUSe/yCMHMMgeCmWUOBDPLHAhmljkQ\nzCxzIJhZ5kAws8zzEHrIRRddVNp+5plnttT/9ddfX9r+hS98obT99ddfb2n7nXbNNQN+WBcA55xz\nTmn7V77yldL2+fPnN13TUOMRgpllDgQzyxwIZpY5EMwscyCYWeZAMLPMgWBmmechdNERRxxR2n7p\npZe21P9rr71W2n7hhReWtu/YsaOl7XfajBkzStvPPvvs0vZx48a1sZrhySMEM8scCGaWORDMLHMg\nmFnmQDCzzIFgZpkDwcwyz0Pooosvvri0/YADDihtrzdP4JRTTmlp/V5X7/Maxo8fX9q+ffv20vZ6\n/5dib1B3hCBpsaRNkp6qeW68pCWSVqevnvFhNgw0cshwE3Dibs9dDCyNiMOApemxmQ1xdQMhIh4G\ntuz29KnAzen+zcBpba7LzCow2JOKEyNiY7r/AlD+T/PMbEho+aRiRISkGKhd0hxgTqvbMbPOG+wI\noU/SJID0ddNAC0bEwoiYERHlf6pmZpUbbCDcC5yV7p8F3NOecsysSnUPGSTdAswGJkhaB3wJmAfc\nJumTwPPAxzpZ5HBx1FFHtbT+/fffX9r+0EMPtdT/vvvuW9o+YsSIlvqvZ9q0aaXtxx9/fEv93377\n7aXta9asaan/4aBuIETEGQM0faDNtZhZxTx12cwyB4KZZQ4EM8scCGaWORDMLHMgmFnmz0MYQkaO\nHNnS+jNnzixtv/LKK0vbP/jBD7a0/U7r6+srbb/qqqu6VMnQ5RGCmWUOBDPLHAhmljkQzCxzIJhZ\n5kAws8yBYGaZ5yF00dVXX13avnjx4tL2E044obT9gQceKG2fNWtWafs++wzt94cbb7yxtH3lypVd\nqmToGto/AWbWVg4EM8scCGaWORDMLHMgmFnmQDCzzIFgZpnnIXTR1KlTW1p/v/3Kv12zZ89uqf/H\nH3+8tP2uu+4qbZ88eXJp+2c/+9mma2rGsmXLOtr/3sAjBDPLHAhmljkQzCxzIJhZ5kAws8yBYGaZ\nA8HMMs9D6KJ6n3ewbdu2jm7/1ltvLW1fu3ZtafvOnTtL27/4xS82XVMzHnvssdL2++67r6Pb3xvU\nHSFIWixpk6Snap6bK2m9pOXpdnJnyzSzbmjkkOEm4MQ9PP/1iJiebo5ms2GgbiBExMPAli7UYmYV\na+Wk4nmSVqRDinFtq8jMKjPYQLgBmAZMBzYC1w60oKQ5kpZJ8l+emPW4QQVCRPRFxM6I2AXcCAz4\nb4UjYmFEzIiIGYMt0sy6Y1CBIGlSzcPTgacGWtbMho668xAk3QLMBiZIWgd8CZgtaToQwBrg3A7W\nOGysW7eutH3evHldqqQztm7d2tH+FyxYUNq+Y8eOjm5/b1A3ECLijD08vagDtZhZxTx12cwyB4KZ\nZQ4EM8scCGaWORDMLHMgmFnmz0Owtqn3eQn17Nq1q7R99erVLfVv9XmEYGaZA8HMMgeCmWUOBDPL\nHAhmljkQzCxzIJhZ5nkI1jbnntvax2IsWbKktH358uUt9W/1eYRgZpkDwcwyB4KZZQ4EM8scCGaW\nORDMLHMgmFnmeQjWsLFjx5a2jxkzpqX+58+f39L61jqPEMwscyCYWeZAMLPMgWBmmQPBzDIHgpll\nDgQzyzwPwRo2c+bM0vapU6eWtm/fvr20ffPmzU3XZO1Vd4QgaYqkByU9LWmlpPPT8+MlLZG0On0d\n1/lyzayTGjlk2AFcFBGHA0cDn5F0OHAxsDQiDgOWpsdmNoTVDYSI2BgRP0v3XwWeASYDpwI3p8Vu\nBk7rVJFm1h1NnVSUdCjwPuBxYGJEbExNLwAT21qZmXVdwycVJY0C7gAuiIhXJOW2iAhJMcB6c4A5\nrRZqZp3X0AhB0v4UYfDdiLgzPd0naVJqnwRs2tO6EbEwImZExIx2FGxmndPIVQYBi4BnIuK6mqZ7\ngbPS/bOAe9pfnpl1kyL2ONJ/cwHpWOAR4ElgV3r6EorzCLcBU4HngY9FxJY6fZVvzHraqlWrStvf\n/e53l7Zv2VL648GECROarskaFxGqt0zdcwgR8SgwUEcfaLYoM+tdnrpsZpkDwcwyB4KZZQ4EM8sc\nCGaWORDMLPPnIVjDRo4c2dL6K1asaFMl1ikeIZhZ5kAws8yBYGaZA8HMMgeCmWUOBDPLHAhmlnke\ngnXNzp07qy7B6vAIwcwyB4KZZQ4EM8scCGaWORDMLHMgmFnmQDCzzPMQrGtmzZpV2n755ZeXtl9x\nxRXtLMf2wCMEM8scCGaWORDMLHMgmFnmQDCzzIFgZpkDwcwyz0Owhi1YsKC0/bLLLittP+igg0rb\nd+3a1XRN1l51RwiSpkh6UNLTklZKOj89P1fSeknL0+3kzpdrZp3UyAhhB3BRRPxM0mjgCUlLUtvX\nI+KazpVnZt1UNxAiYiOwMd1/VdIzwOROF2Zm3dfUSUVJhwLvAx5PT50naYWkxZLGDbDOHEnLJC1r\nqVIz67iGA0HSKOAO4IKIeAW4AZgGTKcYQVy7p/UiYmFEzIiIGW2o18w6qKFAkLQ/RRh8NyLuBIiI\nvojYGRG7gBuBmZ0r08y6oZGrDAIWAc9ExHU1z0+qWex04Kn2l2dm3aSIKF9AOhZ4BHgS6L9QfAlw\nBsXhQgBrgHPTCciyvso3ZmYdExGqt0zdQGgnB4JZdRoJBE9dNrPMgWBmmQPBzDIHgpllDgQzyxwI\nZpY5EMwscyCYWeZAMLPMgWBmmQPBzDIHgpllDgQzyxwIZpZ1+/8yvAg8X/N4QnquV7m+1vRyfb1c\nG7S/vrc3slBXPw/htzYuLevlz1p0fa3p5fp6uTaorj4fMphZ5kAws6zqQFhY8fbrcX2t6eX6erk2\nqKi+Ss8hmFlvqXqEYGY9xIFgZpkDwcwyB4KZZQ4EM8v+Hyp7/AmhJxHPAAAAAElFTkSuQmCC\n", 250 | "text/plain": [ 251 | "" 252 | ] 253 | }, 254 | "metadata": {}, 255 | "output_type": "display_data" 256 | } 257 | ], 258 | "source": [ 259 | "trainimg = mnist.train.images\n", 260 | "trainlabel = mnist.train.labels\n", 261 | "nsample = 1\n", 262 | "randidx = np.random.randint(trainimg.shape[0], size=nsample)\n", 263 | "\n", 264 | "for i in [0, 1, 2]:\n", 265 | " curr_img = np.reshape(trainimg[i, :], (28, 28)) # 28 by 28 matrix \n", 266 | " curr_label = np.argmax(trainlabel[i, :] ) # Label\n", 267 | " plt.matshow(curr_img, cmap=plt.get_cmap('gray'))\n", 268 | " plt.title(\"\" + str(i + 1) + \"th Training Data \" \n", 269 | " + \"Label is \" + str(curr_label))" 270 | ] 271 | }, 272 | { 273 | "cell_type": "markdown", 274 | "metadata": { 275 | "deletable": true, 276 | "editable": true 277 | }, 278 | "source": [ 279 | "## Softmax Regressions" 280 | ] 281 | }, 282 | { 283 | "cell_type": "markdown", 284 | "metadata": { 285 | "deletable": true, 286 | "editable": true 287 | }, 288 | "source": [ 289 | "我們知道 MNIST 裡面的手寫數字圖片都是從 0 到 9 的數字，因此我們想要知道給定一張圖片的時候對應到各個數字的機率是多少．例如我們的模型可能看了一張數字為 9 的圖片，模型不可能百分之百的確定他是某個數字，可能覺得有 80% 的機率為 9．然後有 5% 的機率是 8．剩餘的機率為其他數字．\n", 290 | "\n", 291 | "`softmax regression` 是一個簡單且自然的模型，而 MNIST 就是一個經典的例子．`softmax` 的作用就是把機率分配給許多的類別，並且這些類別機率的加總為 1．即使是之後更複雜的模型，在模型最後一個階段仍然是使用 `softmax` 來分配機率．\n", 292 | "\n", 293 | "`softmax regression` 有兩個步驟，第一個我們必須先蒐集對應類別的**證據** (`evidence`)，然後再把 `evidence` 轉化成機率．\n", 294 | "\n", 295 | "為了得到一張給定圖片屬於哪一個數字的證據 (`evidence`) ，我們把圖片的 pixenl 加權求和．權重是負的表示這部分的 pixel 有很強的證據顯示突變並不屬於該數字，而如果很可能是這個數字，權重就會是正的．\n", 296 | "\n", 297 | "下面的圖片顯示了模型從每個數字學習到的權值．紅色表示負權重，藍色表示正權重．\n", 298 | "\n", 299 | "![](https://www.tensorflow.org/images/softmax-weights.png)\n", 300 | "\n", 301 | "我們同樣加入了額外的 evidence 稱作**偏移量** (bias)．基本上我們會說這個證據是和輸入比較沒有關係的．結果是給定一個輸入 **x** 對應到的數字 **i** 他的證據如下:\n", 302 | "\n", 303 | "$$\\text{evidence}_i = \\sum_j W_{i,~ j} x_j + b_i$$\n", 304 | "\n", 305 | "$$\\text{y= softmax(evidence)}$$\n", 306 | " \n", 307 | "在這裡 `softmax` 代表著一種`激勵函數`或是`連結函數`，把線性輸出轉化成我們需要的形式，在這裡則是把它轉成 10 個類別．你可以想把 `evidence` 轉化成每個數字類別的機率．它的定義如下：\n", 308 | "\n", 309 | "$$\\text{softmax}(x)= \\text{normalize(exp(x))}$$\n", 310 | "\n", 311 | "如果我們把這個等式展開會得到：\n", 312 | "\n", 313 | "![](http://imgur.com/E2qHl47.jpg)\n", 314 | " \n", 315 | "\n", 316 | "通常如果我們把 softmax 函數想像成這樣的形式會幫助我們理解：把輸入指數化並且正規化．指數話的意思為如果一個 `evidence` 權重值增加的話，會對輸出的 hypothesis 貢獻更多的比重．相對的，如果權重值減少，在 hypothesis 中會減少得更多．注意到沒有一個 hypothesis 有 0 或是負數的值．Softmax 同時會正規化這些權重，讓他加總會是 1，成為一個機率分佈（如果想要用更直覺的方式了解 softmax，可以參考 Michael Nielsen 的書，裡面有完整的互動式視覺化．\n", 317 | "\n", 318 | "現在可以把 softmax 回歸表示成以下的圖像．對於每個輸出，我們計算其權重和並且加上偏移量，最後用上 softmax．\n", 319 | "\n", 320 | "![](https://www.tensorflow.org/images/softmax-regression-scalargraph.png)\n", 321 | "\n", 322 | "如果把它寫成等式，我們會得到：\n", 323 | "\n", 324 | "![](https://www.tensorflow.org/images/softmax-regression-scalarequation.png)\n", 325 | "\n", 326 | "我們可比這些過程向量化，把它變成一系列的矩陣相乘，還有相加．這些過程對於計算效率有很大的幫助（同時也幫助我們思考）\n", 327 | "\n", 328 | "更精簡地，我們可以直接寫下：\n", 329 | "\n", 330 | "$$y = \\text{softmax}(Wx+b)$$\n", 331 | "\n", 332 | "現在我們可以來把這些過程用 Tensorflow 來實現．" 333 | ] 334 | }, 335 | { 336 | "cell_type": "markdown", 337 | "metadata": { 338 | "collapsed": false, 339 | "deletable": true, 340 | "editable": true 341 | }, 342 | "source": [ 343 | "## 實現 Regression" 344 | ] 345 | }, 346 | { 347 | "cell_type": "markdown", 348 | "metadata": { 349 | "deletable": true, 350 | "editable": true 351 | }, 352 | "source": [ 353 | "我們通常會用 Numpy 這類的套件來在 Python 中更有效率地處理像是矩陣相乘這樣的數值運算，而它會把這些計算移到 Python 外面，使用別種的程式語言還更有效率的實現方法．很不幸的是這樣的方法當把結果移回 Python 的時候會有 overhead 的情形．特別是在程式執行在 GPUs 或者分散式系統的時候，移動資料的成本會變得非常的高．\n", 354 | "\n", 355 | "Tensorflow 同樣的也把這些計算移到 Python 外，但是它用了一些方法來避免 overhead．它先讓我們先敘述一個交互操作的圖，然後再把所有交互計算的過程移到 Python 外面，而不是只是在 Python 外面執行單一個昂貴的操作．(這樣的方式可以在一些機器學習套件中看到)\n", 356 | "\n", 357 | "要開始執行 tensorflow 之前先讓我們 import 它．" 358 | ] 359 | }, 360 | { 361 | "cell_type": "code", 362 | "execution_count": 11, 363 | "metadata": { 364 | "collapsed": false, 365 | "deletable": true, 366 | "editable": true 367 | }, 368 | "outputs": [], 369 | "source": [ 370 | "import tensorflow as tf" 371 | ] 372 | }, 373 | { 374 | "cell_type": "markdown", 375 | "metadata": { 376 | "deletable": true, 377 | "editable": true 378 | }, 379 | "source": [ 380 | "我們用操作符號變數來描述這一些交互操作單元．讓我們來建立一個範例:" 381 | ] 382 | }, 383 | { 384 | "cell_type": "code", 385 | "execution_count": 12, 386 | "metadata": { 387 | "collapsed": true, 388 | "deletable": true, 389 | "editable": true 390 | }, 391 | "outputs": [], 392 | "source": [ 393 | "x = tf.placeholder(tf.float32, [None, 784])" 394 | ] 395 | }, 396 | { 397 | "cell_type": "markdown", 398 | "metadata": { 399 | "deletable": true, 400 | "editable": true 401 | }, 402 | "source": [ 403 | "`x` 不是一個特定的數值．他是一個佔位子 (`placeholder`)，是一個先要求 Tensorflow 預先保留的數值，在真正計算的時候才把數值輸入進去．我們想要可以輸入任意數量的 MNIST 圖片，每一張圖都會先轉化成 784 維的向量．用 2-D 的浮點數 tensor 來表現它．它的形狀是 `[None, 784]`．(這裡的 `None` 意味著它第一個維度可以是任一長度的．)\n", 404 | "\n", 405 | "我們的模型同時需要權重還有偏移值．可以把它們看作一個額外的輸入，但是 Tensorflow有定義更好的方法來表示他們: `Variable`．一個 `Variable` 就是一個在 Tensorflow 交互操作圖中可以被變更的 tensor．他可以在計算中被取用和變更．在機器學習中，通常模型參數就會用 `Variables` 來表示．" 406 | ] 407 | }, 408 | { 409 | "cell_type": "code", 410 | "execution_count": 13, 411 | "metadata": { 412 | "collapsed": true, 413 | "deletable": true, 414 | "editable": true 415 | }, 416 | "outputs": [], 417 | "source": [ 418 | "W = tf.Variable(tf.zeros([784, 10]))\n", 419 | "b = tf.Variable(tf.zeros([10]))" 420 | ] 421 | }, 422 | { 423 | "cell_type": "markdown", 424 | "metadata": { 425 | "deletable": true, 426 | "editable": true 427 | }, 428 | "source": [ 429 | "我們經由給予 `tf.Variable` 初始值的方式來建立 Variable: 在這個例子中我們同時建立了 `W` 和 `b` 這兩個都為 0 的 tensor．因為 `W` 和 `b` 都會在學習過程中被學習出來，所以我們並不需要特別考慮他的初始值是多少．\n", 430 | "\n", 431 | "值得注意的是 `W` 的形狀為 [784, 10] 因為我們想要把一個 784 維的向量經由矩陣相乘後產生一個 10 維的證據 (evidence）向量來表示不同的數字．b 則是一個長度為 10 的向量，然後我們可以把他加入最後的輸出中．\n", 432 | "\n", 433 | "我們現在可以來實現我們的模型了．它僅僅需要一行來定義它!" 434 | ] 435 | }, 436 | { 437 | "cell_type": "code", 438 | "execution_count": 14, 439 | "metadata": { 440 | "collapsed": true, 441 | "deletable": true, 442 | "editable": true 443 | }, 444 | "outputs": [], 445 | "source": [ 446 | "y = tf.nn.softmax(tf.matmul(x, W) + b)" 447 | ] 448 | }, 449 | { 450 | "cell_type": "markdown", 451 | "metadata": { 452 | "deletable": true, 453 | "editable": true 454 | }, 455 | "source": [ 456 | "首先我們把 `x` 乘上 `W` 這裡的表示方式就是 `tf.matmul(x, W)` 對應了之前的矩陣相乘的數學式 **Wx**，`x` 是一個代表著多個輸入的 2D tensor．我們接下來把它加上 `b`，最後把它輸入 `tf.nn.softmax`．\n", 457 | "\n", 458 | "到這裡，我們用了幾行程式來設定變數，然後再一行就建立起我們自己的模型了．Tensorflow 不僅僅可以讓 `softmax regression` 變得如此簡單，像是其他的數值運算，機器學習模型，甚至是物理學的模擬都可以利用 Tensorflow 來描述．而只要這些模型被定義後，它就可以在任何的設備上使用，例如你的電腦的 CPU 和 GPU，甚至是手機呢！" 459 | ] 460 | }, 461 | { 462 | "cell_type": "markdown", 463 | "metadata": { 464 | "deletable": true, 465 | "editable": true 466 | }, 467 | "source": [ 468 | "## 模型訓練" 469 | ] 470 | }, 471 | { 472 | "cell_type": "markdown", 473 | "metadata": { 474 | "deletable": true, 475 | "editable": true 476 | }, 477 | "source": [ 478 | "為了要訓練的我們的模型，我們必須先定義一下怎樣的模型才是好的模型．事實上在機器學習中，通常是定義一個模型怎樣算是不好的．我們把這個定義稱作成本 (cost) 或是損失 (loss)．它代表我們的模型和預期的結果間的差距．我們會嘗試要最小化這些成本，當這些成本或損失越低的時候，就代表著我們的模型越好．\n", 479 | "\n", 480 | "有一個非常常見而且很棒的成本函數稱作 `cross-entropy`．它原先產生於通訊理論中的通訊壓縮編碼，但從賭博到機器學習等領域都有著很重要的地位．它的定義如下:\n", 481 | "\n", 482 | "$$H_{y'}(y) = -\\sum_i y'_i \\log(y_i)$$\n", 483 | "\n", 484 | "`y` 是預測的機率分佈，而 `y'` 是真實的機率分佈 (one-hot 數字向量)．概略地來說 `cross-entropy` 用來量測我們的預測和真相的之間的差距．更多的探討 `cross-entropy` 有點超出這裡這份的指引的範圍，但很推薦你好好地理解[它](http://colah.github.io/posts/2015-09-Visual-Information/)．\n", 485 | "\n", 486 | "為了實現 `cross-entropy` 我們必須先加入一個新的佔位子 (placeholder) 來放置正確的答案．" 487 | ] 488 | }, 489 | { 490 | "cell_type": "code", 491 | "execution_count": 15, 492 | "metadata": { 493 | "collapsed": true, 494 | "deletable": true, 495 | "editable": true 496 | }, 497 | "outputs": [], 498 | "source": [ 499 | "y_ = tf.placeholder(tf.float32, [None, 10])" 500 | ] 501 | }, 502 | { 503 | "cell_type": "markdown", 504 | "metadata": { 505 | "deletable": true, 506 | "editable": true 507 | }, 508 | "source": [ 509 | "然後我們可以來實現 cross-entropy 函數, $-\\sum y'\\log(y)$:" 510 | ] 511 | }, 512 | { 513 | "cell_type": "code", 514 | "execution_count": 16, 515 | "metadata": { 516 | "collapsed": true, 517 | "deletable": true, 518 | "editable": true 519 | }, 520 | "outputs": [], 521 | "source": [ 522 | "cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))" 523 | ] 524 | }, 525 | { 526 | "cell_type": "markdown", 527 | "metadata": { 528 | "deletable": true, 529 | "editable": true 530 | }, 531 | "source": [ 532 | "首先 `tf.log` 會先對每個 `y` 的元素取 log．接下來我們把每個 `y_` 中的元素乘上 `tf.log(y)` 中對應的元素．接下來使用 `tf.reduce_sum` 把第二個維度的元素加總起來，(reduction_indices=[1]，這個參數)．最後 `tf.reduce_mean` 計算出這一輪的平均值．\n", 533 | "\n", 534 | "(在程式碼裡面我們並沒有直接使用這段程式碼，因為它是 numerically unstable．取而代之的我們使用 `tf.nn.softmax_cross_entropy_with_logits`．並把 `tf.matmul(x, W) + b` 當作函數輸入．在你自己的程式裡面請考慮使用 `tf.nn(sparse_)softmax_cross_entropy_with_logits`．\n", 535 | "\n", 536 | "好的，現在我們已經知道我們要我們的模型做什麼了，而且 Tensorflow 也已經知道整個模型的計算流程圖了，現在就讓 Tensorflow 來幫你訓練模型吧．它可以自動的計算[反向傳遞](http://colah.github.io/posts/2015-08-Backprop/) (backpropagation algorithm)並且調整參數來讓成本 (lost) 最小化．當然的你可以自己選擇要使用哪一個調整參數的最佳化演算法．" 537 | ] 538 | }, 539 | { 540 | "cell_type": "code", 541 | "execution_count": 17, 542 | "metadata": { 543 | "collapsed": false, 544 | "deletable": true, 545 | "editable": true 546 | }, 547 | "outputs": [], 548 | "source": [ 549 | "train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)" 550 | ] 551 | }, 552 | { 553 | "cell_type": "markdown", 554 | "metadata": { 555 | "deletable": true, 556 | "editable": true 557 | }, 558 | "source": [ 559 | "在這個例子中，我們要 Tensorflow 使用梯度下降法 [gradient descent algorithm](https://en.wikipedia.org/wiki/Gradient_descent) 來最小化 `cross_entropy`，而它的學習速率 (learning rate) 是 0.5．梯度下降法 (Gradient descent) 是一個簡單的學習方法，Tensorflow 會把每個參數往最小化 cost 的方向調整．不過 Tensorflow 同時也提供了許多[最佳化](https://www.tensorflow.org/api_docs/python/train#optimizers)的演算法：而且只要調整一行的程式碼就可以使用這些演算法了．\n", 560 | "\n", 561 | "實際上 Tensorflow 在這裡做的事情是在你所定義的計算圖用一系列後台的計算來實現反向傳遞以及梯度下降法．最後它給你的只是一個單一簡單的函數，當運行的時候，他就會利用梯度下降法來訓練你的模型參數，不斷地減低 cost．" 562 | ] 563 | }, 564 | { 565 | "cell_type": "markdown", 566 | "metadata": { 567 | "deletable": true, 568 | "editable": true 569 | }, 570 | "source": [ 571 | "現在我們已經設置好我們的模型了，但在執行之前還有最後一件事情是我們要先來初始化我們所建立的變數．注意一下這時候只是定義而已還沒有真正的執行．" 572 | ] 573 | }, 574 | { 575 | "cell_type": "code", 576 | "execution_count": 18, 577 | "metadata": { 578 | "collapsed": false, 579 | "deletable": true, 580 | "editable": true 581 | }, 582 | "outputs": [], 583 | "source": [ 584 | "init = tf.global_variables_initializer()" 585 | ] 586 | }, 587 | { 588 | "cell_type": "markdown", 589 | "metadata": { 590 | "deletable": true, 591 | "editable": true 592 | }, 593 | "source": [ 594 | "我們現在可以利用 `Session` 來初始化我們的參數以及啟動我們的模型了．" 595 | ] 596 | }, 597 | { 598 | "cell_type": "code", 599 | "execution_count": 19, 600 | "metadata": { 601 | "collapsed": false, 602 | "deletable": true, 603 | "editable": true 604 | }, 605 | "outputs": [], 606 | "source": [ 607 | "sess = tf.Session()\n", 608 | "sess.run(init)" 609 | ] 610 | }, 611 | { 612 | "cell_type": "markdown", 613 | "metadata": { 614 | "deletable": true, 615 | "editable": true 616 | }, 617 | "source": [ 618 | "開始訓練模型！我們會執行 1000 次的訓練" 619 | ] 620 | }, 621 | { 622 | "cell_type": "code", 623 | "execution_count": 20, 624 | "metadata": { 625 | "collapsed": false, 626 | "deletable": true, 627 | "editable": true 628 | }, 629 | "outputs": [], 630 | "source": [ 631 | "for i in range(1000):\n", 632 | " batch_xs, batch_ys = mnist.train.next_batch(100)\n", 633 | " sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})" 634 | ] 635 | }, 636 | { 637 | "cell_type": "markdown", 638 | "metadata": { 639 | "deletable": true, 640 | "editable": true 641 | }, 642 | "source": [ 643 | "在每一次 loop 中我們會從訓練數據中隨機抓取一批 100 筆數據，然後把這些數據去替換掉之前我們設定的站位子 (`placeholder`)來進行訓練．\n", 644 | "\n", 645 | "使用一小部分的隨機數據稱作隨機訓練 (stochastic training)，更精確地說是隨機梯度下降．理想上我們希望用所有的數據來訓練，這樣會有更好的訓練結果，但這樣需要很大的計算消耗．所以每一次使用不同的訓練子集，這樣做可以有一樣的效果但是比較少的計算消耗．" 646 | ] 647 | }, 648 | { 649 | "cell_type": "markdown", 650 | "metadata": { 651 | "collapsed": false, 652 | "deletable": true, 653 | "editable": true 654 | }, 655 | "source": [ 656 | "## 評估我們的模型\n", 657 | "\n", 658 | "我們的模型表現的如何呢？\n", 659 | "讓我們看看我們預測的數字是否正確．`tf.argmax` 是一個特別有用的函數，它可以讓我們找到在某一維的 tensor 中找到最大的數值的索引值 (index)．例如 `tf.argmax(y, 1)` 代表著模型對於每一筆輸入認為最有可能的數字，`tf.argmax(y_, 1)` 則是代表著正確的數字．我們可以使用 `tf.equal` 來確認我們的預測是否正確．\n" 660 | ] 661 | }, 662 | { 663 | "cell_type": "code", 664 | "execution_count": 21, 665 | "metadata": { 666 | "collapsed": false, 667 | "deletable": true, 668 | "editable": true 669 | }, 670 | "outputs": [], 671 | "source": [ 672 | "correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))" 673 | ] 674 | }, 675 | { 676 | "cell_type": "markdown", 677 | "metadata": { 678 | "deletable": true, 679 | "editable": true 680 | }, 681 | "source": [ 682 | "這列出了一系列的布林值．為了來看看有多少比重的預測是正確的，我們把布林值轉化成福點數然後取平均值．例如 `[True, False, True, True]` 會變成 `[1, 0, 1, 1]`平均值是 `0.75`．" 683 | ] 684 | }, 685 | { 686 | "cell_type": "code", 687 | "execution_count": 22, 688 | "metadata": { 689 | "collapsed": false, 690 | "deletable": true, 691 | "editable": true 692 | }, 693 | "outputs": [], 694 | "source": [ 695 | "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))" 696 | ] 697 | }, 698 | { 699 | "cell_type": "markdown", 700 | "metadata": { 701 | "collapsed": true, 702 | "deletable": true, 703 | "editable": true 704 | }, 705 | "source": [ 706 | "終於我們可以來印出我們的測試資料執行出來的準度了．" 707 | ] 708 | }, 709 | { 710 | "cell_type": "code", 711 | "execution_count": 23, 712 | "metadata": { 713 | "collapsed": false, 714 | "deletable": true, 715 | "editable": true 716 | }, 717 | "outputs": [ 718 | { 719 | "name": "stdout", 720 | "output_type": "stream", 721 | "text": [ 722 | "0.9189\n" 723 | ] 724 | } 725 | ], 726 | "source": [ 727 | "print(sess.run(accuracy, feed_dict = {x: mnist.test.images, y_: mnist.test.labels}))" 728 | ] 729 | }, 730 | { 731 | "cell_type": "markdown", 732 | "metadata": { 733 | "deletable": true, 734 | "editable": true 735 | }, 736 | "source": [ 737 | "出來的結果大概是 92%\n", 738 | "\n", 739 | "這樣的結果算是好的嗎？其實是非常差的．這是因為我們用的是非常簡單的模型．如果做一些小調整，可以得到 97% 的精準度．而最好的模型可以達到 99.7%！(更多的資訊可以看一下[一些結果](http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html)．\n", 740 | "\n", 741 | "重要的是我們從這個模型學到了什麼，如果你覺得這樣的結果很令人沮喪的話看一下接下來的[教材](https://www.tensorflow.org/tutorials/mnist/pros/index)吧！學習用 Tensorflow 來建立更好的模型！" 742 | ] 743 | }, 744 | { 745 | "cell_type": "code", 746 | "execution_count": 24, 747 | "metadata": { 748 | "collapsed": false, 749 | "deletable": true, 750 | "editable": true 751 | }, 752 | "outputs": [ 753 | { 754 | "name": "stdout", 755 | "output_type": "stream", 756 | "text": [ 757 | "[ 3.29472641e-05 7.88767363e-09 1.11483721e-04 2.53725634e-03\n", 758 | " 1.26105158e-06 2.56959665e-05 2.00715462e-08 9.96722996e-01\n", 759 | " 1.77862657e-05 5.50678174e-04]\n" 760 | ] 761 | } 762 | ], 763 | "source": [ 764 | "print(sess.run(y[0,:], feed_dict = {x: mnist.test.images, y_: mnist.test.labels}))" 765 | ] 766 | }, 767 | { 768 | "cell_type": "code", 769 | "execution_count": null, 770 | "metadata": { 771 | "collapsed": true, 772 | "deletable": true, 773 | "editable": true 774 | }, 775 | "outputs": [], 776 | "source": [] 777 | } 778 | ], 779 | "metadata": { 780 | "kernelspec": { 781 | "display_name": "Python 2", 782 | "language": "python", 783 | "name": "python2" 784 | }, 785 | "language_info": { 786 | "codemirror_mode": { 787 | "name": "ipython", 788 | "version": 2 789 | }, 790 | "file_extension": ".py", 791 | "mimetype": "text/x-python", 792 | "name": "python", 793 | "nbconvert_exporter": "python", 794 | "pygments_lexer": "ipython2", 795 | "version": "2.7.12" 796 | } 797 | }, 798 | "nbformat": 4, 799 | "nbformat_minor": 2 800 | } 801 | -------------------------------------------------------------------------------- /notebooks/4_Save_Restore_Model.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 3. Save and Restore model\n", 8 | "\n", 9 | "\n", 10 | "\n", 11 | "還記得當我們在定義模型的時候，會使用 [Variables](https://www.tensorflow.org/api_docs/python/state_ops) 來儲存模型參數，而在訓練的時候則會一次又一次的更新它．這些參數在這些過程中都是儲存在記憶體裡．不過我們仍然需要在訓練過後把訓練完成的參數儲存在硬碟裡面，以便之後來使用或者對模型對進一步分析．\n", 12 | "\n", 13 | "在 Tensorflow 裡面最簡單的方法是使用 `tf.train.Saver` 這個物件 (object) 來儲存模型，在初始化這個物件的時候，裡面包含了計算圖 (graph) 中對於變量 (variables) 的許多種操作方法 (ops) ．而物件中又提供了許多方法 (method) 來執行這些操作方法 (ops)．它可以幫助我們簡單的把變量 (variables) 儲存成檔案，以提供之後使用．以下先附上兩個 class 的 api 連結，可以在裡面看到更為完整的說明．\n", 14 | "\n", 15 | "* [tf.Variable](https://www.tensorflow.org/api_docs/python/state_ops/variables#Variable) class\n", 16 | "* [tf.train.Saver](https://www.tensorflow.org/api_docs/python/state_ops/saving_and_restoring_variables#Saver) class\n", 17 | "\n", 18 | "## 儲存模型\n", 19 | "\n", 20 | "### Checkpoint 檔案\n", 21 | "\n", 22 | "變量 (Variables) 是被以 binary 的方式儲存成一個 checkpoint 檔 (.ckpt)，簡單的說它儲存了變量 (variable) 的名字和對應的張量 (tensor) 數值．\n", 23 | "\n", 24 | "當你建立了一個 Saver 物件以後，是可以選擇每個 variable 要用什麼名字儲存在 checkpoint 檔裡的．預設上則是會使用 [Variable.name](https://www.tensorflow.org/api_docs/python/state_ops/variables#Variable.name) 的值．\n", 25 | "\n", 26 | "為了要來了解到底儲存了什麼 variable 在 checkpoint 檔裡，你可以使用 [inpect_checkpoint](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/inspect_checkpoint.py) 中的 `print_tensors_in_checkpoint_file` 函數來檢測．\n", 27 | "\n", 28 | "### 儲存變量 (Variables)\n", 29 | "\n", 30 | "以下是一個利用 `tf.train.Saver()` 來建立物件來儲存變量 (variables) 的範例．" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": 1, 36 | "metadata": { 37 | "collapsed": false 38 | }, 39 | "outputs": [], 40 | "source": [ 41 | "import tensorflow as tf\n", 42 | "import os\n", 43 | "os.makedirs(\"/tmp/model\")\n", 44 | "os.makedirs(\"/tmp/model-subset\")" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": 2, 50 | "metadata": { 51 | "collapsed": false 52 | }, 53 | "outputs": [ 54 | { 55 | "name": "stdout", 56 | "output_type": "stream", 57 | "text": [ 58 | "[array([ 0.1, 0.1], dtype=float32), array([ 0.30000001, 0.30000001], dtype=float32)]\n" 59 | ] 60 | } 61 | ], 62 | "source": [ 63 | "# 建立一些變數以及對應的名字．\n", 64 | "\n", 65 | "v1 = tf.Variable([0.1, 0.1], name=\"v1\")\n", 66 | "v2 = tf.Variable([0.2, 0.2], name=\"v2\")\n", 67 | "\n", 68 | "# 建立所有 variables 的初始化 ops\n", 69 | "init_op = tf.global_variables_initializer()\n", 70 | "\n", 71 | "# 建立 saver 物件\n", 72 | "saver = tf.train.Saver()\n", 73 | "\n", 74 | "with tf.Session() as sess:\n", 75 | " \n", 76 | " # 執行初始化\n", 77 | " sess.run(init_op)\n", 78 | " \n", 79 | " #重新指定 v2 的值\n", 80 | " ops = tf.assign(v2, [0.3, 0.3])\n", 81 | " sess.run(ops)\n", 82 | " \n", 83 | " print sess.run(tf.global_variables())\n", 84 | " # ... 中間略去許多模型定義以及訓練，例如可以是 MNIST 的定義以及訓練\n", 85 | " \n", 86 | " save_path = saver.save(sess, \"/tmp/model/model.ckpt\") # 儲存模型到 /tmp/model.ckpt" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": 3, 92 | "metadata": { 93 | "collapsed": false 94 | }, 95 | "outputs": [ 96 | { 97 | "name": "stdout", 98 | "output_type": "stream", 99 | "text": [ 100 | "v1 (DT_FLOAT) [2]\r\n", 101 | "v2 (DT_FLOAT) [2]\r\n", 102 | "\r\n" 103 | ] 104 | } 105 | ], 106 | "source": [ 107 | "# 使用 inspect_checkpoint.py tool 來印出 ckpt 檔\n", 108 | "!python /usr/local/lib/python2.7/dist-packages/tensorflow/python/tools/inspect_checkpoint.py --file_name=/tmp/model/model.ckpt" 109 | ] 110 | }, 111 | { 112 | "cell_type": "markdown", 113 | "metadata": {}, 114 | "source": [ 115 | "### 儲存特定變量 (Variables)\n", 116 | "\n", 117 | "如果我們沒有指定特定的參數給 `tf.train.Saver()`，它會自動把計算圖裡的所有變量都儲存進去．\n", 118 | "\n", 119 | "但有時候在儲存的時候我們會想要指定不一樣的名字，例如建立了一個變量叫做 \"weights\" 但是在儲存的時候我們會希望他的名字叫做 \"params\"\n", 120 | "\n", 121 | "也有些時候我們只想儲存部分的變量．例如訓練了一個 5 層的神經網路，但是我們只需要前 4 層的變量．\n", 122 | "\n", 123 | "我們可以很簡單地在 `Python dictionary` 裡指定變量以及對應的名字來給 `tf.train.Saver()` 當參數，就可以完成上面的目標了．範例如下\n", 124 | "\n", 125 | "註: \n", 126 | "- 你可以建立許多個 `saver` 物件，並且在這些物件之間，儲存或者載入部分的模型變量．同樣的變量會在不同的 `saver` 物件中存在，而且只有在執行 `restore()` 的時候才會被改變．\n", 127 | "- 如果你只在一個 session 裡載入了一個模型中一部分的變量，你必須要先用 `tf.global_variables_initializer()` 的方法先把其他的沒有載入的變量先初始化起來．" 128 | ] 129 | }, 130 | { 131 | "cell_type": "code", 132 | "execution_count": 4, 133 | "metadata": { 134 | "collapsed": false 135 | }, 136 | "outputs": [], 137 | "source": [ 138 | "tf.reset_default_graph()\n", 139 | "v1 = tf.Variable([0.1, 0.1], name=\"v1\")\n", 140 | "v2 = tf.Variable([0.4, 0.4], name=\"v2\")\n", 141 | "init_ops = tf.global_variables_initializer()\n", 142 | "\n", 143 | "with tf.Session() as sess:\n", 144 | " sess.run(init_ops)\n", 145 | " saver = tf.train.Saver({\"my_v2\": v2})\n", 146 | " save_path = saver.save(sess, \"/tmp/model-subset/model.ckpt\")" 147 | ] 148 | }, 149 | { 150 | "cell_type": "code", 151 | "execution_count": 5, 152 | "metadata": { 153 | "collapsed": false 154 | }, 155 | "outputs": [ 156 | { 157 | "name": "stdout", 158 | "output_type": "stream", 159 | "text": [ 160 | "my_v2 (DT_FLOAT) [2]\r\n", 161 | "\r\n" 162 | ] 163 | } 164 | ], 165 | "source": [ 166 | "# 使用 inspect_checkpoint.py tool 來印出 ckpt 檔\n", 167 | "!python /usr/local/lib/python2.7/dist-packages/tensorflow/python/tools/inspect_checkpoint.py --file_name=/tmp/model-subset/model.ckpt" 168 | ] 169 | }, 170 | { 171 | "cell_type": "markdown", 172 | "metadata": {}, 173 | "source": [ 174 | "## 載入模型\n", 175 | "\n", 176 | "### 1. 預先定義好模型\n", 177 | "第一種方法來載入模型的時候，你必須先定義好原本的計算圖 (模型)，例如這裡我們先定義了 `v1` 和 `v2` 然後用 `sess.restore` 的方法載入之前被更改過的值．\n", 178 | "同樣的 Saver 物件也是可以拿來載入變量 (variables)．值得注意的是當要從檔案中載入變量 (variables) 的時候是不用先把它初始化的．\n", 179 | "\n", 180 | "\n" 181 | ] 182 | }, 183 | { 184 | "cell_type": "code", 185 | "execution_count": 6, 186 | "metadata": { 187 | "collapsed": false 188 | }, 189 | "outputs": [ 190 | { 191 | "name": "stdout", 192 | "output_type": "stream", 193 | "text": [ 194 | "Model restored.\n", 195 | "all values [array([ 0.1, 0.1], dtype=float32), array([ 0.30000001, 0.30000001], dtype=float32)]\n", 196 | "v2 value : [ 0.30000001 0.30000001]\n" 197 | ] 198 | } 199 | ], 200 | "source": [ 201 | "tf.reset_default_graph()\n", 202 | "\n", 203 | "v1 = tf.Variable(tf.constant(0.1, shape = [2]), name=\"v1\")\n", 204 | "v2 = tf.Variable(tf.constant(0.2, shape = [2]), name=\"v2\")\n", 205 | "\n", 206 | "saver = tf.train.Saver()\n", 207 | "\n", 208 | "with tf.Session() as sess:\n", 209 | " saver.restore(sess, \"/tmp/model/model.ckpt\")\n", 210 | " print(\"Model restored.\")\n", 211 | " print(\"all values %s\" % sess.run(tf.global_variables()))\n", 212 | " print(\"v2 value : %s\" % sess.run(v2))" 213 | ] 214 | }, 215 | { 216 | "cell_type": "markdown", 217 | "metadata": {}, 218 | "source": [ 219 | "### 2. 載入 meta\n", 220 | "\n", 221 | "另一種方法則是在 `0.11.0RC1` 之後可以用載入 `meta` 的方式，就不用預先定義計算圖了．但是需要從重新拿取每一個參數的名字，以下就是範例．" 222 | ] 223 | }, 224 | { 225 | "cell_type": "code", 226 | "execution_count": 9, 227 | "metadata": { 228 | "collapsed": false 229 | }, 230 | "outputs": [ 231 | { 232 | "name": "stdout", 233 | "output_type": "stream", 234 | "text": [ 235 | "v1:0 with value [ 0.1 0.1]\n", 236 | "v2:0 with value [ 0.40000001 0.40000001]\n" 237 | ] 238 | } 239 | ], 240 | "source": [ 241 | "tf.reset_default_graph()\n", 242 | "with tf.Session() as sess:\n", 243 | " saver = tf.train.import_meta_graph('/tmp/model-subset/model.ckpt.meta')\n", 244 | " saver.restore(sess, tf.train.latest_checkpoint('/tmp/model-subset/'))\n", 245 | " sess.run(tf.global_variables_initializer())\n", 246 | " all_vars = tf.trainable_variables()\n", 247 | " for v in all_vars:\n", 248 | " print(\"%s with value %s\" % (v.name, sess.run(v)))" 249 | ] 250 | }, 251 | { 252 | "cell_type": "markdown", 253 | "metadata": {}, 254 | "source": [ 255 | "## 學習資源連結\n", 256 | "- [[tensorflow 官方文件] variables](https://www.tensorflow.org/how_tos/variables/)\n", 257 | "- [[stackoverflow] How to restore model](http://stackoverflow.com/questions/33759623/tensorflow-how-to-restore-a-previously-saved-model-python)" 258 | ] 259 | } 260 | ], 261 | "metadata": { 262 | "kernelspec": { 263 | "display_name": "Python 2", 264 | "language": "python", 265 | "name": "python2" 266 | }, 267 | "language_info": { 268 | "codemirror_mode": { 269 | "name": "ipython", 270 | "version": 2 271 | }, 272 | "file_extension": ".py", 273 | "mimetype": "text/x-python", 274 | "name": "python", 275 | "nbconvert_exporter": "python", 276 | "pygments_lexer": "ipython2", 277 | "version": "2.7.6" 278 | } 279 | }, 280 | "nbformat": 4, 281 | "nbformat_minor": 2 282 | } 283 | -------------------------------------------------------------------------------- /notebooks/5_Autoencoder.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# AutoEncoder\n" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "Autoencoder 是一種無監督 (unsupervised) 的訓練方式，也就是說在訓練的時候是不需要給定目標的，那它是怎麼做到的呢？\n", 15 | "\n", 16 | "在 Hinton 的開山 [paper]( [Reducing the Dimensionality of Data with Neural Networks](https://www.cs.toronto.edu/~hinton/science.pdf)) ．它的概念可以從以下的圖來說明．可以看到有兩個部分 **Encoder** 以及 **Decoder**，而中間有一個重疊的 **code layer**．模型輸入是從 Encoder 輸入，輸出則是 Decoder 輸出．而 Encoder 中有許多的 hidden layer (隱層)，其中節點數量是遞減的；相對應的 Decoder 則是遞增的．\n", 17 | "\n", 18 | "在這樣對稱的網路結構中，給定的訓練條件是輸出越接近輸入越好，\n", 19 | "$$\n", 20 | "x : input of network \\\\\n", 21 | "\\hat{x} : output of network \\\\\n", 22 | "h(x) : hidden layer function \\\\\n", 23 | "x \\equiv \\hat{x} = h(x)\n", 24 | "$$\n", 25 | "\n", 26 | "\n", 27 | "![](http://imgur.com/WnzC82f.jpg) \n", 28 | "\n", 29 | "圖片來自 Hinton 的 paper [Reducing the Dimensionality of Data with Neural Networks](https://www.cs.toronto.edu/~hinton/science.pdf)\n", 30 | "\n", 31 | "訓練的過程用 `backpropagation` 即可，而當訓練完成以後，我們把 Decoder 移除，只剩下 Encoder (包含 Code layer)．\n", 32 | "\n", 33 | "## 無監督學習\n", 34 | "\n", 35 | "很有趣的如果只剩下 Encoder，可以看到它有很類似 `PCA` 的 `Dimension Reduction` 效果．因為 2000 維輸入變成了 30 維．不同的是它是一個非線性的降維方法取出了原始輸入的必要特徵．\n", 36 | "\n", 37 | "來看看取出的特徵和 `PCA` 的做對照：\n", 38 | "\n", 39 | "第一列為原始圖片\n", 40 | "\n", 41 | "第二列為 Autoencoder 降維結果\n", 42 | "\n", 43 | "第三列為 PCA 降維結果\n", 44 | "\n", 45 | "![](http://imgur.com/9QNBD27.jpg)\n", 46 | "\n", 47 | "圖片來自 Hinton 的 paper [Reducing the Dimensionality of Data with Neural Networks](https://www.cs.toronto.edu/~hinton/science.pdf)\n", 48 | "\n", 49 | "可以看到 `Autoencoder` 的效果比 `PCA` 來得好．\n", 50 | "\n", 51 | "那如果用 MNIST 的資料投影到二維的視覺化圖形來看，左邊是 `PCA`，右邊是 `Autoencoder`，也可以看到它比較成功的輸入依據特徵不同做分類．但這段過程都是無監督的情下完成的，也可以說是 \n", 52 | "\n", 53 | "> **它自動地學會了如何區分每個手寫數字**\n", 54 | "\n", 55 | "![](http://imgur.com/VX4oeSa.jpg)\n", 56 | "\n", 57 | "而在實用的時候，網路的最後面會接一個 softmax 之類的分類器，因為理論上已經找出了輸入的特徵，這時候加上分類器，才是人類真正告訴它你分類出的結果是什麼．\n", 58 | "\n", 59 | "## 實作 Autoencoder" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": null, 65 | "metadata": { 66 | "collapsed": false 67 | }, 68 | "outputs": [], 69 | "source": [ 70 | "import tensorflow as tf\n", 71 | "from tensorflow.examples.tutorials.mnist import input_data\n", 72 | "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot = True)" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": 2, 78 | "metadata": { 79 | "collapsed": true 80 | }, 81 | "outputs": [], 82 | "source": [ 83 | "def weight_variable(shape, name):\n", 84 | " return tf.Variable(tf.truncated_normal(shape = shape, stddev = 0.1), name)\n", 85 | "def bias_variable(shape, name):\n", 86 | " return tf.Variable(tf.constant(0.1, shape = shape), name)" 87 | ] 88 | }, 89 | { 90 | "cell_type": "markdown", 91 | "metadata": {}, 92 | "source": [ 93 | "初步的構想是建立一個七層的神經網路，在 encoder 維度會從 784 逐步變成 300, 100, 5 (code layer)， decoder 的時候再逐步轉回 100, 300, 784．而這個想法在現在對 tensorflow 比較熟悉的情形下是很容易實現的，以下就是我實現的程式碼片段\n", 94 | "\n", 95 | "**input -> 784 -> 300 -> 100 -> 5 (code layer) -> 100 -> 300 -> 784 -> output**" 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": 3, 101 | "metadata": { 102 | "collapsed": false 103 | }, 104 | "outputs": [], 105 | "source": [ 106 | "x = tf.placeholder(tf.float32, shape = [None, 784])\n", 107 | "e_W_1 = weight_variable([784, 300], \"e_W_1\")\n", 108 | "e_b_1 = bias_variable([300], \"e_b_1\")\n", 109 | "e_layer1 = tf.nn.relu(tf.matmul(x, e_W_1) + e_b_1)\n", 110 | "e_W_2 = weight_variable([300, 100], \"e_W_2\")\n", 111 | "e_b_2 = bias_variable([100], \"e_b_2\")\n", 112 | "e_layer2 = tf.nn.relu(tf.matmul(e_layer1, e_W_2) + e_b_2)\n", 113 | "e_W_3 = weight_variable([100, 20], \"e_W_3\")\n", 114 | "e_b_3 = bias_variable([20], \"e_b_3\")\n", 115 | "code_layer = tf.nn.relu(tf.matmul(e_layer2, e_W_3) + e_b_3)\n", 116 | "d_W_1 = weight_variable([20, 100], \"d_W_1\")\n", 117 | "d_b_1 = bias_variable([100], \"d_b_1\")\n", 118 | "d_layer1 = tf.nn.relu(tf.matmul(code_layer, d_W_1) + d_b_1)\n", 119 | "d_W_2 = weight_variable([100, 300], \"d_W_2\")\n", 120 | "d_b_2 = bias_variable([300], \"d_b_2\")\n", 121 | "d_layer2 = tf.nn.relu(tf.matmul(d_layer1, d_W_2) + d_b_2)\n", 122 | "d_W_3 = weight_variable([300, 784], \"d_W_3\")\n", 123 | "d_b_3 = bias_variable([784], \"d_b_3\")\n", 124 | "output_layer = tf.nn.relu(tf.matmul(d_layer2, d_W_3) + d_b_3)" 125 | ] 126 | }, 127 | { 128 | "cell_type": "markdown", 129 | "metadata": {}, 130 | "source": [ 131 | "loss 函數我使用了 `mean square error`，而 optimizer 原本是使用 `GradientDescentOptimizer`，但是做出來的 decode 結果變得非常糟糕 (如下)．後來上網搜尋了以後改用 `RMSPropOptimizer`，以下會隨機選出五個數字來看看所做出來的結果．" 132 | ] 133 | }, 134 | { 135 | "cell_type": "code", 136 | "execution_count": 4, 137 | "metadata": { 138 | "collapsed": false 139 | }, 140 | "outputs": [], 141 | "source": [ 142 | "loss = tf.reduce_mean(tf.pow(output_layer - x, 2))\n", 143 | "optimizer = tf.train.RMSPropOptimizer(0.01).minimize(loss)\n", 144 | "init_op = tf.global_variables_initializer()" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": 5, 150 | "metadata": { 151 | "collapsed": false 152 | }, 153 | "outputs": [ 154 | { 155 | "name": "stdout", 156 | "output_type": "stream", 157 | "text": [ 158 | "step 0, loss 0.154249\n", 159 | "step 100, loss 0.0938953\n", 160 | "step 200, loss 0.0631583\n", 161 | "step 300, loss 0.0638728\n", 162 | "step 400, loss 0.0447373\n", 163 | "step 500, loss 0.0425786\n", 164 | "step 600, loss 0.0370292\n", 165 | "step 700, loss 0.0394285\n", 166 | "step 800, loss 0.0413142\n", 167 | "step 900, loss 0.0367071\n", 168 | "step 1000, loss 0.0392272\n", 169 | "step 1100, loss 0.0350341\n", 170 | "step 1200, loss 0.036414\n", 171 | "step 1300, loss 0.0374863\n", 172 | "step 1400, loss 0.0331814\n", 173 | "step 1500, loss 0.0381974\n", 174 | "step 1600, loss 0.0351127\n", 175 | "step 1700, loss 0.0374433\n", 176 | "step 1800, loss 0.0347769\n", 177 | "step 1900, loss 0.0375457\n", 178 | "step 2000, loss 0.0378837\n", 179 | "step 2100, loss 0.0406829\n", 180 | "step 2200, loss 0.0348483\n", 181 | "step 2300, loss 0.0344163\n", 182 | "step 2400, loss 0.0335558\n", 183 | "step 2500, loss 0.0333809\n", 184 | "step 2600, loss 0.030228\n", 185 | "step 2700, loss 0.0333538\n", 186 | "step 2800, loss 0.0300326\n", 187 | "step 2900, loss 0.0313007\n", 188 | "step 3000, loss 0.0325943\n", 189 | "step 3100, loss 0.0291298\n", 190 | "step 3200, loss 0.029259\n", 191 | "step 3300, loss 0.029667\n", 192 | "step 3400, loss 0.0284405\n", 193 | "step 3500, loss 0.0336205\n", 194 | "step 3600, loss 0.0346183\n", 195 | "step 3700, loss 0.0361299\n", 196 | "step 3800, loss 0.036426\n", 197 | "step 3900, loss 0.0304675\n", 198 | "step 4000, loss 0.0318735\n", 199 | "step 4100, loss 0.0310783\n", 200 | "step 4200, loss 0.0333465\n", 201 | "step 4300, loss 0.030751\n", 202 | "step 4400, loss 0.0293944\n", 203 | "step 4500, loss 0.0314036\n", 204 | "step 4600, loss 0.035683\n", 205 | "step 4700, loss 0.0282408\n", 206 | "step 4800, loss 0.030579\n", 207 | "step 4900, loss 0.0262065\n", 208 | "step 5000, loss 0.033858\n", 209 | "step 5100, loss 0.0319152\n", 210 | "step 5200, loss 0.0313285\n", 211 | "step 5300, loss 0.0335166\n", 212 | "step 5400, loss 0.0321645\n", 213 | "step 5500, loss 0.0298387\n", 214 | "step 5600, loss 0.0291673\n", 215 | "step 5700, loss 0.0369572\n", 216 | "step 5800, loss 0.0345598\n", 217 | "step 5900, loss 0.0326751\n", 218 | "step 6000, loss 0.0332971\n", 219 | "step 6100, loss 0.0305661\n", 220 | "step 6200, loss 0.0356756\n", 221 | "step 6300, loss 0.0339457\n", 222 | "step 6400, loss 0.0300257\n", 223 | "step 6500, loss 0.0301127\n", 224 | "step 6600, loss 0.0316595\n", 225 | "step 6700, loss 0.0304899\n", 226 | "step 6800, loss 0.0319861\n", 227 | "step 6900, loss 0.032973\n", 228 | "step 7000, loss 0.0308691\n", 229 | "step 7100, loss 0.0319135\n", 230 | "step 7200, loss 0.0323307\n", 231 | "step 7300, loss 0.0345884\n", 232 | "step 7400, loss 0.0305472\n", 233 | "step 7500, loss 0.0304301\n", 234 | "step 7600, loss 0.0383766\n", 235 | "step 7700, loss 0.0345033\n", 236 | "step 7800, loss 0.0308742\n", 237 | "step 7900, loss 0.0333693\n", 238 | "step 8000, loss 0.0319904\n", 239 | "step 8100, loss 0.0318597\n", 240 | "step 8200, loss 0.0333005\n", 241 | "step 8300, loss 0.0300832\n", 242 | "step 8400, loss 0.0325518\n", 243 | "step 8500, loss 0.0332775\n", 244 | "step 8600, loss 0.0306761\n", 245 | "step 8700, loss 0.0316801\n", 246 | "step 8800, loss 0.0308505\n", 247 | "step 8900, loss 0.0330996\n", 248 | "step 9000, loss 0.0341253\n", 249 | "step 9100, loss 0.0341319\n", 250 | "step 9200, loss 0.0334614\n", 251 | "step 9300, loss 0.0290531\n", 252 | "step 9400, loss 0.0323436\n", 253 | "step 9500, loss 0.0331529\n", 254 | "step 9600, loss 0.0308602\n", 255 | "step 9700, loss 0.0326363\n", 256 | "step 9800, loss 0.0344703\n", 257 | "step 9900, loss 0.0323956\n", 258 | "final loss 0.0315639\n" 259 | ] 260 | } 261 | ], 262 | "source": [ 263 | "sess = tf.InteractiveSession()\n", 264 | "sess.run(init_op)\n", 265 | "for i in range(10000):\n", 266 | " batch = mnist.train.next_batch(50)\n", 267 | " if i%100 == 0:\n", 268 | " print(\"step %d, loss %g\"%(i, loss.eval(feed_dict={x:batch[0]})))\n", 269 | " optimizer.run(feed_dict={x: batch[0]})\n", 270 | " \n", 271 | "print(\"final loss %g\" % loss.eval(feed_dict={x: mnist.test.images}))" 272 | ] 273 | }, 274 | { 275 | "cell_type": "code", 276 | "execution_count": 6, 277 | "metadata": { 278 | "collapsed": false 279 | }, 280 | "outputs": [ 281 | { 282 | "data": { 283 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWEAAAFdCAYAAADSR9wBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAE4JJREFUeJzt3XusnHWdx/H3l+0FrCGNZWl1uRVqN9ZFQutSykUrLemC\nSXHVuFuNLkqiRNxUkl2NRresBowXGi5aI9EFjdUEFaJG2moN1EVsu4FAuNaUBYpCr5iWlEK5/PaP\nZ5odDm3PM3NmzndmzvuVTMI88z3zfB+e08/5ze+5TJRSkCTlOCK7AUkaywxhSUpkCEtSIkNYkhIZ\nwpKUyBCWpESGsCQlMoQlKZEhLEmJDGFJStSTIRwRl0XEYxGxLyLWR8TfZ/fUCRGxLCJeGfJ4KLuv\ndkTEuRHxi4j4c2M7Fh+k5ksR8VREPBcRv4mIGRm9tmO47YuIGw+yL2/L6reuiPhcRGyMiD0RsS0i\nbo2ImUNqJkbEtyJiZ0Q8GxE/jYhjs3puRc3tu2PIfns5IlZk9dxzIRwR/wRcDSwDTgfuA9ZExDGp\njXXOA8BUYFrjcU5uO22bBNwLXAa85gYkEfFZ4FPAJ4AzgL1U+3HCaDY5AofdvoZVvHpfLhmd1kbk\nXOB6YC6wEBgP/DoijmqquQZ4N/A+4B3Am4CfjXKf7aqzfQW4gf/fd28EPjPKfTZ1U0pPPYD1wLVN\nzwP4E/CZ7N46sG3LgHuy++jCdr0CLB6y7Cng8qbnRwP7gA9k99uh7bsRuCW7tw5s2zGN7TunaT+9\nAPxjU83fNmrOyO53pNvXWHY7sDy7twOPnhoJR8R4YA7w2wPLSvV/bS0wL6uvDntz4yPuoxHxw4g4\nPruhTouI6VQjjOb9uAfYwODsR4D5jY+8j0TEioh4Q3ZDbZhMNTJ8pvF8DjCOV++7TcAW+nPfDd2+\nAz4UETsi4v6IuGrISHlUjcta8SEcA/wVsG3I8m1Uf4373XrgYmAT1UegK4DfRcTflVL2JvbVadOo\nfvEPth+njX47XbGK6iP6Y8ApwFeA2yJiXmPg0PMiIqimHu4spRw4NjEN2N/4o9ms7/bdIbYPYCXw\nBNWntbcBXwNmAu8f9SbpvRA+lODQ83J9o5SypunpAxGxkeqX4QNUH28H3UDsR4BSys1NTx+MiPuB\nR4H5VB93+8EKYBb1jkv04747sH1nNy8spXy36emDEbEVWBsR00spj41mg9B7B+Z2Ai9TTZg3O5bX\njqr6XillN/BHoG/OGqhpK9U/2jGxHwEa/3h30if7MiK+CVwIzC+lPNX00lZgQkQcPeRH+mrfDdm+\np4cp30D1+5qy73oqhEspLwJ3AwsOLGt8pFgA3JXVV7dExOupPsoO90vSVxqBtJVX78ejqY5YD9x+\nBIiI44Ap9MG+bATURcC7Silbhrx8N/ASr953M4ETgD+MWpMjMMz2HczpVKP8lH3Xi9MRy4HvR8Td\nwEbgcuB1wE2ZTXVCRHwd+CXVFMTfAP9J9Qv/48y+2hERk6hGDtFYdHJEnAY8U0p5kmou7gsRsRl4\nHPgy1VkuP09ot2WH277GYxnVnPDWRt1XqT7VrHntu/WOxvmwS4DFwN6IOPBpZXcp5flSyp6I+B6w\nPCL+AjwLXAf8vpSyMafr+obbvog4GfggcBuwCziNKnPWlVIeyOg5/fSMQ5xW8kmqf7j7qP76vj27\npw5t14+pgmgf1dHmHwHTs/tqc1veSXXqz8tDHv/VVHMF1cGP56jCaUZ2353YPuBIYDVVAD8P/C/w\nbeCvs/uusV0H26aXgY801UykOtd2J1UI/wQ4Nrv3TmwfcBxwB7Cj8Xu5ieqg6uuzeo5GY5KkBD01\nJyxJY40hLEmJDGFJSmQIS1IiQ1iSEhnCkpQo/WKNiJgCLKI6L/j53G4kqSOOBE4C1pRSdh2usGsh\nHBGXAf9Gdeel+4B/LaX8z0FKF1Hd1UiSBs2HqC7KOqSuTEe0+O0Yj3ejB0nqAY8PV9CtOeHLge+U\nUn5QSnkEuJTqEsGPHaTWKQhJg2rYfOt4CI+Rb8eQpI7oxkj4cN+O0Vd35pekbhvNU9T68c78ktRV\n3QjhMfXtGJI0Eh0P4TLGvh1DkkaiW+cJD+y3Y0hSJ3UlhEspNzfOCf4S1bTEvcCiUsqObqxPkvpV\n+jdrRMRsqukLSRo0c0op9xyuwBv4SFIiQ1iSEhnCkpTIEJakRIawJCUyhCUpkSEsSYkMYUlKZAhL\nUiJDWJISGcKSlMgQlqREhrAkJTKEJSmRISxJiQxhSUpkCEtSIkNYkhIZwpKUyBCWpESGsCQlMoQl\nKZEhLEmJDGFJSmQIS1IiQ1iSEhnCkpTIEJakRIawJCUyhCUpkSEsSYkMYUlKNC67AY2OxYsX1669\n9tpra9eedNJJtWtLKbVrI6J27fbt22vXXnXVVbVrAVauXFm7dufOnS29twSOhCUplSEsSYkMYUlK\nZAhLUiJDWJISGcKSlMgQlqREhrAkJTKEJSmRISxJiQxhSUrU8XtHRMQyYNmQxY+UUmZ1el2q761v\nfWvt2nXr1tWu3bVrV+3a008/vXZtK/eZOOaYY2rXLl++vHYttHbPjQULFrT03hJ07wY+DwALgAN3\nYXmpS+uRpL7WrRB+qZSyo0vvLUkDo1tzwm+OiD9HxKMR8cOIOL5L65GkvtaNEF4PXAwsAi4FpgO/\ni4hJXViXJPW1jk9HlFLWND19ICI2Ak8AHwBu7PT6JKmfdf0UtVLKbuCPwIxur0uS+k3XQzgiXg+c\nAjzd7XVJUr/peAhHxNcj4h0RcWJEnAXcSnWK2o87vS5J6nfdOEXtOOBHwBRgB3AncGYppf5Z/ZI0\nRnTjwNySTr+nJA2qaOXy0K40EDEbuDu1iTFg3Lj6f29feqn+BY7jx4/vSu2pp55au/bzn/987drz\nzz+/di3AhAkTatdedNFFtWt/9atftdSH+tacUso9hyvwBj6SlMgQlqREhrAkJTKEJSmRISxJiQxh\nSUpkCEtSIkNYkhIZwpKUyBCWpETd+o459ZhWLkVuxYsvvtiV2g0bNtSubeVy4Ztuuql2LcCHP/zh\n2rVnnnlm7VovW9YBjoQlKZEhLEmJDGFJSmQIS1IiQ1iSEhnCkpTIEJakRIawJCUyhCUpkSEsSYm8\nbFl97+yzz65du2TJkpbeu5XLvb0UWe1wJCxJiQxhSUpkCEtSIkNYkhIZwpKUyBCWpESGsCQlMoQl\nKZEhLEmJDGFJSuRly+pJRxxRf3zwxS9+sXbtuHGt/co/9NBDtWvXr1/f0ntL4EhYklIZwpKUyBCW\npESGsCQlMoQlKZEhLEmJDGFJSmQIS1IiQ1iSEhnCkpTIy5b1Gqecckrt2iuvvLJ27dy5c2vXRkTt\n2uOPP752bas2bdpUu3bSpEm1a/fu3dtOOxpALY+EI+LciPhFRPw5Il6JiMUHqflSRDwVEc9FxG8i\nYkZn2pWkwdLOdMQk4F7gMqAMfTEiPgt8CvgEcAawF1gTERNG0KckDaSWpyNKKauB1QBx8M+MS4Ev\nl1J+2aj5CLANeA9wc/utStLg6eiBuYiYDkwDfntgWSllD7ABmNfJdUnSIOj02RHTqKYotg1Zvq3x\nmiSpyWidohYcZP5Yksa6TofwVqrAnTpk+bG8dnQsSWNeR0O4lPIYVRAvOLAsIo4G5gJ3dXJdkjQI\nWj47IiImATOoRrwAJ0fEacAzpZQngWuAL0TEZuBx4MvAn4Cfd6RjSRog7Vwx93bgdqo53gJc3Vj+\nfeBjpZSvRcTrgO8Ak4H/Bi4opezvQL+SNFCilNzjZRExG7g7tQm9ytVXXz18UcOnP/3pLnbSX7Zs\n2VK79vzzz69du3nz5nbaUW+YU0q553AF3sBHkhIZwpKUyBCWpESGsCQlMoQlKZEhLEmJDGFJSmQI\nS1IiQ1iSEhnCkpTIb1seI1r5JuAFCxYMX9SGVi6RX7lyZe3au+6qf4O+9773vbVrAc4777zatSec\ncELt2lWrVtWuXbhwYe3aJ554onateoMjYUlKZAhLUiJDWJISGcKSlMgQlqREhrAkJTKEJSmRISxJ\niQxhSUpkCEtSIr9tWa9xySWX1K79+Mc/Xru2lUuRr7vuutq13XTBBRfUrr3++utr106fPr127Q03\n3FC7dunSpbVr9+/fX7tWbfPbliWplxnCkpTIEJakRIawJCUyhCUpkSEsSYkMYUlKZAhLUiJDWJIS\nGcKSlMgQlqRE3jtC6pC5c+fWrl23bl3t2vHjx9euXbhwYe3a22+/vXat2ua9IySplxnCkpTIEJak\nRIawJCUyhCUpkSEsSYkMYUlKZAhLUiJDWJISGcKSlGhcqz8QEecC/w7MAd4IvKeU8oum128E/mXI\nj60upVw4kkalXrdhw4batfv27atd28ply/Pnz69d62XLvaGdkfAk4F7gMuBQN55YBUwFpjUeS9rq\nTpIGXMsj4VLKamA1QETEIcpeKKXsGEljkjQWdGtOeH5EbIuIRyJiRUS8oUvrkaS+1vJIuIZVwM+A\nx4BTgK8At0XEvJJ930xJ6jEdD+FSys1NTx+MiPuBR4H5gEcCJKlJ109RK6U8BuwEZnR7XZLUb7oe\nwhFxHDAFeLrb65KkftPOecKTqEa1B86MODkiTgOeaTyWUc0Jb23UfRX4I7CmEw1L0iBpZ0747VRz\nu6XxuLqx/PvAJ4G3AR8BJgNPUYXvf5RSXhxxt5I0YNo5T3gdh5/G+If225GksaUbp6hJGsbmzZtr\n186ePbuLnSibN/CRpESGsCQlMoQlKZEhLEmJDGFJSmQIS1IiQ1iSEhnCkpTIEJakRIawJCXysmUp\nwdq1a2vXetnyYHMkLEmJDGFJSmQIS1IiQ1iSEhnCkpTIEJakRIawJCUyhCUpkSEsSYkMYUlK5GXL\nUoJZs2Zlt6Ae4UhYkhIZwpKUyBCWpESGsCQlMoQlKZEhLEmJDGFJSmQIS1IiQ1iSEhnCkpTIy5Y1\nIkccUf/v+Kmnnlq7dt++fbVrn3zyya68bzcdddRR2S2oRzgSlqREhrAkJTKEJSmRISxJiQxhSUpk\nCEtSIkNYkhIZwpKUyBCWpESGsCQl8rJljcjSpUtr137jG9+oXdvKpciXXnpp7drVq1fXrm3VtGnT\natfOmzeva32ov7Q0Eo6Iz0XExojYExHbIuLWiJg5pGZiRHwrInZGxLMR8dOIOLazbUvSYGh1OuJc\n4HpgLrAQGA/8OiKa70ZyDfBu4H3AO4A3AT8beauSNHhamo4opVzY/DwiLga2A3OAOyPiaOBjwD+X\nUtY1aj4KPBwRZ5RSNnaka0kaECM9MDcZKMAzjedzqIL9twcKSimbgC2Ak2CSNETbIRwRQTX1cGcp\n5aHG4mnA/lLKniHl2xqvSZKajOTsiBXALOCcGrVBNWKWJDVpayQcEd8ELgTml1KeanppKzChMTfc\n7Fiq0bAkqUnLIdwI4IuAd5VStgx5+W7gJWBBU/1M4ATgDyPoU5IGUkvTERGxAlgCLAb2RsTUxku7\nSynPl1L2RMT3gOUR8RfgWeA64PeeGSFJr9XqnPClVHO7dwxZ/lHgB43/vhx4GfgpMBFYDVzWfouS\nNLiilNzjZRExm2oaQz3inHPqHGutrF27tnbt+PHja9fu3r27du3DDz9cu/aWW26pXQswefLk2rWX\nXHJJ7dqpU6cOX9SwadOm2rWLFi2qXdvKpeFq25xSyj2HK/AGPpKUyBCWpESGsCQlMoQlKZEhLEmJ\nDGFJSmQIS1IiQ1iSEhnCkpTIEJakRF62rBFZsmRJ7dprr722du2UKVPaaadvvPDCC7VrzzvvvNq1\n69evb6cddY+XLUtSLzOEJSmRISxJiQxhSUpkCEtSIkNYkhIZwpKUyBCWpESGsCQlMoQlKZGXLWvU\nnHjiibVrW/nG57POOqt27Vve8pbatQDbt2+vXTtx4sTatVdccUXt2vvuu692rXqOly1LUi8zhCUp\nkSEsSYkMYUlKZAhLUiJDWJISGcKSlMgQlqREhrAkJTKEJSmRISxJibx3hCR1j/eOkKReZghLUiJD\nWJISGcKSlMgQlqREhrAkJTKEJSmRISxJiQxhSUpkCEtSopZCOCI+FxEbI2JPRGyLiFsjYuaQmjsi\n4pWmx8sRsaKzbUvSYGh1JHwucD0wF1gIjAd+HRFHNdUU4AZgKjANeCPwmZG3KkmDZ1wrxaWUC5uf\nR8TFwHZgDnBn00vPlVJ2jLg7SRpwI50Tnkw18n1myPIPRcSOiLg/Iq4aMlKWJDW0NBJuFhEBXAPc\nWUp5qOmllcATwFPA24CvATOB94+gT0kaSG2HMLACmAWc3bywlPLdpqcPRsRWYG1ETC+lPDaC9UnS\nwGlrOiIivglcCMwvpTw9TPkGIIAZ7axLkgZZyyPhRgBfBLyzlLKlxo+cTjVvPFxYS9KY01IIN873\nXQIsBvZGxNTGS7tLKc9HxMnAB4HbgF3AacByYF0p5YHOtS1Jg6HVkfClVKPaO4Ys/yjwA2A/1fnD\nS4FJwJPAT4ArR9SlJA2oVs8TPuwccinlT8D8kTQkSWOJ946QpESGsCQlMoQlKZEhLEmJDGFJSmQI\nS1IiQ1iSEhnCkpTIEJakRIawJCUyhCUpkSEsSYkMYUlKZAhLUiJDWJISGcKSlMgQlqREhrAkJTKE\nJSmRISxJiQxhSUrUCyF8ZHYDktQlw+ZbL4TwSdkNSFKXnDRcQZRSRqGPwzQQMQVYBDwOPJ/ajCR1\nxpFUAbymlLLrcIXpISxJY1kvTEdI0phlCEtSIkNYkhIZwpKUyBCWpESGsCQlMoQlKdH/Af6VON9h\n4yY5AAAAAElFTkSuQmCC\n", 284 | "text/plain": [ 285 | "" 286 | ] 287 | }, 288 | "metadata": {}, 289 | "output_type": "display_data" 290 | }, 291 | { 292 | "data": { 293 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWEAAAFdCAYAAADSR9wBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAFVBJREFUeJzt3X/sXXV9x/Hnm59KsTTQtWUWpPyoUrG1lCEMcSU1MqtB\nzBa3jsSp/8yMEUOyuZgsw2m2ZRoJU8fiwub8HUFn1ASBocBWtlItP7S2aiiFlrbfCu1CK20pbT/7\n49xuly9tv5/z/d7b9/fePh/JTbj3vvq978P59tVzzz3n3CilIEnKcVz2AJJ0LLOEJSmRJSxJiSxh\nSUpkCUtSIktYkhJZwpKUyBKWpESWsCQlsoQlKdGkLOGIuD4i1kfE7ohYERG/kT1TL0TETRFxYNRt\nTfZc4xERV0bEdyJiU2c5rjlE5mMRsTkidkXEv0fE+RmzjsdYyxcRnz/Eurwza95aEfGRiFgZETsi\nYmtEfCsi5o7KnBwR/xARz0bEzoj4RkTMyJq5jcrlu3/UetsfEbdmzTzpSjgifg/4FHATsBB4DLg7\nIqanDtY7q4GZwKzO7c2544zbFOBR4HrgZRcgiYg/B/4E+CPgUuB5mvV40tEccgKOuHwd3+Ol63LZ\n0RltQq4EPgO8CXgrcCJwT0S8sitzC/AO4HeAtwC/DnzzKM85XjXLV4B/4v/X3ZnAh4/ynF3TlDKp\nbsAK4O+77gfwNPDh7Nl6sGw3AQ9nz9GH5ToAXDPqsc3AjV33pwK7gfdkz9uj5fs88G/Zs/Vg2aZ3\nlu/NXevpBeDdXZnXdjKXZs870eXrPHYfcHP2bAdvk2pLOCJOBBYB3z/4WGn+r90LXJ41V49d0HmL\nuy4ivhwRZ2UP1GsRMYdmC6N7Pe4AHmJ41iPA4s5b3p9FxK0RcXr2QOMwjWbLcHvn/iLgBF667n4O\nbGAw193o5Tvouoh4JiJ+EhF/M2pL+ag6IeuFD2M6cDywddTjW2n+NR50K4D3AT+neQv0UeA/IuKi\nUsrziXP12iyaX/xDrcdZR3+cvvgezVv09cB5wN8Cd0bE5Z0Nh0kvIoJm18PyUsrBzyZmAXs7/2h2\nG7h1d5jlA/gK8BTNu7X5wCeAucDvHvUhmXwlfDjB4ffLDYxSyt1dd1dHxEqaX4b30Ly9HXZDsR4B\nSim3d939aUT8BFgHLKZ5uzsIbgXmUfe5xCCuu4PLd0X3g6WU27ru/jQiRoB7I2JOKWX90RwQJt8H\nc88C+2l2mHebwcu3qgZeKeU54BfAwBw1UGmE5i/tMbEeATp/eZ9lQNZlRHwWWAosLqVs7npqBDgp\nIqaO+iMDte5GLd+WMeIP0fy+pqy7SVXCpZQXgVXAkoOPdd5SLAH+K2uufomIU2neyo71SzJQOoU0\nwkvX41SaT6yHbj0CRMRs4AwGYF12CupdwFWllA2jnl4F7OOl624ucDbw30dtyAkYY/kOZSHNVn7K\nupuMuyNuBr4QEauAlcCNwCnAv2YO1QsR8UnguzS7IF4N/BXNL/zXMucaj4iYQrPlEJ2Hzo2IBcD2\nUspGmn1xfxERjwNPAh+nOcrl2wnjtnak5evcbqLZJzzSyf0dzbuau1/+0yaPzvGwy4BrgOcj4uC7\nledKKXtKKTsi4p+BmyPif4CdwKeBB0spK3OmrjfW8kXEucAfAHcC24AFNJ3zQClldcbM6YdnHOaw\nkj+m+Yu7m+Zf30uyZ+rRcn2Npoh203za/FVgTvZc41yW36I59Gf/qNu/dGU+SvPhxy6acjo/e+5e\nLB/wCuAumgLeAzwB/CPwa9lzVyzXoZZpP/DerszJNMfaPktTwncAM7Jn78XyAbOB+4FnOr+XP6f5\nUPXUrJmjM5gkKcGk2icsSccaS1iSElnCkpTIEpakRJawJCWyhCUpUfrJGhFxBnA1zXHBe3KnkaSe\neAVwDnB3KWXbkYJ9K+GIuB74U5orLz0G3FBK+eEholfTXNVIkobNdTQnZR1WX3ZHtPx2jCf7MYMk\nTQJPjhXo1z7hG4HPlVK+WEr5GfBBmlMEP3CIrLsgJA2rMfut5yV8jHw7hiT1RD+2hI/07RgDdWV+\nSeq3o3mI2iBemV+S+qofJXxMfTuGJE1Ez0u4HGPfjiFJE9Gv44SH9tsxJKmX+lLCpZTbO8cEf4xm\nt8SjwNWllGf68XqSNKjSv1kjIi6m2X0hScNmUSnl4SMFvICPJCWyhCUpkSUsSYksYUlKZAlLUiJL\nWJISWcKSlMgSlqRElrAkJbKEJSmRJSxJiSxhSUpkCUtSIktYkhJZwpKUyBKWpESWsCQlsoQlKZEl\nLEmJLGFJSmQJS1KivnzlvQbbJZdcUp2dPn16dXbXrl3V2b1791ZnN23aVJ3dsmVLdRZg3759rfJS\nW24JS1IiS1iSElnCkpTIEpakRJawJCWyhCUpkSUsSYksYUlKZAlLUiJLWJISedryAHv3u99dnb3i\niiuqs29/+9urs/PmzavO7t69uzo7MjJSnd2wYUN1ds2aNdVZgEceeaQ6u3bt2ursj370o+rsnj17\nqrMaPG4JS1IiS1iSElnCkpTIEpakRJawJCWyhCUpkSUsSYksYUlKZAlLUiJLWJISWcKSlKjn146I\niJuAm0Y9/LNSSv1FBlTl3HPPrc4uWLCgOnvyySdXZ3fu3FmdPfHEE6uz55xzTnX29NNPr86edtpp\n1VmAiKjO7tixozrb5toYjz/+eHVWg6dfF/BZDSwBDv4G7+vT60jSQOtXCe8rpTzTp58tSUOjX/uE\nL4iITRGxLiK+HBFn9el1JGmg9aOEVwDvA64GPgjMAf4jIqb04bUkaaD1fHdEKeXurrurI2Il8BTw\nHuDzvX49SRpkfT9ErZTyHPAL4Px+v5YkDZq+l3BEnAqcB2zp92tJ0qDpeQlHxCcj4i0R8ZqI+E3g\nWzSHqH2t168lSYOuH4eozQa+CpwBPAMsBy4rpWzrw2tJ0kDrxwdzy3r9MyVpWPmV95NMm9NqH3ro\noers1q1bq7OvetWrqrNtTnF+8cUXq7P79tWfZDlvXv0Z8W1PW25zqvXs2bOrs2vXrm01h4aXF/CR\npESWsCQlsoQlKZElLEmJLGFJSmQJS1IiS1iSElnCkpTIEpakRJawJCWKUkruABEXA6tSh9BLtDm1\n97nnnuvjJHVe//rXV2ff9ra3tfrZr33ta6uzbb6Zefny5dXZL33pS9VZTTqLSikPHynglrAkJbKE\nJSmRJSxJiSxhSUpkCUtSIktYkhJZwpKUyBKWpESWsCQlsoQlKZHftqyXmQynIrfxwgsvVGfnzJnT\n6mdPnz69Ort79+7qbJuZNdzcEpakRJawJCWyhCUpkSUsSYksYUlKZAlLUiJLWJISWcKSlMgSlqRE\nlrAkJfK0ZU1KU6dOrc4uWbKkOjt79uxWcxx3XP12yvbt26uzmzdvbjWHhpdbwpKUyBKWpESWsCQl\nsoQlKZElLEmJLGFJSmQJS1IiS1iSElnCkpTIEpakRJ62rKOmzSnDy5Ytq85eccUV1dkZM2ZUZwFG\nRkaqs6tWrarOLl++vNUcGl6tt4Qj4sqI+E5EbIqIAxFxzSEyH4uIzRGxKyL+PSLO7824kjRcxrM7\nYgrwKHA9UEY/GRF/DvwJ8EfApcDzwN0RcdIE5pSkodR6d0Qp5S7gLoCIiENEPgR8vJTy3U7mvcBW\n4Frg9vGPKknDp6cfzEXEHGAW8P2Dj5VSdgAPAZf38rUkaRj0+uiIWTS7KLaOenxr5zlJUpejdYha\ncIj9x5J0rOt1CY/QFO7MUY/P4OVbx5J0zOtpCZdS1tMU8f9930xETAXeBPxXL19LkoZB66MjImIK\ncD7NFi/AuRGxANheStkI3AL8RUQ8DjwJfBx4Gvh2TyaWpCEynjPmLgHuo9nHW4BPdR7/AvCBUson\nIuIU4HPANOA/gbeXUvb2YF5JGirjOU74AcbYjVFK+Sjw0fGNpEFy3XXXVWeXLl1anT377LOrs2ec\ncUZ1dtu2bdVZgE2bNlVnX3zxxersWWedVZ3duHFjdVaDxwv4SFIiS1iSElnCkpTIEpakRJawJCWy\nhCUpkSUsSYksYUlKZAlLUiJLWJIS+W3Lepl3vvOd1dnFixdXZ+fMmVOdnTZtWnV2z5491dn9+/dX\nZwHmz59fnX3Na15TnX3jG99YnV23bl119rbbbqvOHjhwoDqr/nFLWJISWcKSlMgSlqRElrAkJbKE\nJSmRJSxJiSxhSUpkCUtSIktYkhJZwpKUyNOWNSFPPPFEdbaUUp3dt29fdfa44+q3JdqcDg1w4YUX\nVmcvu+yy6uxVV11Vnf3hD39YnT3llFOqs7fcckt1Vv3jlrAkJbKEJSmRJSxJiSxhSUpkCUtSIktY\nkhJZwpKUyBKWpESWsCQlsoQlKZElLEmJvHaEXubBBx+szq5YsaI6O2PGjOrs5s2bq7Ntrh1x8cUX\nV2cBLrjggurstddeW52dN29edfZ1r3tddXbbtm3V2U2bNlVn77jjjuqs2nFLWJISWcKSlMgSlqRE\nlrAkJbKEJSmRJSxJiSxhSUpkCUtSIktYkhJZwpKUKNp8DTlARFwJ/BmwCDgTuLaU8p2u5z8P/OGo\nP3ZXKWXpYX7excCqVkNIk9Cll15anb3hhhuqs5dddll1du/evdXZe+65pzp74403Vmf1EotKKQ8f\nKTCeLeEpwKPA9cDhGvx7wExgVue2bByvI0lDr/UFfEopdwF3AUREHCb2QinlmYkMJknHgn7tE14c\nEVsj4mcRcWtEnN6n15GkgdaPS1l+D/gmsB44D/hb4M6IuLy03QEtSUOu5yVcSrm96+5PI+InwDpg\nMXBfr19PkgZZ3w9RK6WsB54Fzu/3a0nSoOl7CUfEbOAMYEu/X0uSBk3r3RERMYVmq/bgkRHnRsQC\nYHvndhPNPuGRTu7vgF8Ad/diYEkaJuPZJ3wJzb7d0rl9qvP4F4A/BuYD7wWmAZtpyvcvSykvTnha\nSRoy4zlO+AGOvBvjt8c/jiQdW/y2ZalHVq5cWZ298847q7NnnnlmdXbu3LnV2VNPPbU6e9FFF1Vn\nV69eXZ2VF/CRpFSWsCQlsoQlKZElLEmJLGFJSmQJS1IiS1iSElnCkpTIEpakRJawJCXytOVJ5rjj\n6v9dPHDgQB8nUT9t2LChOrt+/frqbJtTnLdv316dPemkk6qzasctYUlKZAlLUiJLWJISWcKSlMgS\nlqRElrAkJbKEJSmRJSxJiSxhSUpkCUtSIk9bnmRmzZpVnb388surs8cff3x19vbbb6/O6v/NnDmz\nOrtw4cLq7JQpU6qzO3furM5GRHV2165d1Vm145awJCWyhCUpkSUsSYksYUlKZAlLUiJLWJISWcKS\nlMgSlqRElrAkJbKEJSmRpy1PMrNnz67OXnjhhdXZqVOnVmfbfLPuihUrqrMjIyPV2Tan1LY5VfeU\nU06pzkK7U8PbnIq8aNGi6uzcuXOrszt27KjObty4sTr79NNPV2fVjlvCkpTIEpakRJawJCWyhCUp\nkSUsSYksYUlKZAlLUiJLWJISWcKSlMgSlqREnrY8yaxbt646u2XLlurs/Pnzq7MXXXRRdXbp0qXV\n2R//+MfV2fXr11dnp0+fXp1tc1p42/yrX/3q6uzZZ5/dao5aP/jBD6qz9913X3X2V7/61XjGUYVW\nW8IR8ZGIWBkROyJia0R8KyLmjsqcHBH/EBHPRsTOiPhGRMzo7diSNBza7o64EvgM8CbgrcCJwD0R\n8cquzC3AO4DfAd4C/DrwzYmPKknDp9XuiFLKS957RsT7gF8Ci4DlETEV+ADw+6WUBzqZ9wNrI+LS\nUsrKnkwtSUNioh/MTQMKsL1zfxFNsX//YKCU8nNgA1B/TUBJOkaMu4SjueDrLcDyUsqazsOzgL2l\nlNEXNd3aeU6S1GUiR0fcCswD3lyRDZotZklSl3FtCUfEZ4GlwOJSyuaup0aAkzr7hrvNoNkaliR1\naV3CnQJ+F3BVKWXDqKdXAfuAJV35ucDZwH9PYE5JGkqtdkdExK3AMuAa4PmImNl56rlSyp5Syo6I\n+Gfg5oj4H2An8GngQY+MkKSXa7tP+IM0+3bvH/X4+4Evdv77RmA/8A3gZOAu4PrxjyhJw6vtccJj\n7r4opbwA3NC5qaVt27ZVZ9euXVudfcMb3lCdXbJkydihjgULFlRn582bV51t8w3KpdR/5nvaaadV\nZwFmzpw5dqjj+OOPr862+ebp5cuXV2e//vWvV2c3bdpUnVX/eAEfSUpkCUtSIktYkhJZwpKUyBKW\npESWsCQlsoQlKZElLEmJLGFJSmQJS1Iiv215gD366KPV2Tan306bNq062+a05fPOO686u3///urs\nCSfU/xofOHCgOguwffv2sUMdmzdvHjvUce+991Zn25yKvGbNmrFDmlTcEpakRJawJCWyhCUpkSUs\nSYksYUlKZAlLUiJLWJISWcKSlMgSlqRElrAkJfK05QG2a9eu6myb02TXrVtXnV24cGF1dtGiRdXZ\ns846qzq7Y8eO6uwTTzxRnQV47LHHqrOPPPJIdfapp55qNYeGl1vCkpTIEpakRJawJCWyhCUpkSUs\nSYksYUlKZAlLUiJLWJISWcKSlMgSlqRElrAkJYpSSu4AERcDq1KHkKT+WFRKefhIAbeEJSmRJSxJ\niSxhSUpkCUtSIktYkhJZwpKUyBKWpESWsCQlsoQlKZElLEmJWpVwRHwkIlZGxI6I2BoR34qIuaMy\n90fEga7b/oi4tbdjS9JwaLslfCXwGeBNwFuBE4F7IuKVXZkC/BMwE5gFnAl8eOKjStLwOaFNuJSy\ntPt+RLwP+CWwCFje9dSuUsozE55OkobcRPcJT6PZ8t0+6vHrIuKZiPhJRPzNqC1lSVJHqy3hbhER\nwC3A8lLKmq6nvgI8BWwG5gOfAOYCvzuBOSVpKI27hIFbgXnAFd0PllJu67r704gYAe6NiDmllPUT\neD1JGjrj2h0REZ8FlgKLSylbxog/BARw/nheS5KGWest4U4Bvwv4rVLKhoo/spBmv/FYZS1Jx5xW\nJdw53ncZcA3wfETM7Dz1XCllT0ScC/wBcCewDVgA3Aw8UEpZ3buxJWk4tN0S/iDNVu39ox5/P/BF\nYC/N8cMfAqYAG4E7gL+e0JSSNKTaHid8xH3IpZSngcUTGUiSjiVeO0KSElnCkpTIEpakRJawJCWy\nhCUpkSUsSYksYUlKZAlLUiJLWJISWcKSlMgSlqRElrAkJbKEJSmRJSxJiSxhSUpkCUtSIktYkhJZ\nwpKUyBKWpESWsCQlsoQlKdFkKOFXZA8gSX0yZr9NhhI+J3sASeqTc8YKRCnlKMxxhAEizgCuBp4E\n9qQOI0m98QqaAr67lLLtSMH0EpakY9lk2B0hSccsS1iSElnCkpTIEpakRJawJCWyhCUpkSUsSYn+\nF8uUm+WZXEbRAAAAAElFTkSuQmCC\n", 294 | "text/plain": [ 295 | "" 296 | ] 297 | }, 298 | "metadata": {}, 299 | "output_type": "display_data" 300 | }, 301 | { 302 | "data": { 303 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWEAAAFdCAYAAADSR9wBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAEvhJREFUeJzt3XusXWWZx/Hvw3CvEEKFFgYdi8iMOpHYOjCgYA0mUExA\ngzqDJILGRCKOBp3RmJjBkYzGG+l4YdToAMZbkI5RE2hVIjitAiMEQ2EAZVpBeqEFU5qWirTP/LF2\nM5vT29rn7NXn7H2+n2Qn7LWfs9ezWKe/8+53r0tkJpKkGgdUNyBJM5khLEmFDGFJKmQIS1IhQ1iS\nChnCklTIEJakQoawJBUyhCWpkCEsSYWmZQhHxOURsSoino6I2yPib6p7GoaIuDIidkx43F/d12RE\nxJkR8cOIeKy3HefvpubjEbEmIrZGxE8i4qSKXidjX9sXEdfuZl/eVNVvWxHxkYi4MyKeioj1EfH9\niDh5Qs0hEfGliNgYEZsj4saIOLaq50G03L5bJ+y37RFxTVXP0y6EI+LvgM8BVwKvBH4NLIuI55c2\nNjwrgTnA3N7jNbXtTNos4B7gcmCXC5BExIeB9wLvBk4FttDsx4P3Z5NTsNft67mZ5+7Li/ZPa1Ny\nJvAF4DTg9cBBwI8j4rC+msXAG4ALgbOA44El+7nPyWqzfQl8lf/fd8cBH9rPffZ1kzmtHsDtwL/1\nPQ/g98CHqnsbwrZdCdxd3UcH27UDOH/CsjXAFX3PjwSeBt5a3e+Qtu9a4D+rexvCtj2/t32v6dtP\nfwTe1Ffzl72aU6v7ner29Zb9DLi6uredj2k1Eo6Ig4AFwC07l2Xzf+2nwOlVfQ3ZS3ofcR+OiG9G\nxAuqGxq2iJhHM8Lo349PAXcwPvsRYGHvI+8DEXFNRBxd3dAkHEUzMnyy93wBcCDP3XcPAo8wmvtu\n4vbtdHFEbIiIeyPiExNGyvvVgVUr3oPnA38GrJ+wfD3NX+NRdztwKfAgzUegjwE/j4i/zswthX0N\n21yaX/zd7ce5+7+dTtxM8xF9FfBi4JPATRFxem/gMO1FRNBMPSzPzJ3fTcwFnun90ew3cvtuD9sH\n8C3gdzSf1l4BfBo4GXjzfm+S6RfCexLseV5uZGTmsr6nKyPiTppfhrfSfLwdd2OxHwEy84a+p/dF\nxL3Aw8BCmo+7o+Aa4GW0+15iFPfdzu17df/CzPxa39P7ImId8NOImJeZq/ZngzD9vpjbCGynmTDv\ndyy7jqpGXmZuAh4CRuaogZbW0fyjnRH7EaD3j3cjI7IvI+KLwHnAwsxc0/fSOuDgiDhywo+M1L6b\nsH1r91F+B83va8m+m1YhnJl/Au4Czt65rPeR4mzgF1V9dSUinkfzUXZfvyQjpRdI63jufjyS5hvr\nsduPABFxAjCbEdiXvYC6AHhdZj4y4eW7gGd57r47GXgh8Mv91uQU7GP7dueVNKP8kn03Hacjrgau\nj4i7gDuBK4DDgesqmxqGiPgM8COaKYg/B/6F5hf+O5V9TUZEzKIZOURv0YkRcQrwZGY+SjMX99GI\n+C2wGriK5iiXHxS0O7C9bV/vcSXNnPC6Xt2naD7VLNv13aaP3vGwFwHnA1siYuenlU2ZuS0zn4qI\nrwNXR8QfgM3A54EVmXlnTdft7Wv7IuJE4G3ATcATwCk0mXNbZq6s6Ln88Iw9HFbyHpp/uE/T/PV9\nVXVPQ9qu79AE0dM03zZ/G5hX3dckt+W1NIf+bJ/w+I++mo/RfPmxlSacTqruexjbBxwKLKUJ4G3A\n/wL/DhxT3XeL7drdNm0H3t5XcwjNsbYbaUL4e8Cx1b0PY/uAE4BbgQ2938sHab5UfV5Vz9FrTJJU\nYFrNCUvSTGMIS1IhQ1iSChnCklTIEJakQoawJBUqP1kjImYD59AcF7ytthtJGopDgRcByzLzib0V\ndhbCEXE58I80V176NfAPmfnfuyk9h+aqRpI0bi6mOSlrjzqZjhjw7hiru+hBkqaB1fsq6GpO+Arg\nK5n5jcx8ALiM5hTBd+6m1ikISeNqn/k29BCeIXfHkKSh6GIkvLe7Y4zUlfklqWv78xC1UbwyvyR1\nqosQnlF3x5CkqRh6COcMuzuGJE1FV8cJj+3dMSRpmDoJ4cy8oXdM8MdppiXuAc7JzA1drE+SRlX5\nnTUiYj7N9IUkjZsFmXn33gq8gI8kFTKEJamQISxJhQxhSSpkCEtSIUNYkgoZwpJUyBCWpEKGsCQV\nMoQlqZAhLEmFDGFJKmQIS1IhQ1iSChnCklTIEJakQoawJBUyhCWpkCEsSYUMYUkqZAhLUiFDWJIK\nGcKSVMgQlqRChrAkFTKEJamQISxJhQxhSSpkCEtSIUNYkgoZwpJUyBCWpEKGsCQVMoQlqZAhLEmF\nDGFJKmQIS1IhQ1iSChnCklTIEJakQoawJBUyhCWpkCEsSYUMYUkqNPQQjogrI2LHhMf9w16PJI2D\nAzt635XA2UD0nj/b0XokaaR1FcLPZuaGjt5bksZGV3PCL4mIxyLi4Yj4ZkS8oKP1SNJI6yKEbwcu\nBc4BLgPmAT+PiFkdrEuSRtrQpyMyc1nf05URcSfwO+CtwLXDXp8kjbLOD1HLzE3AQ8BJXa9LkkZN\n5yEcEc8DXgys7XpdkjRqujhO+DMRcVZE/EVEnAF8n+YQte8Me12SNOq6OETtBODbwGxgA7Ac+NvM\nfKKDdUnSSOvii7mLhv2ekjSuvHaEJBUyhCWpkCEsSYUMYUkqZAhLUiFDWJIKGcKSVMgQlqRChrAk\nFTKEJamQISxJhQxhSSpkCEtSIUNYkgoZwpJUyBCWpEKGsCQVMoQlqZAhLEmFDGFJKmQIS1IhQ1iS\nCg39lvfaf4444ojWtbNmzeqwk+H74Ac/2Lr23HPPbV17//33D9TH0qVLW9fefPPNA713W4sWLeqk\ndsuWLa1r3/e+97Wu3bx5c+taORKWpFKGsCQVMoQlqZAhLEmFDGFJKmQIS1IhQ1iSChnCklTIEJak\nQoawJBWKzKxtIGI+cFdpE8CCBQta1w5yCmeX5s+f37r2pS99aYedDF9EtK6t/h2ejFHbvne9612t\na6+77rruGhk9CzLz7r0VOBKWpEKGsCQVMoQlqZAhLEmFDGFJKmQIS1IhQ1iSChnCklTIEJakQoaw\nJBUa67stX3XVVa1rL7300ta1xx133CS6Gb5RO/VVo+uwww6rbmFsDTwSjogzI+KHEfFYROyIiPN3\nU/PxiFgTEVsj4icRcdJw2pWk8TKZ6YhZwD3A5cAuw6uI+DDwXuDdwKnAFmBZRBw8hT4laSwNPB2R\nmUuBpQCx+8/D7weuyswf9WreDqwH3gjcMPlWJWn8DPWLuYiYB8wFbtm5LDOfAu4ATh/muiRpHAz7\n6Ii5NFMU6ycsX997TZLUZ38dohbsZv5Ykma6YYfwOprAnTNh+bHsOjqWpBlvqCGcmatogvjsncsi\n4kjgNOAXw1yXJI2DgY+OiIhZwEk0I16AEyPiFODJzHwUWAx8NCJ+C6wGrgJ+D/xgKB1L0hiZzBlz\nrwJ+RjPHm8DnesuvB96ZmZ+OiMOBrwBHAf8FLMrMZ4bQrySNlbG+2/K9997buna63I34xhtvbF37\nlre8pXXt8uXLW9euWrWqde0gvvzlL7eu3bp1a+vaHTt2TKadVi655JLWtcccc0zr2kFOOT/jjDNa\n186bN6917SCOP/741rWPP/54Jz2MKO+2LEnTmSEsSYUMYUkqZAhLUiFDWJIKGcKSVMgQlqRChrAk\nFTKEJamQISxJhcb6tOU5cyZeUXPPPvCBD7SuXb169UB9LFmypHXtxo0bW9cOcprspk2bWtdu27at\nda26d/3117euvfjiizvpwdOWJ83TliVpOjOEJamQISxJhQxhSSpkCEtSIUNYkgoZwpJUyBCWpEKG\nsCQVMoQlqdBYn7YsTVdz585tXfvYY4912Ek7g5y2vH79+g47GTmetixJ05khLEmFDGFJKmQIS1Ih\nQ1iSChnCklTIEJakQoawJBUyhCWpkCEsSYUMYUkqdGB1A9JMtGjRota1XV3fZcWKFa1rN23a1EkP\nciQsSaUMYUkqZAhLUiFDWJIKGcKSVMgQlqRChrAkFTKEJamQISxJhQxhSSo08GnLEXEm8E/AAuA4\n4I2Z+cO+168FLpnwY0sz87ypNCqNk0FOW+7KZz/72da127Zt67CTmW0yI+FZwD3A5cCeTmq/GZgD\nzO09LppUd5I05gYeCWfmUmApQETEHsr+mJkbptKYJM0EXc0JL4yI9RHxQERcExFHd7QeSRppXVzK\n8mZgCbAKeDHwSeCmiDg9u7omnySNqKGHcGbe0Pf0voi4F3gYWAj8bNjrk6RR1vkhapm5CtgInNT1\nuiRp1HQewhFxAjAbWNv1uiRp1EzmOOFZNKPanUdGnBgRpwBP9h5X0swJr+vVfQp4CFg2jIYlaZxM\nZk74VTRzu9l7fK63/HrgPcArgLcDRwFraML3nzPzT1PuVpLGzGSOE76NvU9jnDv5diRpZvFuy1KB\nCy+8sHXtIEd2rl3b/quXe+65p3WtuuMFfCSpkCEsSYUMYUkqZAhLUiFDWJIKGcKSVMgQlqRChrAk\nFTKEJamQISxJhTxtWRojK1eubF376KOPdtiJ2nIkLEmFDGFJKmQIS1IhQ1iSChnCklTIEJakQoaw\nJBUyhCWpkCEsSYUMYUkq5GnLUoEDDmg//tmxY0fr2oiYTDsq5EhYkgoZwpJUyBCWpEKGsCQVMoQl\nqZAhLEmFDGFJKmQIS1IhQ1iSChnCklTI05alIVm0aFHr2kFORc7MTmo1PTgSlqRChrAkFTKEJamQ\nISxJhQxhSSpkCEtSIUNYkgoZwpJUyBCWpEKGsCQV8rRlaUgGOW25K0uWLKluQQMaaCQcER+JiDsj\n4qmIWB8R34+IkyfUHBIRX4qIjRGxOSJujIhjh9u2JI2HQacjzgS+AJwGvB44CPhxRBzWV7MYeANw\nIXAWcDzgn2dJ2o2BpiMy87z+5xFxKfA4sABYHhFHAu8E/j4zb+vVvAP4n4g4NTPvHErXkjQmpvrF\n3FFAAk/2ni+gCfZbdhZk5oPAI8DpU1yXJI2dSYdwRATN1MPyzLy/t3gu8ExmPjWhfH3vNUlSn6kc\nHXEN8DLgNS1qg2bELEnqM6mRcER8ETgPWJiZa/peWgcc3Jsb7ncszWhYktRn4BDuBfAFwOsy85EJ\nL98FPAuc3Vd/MvBC4JdT6FOSxtJA0xERcQ1wEXA+sCUi5vRe2pSZ2zLzqYj4OnB1RPwB2Ax8Hljh\nkRGStKtB54Qvo5nbvXXC8ncA3+j99xXAduBG4BBgKXD55FuUpPEV1XdnjYj5NNMY0khbs2bNvot6\n5syZs++ingcffLB17Wmnnda6dvPmza1rNWkLMvPuvRV4AR9JKmQIS1IhQ1iSChnCklTIEJakQoaw\nJBUyhCWpkCEsSYUMYUkqZAhLUiHvtiztxctf/vLWtYcffngnPSxevLh1racijx5HwpJUyBCWpEKG\nsCQVMoQlqZAhLEmFDGFJKmQIS1IhQ1iSChnCklTIEJakQp62LO3F/PnzW9ceccQRrWsPOKD9+Gfr\n1q2tazV6HAlLUiFDWJIKGcKSVMgQlqRChrAkFTKEJamQISxJhQxhSSpkCEtSIUNYkgoZwpJUyGtH\nSEOSma1rd+zY0WEnGiWOhCWpkCEsSYUMYUkqZAhLUiFDWJIKGcKSVMgQlqRChrAkFTKEJamQISxJ\nhQY6bTkiPgK8Cfgr4GngF8CHM/OhvppbgbP6fiyBr2Tme6bcrTRFs2fPHqj+kksu6aSPZ555pnXt\n2rVrO+lB08OgI+EzgS8ApwGvBw4CfhwRh/XVJPBVYA4wFzgO+NDUW5Wk8TPQSDgzz+t/HhGXAo8D\nC4DlfS9tzcwNU+5OksbcVOeEj6IZ+T45YfnFEbEhIu6NiE9MGClLknomfSnLiAhgMbA8M+/ve+lb\nwO+ANcArgE8DJwNvnkKfkjSWpnI94WuAlwGv7l+YmV/re3pfRKwDfhoR8zJz1RTWJ0ljZ1LTERHx\nReA8YGFm7uur2zuAAE6azLokaZwNPBLuBfAFwGsz85EWP/JKmnljj7ORpAkGPU74GuAi4HxgS0TM\n6b20KTO3RcSJwNuAm4AngFOAq4HbMnPl8NqWpPEw6Ej4MppR7a0Tlr8D+AbwDM3xw+8HZgGPAt8D\n/nVKXUrSmBr0OOG9ziFn5u+BhVNpSJJmEu+2rBll8eLFA9UvXLiwkz5WrFjRuvaWW27ppAdND17A\nR5IKGcKSVMgQlqRChrAkFTKEJamQISxJhQxhSSpkCEtSIUNYkgoZwpJUyNOWNaMcc8wxnb33r371\nq9a1F1xwQWd9aLQ4EpakQoawJBUyhCWpkCEsSYUMYWlA3/3ud6tb0BgxhKUBGcIaJkNYkgoZwpJU\nyBCWpELT4Yy5Q6sb0Mzxm9/8ZqD6o48+epdlmzZt4u67795l+QMPPND6fbdv3z5QHxpZ+8y3yMz9\n0cieG4h4G/Ct0iYkqRsXZ+a391YwHUJ4NnAOsBrYVtqMJA3HocCLgGWZ+cTeCstDWJJmMr+Yk6RC\nhrAkFTKEJamQISxJhQxhSSpkCEtSIUNYkgr9H0/9KubczM4qAAAAAElFTkSuQmCC\n", 304 | "text/plain": [ 305 | "" 306 | ] 307 | }, 308 | "metadata": {}, 309 | "output_type": "display_data" 310 | }, 311 | { 312 | "data": { 313 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWEAAAFdCAYAAADSR9wBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAFD5JREFUeJzt3XusXWWdxvHvz1IKtOXqlEO5FiqDZBSRAtPRMjU16QxG\n0GicQRMH/UciGkMyozExA6MZjRoJ44WJRofReElQxygJUMFwmYrQEYOhQKFASy292FJsS29A+84f\nazduDr2865y9+zv7nO8n2Ql77adnv4t1+nTtd69LlFKQJOV4TfYAJGkis4QlKZElLEmJLGFJSmQJ\nS1IiS1iSElnCkpTIEpakRJawJCWyhCUp0Zgs4Yi4OiJWRMSOiLg/Ii7MHlMvRMS1EbFn2OPR7HGN\nRETMi4hfRMSznfW4bB+Zz0bEmojYHhF3RMTsjLGOxMHWLyJu2se2vDVrvLUi4tMRsSQitkTE+oj4\nWUScPSwzJSK+EREbI2JrRPwkImZkjbmNyvW7e9h22x0RN2aNecyVcET8A/AV4FrgfOD3wKKIeG3q\nwHpnKXAiMNR5vDV3OCM2FXgIuBp41QVIIuJTwMeAjwAXAdtotuPhh3KQo3DA9eu4jVduyysOzdBG\nZR7wNeBi4O3AZOCXEXFkV+YG4B3Ae4BLgJnATw/xOEeqZv0K8C3+vO1OAj55iMfZNZpSxtQDuB/4\nj67nAawGPpk9th6s27XA77LH0Yf12gNcNmzZGuCarudHAzuA92WPt0frdxPwP9lj68G6vbazfm/t\n2k67gHd3Zf6yk7koe7yjXb/OsruA67PHtvcxpvaEI2IycAHwq73LSvN/7U5gbta4eux1nY+4T0XE\n9yPi1OwB9VpEzKLZw+jejluABxg/2xFgfucj77KIuDEijs8e0AgcS7NnuKnz/ALgMF657R4HVjGY\n2274+u31gYjYEBEPR8Tnh+0pH1KHZb3xfrwWmASsH7Z8Pc2/xoPufuBK4HGaj0DXAfdGxF+VUrYl\njqvXhmh+8fe1HYcO/XD64jaaj+grgLOALwC3RsTczo7DmBcRQTP1sLiUsve7iSHgxc4/mt0Gbtvt\nZ/0AfgA8Q/Np7Y3Al4Czgfce8kEy9kp4f4L9z8sNjFLKoq6nSyNiCc0vw/toPt6Od+NiOwKUUm7u\nevpIRDwMPAXMp/m4OwhuBM6l7nuJQdx2e9fvLd0LSynf7nr6SESsA+6MiFmllBWHcoAw9r6Y2wjs\nppkw7zaDV+9VDbxSymbgCWBgjhqotI7mL+2E2I4Anb+8GxmQbRkRXwcuBeaXUtZ0vbQOODwijh72\nRwZq2w1bv7UHiT9A8/uasu3GVAmXUl4CHgQW7F3W+UixALgva1z9EhHTaD7KHuyXZKB0Cmkdr9yO\nR9N8Yz3utiNARJwCnMAAbMtOQV0OvK2UsmrYyw8CL/PKbXc2cBrwm0M2yFE4yPrty/k0e/kp224s\nTkdcD3w3Ih4ElgDXAEcB/505qF6IiC8Dt9BMQZwM/BvNL/yPMsc1EhExlWbPITqLzoyI84BNpZQ/\n0MzFfSYingRWAp+jOcrl5wnDbe1A69d5XEszJ7yuk/sizaeaRa/+aWNH53jYK4DLgG0RsffTyuZS\nys5SypaI+A5wfUQ8D2wFvgr8upSyJGfU9Q62fhFxJvB+4FbgOeA8ms65p5SyNGPM6Ydn7Oewko/S\n/MXdQfOv75zsMfVovX5EU0Q7aL5t/iEwK3tcI1yXv6U59Gf3sMd/dWWuo/nyYztNOc3OHncv1g84\nAridpoB3Ak8D/wn8Rfa4K9ZrX+u0G/hgV2YKzbG2G2lK+MfAjOyx92L9gFOAu4ENnd/Lx2m+VJ2W\nNeboDEySlGBMzQlL0kRjCUtSIktYkhJZwpKUyBKWpESWsCQlSj9ZIyJOABbSHBe8M3c0ktQTRwBn\nAItKKc8dKNi3Eo6Iq4F/prny0u+Bj5dS/m8f0YU0VzWSpPHmAzQnZe1XX6YjWt4dY2U/xiBJY8DK\ngwX6NSd8DfDNUsr3SinLgKtoThH88D6yTkFIGq8O2m89L+EJcncMSeqJfuwJH+juGAN1ZX5J6rdD\neYjaIF6ZX5L6qh8lPKHujiFJo9HzEi4T7O4YkjQa/TpOeNzeHUOSeqkvJVxKublzTPBnaaYlHgIW\nllI29OP9JGlQpd9ZIyLeTDN9IUnjzQWllN8dKOAFfCQpkSUsSYksYUlKZAlLUiJLWJISWcKSlMgS\nlqRElrAkJbKEJSmRJSxJiSxhSUpkCUtSIktYkhJZwpKUyBKWpESWsCQlsoQlKZElLEmJLGFJSmQJ\nS1IiS1iSElnCkpTIEpakRJawJCWyhCUpkSUsSYksYUlKZAlLUiJLWJISWcKSlMgSlqRElrAkJbKE\nJSmRJSxJiSxhSUpkCUtSIktYkhJZwpKUyBKWpESWsCQlsoQlKZElLEmJLGFJSmQJS1KinpdwRFwb\nEXuGPR7t9ftI0nhwWJ9+7lJgARCd5y/36X0kaaD1q4RfLqVs6NPPlqRxo19zwq+LiGcj4qmI+H5E\nnNqn95GkgdaPEr4fuBJYCFwFzALujYipfXgvSRpoPZ+OKKUs6nq6NCKWAM8A7wNu6vX7SdIg6/sh\naqWUzcATwOx+v5ckDZq+l3BETAPOAtb2+70kadD04zjhL0fEJRFxekT8DfAzmkPUftTr95KkQdeP\nQ9ROAX4InABsABYDf11Kea4P7yVJA60fX8xd0eufKUnjldeOkKRElrAkJbKEJSmRJSxJiSxhSUpk\nCUtSIktYkhJZwpKUyBKWpESWsCQlsoQlKZElLEmJLGFJSmQJS1IiS1iSElnCkpTIEpakRJawJCWy\nhCUpkSUsSYksYUlKZAlLUqKe3/JerzZjxozq7Jve9Kbq7PTp06uzW7Zsqc5OmzatOtsvU6dOrc6+\n5jX1+xJ79uxpNY7du3f3Jbtr167q7PLly6uzK1eurM62sX379r78XLknLEmpLGFJSmQJS1IiS1iS\nElnCkpTIEpakRJawJCWyhCUpkSUsSYksYUlK5GnLHXPmzKnOtjldGGDevHnV2VNPPbU6e9RRR1Vn\nTzrppOpsRFRn25wOPWXKlOpsm1OAJ02aVJ097LB2v/Jt/l+0+dnHHHNMdXbr1q3V2RUrVlRnH3zw\nwerszTffXJ1du3ZtdVbuCUtSKktYkhJZwpKUyBKWpESWsCQlsoQlKZElLEmJLGFJSmQJS1IiS1iS\nEnnackebU5EnT57c6mcPDQ1VZy+88MLq7Mknn1ydbXOa7ObNm6uzmzZtqs6+/PLL1dkdO3ZUZ488\n8sjqbNu7LW/btq062+b06Tankc+cObM622a8c+fOrc4+8sgj1VlPW26n9Z5wRMyLiF9ExLMRsSci\nLttH5rMRsSYitkfEHRExuzfDlaTxZSTTEVOBh4CrgTL8xYj4FPAx4CPARcA2YFFEHD6KcUrSuNR6\nOqKUcjtwO0Ds+xJTnwA+V0q5pZP5ILAeeBdQfykmSZoAevrFXETMAoaAX+1dVkrZAjwA1E9ASdIE\n0eujI4ZopijWD1u+vvOaJKnLoTpELdjH/LEkTXS9LuF1NIV74rDlM3j13rEkTXg9LeFSygqaIl6w\nd1lEHA1cDNzXy/eSpPGg9dERETEVmE2zxwtwZkScB2wqpfwBuAH4TEQ8CawEPgesBn7ekxFL0jgy\nkjPm5gB30czxFuArneXfBT5cSvlSRBwFfBM4Fvhf4O9LKS/2YLySNK6M5DjhezjINEYp5TrgupEN\nKcddd91VnZ0/f36rn/3YY49VZ08//fTq7BFHHFGd3bBhQ3V2yZIl1dlnnnmmOtvmrsFt7lzc5i7O\n69e3+2qizV2f23j9619fnT3llFOqs21+J44//vjq7DnnnFOdvfPOO6uz8gI+kpTKEpakRJawJCWy\nhCUpkSUsSYksYUlKZAlLUiJLWJISWcKSlMgSlqRE3m15BFavXt0qf/jh9bfXO+qoo6qzK1eurM4+\n/fTT1dlbbrmlOvvEE09UZ/VnF110UXX2ne98Z3V2wYIFBw91rFmzpjq7ZcuW6qzacU9YkhJZwpKU\nyBKWpESWsCQlsoQlKZElLEmJLGFJSmQJS1IiS1iSElnCkpTI05ZH4Mknn2yVf/bZZ6uzO3furM5u\n3LixOvvcc89VZ9uMVyOzbNmy6uzll19enZ08eXJ1ts2pyJs2barOqh33hCUpkSUsSYksYUlKZAlL\nUiJLWJISWcKSlMgSlqRElrAkJbKEJSmRJSxJiSxhSUrktSNGoM1t6aHd+fzbt2+vzra55f1TTz1V\nnd22bVt1ViMzNDRUnZ0+fXp1dtq0adXZ1atXV2eXLl1anVU77glLUiJLWJISWcKSlMgSlqRElrAk\nJbKEJSmRJSxJiSxhSUpkCUtSIktYkhK1Pm05IuYB/wJcAJwEvKuU8ouu128C/mnYH7u9lHLpaAY6\nlrQ5tRhg0qRJ1dk2t0KfMWNGdXbDhg3VWfXfrFmzqrPnn39+dbbN79qKFSuqs21OkVc7I9kTngo8\nBFwNlP1kbgNOBIY6jytGNDpJGuda7wmXUm4HbgeIiNhPbFcpxV0vSTqIfs0Jz4+I9RGxLCJujIjj\n+/Q+kjTQ+nEpy9uAnwIrgLOALwC3RsTcUsr+pi8kaULqeQmXUm7uevpIRDwMPAXMB+7q9ftJ0iDr\n+yFqpZQVwEZgdr/fS5IGTd9LOCJOAU4A1vb7vSRp0IzkOOGpNHu1e4+MODMizgM2dR7X0swJr+vk\nvgg8ASzqxYAlaTwZyZzwHJq53dJ5fKWz/LvAR4E3Ah8EjgXW0JTvv5ZSXhr1aCVpnBnJccL3cOBp\njL8b+XAkaWLxbsuHwO7du6uzL7zwQl+yGlvmz59fnT3jjDOqs21OL161alV1Vv3jBXwkKZElLEmJ\nLGFJSmQJS1IiS1iSElnCkpTIEpakRJawJCWyhCUpkSUsSYk8bVlKcO6551Znp0+fXp1dvnx5X7Lq\nH/eEJSmRJSxJiSxhSUpkCUtSIktYkhJZwpKUyBKWpESWsCQlsoQlKZElLEmJPG1Z6pGFCxdWZ2fO\nnFmdPeaYY6qzGzdurM56t+WxwT1hSUpkCUtSIktYkhJZwpKUyBKWpESWsCQlsoQlKZElLEmJLGFJ\nSmQJS1IiT1uWeuQNb3hDdfb000+vzv7pT3+qzv72t7+tzmpscE9YkhJZwpKUyBKWpESWsCQlsoQl\nKZElLEmJLGFJSmQJS1IiS1iSElnCkpTI05alHpk7d2519rjjjqvOLl68uDp73333VWc1NrTaE46I\nT0fEkojYEhHrI+JnEXH2sMyUiPhGRGyMiK0R8ZOImNHbYUvS+NB2OmIe8DXgYuDtwGTglxFxZFfm\nBuAdwHuAS4CZwE9HP1RJGn9aTUeUUi7tfh4RVwJ/BC4AFkfE0cCHgX8spdzTyXwIeCwiLiqlLOnJ\nqCVpnBjtF3PHAgXY1Hl+AU2x/2pvoJTyOLAKqJ8wk6QJYsQlHBFBM/WwuJTyaGfxEPBiKWXLsPj6\nzmuSpC6jOTriRuBc4K0V2aDZY5YkdRnRnnBEfB24FJhfSlnT9dI64PDO3HC3GTR7w5KkLq1LuFPA\nlwNvK6WsGvbyg8DLwIKu/NnAacBvRjFOSRqXWk1HRMSNwBXAZcC2iDix89LmUsrOUsqWiPgOcH1E\nPA9sBb4K/NojIyTp1drOCV9FM7d797DlHwK+1/nva4DdwE+AKcDtwNUjH6IkjV9tjxM+6PRFKWUX\n8PHOQxpoc+bMqc4ODdUfAPT8889XZx977LHq7OrVq6uzGhu8gI8kJbKEJSmRJSxJiSxhSUpkCUtS\nIktYkhJZwpKUyBKWpESWsCQlsoQlKZF3W5YOYN68edXZKVOmVGfXrl1bnV2+fHl1VoPHPWFJSmQJ\nS1IiS1iSElnCkpTIEpakRJawJCWyhCUpkSUsSYksYUlKZAlLUiJPW5YOYPbs2dXZNqctb926tTq7\nbNmy6qwGj3vCkpTIEpakRJawJCWyhCUpkSUsSYksYUlKZAlLUiJLWJISWcKSlMgSlqRElrAkJfLa\nEZpQzjnnnFb5E044oTo7adKk6uz06dOrs/fee291VoPHPWFJSmQJS1IiS1iSElnCkpTIEpakRJaw\nJCWyhCUpkSUsSYksYUlKZAlLUqJWpy1HxKeBdwPnADuA+4BPlVKe6MrcDVzS9ccK8M1SykdHPVpp\nlNrcwh7guOOOq87u2LGjOrtixYrq7Isvvlid1eBpuyc8D/gacDHwdmAy8MuIOLIrU4BvAScCQ8BJ\nwCdHP1RJGn9a7QmXUi7tfh4RVwJ/BC4AFne9tL2UsmHUo5OkcW60c8LH0uz5bhq2/AMRsSEiHo6I\nzw/bU5YkdYz4UpYREcANwOJSyqNdL/0AeAZYA7wR+BJwNvDeUYxTksal0VxP+EbgXOAt3QtLKd/u\nevpIRKwD7oyIWaWU+m8jJGkCGNF0RER8HbgUmF9KWXuQ+ANAAO2+lpakCaD1nnCngC8H/raUsqri\nj5xPM298sLKWpAmn7XHCNwJXAJcB2yLixM5Lm0spOyPiTOD9wK3Ac8B5wPXAPaWUpb0btiSND233\nhK+i2au9e9jyDwHfA16kOX74E8BU4A/Aj4F/H9UoJWmcanuc8AHnkEspq4H5oxmQJE0k3m1ZE8rk\nyZNb5ZctW1adPe2006qzd9xxR3W2zR2f161bV53V2OAFfCQpkSUsSYksYUlKZAlLUiJLWJISWcKS\nlMgSlqRElrAkJbKEJSmRJSxJiaKUkjuAiDcDD6YOQtqPs846qzq7e/fu6mybOyivWbOmOqsx54JS\nyu8OFHBPWJISWcKSlMgSlqRElrAkJbKEpZa2bt2aPQSNI5aw1NILL7yQPQSNI5awJCWyhCUpkSUs\nSYnGwo0+j8gegLQ/u3btetWyPXv27HN5mzPmXnrppVGNSwPjoP02Fkr4jOwBSPuzevXqVsulYc4A\n7jtQYCxcO+IEYCGwEtiZOhhJ6o0jaAp4USnluQMF00tYkiYyv5iTpESWsCQlsoQlKZElLEmJLGFJ\nSmQJS1IiS1iSEv0/mh95z3AapvUAAAAASUVORK5CYII=\n", 314 | "text/plain": [ 315 | "" 316 | ] 317 | }, 318 | "metadata": {}, 319 | "output_type": "display_data" 320 | }, 321 | { 322 | "data": { 323 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWEAAAFdCAYAAADSR9wBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAE+hJREFUeJzt3X2sXHWdx/H3V1pAixWt0OpSlpYChqygtFu2a4s1VdlF\nw0M0LqCy6D8QcUNIdiUmhkrNaipKWB5qNLKsBtRQXOJToYIRXFAoaYPSutQIRVRoaQu2tKXQwm//\nONPscPt0Zu6cfu/Mfb+SSZgz397zPZy5n/ub33mYKKUgScrxmuwGJGk0M4QlKZEhLEmJDGFJSmQI\nS1IiQ1iSEhnCkpTIEJakRIawJCUyhCUp0YgM4Yi4JCLWRMQLEfFARPxtdk+9EBHzI+KVIY/fZvfV\njYiYExE/jIg/t7bjzD3ULIiIpyJiW0TcFRHTMnrtxv62LyJu2sO+XJLVb10R8dmIWBYRmyNiXUTc\nHhHHD6k5JCJuiIgNEfF8RNwWEUdm9dyJmtt3z5D99nJELMrqecSFcET8E/BVYD7wTuDXwNKIeHNq\nY72zEpgITGo9Zue207VxwMPAJcBuNyCJiMuBTwMXATOBrVT78eAD2eQw7HP7Wu7g1fvyvAPT2rDM\nAa4DTgXeC4wFfhoRr22ruQb4APAh4DTgrcD3D3Cf3aqzfQX4Bv+/794CfOYA99nWTSkj6gE8APxH\n2/MA/gR8Jru3HmzbfGBFdh8NbNcrwJlDlj0FXNb2fDzwAvCR7H57tH03Af+d3VsPtu3Nre2b3baf\nXgTOaas5oVUzM7vf4W5fa9nPgauze9v1GFEj4YgYC0wHfrZrWan+r90NzMrqq8eOa33EfSwibo6I\nydkN9VpETKEaYbTvx83AgwzOfgSY2/rI+2hELIqIN2U31IXDqUaGz7aeTwfG8Op9txp4kv7cd0O3\nb5ePRsT6iHgkIr44ZKR8QI3JWvFevBk4CFg3ZPk6qr/G/e4B4EJgNdVHoM8Dv4iIvymlbE3sq9cm\nUb3x97QfJx34dhpxB9VH9DXAscCXgCURMas1cBjxIiKoph7uK6XsOjYxCXip9UezXd/tu71sH8At\nwB+oPq2dBHwZOB748AFvkpEXwnsT7H1erm+UUpa2PV0ZEcuo3gwfofp4O+gGYj8ClFJubXu6KiIe\nAR4D5lJ93O0Hi4ATqXdcoh/33a7te1f7wlLKN9ueroqItcDdETGllLLmQDYII+/A3AbgZaoJ83ZH\nsvuoqu+VUjYBvwP65qyBmtZS/dKOiv0I0Prl3UCf7MuIuB44A5hbSnmq7aW1wMERMX7IP+mrfTdk\n+57eT/mDVO/XlH03okK4lLIDWA7M27Ws9ZFiHvDLrL6aEhGHUX2U3d+bpK+0Amktr96P46mOWA/c\nfgSIiKOACfTBvmwF1FnAe0opTw55eTmwk1fvu+OBo4FfHbAmh2E/27cn76Qa5afsu5E4HXE18K2I\nWA4sAy4DXgf8V2ZTvRARVwE/opqC+CvgSqo3/Hcz++pGRIyjGjlEa9HUiDgZeLaU8kequbjPRcTv\ngSeAL1Cd5fKDhHY7tq/taz3mU80Jr23VLaT6VLN09582crTOhz0POBPYGhG7Pq1sKqVsL6Vsjogb\ngasj4jngeeBa4P5SyrKcruvb3/ZFxFTgfGAJsBE4mSpz7i2lrMzoOf30jL2cVvIpql/cF6j++s7I\n7qlH2/VdqiB6gepo83eAKdl9dbkt76Y69eflIY//bKv5PNXBj21U4TQtu+9ebB9wKHAnVQBvBx4H\nvgYckd13je3a0za9DFzQVnMI1bm2G6hCeDFwZHbvvdg+4CjgHmB96325muqg6mFZPUerMUlSghE1\nJyxJo40hLEmJDGFJSmQIS1IiQ1iSEhnCkpQo/WKNiJgAnE51XvD23G4kqScOBY4BlpZSNu6rsLEQ\njohLgH+luvPSr4F/KaU8tIfS06nuaiRJg+ajVBdl7VUj0xEdfjvGE030IEkjwBP7K2hqTvgy4Oul\nlG+XUh4FLqa6RPCTe6h1CkLSoNpvvvU8hEfJt2NIUk80MRLe17dj9NWd+SWpaQfyFLV+vDO/JDWq\niRAeVd+OIUnD0fMQLqPs2zEkaTiaOk94YL8dQ5J6qZEQLqXc2joneAHVtMTDwOmllPVNrE+S+lX6\nN2tExClU0xeSNGiml1JW7KvAG/hIUiJDWJISGcKSlMgQlqREhrAkJTKEJSmRISxJiQxhSUpkCEtS\nIkNYkhIZwpKUyBCWpESGsCQlMoQlKZEhLEmJDGFJSmQIS1IiQ1iSEhnCkpTIEJakRIawJCUyhCUp\nkSEsSYkMYUlKZAhLUiJDWJISGcKSlMgQlqREhrAkJTKEJSmRISxJiQxhSUo0JrsB9bexY8fWrn3N\na+r/zZ88eXLt2gsuuKB27XHHHVe7FuDcc8/tqL4J1157be3a+fPn1679y1/+0k076jFHwpKUyBCW\npESGsCQlMoQlKZEhLEmJDGFJSmQIS1IiQ1iSEhnCkpTIEJakRIawJCWKUkpvf2DEfGDoBeyPllJO\n3Ev9KcDynjahYTn66KNr19511121a6dNm9ZNO+rAww8/XLt2zpw5tWu3bdvWTTuC6aWUFfsqaOoG\nPiuBeUC0nu9saD2S1NeaCuGdpZT1Df1sSRoYTc0JHxcRf46IxyLi5oiof19CSRpFmgjhB4ALgdOB\ni4EpwC8iYlwD65Kkvtbz6YhSytK2pysjYhnwB+AjwE29Xp8k9bPGT1ErpWwCfgd4aFyShmg8hCPi\nMOBY4Omm1yVJ/abnIRwRV0XEaRHx1xHx98DtVKeofbfX65KkftfEKWpHAd8BJgDrgfuAvyulbGxg\nXZLU15o4MHder3+mJA0qv/Jeu1m4cGHtWi9FHlne8Y531K499NBDa9d62XJzvIGPJCUyhCUpkSEs\nSYkMYUlKZAhLUiJDWJISGcKSlMgQlqREhrAkJTKEJSmRly1rN1u2bMlugZdeeql27YYNGxrr48Yb\nb6xdO2HChNq1F110Ue3agw46qHat+o8jYUlKZAhLUiJDWJISGcKSlMgQlqREhrAkJTKEJSmRISxJ\niQxhSUpkCEtSIi9b1m6+8pWv1K6dN29e7drvfe97tWvvv//+2rU/+clPatc2aebMmbVrP/axj9Wu\nHT9+fDftqE84EpakRIawJCUyhCUpkSEsSYkMYUlKZAhLUiJDWJISGcKSlMgQlqREhrAkJfKyZe1m\n9erVtWunTp3aYCf5ZsyYUbv2qquuql3rpcjaxZGwJCUyhCUpkSEsSYkMYUlKZAhLUiJDWJISGcKS\nlMgQlqREhrAkJTKEJSmRly1rVDn22GM7qr/88str186ePbvTdnrulltuqV27ZcuWBjtRXR2PhCNi\nTkT8MCL+HBGvRMSZe6hZEBFPRcS2iLgrIqb1pl1JGizdTEeMAx4GLgHK0Bcj4nLg08BFwExgK7A0\nIg4eRp+SNJA6no4opdwJ3AkQEbGHkkuBL5RSftSquQBYB5wN3Np9q5I0eHp6YC4ipgCTgJ/tWlZK\n2Qw8CMzq5bokaRD0+uyISVRTFOuGLF/Xek2S1OZAnaIW7GH+WJJGu16H8FqqwJ04ZPmR7D46lqRR\nr6chXEpZQxXE83Yti4jxwKnAL3u5LkkaBB2fHRER44BpVCNegKkRcTLwbCnlj8A1wOci4vfAE8AX\ngD8BP+hJx5I0QLq5Ym4G8HOqOd4CfLW1/FvAJ0spX46I1wFfBw4H/gf4x1LKSz3oV5IGSpSSe7ws\nIk4Blqc2oQNi3LhxtWvf//731679+Mc/Xrv2tNNOq10L8MY3vrGj+ibcfffdtWvPOuus2rXbt2/v\nph11ZnopZcW+CryBjyQlMoQlKZEhLEmJDGFJSmQIS1IiQ1iSEhnCkpTIEJakRIawJCUyhCUpkd+2\nrGHp5FLkG264oXZtJ5ciD7oTTjihdu20afW/U3flypXdtKMecyQsSYkMYUlKZAhLUiJDWJISGcKS\nlMgQlqREhrAkJTKEJSmRISxJiQxhSUrkZcsalgULFtSu9VLk7kyePLl27ZVXXlm79txzz61du2PH\njtq16owjYUlKZAhLUiJDWJISGcKSlMgQlqREhrAkJTKEJSmRISxJiQxhSUpkCEtSIkNYkhJ57wgN\nSyklu4WOPPTQQx3Vb9q0qXbtxIkTa9e+/e1v76iPus4+++zatSeddFLt2uXLl3fTjmpwJCxJiQxh\nSUpkCEtSIkNYkhIZwpKUyBCWpESGsCQlMoQlKZEhLEmJDGFJStTxZcsRMQf4N2A68Bbg7FLKD9te\nvwn45yH/7M5SyhnDaVQj0/z582vXHnTQQbVrx4yp/9a8+eaba9euWrWqdi3Ali1batdOmDChdu2K\nFStq1x511FG1aztx/vnn1671suXmdDMSHgc8DFwC7O3GAXcAE4FJrcd5XXUnSQOu45FwKeVO4E6A\niIi9lL1YSlk/nMYkaTRoak54bkSsi4hHI2JRRLypofVIUl9r4laWdwDfB9YAxwJfApZExKzSb/c9\nlKSG9TyESym3tj1dFRGPAI8Bc4Gf93p9ktTPGj9FrZSyBtgATGt6XZLUbxoP4Yg4CpgAPN30uiSp\n33RznvA4qlHtrjMjpkbEycCzrcd8qjnhta26hcDvgKW9aFiSBkk3c8IzqOZ2S+vx1dbybwGfAk4C\nLgAOB56iCt8rSik7ht2tJA2Ybs4Tvpd9T2P8Q/ftSNLo4rcta1i2bt1au/ayyy5rsJN8GzdurF27\nffv2BjupZ9q0+sfKO7mMfOfOnd20M2p5Ax9JSmQIS1IiQ1iSEhnCkpTIEJakRIawJCUyhCUpkSEs\nSYkMYUlKZAhLUiIvW5ZGqeeee6527SuvvNJgJ6ObI2FJSmQIS1IiQ1iSEhnCkpTIEJakRIawJCUy\nhCUpkSEsSYkMYUlKZAhLUiIvWz4AZsyYUbt2ypQptWsXL17cTTsS0Nm3Q3vZcnMcCUtSIkNYkhIZ\nwpKUyBCWpESGsCQlMoQlKZEhLEmJDGFJSmQIS1IiQ1iSEnnZchdmz57dUf2Pf/zj2rWdXB76m9/8\npnbt6tWra9eqO1dccUXt2mOOOaa5Rmq67rrrslsQjoQlKZUhLEmJDGFJSmQIS1IiQ1iSEhnCkpTI\nEJakRIawJCUyhCUpkSEsSYm8bLkLH/zgBzuqf/3rX99IH0uXLq1du3DhwvQeHn/88UZ6mDx5cu3a\nTvfd2972ttq1F198ce3aMWOa+dVbuXJl7dr169c30oM609FIOCI+GxHLImJzRKyLiNsj4vghNYdE\nxA0RsSEino+I2yLiyN62LUmDodPpiDnAdcCpwHuBscBPI+K1bTXXAB8APgScBrwV+P7wW5WkwdPR\nZ6JSyhntzyPiQuAZYDpwX0SMBz4JnFtKubdV8wngfyNiZillWU+6lqQBMdwDc4cDBXi29Xw6VbD/\nbFdBKWU18CQwa5jrkqSB03UIR0RQTT3cV0r5bWvxJOClUsrmIeXrWq9JktoM5xDtIuBEoM4dzoNq\nxCxJatPVSDgirgfOAOaWUp5qe2ktcHBrbrjdkVSjYUlSm45DuBXAZwHvKaU8OeTl5cBOYF5b/fHA\n0cCvhtGnJA2kjqYjImIRcB5wJrA1Iia2XtpUStleStkcETcCV0fEc8DzwLXA/Z4ZIUm763RO+GKq\nud17hiz/BPDt1n9fBrwM3AYcAtwJXNJ9i5I0uKKU3ONlEXEK1TRG3zjnnHM6qr/tttsa6iTfli1b\natfu2LGjkR7Gjh1bu/awww5rpIcmdXIp8vve977atc8880w37agz00spK/ZV4A18JCmRISxJiQxh\nSUpkCEtSIkNYkhIZwpKUyBCWpESGsCQlMoQlKZEhLEmJ/LblLixZsqSj+uXL61+VPX369E7bSdWP\nlwGPBKtWrapdu2DBgtq1XorcfxwJS1IiQ1iSEhnCkpTIEJakRIawJCUyhCUpkSEsSYkMYUlKZAhL\nUiJDWJISedlyF1588cWO6mfNmlW79g1veEPt2ksvvbR27Zw5c2rXHnHEEbVrTzzxxNq1/Wj16tW1\naxcvXly79vrrr69du379+tq16j+OhCUpkSEsSYkMYUlKZAhLUiJDWJISGcKSlMgQlqREhrAkJTKE\nJSmRISxJiQxhSUoUpZTcBiJOAep/J7wk9Y/ppZQV+ypwJCxJiQxhSUpkCEtSIkNYkhIZwpKUyBCW\npESGsCQlMoQlKZEhLEmJDGFJStRRCEfEZyNiWURsjoh1EXF7RBw/pOaeiHil7fFyRCzqbduSNBg6\nHQnPAa4DTgXeC4wFfhoRr22rKcA3gInAJOAtwGeG36okDZ4xnRSXUs5ofx4RFwLPANOB+9pe2lZK\nWT/s7iRpwA13TvhwqpHvs0OWfzQi1kfEIxHxxSEjZUlSS0cj4XYREcA1wH2llN+2vXQL8AfgKeAk\n4MvA8cCHh9GnJA2krkMYWAScCLyrfWEp5ZttT1dFxFrg7oiYUkpZM4z1SdLA6Wo6IiKuB84A5pZS\nnt5P+YNAANO6WZckDbKOR8KtAD4LeHcp5cka/+SdVPPG+wtrSRp1Ogrh1vm+5wFnAlsjYmLrpU2l\nlO0RMRU4H1gCbAROBq4G7i2lrOxd25I0GDodCV9MNaq9Z8jyTwDfBl6iOn/4UmAc8EdgMfDvw+pS\nkgZUp+cJ73MOuZTyJ2DucBqSpNHEe0dIUiJDWJISGcKSlMgQlqREhrAkJTKEJSmRISxJiQxhSUpk\nCEtSIkNYkhIZwpKUyBCWpESGsCQlMoQlKZEhLEmJDGFJSmQIS1IiQ1iSEhnCkpTIEJakRIawJCUa\nCSF8aHYDktSQ/ebbSAjhY7IbkKSGHLO/giilHIA+9tFAxATgdOAJYHtqM5LUG4dSBfDSUsrGfRWm\nh7AkjWYjYTpCkkYtQ1iSEhnCkpTIEJakRIawJCUyhCUpkSEsSYn+D53vLloBOZ4sAAAAAElFTkSu\nQmCC\n", 324 | "text/plain": [ 325 | "" 326 | ] 327 | }, 328 | "metadata": {}, 329 | "output_type": "display_data" 330 | }, 331 | { 332 | "data": { 333 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWEAAAFdCAYAAADSR9wBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAFXJJREFUeJzt3X3MnXV9x/H3l0J5uKGUhz7Kg6UVDCVtpIzaQbVYgQ0T\ncNG4IYlT/xkZWxaTzcVkGU6zLXOROHUsMzqn8SEBnREjgkoEeWohYFHqaJ22pZS2tBRb21Ja2t/+\nuE63400fftd9n9Pv/fB+JSfhnPPpOb+L6+6nv3Pd1+9cUUpBkpTjuOwBSNJ4ZglLUiJLWJISWcKS\nlMgSlqRElrAkJbKEJSmRJSxJiSxhSUpkCUtSohFZwhFxS0SsiYiXI2JZRPxO9ph6ISJujYgDg24/\nzx7XUETE4oi4KyI2dLbj+kNkPhYRz0fE7oj4QUTMyRjrUBxt+yLii4fYl3dnjbdWRHwkIh6LiB0R\nsTkivhURFw7KnBgR/xoRWyPiNxHxjYiYmjXmNiq37/5B+21/RNyeNeYRV8IR8YfAJ4FbgTcBTwH3\nRsTZqQPrnaeBacD0zu3K3OEM2QCwArgFeM0XkETEXwN/BvwJcDmwi2Y/TjyWgxyGI25fx/f47X15\n47EZ2rAsBj4DLATeDpwAfD8iTu7KfAp4B/Au4C3ATOCbx3icQ1WzfQX4HP+/72YAHz7G4+waTSkj\n6gYsA/6l634AzwEfzh5bD7btVuDJ7HH0YbsOANcPeux54ENd9ycBLwPvyR5vj7bvi8B/ZY+tB9t2\ndmf7ruzaT68Af9CVuaiTuTx7vMPdvs5jPwJuyx7bwduImglHxAnAAuC+g4+V5v/aD4FFWePqsTd0\nPuL+MiK+EhHnZg+o1yJiFs0Mo3s/7gCWM3b2I8CSzkfeZyLi9og4M3tAQzCZZma4rXN/AXA8v73v\nVgHPMjr33eDtO+imiNgSET+LiH8YNFM+po7PeuPDOBuYAGwe9Phmmn+NR7tlwPuBVTQfgT4K/Dgi\nLiml7EocV69Np/nBP9R+nH7sh9MX36P5iL4GmA38I3B3RCzqTBxGvIgImkMPD5VSDv5uYjqwt/OP\nZrdRt+8Os30AXwXW0Xxamwd8ArgQePcxHyQjr4QPJzj8cblRo5Ryb9fdpyPiMZofhvfQfLwd68bE\nfgQopdzRdXdlRPwM+CWwhObj7mhwO3Axdb+XGI377uD2XdH9YCnl8113V0bEJuCHETGrlLLmWA4Q\nRt4v5rYC+2kOmHebymtnVaNeKWU7sBoYNWcNVNpE85d2XOxHgM5f3q2Mkn0ZEZ8FrgOWlFKe73pq\nEzAxIiYN+iOjat8N2r6NR4kvp/l5Tdl3I6qESyn7gCeApQcf63ykWAo8kjWufomIU2k+yh7th2RU\n6RTSJn57P06i+Y31mNuPABFxDnAWo2BfdgrqBuCqUsqzg55+AniV3953FwLnAY8es0EOw1G271De\nRDPLT9l3I/FwxG3AlyLiCeAx4EPAKcB/Zg6qFyLin4Hv0ByCeB3wdzQ/8F/PHNdQRMQAzcwhOg9d\nEBHzgW2llPU0x+L+JiL+B1gLfJzmLJdvJwy3tSNtX+d2K80x4U2d3D/RfKq597WvNnJ0zoe9Ebge\n2BURBz+tbC+l7Cml7IiILwC3RcRLwG+ATwMPl1Ieyxl1vaNtX0RcALwXuBt4EZhP0zkPlFKezhhz\n+ukZhzmt5E9p/uK+TPOv72XZY+rRdn2dpohepvlt89eAWdnjGuK2vJXm1J/9g27/0ZX5KM0vP3bT\nlNOc7HH3YvuAk4B7aAp4D/Ar4N+AKdnjrtiuQ23TfuB9XZkTac613UpTwncCU7PH3ovtA84B7ge2\ndH4uV9H8UvXUrDFHZ2CSpAQj6piwJI03lrAkJbKEJSmRJSxJiSxhSUpkCUtSovTFGhFxFnAtzXnB\ne3JHI0k9cRLweuDeUsqLRwr2rYQj4hbgL2m+eekp4M9LKY8fInotzbcaSdJYcxPNoqzD6svhiJZX\nx1jbjzFI0giw9miBfh0T/hDw76WUL5dSngFuplki+MFDZD0EIWmsOmq/9byEx8nVMSSpJ/oxEz7S\n1TFG1TfzS1K/HctT1EbjN/NLUl/1o4TH1dUxJGk4el7CZZxdHUOShqNf5wmP2atjSFIv9aWESyl3\ndM4J/hjNYYkVwLWllC39eD9JGq3Sr6wREZfSHL6QpLFmQSnlySMF/AIfSUpkCUtSIktYkhJZwpKU\nyBKWpESWsCQlsoQlKZElLEmJLGFJSmQJS1IiS1iSElnCkpTIEpakRJawJCWyhCUpkSUsSYksYUlK\nZAlLUiJLWJISWcKSlMgSlqRElrAkJbKEJSmRJSxJiSxhSUpkCUtSIktYkhJZwpKUyBKWpESWsCQl\nsoQlKZElLEmJjs8egIbulFNOqc5OnTq1Ojt37tzq7OzZs6uzkyZNqs5Onjy5OjthwoTqbCmlOguw\nf//+6mxEVGc3bNhQnV25cmV19sEHH6zOvvzyy9VZ9Y8zYUlKZAlLUiJLWJISWcKSlMgSlqRElrAk\nJbKEJSmRJSxJiSxhSUpkCUtSIktYkhL1/LsjIuJW4NZBDz9TSrm41+813s2bN686O2vWrOrsOeec\n05cxXHrppdXZ0047rTp7/PH1P8Ztvt8BYPv27dXZF198sS/ZKVOmVGd37NhRnV22bFl1Vv3Try/w\neRpYChz8iX+1T+8jSaNav0r41VLKlj69tiSNGf06JvyGiNgQEb+MiK9ExLl9eh9JGtX6UcLLgPcD\n1wI3A7OAH0fEQB/eS5JGtZ4fjiil3Nt19+mIeAxYB7wH+GKv30+SRrO+n6JWStkOrAbm9Pu9JGm0\n6XsJR8SpwGxgY7/fS5JGm56XcET8c0S8JSLOj4jfBb5Fc4ra13v9XpI02vXjFLVzgK8BZwFbgIeA\nN5dS6s9Ol6Rxoh+/mLux168pSWOVl7wfYSZOnFid3bdvX3V29+7d1dk2S1+fe+656uyBAweqs22W\nF59xxhnV2baXvD/llFOqs5MnT67Onn766dXZ2bNnV2eXLl1anV2/fn11dsOGDdVZteMX+EhSIktY\nkhJZwpKUyBKWpESWsCQlsoQlKZElLEmJLGFJSmQJS1IiS1iSErlseYTZu3dvdfaJJ56ozq5atao6\ne8kll1Rnly9fXp3dtm1bdfa44+rnB+eff351ts0SZ4AZM2ZUZxcvXlydXbRoUXV26tSp1dk2S87b\nXIHbZcv940xYkhJZwpKUyBKWpESWsCQlsoQlKZElLEmJLGFJSmQJS1IiS1iSElnCkpTIZcvjxM6d\nO6uzy5Yt6+NIem/t2rV9e+2ZM2dWZ9ssL164cGFfXrfNcu89e/ZUZ9U/zoQlKZElLEmJLGFJSmQJ\nS1IiS1iSElnCkpTIEpakRJawJCWyhCUpkSUsSYlctiwdwZlnnlmdnTt3bnX23HPPrc7u27evOrt9\n+/bqbJurX6t/nAlLUiJLWJISWcKSlMgSlqRElrAkJbKEJSmRJSxJiSxhSUpkCUtSIktYkhK5bFmj\n3sknn1ydveaaa1q99k033VSdfdvb3ladPeGEE6qz9913X3W2zZWyf/WrX1Vn1T+tZ8IRsTgi7oqI\nDRFxICKuP0TmYxHxfETsjogfRMSc3gxXksaWoRyOGABWALcAZfCTEfHXwJ8BfwJcDuwC7o2IicMY\npySNSa0PR5RS7gHuAYiIOETkL4CPl1K+08m8D9gMvBO4Y+hDlaSxp6e/mIuIWcB04P8OYpVSdgDL\ngUW9fC9JGgt6fXbEdJpDFJsHPb6585wkqcuxOkUtOMTxY0ka73pdwptoCnfaoMen8trZsSSNez0t\n4VLKGpoiXnrwsYiYBCwEHunle0nSWND67IiIGADm0Mx4AS6IiPnAtlLKeuBTwN9ExP8Aa4GPA88B\n3+7JiCVpDBnKirnLgB/RHOMtwCc7j38J+GAp5RMRcQrw78Bk4EHg90spe3swXkkaU4ZynvADHOUw\nRinlo8BHhzYkCa666qrqbJvlwldeeWWrccyfP78622Yp8rPPPludffzxx6uzDz/8cHVWI4Nf4CNJ\niSxhSUpkCUtSIktYkhJZwpKUyBKWpESWsCQlsoQlKZElLEmJLGFJSuTVljUsF198cXX26quvrs5e\nccUV1dmLLrqoOtvmyswA+/btq85u3769Ottm2XKbMZ922mnVWY0MzoQlKZElLEmJLGFJSmQJS1Ii\nS1iSElnCkpTIEpakRJawJCWyhCUpkSUsSYlctqzXaLMM+JprrqnOLl68uDp79tlnV2e3bNlSnR0Y\nGKjOQrulyG2WOLe5MvPrXve66myb5d5r1qypzr7yyivVWbXjTFiSElnCkpTIEpakRJawJCWyhCUp\nkSUsSYksYUlKZAlLUiJLWJISWcKSlMgSlqREfnfEODFz5szq7Kmnnlqd3bBhQ3V29erV1dk232vw\n6quvVmfbfL9DW5MnT67OTpkypTo7bdq06uySJUuqs5s2barOfve7363Oqh1nwpKUyBKWpESWsCQl\nsoQlKZElLEmJLGFJSmQJS1IiS1iSElnCkpTIEpakRK2XLUfEYuCvgAXADOCdpZS7up7/IvDHg/7Y\nPaWU64YzUA3Pzp07q7MvvfRSdXb58uXV2UcffbQ6u3Xr1ursnj17qrNtHXdc/Txl3rx51dmlS5dW\nZ2+44Ybq7CWXXFKdffOb31ydddly/wxlJjwArABuAcphMt8DpgHTO7cbhzQ6SRrjWs+ESyn3APcA\nREQcJvZKKWXLcAYmSeNBv44JL4mIzRHxTETcHhFn9ul9JGlU68dXWX4P+CawBpgN/CNwd0QsKqUc\n7vCFJI1LPS/hUsodXXdXRsTPgF8CS4Af9fr9JGk06/spaqWUNcBWYE6/30uSRpu+l3BEnAOcBWzs\n93tJ0mgzlPOEB2hmtQfPjLggIuYD2zq3W2mOCW/q5P4JWA3c24sBS9JYMpRjwpfRHNstndsnO49/\nCfhTYB7wPmAy8DxN+f5tKaV/F/eSpFFqKOcJP8CRD2P83tCHI0nji1dbHid27NjRl+xYd+DAgers\nihUrqrOXX355dfb44+v/mp5xxhnV2QkTJlRnzz///OrsunXrqrPyC3wkKZUlLEmJLGFJSmQJS1Ii\nS1iSElnCkpTIEpakRJawJCWyhCUpkSUsSYlctiz1yMknn1ydHRgYqM5OmTKlOnviiSdWZ9tcpXrv\n3r3VWbXjTFiSElnCkpTIEpakRJawJCWyhCUpkSUsSYksYUlKZAlLUiJLWJISWcKSlMhly8fAzJkz\nq7Ntlr62WUq6fv366qyG5uqrr67OXnbZZdXZNldQbnOl7N27d1dnN27cWJ1VO86EJSmRJSxJiSxh\nSUpkCUtSIktYkhJZwpKUyBKWpESWsCQlsoQlKZElLEmJXLbcMX369OrshAkTWr324sWLq7OzZ8+u\nzv7617+uzv7kJz+pzj7yyCPV2dGmzXJhgEWLFlVn58yZU5194xvfWJ1t8/O2bt266uzjjz9enVX/\nOBOWpESWsCQlsoQlKZElLEmJLGFJSmQJS1IiS1iSElnCkpTIEpakRJawJCVy2XLHpk2bqrMzZsxo\n9dptlspOnTq1Onv66adXZxcsWFCdXbhwYXW2zf+3nTt3VmfbXHW6zVLvtvtu2rRpfclOnDixOrt2\n7drq7F133VWdXb58eXVW/dNqJhwRH4mIxyJiR0RsjohvRcSFgzInRsS/RsTWiPhNRHwjIuqbRZLG\nkbaHIxYDnwEWAm8HTgC+HxHd05ZPAe8A3gW8BZgJfHP4Q5WksafV4YhSynXd9yPi/cALwALgoYiY\nBHwQ+KNSygOdzAeA/46Iy0spj/Vk1JI0Rgz3F3OTgQJs69xfQFPs9x0MlFJWAc8C9d8JKEnjxJBL\nOCKC5tDDQ6WUn3ceng7sLaXsGBTf3HlOktRlOGdH3A5cDFxZkQ2aGbMkqcuQZsIR8VngOmBJKeX5\nrqc2ARM7x4a7TaWZDUuSurQu4U4B3wBcVUp5dtDTTwCvAku78hcC5wGPDmOckjQmtTocERG3AzcC\n1wO7IuLg2enbSyl7Sik7IuILwG0R8RLwG+DTwMOeGSFJr9X2mPDNNMd27x/0+AeAL3f++0PAfuAb\nwInAPcAtQx+iJI1dbc8TPurhi1LKK8Cfd26jxty5c6uzK1eubPXaba6KPH/+/Opsmyv27t27tzrb\nZpn1hg0bqrMbN26szp500knV2YGBgepsm6Xe0G55cZv/xytWrKjOtrkq8p133lmd3bNnT3VW/eMX\n+EhSIktYkhJZwpKUyBKWpESWsCQlsoQlKZElLEmJLGFJSmQJS1IiS1iSEnm15Y62S5HbWL16dXX2\n3HPPrc5On17/PfltluvOmjWrOtvmCsPr1q2rzrZZUjthwoTq7K5du6qzAL/4xS+qs0899VR19qc/\n/Wl19uGHH67O7t+/vzqrkcGZsCQlsoQlKZElLEmJLGFJSmQJS1IiS1iSElnCkpTIEpakRJawJCWy\nhCUpkcuWj4E77rijOvvCCy/0JXveeedVZydNmlSdLaVUZ/ft21ed3bx5c3V21apV1dk2y5ABnnzy\nyers1q1bW722BM6EJSmVJSxJiSxhSUpkCUtSIktYkhJZwpKUyBKWpESWsCQlsoQlKZElLEmJLGFJ\nShRt1v73ZQARlwJPpA5CkvpjQSnliF9A4kxYkhJZwpKUyBKWpESWsCQlsoQlKZElLEmJLGFJSmQJ\nS1IiS1iSElnCkpSoVQlHxEci4rGI2BERmyPiWxFx4aDM/RFxoOu2PyJu7+2wJWlsaDsTXgx8BlgI\nvB04Afh+RJzclSnA54BpwHRgBvDh4Q9Vksae49uESynXdd+PiPcDLwALgIe6ntpdStky7NFJ0hg3\n3GPCk2lmvtsGPX5TRGyJiJ9FxD8MmilLkjpazYS7RUQAnwIeKqX8vOuprwLrgOeBecAngAuBdw9j\nnJI0Jg25hIHbgYuBK7ofLKV8vuvuyojYBPwwImaVUtYM4/0kacwZ0uGIiPgscB2wpJSy8Sjx5UAA\nc4byXpI0lrWeCXcK+AbgraWUZyv+yJtojhsfrawladxpVcKd831vBK4HdkXEtM5T20speyLiAuC9\nwN3Ai8B84DbggVLK070btiSNDW1nwjfTzGrvH/T4B4AvA3tpzh/+C2AAWA/cCfz9sEYpSWNU2/OE\nj3gMuZTyHLBkOAOSpPHE746QpESWsCQlsoQlKZElLEmJLGFJSmQJS1IiS1iSElnCkpTIEpakRJaw\nJCWyhCUpkSUsSYksYUlKZAlLUiJLWJISWcKSlMgSlqRElrAkJbKEJSmRJSxJiSxhSUo0Ekr4pOwB\nSFKfHLXfRkIJvz57AJLUJ68/WiBKKcdgHEcYQMRZwLXAWmBP6mAkqTdOoinge0spLx4pmF7CkjSe\njYTDEZI0blnCkpTIEpakRJawJCWyhCUpkSUsSYksYUlK9L9fg67/hYNt6wAAAABJRU5ErkJggg==\n", 334 | "text/plain": [ 335 | "" 336 | ] 337 | }, 338 | "metadata": {}, 339 | "output_type": "display_data" 340 | }, 341 | { 342 | "data": { 343 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWEAAAFdCAYAAADSR9wBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAE3xJREFUeJzt3XvMXHWdx/H3VygXq4hUaIUigm2XkKYN1OUqUlMToE2Q\nVeMumrioMVaQGBJWNIGtS4OkGgjrpUbFBY2IAd3GC9CKBnCplG5oUC5LG9giYi+WYmjLRaT97R9n\nmh2e3s7MM+f5PjPP+5VMwpz5ds73cKaf/uY35xKlFCRJOV6X3YAkjWWGsCQlMoQlKZEhLEmJDGFJ\nSmQIS1IiQ1iSEhnCkpTIEJakRIawJCUalSEcERdHxNqIeCkiVkTE32f31AsRsSAidgx5PJbdVzci\n4syI+FlE/Km1HeftpuaqiFgXES9GxF0RMSWj127sa/si4sbd7Ms7svqtKyK+EBErI2JLRGyMiCUR\nMW1IzYER8Y2IeDYitkbEjyPiiKyeO1Fz++4Zst+2R8TirJ5HXQhHxD8C1wILgBOB3wHLIuItqY31\nziPARGBS6/Gu3Ha6Nh54CLgY2OUCJBFxOfAZ4FPAycALVPvxgJFschj2un0td/LafXnByLQ2LGcC\nXwNOAd4LjAN+GREHt9VcD8wDPgC8GzgS+MkI99mtOttXgG/z//vurcDnRrjPtm5KGVUPYAXw723P\nA3gG+Fx2bz3YtgXAquw+GtiuHcB5Q5atAy5te34I8BLwoex+e7R9NwL/md1bD7btLa3te1fbfvor\n8A9tNX/Xqjk5u9/hbl9r2d3Addm97XyMqpFwRIwDZgG/3rmsVP/XfgWcltVXj01tfcV9MiJ+EBFH\nZzfUaxFxLNUIo30/bgEeYHD2I8Ds1lfexyNicUQclt1QFw6lGhk+13o+C9if1+671cDT9Oe+G7p9\nO30kIjZFxMMR8aUhI+URtX/WivfgLcB+wMYhyzdS/Wvc71YAFwKrqb4CfRH4TURML6W8kNhXr02i\n+uDvbj9OGvl2GnEn1Vf0tcA7gGuAOyLitNbAYdSLiKCaerivlLLzt4lJwCutfzTb9d2+28P2AdwM\n/IHq29oM4MvANOCDI94koy+E9yTY87xc3yilLGt7+khErKT6MHyI6uvtoBuI/QhQSrm17emjEfEw\n8CQwm+rrbj9YDJxAvd8l+nHf7dy+M9oXllJuaHv6aERsAH4VEceWUtaOZIMw+n6YexbYTjVh3u4I\ndh1V9b1SyvPAGqBvjhqoaQPVX9oxsR8BWn95n6VP9mVEfB2YC8wupaxre2kDcEBEHDLkj/TVvhuy\nfev3Uf4A1ec1Zd+NqhAupfwNeBCYs3NZ6yvFHOC3WX01JSLeQPVVdl8fkr7SCqQNvHY/HkL1i/XA\n7UeAiJgMTKAP9mUroN4HvKeU8vSQlx8EXuW1+24a8Dbg/hFrchj2sX27cyLVKD9l343G6YjrgO9F\nxIPASuBS4PXATZlN9UJEfAX4OdUUxFHAv1F94G/J7KsbETGeauQQrUXHRcRM4LlSyh+p5uKuiIgn\ngKeAhVRHufw0od2O7W37Wo8FVHPCG1p1i6i+1Szb9d1Gj9bxsBcA5wEvRMTObyvPl1JeLqVsiYjv\nAtdFxF+ArcBXgeWllJU5Xde3r+2LiOOADwN3AJuBmVSZc28p5ZGMntMPz9jDYSUXUf3FfYnqX993\nZvfUo+26hSqIXqL6tfmHwLHZfXW5LWdRHfqzfcjjP9pqvkj148eLVOE0JbvvXmwfcBCwlCqAXwb+\nF/gmcHh23zW2a3fbtB34aFvNgVTH2j5LFcK3AUdk996L7QMmA/cAm1qfy9VUP6q+IavnaDUmSUow\nquaEJWmsMYQlKZEhLEmJDGFJSmQIS1IiQ1iSEqWfrBERE4CzqY4Lfjm3G0nqiYOAtwPLSimb91bY\nWAhHxMXAZVRXXvodcEkp5b93U3o21VWNJGnQfITqpKw9amQ6osO7YzzVRA+SNAo8ta+CpuaELwW+\nVUr5finlcWA+1SmCH99NrVMQkgbVPvOt5yE8Ru6OIUk90cRIeG93x+irK/NLUtNG8hC1frwyvyQ1\nqokQHlN3x5Ck4eh5CJcxdncMSRqOpo4THti7Y0hSLzUSwqWUW1vHBF9FNS3xEHB2KWVTE+uTpH6V\nfmeNiDiJavpCkgbNrFLKqr0VeAEfSUpkCEtSIkNYkhIZwpKUyBCWpESGsCQlMoQlKZEhLEmJDGFJ\nSmQIS1Ki9LstS8O1//71P8ZLlizp6L3nzp3baTu17Lfffo28r/qPI2FJSmQIS1IiQ1iSEhnCkpTI\nEJakRIawJCUyhCUpkSEsSYkMYUlKZAhLUiJPW1bfu+KKK2rXnnvuuR29d/bdyDX4HAlLUiJDWJIS\nGcKSlMgQlqREhrAkJTKEJSmRISxJiQxhSUpkCEtSIkNYkhJ52rJGpWnTptWuvfLKK2vXNnka8po1\naxp7bw0uR8KSlMgQlqREhrAkJTKEJSmRISxJiQxhSUpkCEtSIkNYkhIZwpKUyBCWpESGsCQl6vm1\nIyJiAbBgyOLHSykn9Hpd6i+dXA/irrvuarCT+jq5HsTnP//5BjvRoGrqAj6PAHOAaD1/taH1SFJf\nayqEXy2lbGrovSVpYDQ1Jzw1Iv4UEU9GxA8i4uiG1iNJfa2JEF4BXAicDcwHjgV+ExHjG1iXJPW1\nnk9HlFKWtT19JCJWAn8APgTc2Ov1SVI/a/wQtVLK88AaYErT65KkftN4CEfEG4B3AOubXpck9Zue\nh3BEfCUi3h0Rx0TE6cASqkPUbun1uiSp3zVxiNpk4IfABGATcB9waillcwPrkqS+1sQPcxf0+j0l\naVB5y3uNmDPOOKN27eTJk2vXvu519WfVfv/739euBTjnnHNq165f788e6pwX8JGkRIawJCUyhCUp\nkSEsSYkMYUlKZAhLUiJDWJISGcKSlMgQlqREhrAkJfK0ZQ3LWWedVbt20aJFtWtLKbVrd+zYUbt2\n4cKFtWvBU5HVPEfCkpTIEJakRIawJCUyhCUpkSEsSYkMYUlKZAhLUiJDWJISGcKSlMgQlqREnras\nYbnoootq1x522GGN9LB8+fLatXfffXcjPUjdciQsSYkMYUlKZAhLUiJDWJISGcKSlMgQlqREhrAk\nJTKEJSmRISxJiQxhSUrkacvaxRvf+MbatVOnTm2wk3rmzZtXu3br1q0NdiJ1zpGwJCUyhCUpkSEs\nSYkMYUlKZAhLUiJDWJISGcKSlMgQlqREhrAkJTKEJSmRpy1rF9dee23t2pkzZzbSw7Zt22rXeiqy\n+lnHI+GIODMifhYRf4qIHRFx3m5qroqIdRHxYkTcFRFTetOuJA2WbqYjxgMPARcDZeiLEXE58Bng\nU8DJwAvAsog4YBh9StJA6ng6opSyFFgKEBGxm5LPAgtLKT9v1XwU2AicD9zafauSNHh6+sNcRBwL\nTAJ+vXNZKWUL8ABwWi/XJUmDoNdHR0yimqLYOGT5xtZrkqQ2I3WIWrCb+WNJGut6HcIbqAJ34pDl\nR7Dr6FiSxryehnApZS1VEM/ZuSwiDgFOAX7by3VJ0iDo+OiIiBgPTKEa8QIcFxEzgedKKX8Ergeu\niIgngKeAhcAzwE970rEkDZBuzph7J3A31RxvAXaeXvU94OOllC9HxOuBbwGHAv8FnFtKeaUH/UrS\nQIlScn8vi4iTgAdTm9BrbN++vXZtU5+fyy+/vHZtJ6dZSyNsVill1d4KvICPJCUyhCUpkSEsSYkM\nYUlKZAhLUiJDWJISGcKSlMgQlqREhrAkJTKEJSmRd1vWiHnmmWdq1950003NNdJnJkyYULt2/vz5\ntWsnTap/n4VLLrmkdq0640hYkhIZwpKUyBCWpESGsCQlMoQlKZEhLEmJDGFJSmQIS1IiQ1iSEhnC\nkpTI05bHiE9+8pPZLbBixYratZs3b26wk2bMmjWrdu2SJUtq144bN6527eGHH167thNz5sypXXv+\n+efXrl2zZk037QwUR8KSlMgQlqREhrAkJTKEJSmRISxJiQxhSUpkCEtSIkNYkhIZwpKUyBCWpESG\nsCQl8toRY8SVV15ZuzYiGulh27ZtjbxvJzq5fTzANddcU7v2E5/4RO3aTv4fl1Jq1zbl+OOPr107\nY8aM2rVeO8KRsCSlMoQlKZEhLEmJDGFJSmQIS1IiQ1iSEhnCkpTIEJakRIawJCUyhCUpUcenLUfE\nmcC/ALOAtwLnl1J+1vb6jcA/D/ljS0spc4fTqHZ16qmn1q5985vfXLu2k9NkO6m94YYbatd2Yvr0\n6bVrb7/99o7e+6ijjqpd29TpxatXr65d28np0FOnTu2mHfVYNyPh8cBDwMXAnj51dwITgUmtxwVd\ndSdJA67jkXApZSmwFCD2/M/uX0spm4bTmCSNBU3NCc+OiI0R8XhELI6IwxpajyT1tSYuZXkn8BNg\nLfAO4Brgjog4rYyGa/JJ0ijS8xAupdza9vTRiHgYeBKYDdzd6/VJUj9r/BC1Uspa4FlgStPrkqR+\n03gIR8RkYAKwvul1SVK/6eY44fFUo9qdR0YcFxEzgedajwVUc8IbWnWLgDXAsl40LEmDpJs54XdS\nze2W1uPa1vLvARcBM4CPAocC66jC919LKX8bdreSNGC6OU74XvY+jXFO9+1I0tji3Zb72NFHH127\n9uCDD26wk96bNm1a7dply+rPdE2cOLGbdnpu+fLltWsXLVpUu/ayyy6rXetpy6ODF/CRpESGsCQl\nMoQlKZEhLEmJDGFJSmQIS1IiQ1iSEhnCkpTIEJakRIawJCXytOU+1smdg9evr38l0SOPPLKbdvbp\nxBNPrF17+umn166dNGlS7domb+7yne98p3ZtJ/tj4cKFtWtnzJhRu7YTnXzWOr2j9VjnSFiSEhnC\nkpTIEJakRIawJCUyhCUpkSEsSYkMYUlKZAhLUiJDWJISGcKSlCiaPI2zVgMRJwEPpjYxBjz22GO1\na48//vjatdmfH4CIqF07GvqF0dHztm3bate+6U1vaqSHMWBWKWXV3gocCUtSIkNYkhIZwpKUyBCW\npESGsCQlMoQlKZEhLEmJDGFJSmQIS1IiQ1iSEnm35THiqquuql178803164dLacB19Vv/UJnPW/e\nvLl27fvf//5u2lGPORKWpESGsCQlMoQlKZEhLEmJDGFJSmQIS1IiQ1iSEhnCkpTIEJakRIawJCXy\ntOUx4kc/+lHt2mOOOaZ27ac//enatZMnT65dO+huu+222rW333577dr777+/du0TTzxRu1bN6Wgk\nHBFfiIiVEbElIjZGxJKImDak5sCI+EZEPBsRWyPixxFxRG/blqTB0Ol0xJnA14BTgPcC44BfRsTB\nbTXXA/OADwDvBo4EfjL8ViVp8HQ0HVFKmdv+PCIuBP4MzALui4hDgI8D/1RKubdV8zHgfyLi5FLK\nyp50LUkDYrg/zB0KFOC51vNZVMH+650FpZTVwNPAacNclyQNnK5DOCKCaurhvlLKY63Fk4BXSilb\nhpRvbL0mSWoznKMjFgMnAO+qURtUI2ZJUpuuRsIR8XVgLjC7lLKu7aUNwAGtueF2R1CNhiVJbToO\n4VYAvw94Tynl6SEvPwi8Csxpq58GvA2ofwCjJI0RHU1HRMRi4ALgPOCFiJjYeun5UsrLpZQtEfFd\n4LqI+AuwFfgqsNwjIyRpV53OCc+nmtu9Z8jyjwHfb/33pcB24MfAgcBS4OLuW5SkwRXZd5+NiJOo\npjHUh6ZPn1679uqrr65dO2/evNq11YE69fziF7+oXQuwZs2a2rWrVq2qXXvLLbd01If61qxSyl4/\nGF7AR5ISGcKSlMgQlqREhrAkJTKEJSmRISxJiQxhSUpkCEtSIkNYkhIZwpKUyNOWJak5nrYsSaOZ\nISxJiQxhSUpkCEtSIkNYkhIZwpKUyBCWpESGsCQlMoQlKZEhLEmJDGFJSmQIS1IiQ1iSEhnCkpTI\nEJakRIawJCUyhCUpkSEsSYkMYUlKZAhLUiJDWJISGcKSlMgQlqREhrAkJTKEJSmRISxJiQxhSUpk\nCEtSIkNYkhIZwpKUyBCWpESGsCQlMoQlKVFHIRwRX4iIlRGxJSI2RsSSiJg2pOaeiNjR9tgeEYt7\n27YkDYZOR8JnAl8DTgHeC4wDfhkRB7fVFODbwERgEvBW4HPDb1WSBs/+nRSXUua2P4+IC4E/A7OA\n+9peerGUsmnY3UnSgBvunPChVCPf54Ys/0hEbIqIhyPiS0NGypKklo5Gwu0iIoDrgftKKY+1vXQz\n8AdgHTAD+DIwDfjgMPqUpIHUdQgDi4ETgDPaF5ZSbmh7+mhEbAB+FRHHllLWDmN9kjRwupqOiIiv\nA3OB2aWU9fsofwAIYEo365KkQdbxSLgVwO8DziqlPF3jj5xINW+8r7CWpDGnoxBuHe97AXAe8EJE\nTGy99Hwp5eWIOA74MHAHsBmYCVwH3FtKeaR3bUvSYOh0JDyfalR7z5DlHwO+D7xCdfzwZ4HxwB+B\n24Crh9WlJA2oTo8T3usccinlGWD2cBqSpLHEa0dIUiJDWJISGcKSlMgQlqREhrAkJTKEJSmRISxJ\niQxhSUpkCEtSIkNYkhIZwpKUyBCWpESGsCQlMoQlKZEhLEmJDGFJSmQIS1IiQ1iSEhnCkpTIEJak\nRIawJCUaDSF8UHYDktSQfebbaAjht2c3IEkNefu+CqKUMgJ97KWBiAnA2cBTwMupzUhSbxxEFcDL\nSimb91aYHsKSNJaNhukISRqzDGFJSmQIS1IiQ1iSEhnCkpTIEJakRIawJCX6P5niKG9Yvm76AAAA\nAElFTkSuQmCC\n", 344 | "text/plain": [ 345 | "" 346 | ] 347 | }, 348 | "metadata": {}, 349 | "output_type": "display_data" 350 | }, 351 | { 352 | "data": { 353 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWEAAAFdCAYAAADSR9wBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAFOxJREFUeJzt3XusXWWdxvHvjyIFilCo0MtwabG0WgxXh5tFEDTMgAEn\nGAY0Xv+RiBNjMqMxMQOjGY0aCaMOo0aH0XhJvIxREyi3SJmCUAKBAEpLS5FCaekFKZZWaHnnj7U7\nszmU9l3n7H1+5+zz/SQ7Ye/99Kx3sU6fvnvtdYlSCpKkHHtlD0CSJjJLWJISWcKSlMgSlqRElrAk\nJbKEJSmRJSxJiSxhSUpkCUtSIktYkhKNyRKOiCsiYlVEbI2IuyLir7PH1AsRcWVEvDzk8fvscQ1H\nRJwZEb+OiKc663HhLjKfj4g1EfFCRNwcEXMzxjoce1q/iLhuF9vy+qzx1oqIz0bE0ojYHBHrIuKX\nETFvSGZyRPx7RGyIiOcj4ucRcVjWmNuoXL/bhmy3HRFxbdaYx1wJR8TfA18DrgROBB4AboyIN6QO\nrHceAqYDMzqPhbnDGbYpwP3AFcCrLkASEZ8BPgF8DDgF2EKzHfcZzUGOwG7Xr+MGXrktLxudoY3I\nmcA3gFOBdwKvA26KiP26MtcAFwAXA28HZgG/GOVxDlfN+hXgO/z/tpsJfHqUx9k1mlLG1AO4C/i3\nrucBPAl8OntsPVi3K4H7ssfRh/V6GbhwyGtrgE91PT8Q2Apckj3eHq3fdcB/Z4+tB+v2hs76Leza\nTn8B/q4rM7+TOSV7vCNdv85rvwWuzh7bzseYmglHxOuAk4Fbd75Wmv9rtwCnZ42rx47pfMRdGRE/\njIgjsgfUaxExh2aG0b0dNwN3MzjbEeDszkfeRyLi2og4JHtAwzCVZma4qfP8ZGBvXrntlgFPMD63\n3dD12+n9EbE+Ih6MiC8OmSmPqr2zFvwa3gBMAtYNeX0dzb/G491dwIeBZTQfga4Cbo+It5RStiSO\nq9dm0Pzi72o7zhj94fTFDTQf0VcBbwS+BFwfEad3Jg5jXkQEza6HJaWUnd9NzABe7Pyj2W3cbbvX\nWD+AHwF/pPm0dhzwFWAe8N5RHyRjr4RfS/Da++XGjVLKjV1PH4qIpTS/DJfQfLwddAOxHQFKKT/t\nevpwRDwIrATOpvm4Ox5cCyyg7nuJ8bjtdq7f27pfLKV8t+vpwxGxFrglIuaUUlaN5gBh7H0xtwHY\nQbPDvNthvHpWNe6VUp4DlgPj5qiBSmtp/tJOiO0I0PnLu4Fxsi0j4pvA+cDZpZQ1XW+tBfaJiAOH\n/JFxte2GrN/Te4jfTfP7mrLtxlQJl1JeAu4Fzt35WucjxbnAnVnj6peIOIDmo+yefknGlU4hreWV\n2/FAmm+sB247AkTE4cA0xsG27BTURcA7SilPDHn7XmA7r9x284Ajgd+N2iBHYA/rtysn0szyU7bd\nWNwdcTXw/Yi4F1gKfArYH/ivzEH1QkR8FfgNzS6IvwL+heYX/ieZ4xqOiJhCM3OIzktHR8TxwKZS\nymqafXGfi4gVwOPAF2iOcvlVwnBb2936dR5X0uwTXtvJfZnmU82Nr/5pY0fneNjLgAuBLRGx89PK\nc6WUbaWUzRHxPeDqiHgWeB74OnBHKWVpzqjr7Wn9IuJo4H3A9cBG4HiazllcSnkoY8zph2e8xmEl\nH6f5i7uV5l/ft2aPqUfr9ROaItpK823zj4E52eMa5rqcRXPoz44hj//sylxF8+XHCzTlNDd73L1Y\nP2BfYBFNAW8DHgP+Azg0e9wV67WrddoBfLArM5nmWNsNNCX8M+Cw7LH3Yv2Aw4HbgPWd38tlNF+q\nHpA15ugMTJKUYEztE5akicYSlqRElrAkJbKEJSmRJSxJiSxhSUqUfrJGREwDzqM5Lnhb7mgkqSf2\nBWYDN5ZSNu4u2LcSjogrgH+kufLSA8A/lFLu2UX0PJqrGknSoHk/zUlZr6kvuyNa3h3j8X6MQZLG\ngMf3FOjXPuFPAd8upfyglPIIcDnNKYIf3UXWXRCSBtUe+63nJTxB7o4hST3Rj5nw7u6OMa6uzC9J\n/Taah6iNxyvzS1Jf9aOEJ9TdMSRpJHpewmWC3R1DkkaiX8cJD+zdMSSpl/pSwqWUn3aOCf48zW6J\n+4HzSinr+7E8SRqv0u+sEREn0ey+kKRBc3Ip5b7dBbyAjyQlsoQlKZElLEmJLGFJSmQJS1IiS1iS\nElnCkpTIEpakRJawJCWyhCUpUfrdlqWRestb3lKdnTRpUquf/cADD7QdTpUpU6ZUZ7ds2dKXMWhs\ncCYsSYksYUlKZAlLUiJLWJISWcKSlMgSlqRElrAkJbKEJSmRJSxJiSxhSUrkacsak/bff//q7KxZ\ns6qzxx57bKtxfOhDH6rOPvroo9XZNqdD33nnndVZjT/OhCUpkSUsSYksYUlKZAlLUiJLWJISWcKS\nlMgSlqRElrAkJbKEJSmRJSxJiTxtWWPSoYceWp2dP39+dfaYY45pNY699qqfp2zfvr06+8wzz7Qa\nhwaXM2FJSmQJS1IiS1iSElnCkpTIEpakRJawJCWyhCUpkSUsSYksYUlKZAlLUiJLWJIS9fzaERFx\nJXDlkJcfKaUs6PWyNLgWLlxYnT3nnHOqs6WUVuNYtmxZdfYPf/hDdXbFihWtxqHB1a8L+DwEnAtE\n53n9lU0kaQLpVwlvL6Ws79PPlqSB0a99wsdExFMRsTIifhgRR/RpOZI0rvWjhO8CPgycB1wOzAFu\nj4gpfViWJI1rPd8dUUq5sevpQxGxFPgjcAlwXa+XJ0njWd8PUSulPAcsB+b2e1mSNN70vYQj4gDg\njcDT/V6WJI03PS/hiPhqRLw9Io6KiDOAX9IcovaTXi9Lksa7fhyidjjwY2AasB5YApxWStnYh2VJ\n0rjWjy/mLuv1z5SkQeUt7zVqTjjhhOrsxRdfXJ2dOXNmdfbBBx+szgIsXbq0OrtkyZJWP1sCL+Aj\nSaksYUlKZAlLUiJLWJISWcKSlMgSlqRElrAkJbKEJSmRJSxJiSxhSUrkacsaNZdeeml1dv78+dXZ\nP/3pT9XZtWvXVmcBbr311lZ5qS1nwpKUyBKWpESWsCQlsoQlKZElLEmJLGFJSmQJS1IiS1iSElnC\nkpTIEpakRJ62rBFZuHBhdfa8886rzs6aNas6+9hjj1Vnb7nlluoswObNm1vlpbacCUtSIktYkhJZ\nwpKUyBKWpESWsCQlsoQlKZElLEmJLGFJSmQJS1IiS1iSEnnaskbk3HPPrc4uWLCgOrtp06bq7JIl\nS6qzixcvrs5Ko8GZsCQlsoQlKZElLEmJLGFJSmQJS1IiS1iSElnCkpTIEpakRJawJCWyhCUpkact\n61XmzZtXnb3ooouqs/vss0919q677qrO3nDDDdVZaaxpPROOiDMj4tcR8VREvBwRF+4i8/mIWBMR\nL0TEzRExtzfDlaTBMpzdEVOA+4ErgDL0zYj4DPAJ4GPAKcAW4MaIqJ8GSdIE0Xp3RCllEbAIICJi\nF5FPAl8opfymk/kgsA54D/DT4Q9VkgZPT7+Yi4g5wAzg1p2vlVI2A3cDp/dyWZI0CHp9dMQMml0U\n64a8vq7zniSpy2gdohbsYv+xJE10vS7htTSFO33I64fx6tmxJE14PS3hUsoqmiL+v3veRMSBwKnA\nnb1cliQNgtZHR0TEFGAuzYwX4OiIOB7YVEpZDVwDfC4iVgCPA18AngR+1ZMRS9IAGc4Zc28Ffkuz\nj7cAX+u8/n3go6WUr0TE/sC3ganA/wB/W0p5sQfjlaSBMpzjhBezh90YpZSrgKuGNyRl+8AHPlCd\nnT9/fnV2xYoV1dlFixZVZx9++OHq7KCbOXNmdXbDhg3V2Zdeemk4w1EFL+AjSYksYUlKZAlLUiJL\nWJISWcKSlMgSlqRElrAkJbKEJSmRJSxJiSxhSUrk3ZYniBkz6q+p/653vas6u99++1Vn25y2vHjx\n4ursjh07qrNtTZo0qTp7xBFHVGdPOeWU6uyb3vSm6uxee9XPq1atWlWdvfnmm6uza9asqc7KmbAk\npbKEJSmRJSxJiSxhSUpkCUtSIktYkhJZwpKUyBKWpESWsCQlsoQlKZGnLU8Ql1xySXX2yCOPrM5u\n3ry5Onv77bdXZ1evXl2dbaPNqcUAZ511VnV2wYIF1dlZs2ZVZw844IDq7NSpU6uzRx11VHW2zenp\n3/rWt6qzciYsSaksYUlKZAlLUiJLWJISWcKSlMgSlqRElrAkJbKEJSmRJSxJiSxhSUpkCUtSIq8d\nMUGccMIJ1dlp06ZVZ5ctW1adffTRR6uzbZx00knV2Xe/+92tfvbs2bOrswcddFB19oUXXqjO7tix\nozpbSqnOnnbaadXZZ599tjrb5roYa9asqc4OKmfCkpTIEpakRJawJCWyhCUpkSUsSYksYUlKZAlL\nUiJLWJISWcKSlMgSlqRErU9bjogzgX8CTgZmAu8ppfy66/3rgA8N+WOLSinnj2SgerU2t1ifPn16\ndXbSpEnV2VWrVlVn25z6esYZZ1RnzznnnOrscccdV50F2L59e3V2/fr11dmVK1dWZ1966aXq7Jvf\n/Obq7Jw5c6qzbU5lb/NzPW15eDPhKcD9wBXAa52ofgMwHZjReVw2rNFJ0oBrPRMupSwCFgFERLxG\n7C+llPppgSRNUP3aJ3x2RKyLiEci4tqIOKRPy5Gkca0fl7K8AfgFsAp4I/Al4PqIOL20uc6eJE0A\nPS/hUspPu54+HBEPAiuBs4Hf9np5kjSe9f0QtVLKKmADMLffy5Kk8abvJRwRhwPTgKf7vSxJGm+G\nc5zwFJpZ7c4jI46OiOOBTZ3HlTT7hNd2cl8GlgM39mLAkjRIhrNP+K00+3ZL5/G1zuvfBz4OHAd8\nEJgKrKEp338updQfcS5JE8RwjhNezO53Y/zN8IcjSROLd1sexw45pP7w6zbZDRs2VGc3bdpUnT3s\nsMOqs0cddVR19vDDD6/OPvPMM9VZgMcff7w6u3z58urs888/X51tc8rw8ccfX5198cUXq7Nt7vi8\ncePG6qy8gI8kpbKEJSmRJSxJiSxhSUpkCUtSIktYkhJZwpKUyBKWpESWsCQlsoQlKZGnLY9jbU47\n3bp1a3W2zSm1be5G3Ob04qOPPro62+ZuxE888UR1FtqdivznP/+5OtvmNOCDDz64Ojt79uzq7JNP\nPlmdXbZsWXX2qaeeqs7KmbAkpbKEJSmRJSxJiSxhSUpkCUtSIktYkhJZwpKUyBKWpESWsCQlsoQl\nKZGnLY9jK1eurM62uWvw61//+upsmzsoT548uTrb5rTel19+uS9ZgKlTp1ZnDzrooOrszJkzq7PH\nHntsdbbNaeSrVq2qzt5xxx3V2TanvcuZsCSlsoQlKZElLEmJLGFJSmQJS1IiS1iSElnCkpTIEpak\nRJawJCWyhCUpkactj2MbN26szt5zzz3V2TZ3RZ4xY0Z1ts0pzm3uDr333vW/xvvvv391FuCII46o\nzk6bNq06e+CBB1Zn29xN+pFHHqnO3nTTTX35uWrHmbAkJbKEJSmRJSxJiSxhSUpkCUtSIktYkhJZ\nwpKUyBKWpESWsCQlsoQlKZGnLU8QN998c3V2ypQp1dkLLrigOnvwwQf3ZQxt7sx8zDHHVGeh3SnD\nGzZsqM4+/fTT1dn77ruvOrt06dLq7L333lud3bZtW3VW7bSaCUfEZyNiaURsjoh1EfHLiJg3JDM5\nIv49IjZExPMR8fOIqL9ogCRNIG13R5wJfAM4FXgn8DrgpojYrytzDXABcDHwdmAW8IuRD1WSBk+r\n3RGllPO7n0fEh4FngJOBJRFxIPBR4NJSyuJO5iPAHyLilFJK/WclSZoARvrF3FSgAJs6z0+mKfZb\ndwZKKcuAJ4DTR7gsSRo4wy7hiAiaXQ9LSim/77w8A3ixlLJ5SHxd5z1JUpeRHB1xLbAAWFiRDZoZ\nsySpy7BmwhHxTeB84OxSypqut9YC+3T2DXc7jGY2LEnq0rqEOwV8EfCOUsoTQ96+F9gOnNuVnwcc\nCfxuBOOUpIHUandERFwLXAZcCGyJiOmdt54rpWwrpWyOiO8BV0fEs8DzwNeBOzwyQpJere0+4ctp\n9u3eNuT1jwA/6Pz3p4AdwM+BycAi4IrhD1GSBleUkvt9WUScRLMbQ2PE3Llzq7MLF9Z8L9s48cQT\nq7OzZs2qzra5g/Jzzz1XnYV2pyIvX768L9kVK1ZUZx977LHqrEbFyaWU3Z537gV8JCmRJSxJiSxh\nSUpkCUtSIktYkhJZwpKUyBKWpESWsCQlsoQlKZElLEmJPG1ZY9L8+fOrs5MnT67Otr1r8OrVq6uz\nW7dubfWzNSF42rIkjWWWsCQlsoQlKZElLEmJLGFJSmQJS1IiS1iSElnCkpTIEpakRJawJCVqe8t7\naVQsW7YsewjSqHAmLEmJLGFJSmQJS1IiS1iSElnCkpTIEpakRJawJCWyhCUpkSUsSYksYUlKZAlL\nUiJLWJISWcKSlMgSlqRElrAkJbKEJSmRJSxJiSxhSUpkCUtSIktYkhJZwpKUyBKWpEStSjgiPhsR\nSyNic0Ssi4hfRsS8IZnbIuLlrseOiLi2t8OWpMHQdiZ8JvAN4FTgncDrgJsiYr+uTAG+A0wHZgAz\ngU+PfKiSNHj2bhMupZzf/TwiPgw8A5wMLOl664VSyvoRj06SBtxI9wlPpZn5bhry+vsjYn1EPBgR\nXxwyU5YkdbSaCXeLiACuAZaUUn7f9daPgD8Ca4DjgK8A84D3jmCckjSQhl3CwLXAAuBt3S+WUr7b\n9fThiFgL3BIRc0opq0awPEkaOMPaHRER3wTOB84upTy9h/jdQABzh7MsSRpkrWfCnQK+CDirlPJE\nxR85kWa/8Z7KWpImnFYl3Dne9zLgQmBLREzvvPVcKWVbRBwNvA+4HtgIHA9cDSwupTzUu2FL0mBo\nOxO+nGZWe9uQ1z8C/AB4keb44U8CU4DVwM+Afx3RKCVpQLU9Tni3+5BLKU8CZ49kQJI0kXjtCElK\nZAlLUiJLWJISWcKSlMgSlqRElrAkJbKEJSmRJSxJiSxhSUpkCUtSIktYkhJZwpKUyBKWpESWsCQl\nsoQlKZElLEmJLGFJSmQJS1IiS1iSElnCkpTIEpakRGOhhPfNHoAk9cke+20slPDs7AFIUp/M3lMg\nSimjMI7dDCBiGnAe8DiwLXUwktQb+9IU8I2llI27C6aXsCRNZGNhd4QkTViWsCQlsoQlKZElLEmJ\nLGFJSmQJS1IiS1iSEv0vKLCeEK3A/nEAAAAASUVORK5CYII=\n", 354 | "text/plain": [ 355 | "" 356 | ] 357 | }, 358 | "metadata": {}, 359 | "output_type": "display_data" 360 | }, 361 | { 362 | "data": { 363 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWEAAAFdCAYAAADSR9wBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAEfVJREFUeJzt3XvMZHV9x/H3pwVB12wQKKwWLbBIE1JFXa4VZM2a0GKC\nVhEKGov+I3FtDEm7xmgK1VQjZpF62UYjUI2XBLEETID1EsEiwjYQlEXALIVFhL1wCUsWEIFf/ziz\ndXjYy8zzzOz3eeZ5v5JJmDPfPed7OLOf/c1vzjmT1hqSpBp/Ut2AJM1nhrAkFTKEJamQISxJhQxh\nSSpkCEtSIUNYkgoZwpJUyBCWpEKGsCQVmpUhnGR5knuTPJXkpiRHV/c0CknOS/L8lMevq/uajiQn\nJrkqye96+3Hqdmo+leTBJE8m+VGSwyp6nY5d7V+SS7dzLK+u6ndQST6eZE2SLUk2JrkiyeFTavZK\n8pUkDyd5IsnlSQ6o6nkYA+7fdVOO23NJVlX1POtCOMkZwErgPOCNwC+B1Un2L21sdNYCBwKLeo8T\natuZtgXAbcBy4EU3IEnyMeAjwIeAY4CtdMfxJbuzyRnY6f71XMMLj+WZu6e1GTkR+BJwLPA2YE/g\nh0le2ldzEfB24N3AW4BXAd/fzX1O1yD714Cv8cdj90pgxW7us6+b1mbVA7gJ+Pe+5wEeAFZU9zaC\nfTsPuLW6jzHs1/PAqVOWPQic2/d8IfAUcHp1vyPav0uB/6rubQT7tn9v/07oO06/B/6ur+YvezXH\nVPc70/3rLfspcGF1b9ses2oknGRPYAnwk23LWvd/7cfA8VV9jdhrex9x70nyrSSvrm5o1JIcQjfC\n6D+OW4CbmZzjCLC095H3riSrkuxb3dA07EM3Mny093wJsAcvPHZ3A/czN4/d1P3b5r1JNie5Pcln\npoyUd6s9qja8A/sDfwpsnLJ8I92/xnPdTcDZwN10H4HOB36W5K9aa1sL+xq1RXRv/O0dx0W7v52x\nuIbuI/q9wGLgs8DVSY7vDRxmvSShm3q4obW27buJRcAzvX80+825Y7eD/QP4NrCe7tPa64ELgMOB\n03Z7k8y+EN6RsON5uTmjtba67+naJGvo3gyn0328nXQTcRwBWmuX9T29I8ntwD3AUrqPu3PBKuAI\nBvteYi4eu2379+b+ha21r/c9vSPJBuDHSQ5prd27OxuE2ffF3MPAc3QT5v0O4MWjqjmvtfY48Btg\nzpw1MKANdH9p58VxBOj95X2YOXIsk3wZOAVY2lp7sO+lDcBLkiyc8kfm1LGbsn8P7aL8Zrr3a8mx\nm1Uh3Fr7A3ALsGzbst5HimXAjVV9jUuSl9N9lN3Vm2RO6QXSBl54HBfSfWM9cccRIMlBwH7MgWPZ\nC6h3AG9trd0/5eVbgGd54bE7HHgN8Ivd1uQM7GL/tueNdKP8kmM3G6cjLgS+keQWYA1wLvAy4D8r\nmxqFJJ8HfkA3BfHnwL/SveG/W9nXdCRZQDdySG/RoUmOBB5trf2Wbi7uk0nWAfcBn6Y7y+XKgnaH\ntrP96z3Oo5sT3tCr+xzdp5rVL17b7NE7H/ZM4FRga5Jtn1Yeb6093VrbkuRi4MIkjwFPAF8Eft5a\nW1PT9eB2tX9JDgXOAq4GHgGOpMuc61trayt6Lj89YwenlXyY7i/uU3T/+h5V3dOI9uu7dEH0FN23\nzd8BDqnua5r7chLdqT/PTXlc0ldzPt2XH0/ShdNh1X2PYv+AvYFr6QL4aeB/gf8A/qy67wH2a3v7\n9Bzw/r6avejOtX2YLoS/BxxQ3fso9g84CLgO2Nx7X95N96Xqy6t6Tq8xSVKBWTUnLEnzjSEsSYUM\nYUkqZAhLUiFDWJIKGcKSVKj8Yo0k+wEn050X/HRtN5I0EnsDBwOrW2uP7KxwbCGcZDnwT3R3Xvol\n8I+ttf/ZTunJdHc1kqRJ8166i7J2aCzTEUP+OsZ94+hBkmaB+3ZVMK454XOBr7bWvtlauws4h+4S\nwQ9up9YpCEmTapf5NvIQnie/jiFJIzGOkfDOfh1jTt2ZX5LGbXeeojYX78wvSWM1jhCeV7+OIUkz\nMfIQbvPs1zEkaSbGdZ7wxP46hiSN0lhCuLV2We+c4E/RTUvcBpzcWts8ju1J0lxV/ssaSd5EN30h\nSZNmSWvt1p0VeAMfSSpkCEtSIUNYkgoZwpJUyBCWpEKGsCQVMoQlqZAhLEmFDGFJKmQIS1IhQ1iS\nChnCklTIEJakQoawJBUyhCWpkCEsSYUMYUkqZAhLUiFDWJIKGcKSVMgQlqRChrAkFTKEJamQISxJ\nhQxhSSpkCEtSIUNYkgoZwpJUyBCWpEKGsCQVMoQlqZAhLEmF9qhuQJrNjjjiiIFrv/CFLwxcu3z5\n8oFr161bN3Ct5h5HwpJUyBCWpEKGsCQVMoQlqZAhLEmFDGFJKmQIS1IhQ1iSChnCklTIEJakQoaw\nJBUa+b0jkpwHnDdl8V2ttcEvwpdmiRUrVgxce9JJJw1c+653vWvg2gsuuGDgWs0947qBz1pgGZDe\n82fHtB1JmtPGFcLPttY2j2ndkjQxxjUn/Nokv0tyT5JvJXn1mLYjSXPaOEL4JuBs4GTgHOAQ4GdJ\nFoxhW5I0p418OqK1trrv6doka4D1wOnApaPeniTNZWM/Ra219jjwG+CwcW9LkuaasYdwkpcDi4GH\nxr0tSZprRh7CST6f5C1J/iLJXwNX0J2i9t1Rb0uS5rpxnKJ2EPAdYD9gM3ADcFxr7ZExbEuS5rRx\nfDF35qjXKUmTyp+8l3Zi8eLFA9euX79+4NqVK1dOpx1NIG/gI0mFDGFJKmQIS1IhQ1iSChnCklTI\nEJakQoawJBUyhCWpkCEsSYUMYUkq5GXL0k5ccsklA9defPHFA9ceffTRA9fedNNNA9dq7nEkLEmF\nDGFJKmQIS1IhQ1iSChnCklTIEJakQoawJBUyhCWpkCEsSYUMYUkq5GXL0k4cd9xxA9du2rRp4Nph\nfplZk82RsCQVMoQlqZAhLEmFDGFJKmQIS1IhQ1iSChnCklTIEJakQoawJBUyhCWpkJctSzuxcOHC\ngWtvvPHGgWsfeuih6bSjCeRIWJIKGcKSVMgQlqRChrAkFTKEJamQISxJhQxhSSpkCEtSIUNYkgoZ\nwpJUyMuWpZ0444wzBq79xCc+McZONKmGHgknOTHJVUl+l+T5JKdup+ZTSR5M8mSSHyU5bDTtStJk\nmc50xALgNmA50Ka+mORjwEeADwHHAFuB1UleMoM+JWkiDT0d0Vq7FrgWIEm2U/JR4NOttR/0at4P\nbATeCVw2/VYlafKM9Iu5JIcAi4CfbFvWWtsC3AwcP8ptSdIkGPXZEYvopig2Tlm+sfeaJKnP7jpF\nLWxn/liS5rtRh/AGusA9cMryA3jx6FiS5r2RhnBr7V66IF62bVmShcCxwOC//SJJ88TQZ0ckWQAc\nRjfiBTg0yZHAo6213wIXAZ9Msg64D/g08ABw5Ug6lqQJMp0r5o4Cfko3x9uAlb3l3wA+2Fq7IMnL\ngK8C+wD/Dfxta+2ZEfQrSRNlOucJX88upjFaa+cD50+vJWl83ve+91W3IL2AN/CRpEKGsCQVMoQl\nqZAhLEmFDGFJKmQIS1IhQ1iSChnCklTIEJakQoawJBXy15Y1r2zatGls637qqafGtm5NLkfCklTI\nEJakQoawJBUyhCWpkCEsSYUMYUkqZAhLUiFDWJIKGcKSVMgQlqRCXraseWXFihVD1ScZuPbyyy8f\nth3JkbAkVTKEJamQISxJhQxhSSpkCEtSIUNYkgoZwpJUyBCWpEKGsCQVMoQlqZAhLEmFvHeE5ry9\n99574Np99913qHXfeeedA9c+9thjQ61bAkfCklTKEJakQoawJBUyhCWpkCEsSYUMYUkqZAhLUiFD\nWJIKGcKSVMgQlqRCQ1+2nORE4J+BJcArgXe21q7qe/1S4B+m/LFrW2unzKRRaUcWL148cO0b3vCG\noda9cuXKgWu3bt061LolmN5IeAFwG7AcaDuouQY4EFjUe5w5re4kacINPRJurV0LXAuQJDso+31r\nbfNMGpOk+WBcc8JLk2xMcleSVUmGu3WVJM0T47iV5TXA94F7gcXAZ4GrkxzfWtvR9IUkzUsjD+HW\n2mV9T+9IcjtwD7AU+OmotydJc9nYT1Frrd0LPAwcNu5tSdJcM/YQTnIQsB/w0Li3JUlzzXTOE15A\nN6rddmbEoUmOBB7tPc6jmxPe0Kv7HPAbYPUoGpakSTKdOeGj6OZ2W++x7Wz2bwAfBl4PvB/YB3iQ\nLnz/pbX2hxl3K0kTZjrnCV/Pzqcx/mb67UjS/OKvLWvOO+2008a27rVr145t3RJ4Ax9JKmUIS1Ih\nQ1iSChnCklTIEJakQoawJBUyhCWpkCEsSYUMYUkqZAhLUiEvW9acd9ZZZ41t3Q888MDY1i2BI2FJ\nKmUIS1IhQ1iSChnCklTIEJakQoawJBUyhCWpkCEsSYUMYUkqZAhLUiEvW9asdPDBBw9cu//++4+t\nj/Xr149t3RI4EpakUoawJBUyhCWpkCEsSYUMYUkqZAhLUiFDWJIKGcKSVMgQlqRChrAkFfKyZc1K\ny5YtG7j2Fa94xcC1V1111VB9rFu3bqh6aViOhCWpkCEsSYUMYUkqZAhLUiFDWJIKGcKSVMgQlqRC\nhrAkFTKEJamQISxJhbxsWbPSUUcdNZb13nbbbWNZrzRdQ42Ek3w8yZokW5JsTHJFksOn1OyV5CtJ\nHk7yRJLLkxww2rYlaTIMOx1xIvAl4FjgbcCewA+TvLSv5iLg7cC7gbcArwK+P/NWJWnyDDUd0Vo7\npf95krOBTcAS4IYkC4EPAn/fWru+V/MB4M4kx7TW1oyka0maEDP9Ym4foAGP9p4voQv2n2wraK3d\nDdwPHD/DbUnSxJl2CCcJ3dTDDa21X/cWLwKeaa1tmVK+sfeaJKnPTM6OWAUcAZwwQG3oRsySpD7T\nGgkn+TJwCrC0tfZg30sbgJf05ob7HUA3GpYk9Rk6hHsB/A7gra21+6e8fAvwLLCsr/5w4DXAL2bQ\npyRNpKGmI5KsAs4ETgW2Jjmw99LjrbWnW2tbklwMXJjkMeAJ4IvAzz0zQpJebNg54XPo5navm7L8\nA8A3e/99LvAccDmwF3AtsHz6LUrS5Eprtd+XJXkT3TSG9P+2bJl6gs2O7bnnngPXnnDCIN8j/9Et\nt/jW1Iwsaa3durMCb+AjSYUMYUkqZAhLUiFDWJIKGcKSVMgQlqRChrAkFTKEJamQISxJhQxhSSrk\nry1rVrryyisHrj399NMHrn3Pe94zVB9etqxxcyQsSYUMYUkqZAhLUiFDWJIKGcKSVMgQlqRChrAk\nFTKEJamQISxJhQxhSSrkZcualX71q18NXPu6171u4NrVq1dPpx1pbBwJS1IhQ1iSChnCklTIEJak\nQoawJBUyhCWpkCEsSYUMYUkqZAhLUiFDWJIKGcKSVCittdoGkjcB/q64pEm0pLV2684KHAlLUiFD\nWJIKGcKSVMgQlqRChrAkFTKEJamQISxJhQxhSSpkCEtSIUNYkgoNFcJJPp5kTZItSTYmuSLJ4VNq\nrkvyfN/juSSrRtu2JE2GYUfCJwJfAo4F3gbsCfwwyUv7ahrwNeBAYBHwSmDFzFuVpMmzxzDFrbVT\n+p8nORvYBCwBbuh76cnW2uYZdydJE26mc8L70I18H52y/L1JNie5PclnpoyUJUk9Q42E+yUJcBFw\nQ2vt130vfRtYDzwIvB64ADgcOG0GfUrSRJp2CAOrgCOAN/cvbK19ve/pHUk2AD9Ockhr7d4ZbE+S\nJs60piOSfBk4BVjaWntoF+U3AwEOm862JGmSDT0S7gXwO4CTWmv3D/BH3kg3b7yrsJakeWeoEO6d\n73smcCqwNcmBvZceb609neRQ4CzgauAR4EjgQuD61tra0bUtSZNh2JHwOXSj2uumLP8A8E3gGbrz\nhz8KLAB+C3wP+LcZdSlJE2rY84R3OofcWnsAWDqThiRpPvHeEZJUyBCWpEKGsCQVMoQlqZAhLEmF\nDGFJKmQIS1IhQ1iSChnCklTIEJakQoawJBUyhCWpkCEsSYUMYUkqZAhLUiFDWJIKGcKSVMgQlqRC\nhrAkFTKEJamQISxJhWZDCO9d3YAkjcku8202hPDB1Q1I0pgcvKuCtNZ2Qx87aSDZDzgZuA94urQZ\nSRqNvekCeHVr7ZGdFZaHsCTNZ7NhOkKS5i1DWJIKGcKSVMgQlqRChrAkFTKEJamQISxJhf4PdGGu\nmEgwypkAAAAASUVORK5CYII=\n", 364 | "text/plain": [ 365 | "" 366 | ] 367 | }, 368 | "metadata": {}, 369 | "output_type": "display_data" 370 | }, 371 | { 372 | "data": { 373 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWEAAAFdCAYAAADSR9wBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAEl5JREFUeJzt3XusnHWdx/H317UtWmkaCrSwaKHWrpKNBCu31WoNJuxi\nghKNu2jiov9IRGNIdjUmZstiVqNGwnrpRoPLarwkqEvUBFpvgIsK3WCqULlYaEEprb2lrVCu/e4f\nzzQ7HHqZZ87M+Z6Z834lkzDPfDvP9+E553N+85vnEpmJJKnGC6obkKSZzBCWpEKGsCQVMoQlqZAh\nLEmFDGFJKmQIS1IhQ1iSChnCklTIEJakQtMyhCPi8ojYFBH7I+L2iDiruqdBiIhVEXFgwuN31X31\nIyJWRMQPIuKRznZcdIiaqyJiS0Q8HhE/joilFb3242jbFxHXHWJf3ljVb68i4mMRsS4i9kbEtoi4\nISKWTaiZExFfiogdEbEvIr4bESdW9dxGj9t3y4T99mxErK7qedqFcET8PfA5YBVwJvAbYG1EHF/a\n2ODcDSwEFnUer69tp29zgfXA5cDzLkASER8FPgi8HzgbeIxmP86eyiYn4Yjb13ETz92Xl0xNa5Oy\nAvgCcA7wZmAW8KOIeFFXzTXAW4C3A28ATga+N8V99quX7UvgK/z/vjsJ+MgU99nVTea0egC3A//e\n9TyAPwIfqe5tANu2Cvh1dR9D2K4DwEUTlm0Bruh6Pg/YD7yzut8Bbd91wH9X9zaAbTu+s32v79pP\nTwIXd9X8Vafm7Op+J7t9nWU3A1dX93bwMa1GwhExC1gO/PTgsmz+r/0EOK+qrwF7Recj7gMR8Y2I\neGl1Q4MWEafRjDC69+Ne4A7GZz8CrOx85L03IlZHxHHVDfVhPs3IcFfn+XLghTx3390HPMxo7ruJ\n23fQuyNie0TcFRGfnDBSnlIvrFrxYRwP/AWwbcLybTR/jUfd7cClwH00H4GuBH4eEX+dmY8V9jVo\ni2h+8A+1HxdNfTtDcRPNR/RNwMuBTwE3RsR5nYHDtBcRQTP1cFtmHvxuYhHwVOePZreR23eH2T6A\nbwIP0XxaezXwGWAZ8I4pb5LpF8KHExx+Xm5kZObarqd3R8Q6mh+Gd9J8vB13Y7EfATLz+q6nGyLi\nLuABYCXNx91RsBo4nd6+lxjFfXdw+17XvTAzr+16uiEitgI/iYjTMnPTVDYI0++LuR3AszQT5t1O\n5PmjqpGXmXuA+4GROWqgR1tpfmlnxH4E6Pzy7mBE9mVEfBG4EFiZmVu6XtoKzI6IeRP+yUjtuwnb\n9+hRyu+g+Xkt2XfTKoQz82ngTuD8g8s6HynOB35Z1dewRMRLaD7KHu2HZKR0Amkrz92P82i+sR67\n/QgQEacACxiBfdkJqLcCb8rMhye8fCfwDM/dd8uAlwG/mrImJ+Eo23coZ9KM8kv23XScjrga+FpE\n3AmsA64AXgz8V2VTgxARnwV+SDMF8ZfAv9L8wH+7sq9+RMRcmpFDdBYtiYgzgF2Z+QeaubiPR8RG\nYDPwCZqjXL5f0G5rR9q+zmMVzZzw1k7dp2k+1ax9/rtNH53jYS8BLgIei4iDn1b2ZOYTmbk3Ir4K\nXB0Ru4F9wOeBX2Tmupque3e07YuIJcC7gBuBncAZNJlza2beXdFz+eEZhzms5AM0v7j7af76vra6\npwFt17dpgmg/zbfN3wJOq+6rz215I82hP89OePxnV82VNF9+PE4TTkur+x7E9gHHAGtoAvgJ4EHg\nP4ATqvvuYbsOtU3PAu/pqplDc6ztDpoQ/g5wYnXvg9g+4BTgFmB75+fyPpovVV9S1XN0GpMkFZhW\nc8KSNNMYwpJUyBCWpEKGsCQVMoQlqZAhLEmFyk/WiIgFwAU0xwU/UduNJA3EMcCpwNrM3HmkwqGF\ncERcDvwTzZWXfgN8KDP/9xClF9Bc1UiSxs27aU7KOqyhTEe0vDvG5mH0IEnTwOajFQxrTvgK4MuZ\n+fXMvBe4jOYUwfcdotYpCEnj6qj5NvAQniF3x5CkgRjGSPhId8cYqSvzS9KwTeUhaqN4ZX5JGqph\nhPCMujuGJE3GwEM4Z9jdMSRpMoZ1nPDY3h1DkgZpKCGcmdd3jgm+imZaYj1wQWZuH8b6JGlUld9Z\nIyJeQzN9IUnjZnlm/vpIBV7AR5IKGcKSVMgQlqRChrAkFTKEJamQISxJhQxhSSpkCEtSIUNYkgoZ\nwpJUyBCWpEKGsCQVMoQlqZAhLEmFDGFJKmQIS1IhQ1iSChnCklTIEJakQoawJBUyhCWpkCEsSYUM\nYUkqZAhLUiFDWJIKGcKSVMgQlqRChrAkFTKEJamQISxJhQxhSSpkCEtSoRdWNyCNi4svvrjn2p/9\n7Gc91+7Zs6efdjQiHAlLUiFDWJIKGcKSVMgQlqRChrAkFTKEJamQISxJhQxhSSpkCEtSIUNYkgoZ\nwpJUaODXjoiIVcCqCYvvzczTB70uadiuuuqqnmtXrFjRc+2CBQt6rr322mt7rtXoGdYFfO4Gzgei\n8/yZIa1HkkbasEL4mczcPqT3lqSxMaw54VdExCMR8UBEfCMiXjqk9UjSSBtGCN8OXApcAFwGnAb8\nPCLmDmFdkjTSBj4dkZlru57eHRHrgIeAdwLXDXp9kjTKhn6IWmbuAe4Hlg57XZI0aoYewhHxEuDl\nwKPDXpckjZqBh3BEfDYi3hARiyPib4AbaA5R+/ag1yVJo24Yh6idAnwLWABsB24Dzs3MnUNYlySN\ntGF8MXfJoN9TksaVt7zXjLJ48eJW9WeddVbPtSeffHLPtc8840mkangBH0kqZAhLUiFDWJIKGcKS\nVMgQlqRChrAkFTKEJamQISxJhQxhSSpkCEtSIU9b1oyyZMmSVvXz5s3ruXb37t091+7bt69VHxpf\njoQlqZAhLEmFDGFJKmQIS1IhQ1iSChnCklTIEJakQoawJBUyhCWpkCEsSYU8bVkzyvHHH9+qfs6c\nOT3X7t+/v+faRx55pFUfGl+OhCWpkCEsSYUMYUkqZAhLUiFDWJIKGcKSVMgQlqRChrAkFTKEJamQ\nISxJhTxtWTPKwoULW9XPnj2759o///nPQ6nVeHMkLEmFDGFJKmQIS1IhQ1iSChnCklTIEJakQoaw\nJBUyhCWpkCEsSYUMYUkq5GnLmjIR0XNtZg6lh2OPPbZVfZu7M2/evLnn2i1btrTqQ+Or9Ug4IlZE\nxA8i4pGIOBARFx2i5qqI2BIRj0fEjyNi6WDalaTx0s90xFxgPXA58LzhSkR8FPgg8H7gbOAxYG1E\n9H4lFEmaIVpPR2TmGmANQBz68+WHgU9k5g87Ne8BtgFvA67vv1VJGj8D/WIuIk4DFgE/PbgsM/cC\ndwDnDXJdkjQOBn10xCKaKYptE5Zv67wmSeoyVYeoBYeYP5akmW7QIbyVJnAn3r7gRJ4/OpakGW+g\nIZyZm2iC+PyDyyJiHnAO8MtBrkuSxkHroyMiYi6wlGbEC7AkIs4AdmXmH4BrgI9HxEZgM/AJ4I/A\n9wfSsSSNkX7OmHstcDPNHG8Cn+ss/xrwvsz8TES8GPgyMB/4H+DvMvOpAfQrSWOln+OEb+Uo0xiZ\neSVwZX8taVzNmjWr59qnnhrO3+zFixe3qp87d27Ptdu29f61x65du1r1ofHlBXwkqZAhLEmFDGFJ\nKmQIS1IhQ1iSChnCklTIEJakQoawJBUyhCWpkCEsSYW827KmzLBORW5zF+fjjjuu1Xu/4AW9j1P2\n7NnT6r0lcCQsSaUMYUkqZAhLUiFDWJIKGcKSVMgQlqRChrAkFTKEJamQISxJhQxhSSrkacualDZ3\nUH766aeH0kObOyi/6lWvavXes2fP7rl2w4YNrd5bAkfCklTKEJakQoawJBUyhCWpkCEsSYUMYUkq\nZAhLUiFDWJIKGcKSVMgQlqRChrAkFfLaEZqUYV0Poo1XvvKVPdeeeuqprd579+7dPdc++OCDrd5b\nAkfCklTKEJakQoawJBUyhCWpkCEsSYUMYUkqZAhLUiFDWJIKGcKSVMgQlqRCrU9bjogVwD8Dy4GT\ngLdl5g+6Xr8O+McJ/2xNZl44mUalwzn33HN7rp0zZ06r925zKvJDDz3U6r0l6G8kPBdYD1wO5GFq\nbgIWAos6j0v66k6SxlzrkXBmrgHWAEREHKbsyczcPpnGJGkmGNac8MqI2BYR90bE6og4bkjrkaSR\nNoxLWd4EfA/YBLwc+BRwY0Scl5mHm76QpBlp4CGcmdd3Pd0QEXcBDwArgZsHvT5JGmVDP0QtMzcB\nO4Clw16XJI2aoYdwRJwCLAAeHfa6JGnU9HOc8FyaUe3BIyOWRMQZwK7OYxXNnPDWTt2ngfuBtYNo\nWJLGST9zwq+lmdvNzuNzneVfAz4AvBp4DzAf2EITvv+SmfU3I5Okaaaf44Rv5cjTGH/bfzuSNLN4\nt2WNvBNOOKHn2lmzZrV6740bN/Zcu3nz5lbvLYEX8JGkUoawJBUyhCWpkCEsSYUMYUkqZAhLUiFD\nWJIKGcKSVMgQlqRChrAkFfK0ZY28NqctHzhwoNV733///W3bkVpxJCxJhQxhSSpkCEtSIUNYkgoZ\nwpJUyBCWpEKGsCQVMoQlqZAhLEmFDGFJKuRpyxp5J510Us+1Tz75ZKv3/u1vf9u2HakVR8KSVMgQ\nlqRChrAkFTKEJamQISxJhQxhSSpkCEtSIUNYkgoZwpJUyBCWpEKetqxpaenSpT3XLliwoOfa7du3\nt+rj97//fat6qS1HwpJUyBCWpEKGsCQVMoQlqZAhLEmFDGFJKmQIS1IhQ1iSChnCklTIEJakQp62\nrGnpzDPP7Ll29uzZPdfu3LmzVR87duxoVS+11WokHBEfi4h1EbE3IrZFxA0RsWxCzZyI+FJE7IiI\nfRHx3Yg4cbBtS9J4aDsdsQL4AnAO8GZgFvCjiHhRV801wFuAtwNvAE4Gvjf5ViVp/LSajsjMC7uf\nR8SlwJ+A5cBtETEPeB/wD5l5a6fmvcA9EXF2Zq4bSNeSNCYm+8XcfCCBXZ3ny2mC/acHCzLzPuBh\n4LxJrkuSxk7fIRwRQTP1cFtm/q6zeBHwVGbunVC+rfOaJKnLZI6OWA2cDry+h9qgGTFLkrr0NRKO\niC8CFwIrM3NL10tbgdmdueFuJ9KMhiVJXVqHcCeA3wq8KTMfnvDyncAzwPld9cuAlwG/mkSfkjSW\nWk1HRMRq4BLgIuCxiFjYeWlPZj6RmXsj4qvA1RGxG9gHfB74hUdGSNLztZ0TvoxmbveWCcvfC3y9\n899XAM8C3wXmAGuAy/tvUZLGV9vjhI86fZGZTwIf6jykvrS5g/L8+fN7rm179+TNmze3qpfa8gI+\nklTIEJakQoawJBUyhCWpkCEsSYUMYUkqZAhLUiFDWJIKGcKSVMgQlqRC3m1Z09Kxxx7bc21zf4He\nbN26tZ92pKFxJCxJhQxhSSpkCEtSIUNYkgoZwpJUyBCWpEKGsCQVMoQlqZAhLEmFDGFJKuRpy5qW\n5syZ03Pt+vXre67duHFjP+1IQ+NIWJIKGcKSVMgQlqRChrAkFTKEJamQISxJhQxhSSpkCEtSIUNY\nkgoZwpJUyBCWpEJeO0LT0s0339xz7c6dO3uuveeee/ppRxoaR8KSVMgQlqRChrAkFTKEJamQISxJ\nhQxhSSpkCEtSIUNYkgoZwpJUyBCWpEqZ2fMD+BiwDtgLbANuAJZNqLkFOND1eBZYfYT3fA2QPnz4\n8DGGj9ccLVfbjoRXAF8AzgHeDMwCfhQRL+qqSeArwEJgEXAS8JGW65GkGaHVBXwy88Lu5xFxKfAn\nYDlwW9dLj2fm9kl3J0ljbrJzwvNpRr67Jix/d0Rsj4i7IuKTE0bKkqSOvi9lGREBXAPclpm/63rp\nm8BDwBbg1cBngGXAOybRpySNpclcT3g1cDrwuu6FmXlt19MNEbEV+ElEnJaZmyaxPkkaO31NR0TE\nF4ELgZWZ+ehRyu8AAljaz7okaZy1Hgl3AvitwBsz8+Ee/smZNPPGRwtrSZpxWoVwRKwGLgEuAh6L\niIWdl/Zk5hMRsQR4F3AjsBM4A7gauDUz7x5c25I0HtqOhC+jGdXeMmH5e4GvA0/RHD/8YWAu8Afg\nO8C/TapLSRpTbY8TPuIccmb+EVg5mYYkaSbx2hGSVMgQlqRChrAkFTKEJamQISxJhQxhSSpkCEtS\nIUNYkgoZwpJUyBCWpEKGsCQVMoQlqZAhLEmFDGFJKmQIS1IhQ1iSChnCklTIEJakQoawJBUyhCWp\nkCEsSYWmQwgfU92AJA3JUfNtOoTwqdUNSNKQnHq0gsjMKejjCA1ELAAuADYDT5Q2I0mDcQxNAK/N\nzJ1HKiwPYUmayabDdIQkzViGsCQVMoQlqZAhLEmFDGFJKmQIS1IhQ1iSCv0fIz9StSBbayEAAAAA\nSUVORK5CYII=\n", 374 | "text/plain": [ 375 | "" 376 | ] 377 | }, 378 | "metadata": {}, 379 | "output_type": "display_data" 380 | } 381 | ], 382 | "source": [ 383 | "import numpy as np\n", 384 | "import matplotlib.pyplot as plt\n", 385 | "%matplotlib inline\n", 386 | "trainimg = mnist.train.images\n", 387 | "trainlabel = mnist.train.labels\n", 388 | "output_nd = output_layer.eval(feed_dict = {x: mnist.train.images})\n", 389 | "for i in [0, 1, 2, 3, 4]:\n", 390 | " curr_img = np.reshape(trainimg[i, :], (28, 28)) # 28 by 28 matrix \n", 391 | " ae_img = np.reshape(output_nd[i,:], (28, 28))\n", 392 | " curr_label = np.argmax(trainlabel[i, :] ) # Label\n", 393 | " plt.matshow(curr_img, cmap=plt.get_cmap('gray'))\n", 394 | " plt.matshow(ae_img, cmap=plt.get_cmap('gray'))" 395 | ] 396 | }, 397 | { 398 | "cell_type": "markdown", 399 | "metadata": {}, 400 | "source": [ 401 | "而接下來當我想嘗試把 code layer 的維度調整為 2 維來看看\n", 402 | "## 今日心得\n", 403 | "而接下來當我想嘗試把 code layer 的維度調整為 2 維來看看\n", 404 | "\n", 405 | "![](http://imgur.com/1n9Ljnb.jpg)\n", 406 | "\n", 407 | "![](http://imgur.com/cmzYYOS.jpg)\n", 408 | "\n", 409 | "![](http://imgur.com/ICUGyPC.jpg)\n", 410 | "\n", 411 | "![](http://imgur.com/VpnycII.jpg)\n", 412 | "\n", 413 | "我們可以發現完全糊掉了，只用二維來表示輸入數字是不太夠的．更進一步仔細看 code layer 的值，都已經接近零，表示這個網路已經算是死掉了．" 414 | ] 415 | }, 416 | { 417 | "cell_type": "code", 418 | "execution_count": 10, 419 | "metadata": { 420 | "collapsed": false 421 | }, 422 | "outputs": [], 423 | "source": [ 424 | "def plot_scatter(x, labels, title, txt = False):\n", 425 | " plt.title(title)\n", 426 | " ax = plt.subplot()\n", 427 | " ax.scatter(x[:,0], x[:,1], c = labels)\n", 428 | " txts = []\n", 429 | " if txt:\n", 430 | " for i in range(10):\n", 431 | " xtext, ytext = np.median(x[labels == i, :], axis=0)\n", 432 | " txt = ax.text(xtext, ytext, str(i), fontsize=24)\n", 433 | " txt.set_path_effects([\n", 434 | " PathEffects.Stroke(linewidth=5, foreground=\"w\"),\n", 435 | " PathEffects.Normal()])\n", 436 | " txts.append(txt)\n", 437 | " plt.show()" 438 | ] 439 | } 440 | ], 441 | "metadata": { 442 | "kernelspec": { 443 | "display_name": "Python 2", 444 | "language": "python", 445 | "name": "python2" 446 | }, 447 | "language_info": { 448 | "codemirror_mode": { 449 | "name": "ipython", 450 | "version": 2 451 | }, 452 | "file_extension": ".py", 453 | "mimetype": "text/x-python", 454 | "name": "python", 455 | "nbconvert_exporter": "python", 456 | "pygments_lexer": "ipython2", 457 | "version": "2.7.6" 458 | } 459 | }, 460 | "nbformat": 4, 461 | "nbformat_minor": 2 462 | } 463 | -------------------------------------------------------------------------------- /notebooks/images/10_Reccurent_Neural_Network/gradient.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/c1mone/Tensorflow-101/90d1990f923d045bddd2a09886a7fd47d2490f22/notebooks/images/10_Reccurent_Neural_Network/gradient.jpg -------------------------------------------------------------------------------- /notebooks/images/10_Reccurent_Neural_Network/rnn_cell.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/c1mone/Tensorflow-101/90d1990f923d045bddd2a09886a7fd47d2490f22/notebooks/images/10_Reccurent_Neural_Network/rnn_cell.jpg -------------------------------------------------------------------------------- /notebooks/images/10_Reccurent_Neural_Network/rnn_expand.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/c1mone/Tensorflow-101/90d1990f923d045bddd2a09886a7fd47d2490f22/notebooks/images/10_Reccurent_Neural_Network/rnn_expand.jpg -------------------------------------------------------------------------------- /notebooks/images/11_Char_RNN/lstm_inner1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/c1mone/Tensorflow-101/90d1990f923d045bddd2a09886a7fd47d2490f22/notebooks/images/11_Char_RNN/lstm_inner1.jpg -------------------------------------------------------------------------------- /notebooks/images/11_Char_RNN/lstm_inner2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/c1mone/Tensorflow-101/90d1990f923d045bddd2a09886a7fd47d2490f22/notebooks/images/11_Char_RNN/lstm_inner2.jpg -------------------------------------------------------------------------------- /notebooks/images/11_Char_RNN/lstm_inner3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/c1mone/Tensorflow-101/90d1990f923d045bddd2a09886a7fd47d2490f22/notebooks/images/11_Char_RNN/lstm_inner3.jpg -------------------------------------------------------------------------------- /notebooks/images/11_Char_RNN/lstm_inner4.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/c1mone/Tensorflow-101/90d1990f923d045bddd2a09886a7fd47d2490f22/notebooks/images/11_Char_RNN/lstm_inner4.jpg -------------------------------------------------------------------------------- /notebooks/images/12_word2vec/skip_gram_example.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/c1mone/Tensorflow-101/90d1990f923d045bddd2a09886a7fd47d2490f22/notebooks/images/12_word2vec/skip_gram_example.jpg -------------------------------------------------------------------------------- /notebooks/images/12_word2vec/word2vec_example.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/c1mone/Tensorflow-101/90d1990f923d045bddd2a09886a7fd47d2490f22/notebooks/images/12_word2vec/word2vec_example.gif -------------------------------------------------------------------------------- /notebooks/images/13_Generative_Adversarial_Network/GAN_algorithm.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/c1mone/Tensorflow-101/90d1990f923d045bddd2a09886a7fd47d2490f22/notebooks/images/13_Generative_Adversarial_Network/GAN_algorithm.jpg -------------------------------------------------------------------------------- /notebooks/images/13_Generative_Adversarial_Network/GAN_result.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/c1mone/Tensorflow-101/90d1990f923d045bddd2a09886a7fd47d2490f22/notebooks/images/13_Generative_Adversarial_Network/GAN_result.gif -------------------------------------------------------------------------------- /notebooks/images/14_DCGAN/dcgan.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/c1mone/Tensorflow-101/90d1990f923d045bddd2a09886a7fd47d2490f22/notebooks/images/14_DCGAN/dcgan.gif -------------------------------------------------------------------------------- /notebooks/images/14_DCGAN/dcgan.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/c1mone/Tensorflow-101/90d1990f923d045bddd2a09886a7fd47d2490f22/notebooks/images/14_DCGAN/dcgan.jpg -------------------------------------------------------------------------------- /notebooks/libs/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/c1mone/Tensorflow-101/90d1990f923d045bddd2a09886a7fd47d2490f22/notebooks/libs/__init__.py -------------------------------------------------------------------------------- /notebooks/libs/utils.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | from __future__ import division, print_function, absolute_import 3 | 4 | import tensorflow as tf 5 | 6 | def weight_variable(shape, name): 7 | initial = tf.truncated_normal(shape, stddev = 0.1) 8 | return tf.Variable(initial, name) 9 | 10 | def bias_variable(shape, name): 11 | initial = tf.constant(0.1, shape = shape) 12 | return tf.Variable(initial, name) --------------------------------------------------------------------------------