├── .gitignore ├── README.md ├── ale_game.py ├── analysis ├── .layer_view.py.swp ├── Analaze DQN.ipynb ├── fetch_from_s3.sh ├── layer_view.py ├── sync.sh └── translate_weights.py ├── analyze_layers.py ├── dqn.py ├── network.py ├── provision ├── aws_installation.sh ├── gen_run.py ├── mount.sh ├── mount_remote.sh ├── provision.py └── ssh.sh ├── run.py ├── run_dev.py ├── run_play.py ├── run_simple_breakout.py ├── runs ├── run_dqn42.py └── run_dqn43.py ├── simple_breakout.py ├── teacher.py ├── test ├── __init__.py └── test_loss.py ├── updates.py └── video ├── compile-video.sh └── video.webm /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | .DS_Store 3 | .idea 4 | *gz 5 | *npz 6 | *bin 7 | *out 8 | .ipynb_checkpoints 9 | rewards.txt 10 | instance.dns 11 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Deep Q-Learning 2 | 3 | ## Overview 4 | 5 | Our version of the deep q-learning algorithm from [The DQN 6 | paper](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf). This algorithm reads 7 | the screen and the integer score of the Atari 2600 game Space Invaders. The 8 | output is the same control commands as a human would have with a controller 9 | (albeit, without the physical controller). 10 | 11 | ## Installation Dependencies: 12 | * Python 2.7 13 | * Theano 14 | * Lasagne 15 | * pygame 16 | * [Arcade Learning Environment (ALE) 0.5.1](http://arcadelearningenvironment.org) 17 | * Atari 2600 ROM of space_invaders.bin 18 | 19 | ## Amazon Instance Installation 20 | 21 | Look at /provision/aws_installation.sh for a concise shell history to install 22 | the environment. 23 | 24 | ## External References 25 | 26 | [The DQN paper](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf) 27 | 28 | [Human-level control through deep reinforcement 29 | learning](https://storage.googleapis.com/deepmind-data/assets/papers/DeepMindNature14236Paper.pdf) 30 | 31 | [Deep Reinforcement Learning with Double Q-learning](https://arxiv.org/pdf/1509.06461.pdf) - more stable learning through double q-learning 32 | 33 | [Action-Conditional Video Prediction using Deep Networks in Atari Games](https://arxiv.org/pdf/1507.08750.pdf) - predicting future frames 34 | 35 | [Dueling Network Architectures for Deep Q-learning](http://arxiv.org/pdf/1511.06581.pdf) 36 | 37 | [Arcade Learning Environment](http://www.arcadelearningenvironment.org) 38 | 39 | [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](http://arxiv.org/pdf/1604.06057v1.pdf) 40 | 41 | 42 | [Reccurent Model of Visual Attention](http://papers.nips.cc/paper/5542-recurrent-models-of-visual-attention.pdf) - applying q-learning to figure out what part of the image to look at. 43 | 44 | [Prioritized Experience Replay](http://arxiv.org/abs/1511.05952) - drawing from the memory should be more likely if the memory is more shocking 45 | 46 | [Deep Recurrent Q-Learning For Partially Observable MDPs](http://arxiv.org/pdf/1507.06527.pdf) - by using LSTM you can get rid of preprocessing done in DQN paper. 47 | "The recurrent net can better adapt at evaluation time if the quality of observations changes" 48 | 49 | [A fast learning algorithm for deep belief nets](http://www.cs.toronto.edu/~hinton/absps/fastnc.pdf) - Training one layer at a time 50 | 51 | [Reinforcement Learning and Automated Planning: A Survey](http://lpis.csd.auth.gr/publications/rlplan.pdf) 52 | 53 | [Autoregressive Neural Networks](https://opus4.kobv.de/opus4-uni-passau/files/142/Dietz_Sebastian.pdf) - Neural Networks applied to Time Series. 54 | 55 | [Deep Autoregressive Neural Networks](https://www.cs.toronto.edu/~amnih/papers/darn.pdf) - predicting future frames of an Atari Game. 56 | 57 | [Reinforcement Learning: An introduction](https://webdocs.cs.ualberta.ca/~sutton/book/ebook/) - very thorough introduction to Reinforcement Learning. 58 | 59 | [A survey of robot learning by demonstration](http://www.cs.cmu.edu/~mmv/papers/09ras-survey.pdf) Learning by|from demonstration = Learning by watching = Learning from observation = Programming by demonstration = Behaviour cloning|imitation|mimicry 60 | 61 | [DynaQ](https://webdocs.cs.ualberta.ca/~sutton/book/ebook/node96.html) 62 | 63 | [Deep Reinforcement Learning](http://www.iclr.cc/lib/exe/fetch.php?media=iclr2015:silver-iclr2015.pdf) Nice summary of recent advances in Deep Q-learning. 64 | 65 | [Concurrent Q-learning for Autonomous Mapping and Navigation](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.155.616&rep=rep1&type=pdf) One-trial learning??? 66 | 67 | [Using Reinforcement Learning to Adapt an Imitation Task](http://lasa.epfl.ch/publications/uploadedFiles/IROS07.pdf) Overcoming new obstacles ??? 68 | 69 | [On the importance of initialization and momentum in deep learning](http://jmlr.org/proceedings/papers/v28/sutskever13.pdf) - Nesterov Momentum vs Nesterov Accelerated Gradient 70 | 71 | [CNN Features off-the-shelf: an Astounding Baseline for Recognition](http://arxiv.org/pdf/1403.6382v3.pdf) NN generated features are better then manually-made 72 | 73 | [Prioritized Experience Replay](http://arxiv.org/pdf/1511.05952v3.pdf) - on Atari games 74 | 75 | [Network in Network](http://arxiv.org/pdf/1312.4400v3.pdf) - MaxPooling looses information, let's keep some more information. 76 | 77 | [Concurrent Reinforcement Learning](http://www0.cs.ucl.ac.uk/staff/d.silver/web/Publications_files/concurrent-rl.pdf) - RL in time dependent environments 78 | -------------------------------------------------------------------------------- /ale_game.py: -------------------------------------------------------------------------------- 1 | from __future__ import division 2 | from ale_python_interface import ALEInterface 3 | import numpy as np 4 | import skimage.transform 5 | 6 | 7 | def init(game, display_screen=False, record_dir=None): 8 | if display_screen: 9 | import pygame 10 | pygame.init() 11 | ale = ALEInterface() 12 | ale.setBool('display_screen', display_screen) 13 | ale.setInt('random_seed', 123) 14 | if record_dir is not None: 15 | ale.setString("record_screen_dir", record_dir) 16 | ale.loadROM('{game}.bin'.format(game=game)) 17 | ale.setFloat("repeat_action_probability", 0) 18 | 19 | return ale 20 | 21 | 22 | class Phi(object): 23 | 24 | def __init__(self, method="crop_and_resize"): 25 | self.screen_size = (84, 84) 26 | self.method = method 27 | 28 | def __call__(self, frames): 29 | last = frames[3:16:4] 30 | sec_last = frames[2:16:4] 31 | return [self.resize_and_crop(img) for img in np.max([last, sec_last], axis=0)] 32 | 33 | def resize_and_crop(self, im): 34 | # Resize so smallest dim = 256, preserving aspect ratio 35 | if self.method == "resize": 36 | return skimage.transform.resize(im, (84, 84), preserve_range=True).astype(dtype=np.uint8) 37 | else: 38 | im = im[40:-10, :] 39 | h, w = im.shape 40 | if h < w: 41 | im = skimage.transform.resize(im, (84, w*84//h), preserve_range=True) 42 | else: 43 | im = skimage.transform.resize(im, (h*84//w, 84), preserve_range=True) 44 | 45 | # Central crop to 224x224 46 | h, w = im.shape 47 | return im[h//2-42:h//2+42, w//2-42:w//2+42].astype(dtype=np.uint8) 48 | 49 | 50 | class Phi4(object): 51 | 52 | def __init__(self, method): 53 | self.screen_size = (84, 84) 54 | self.method = method 55 | 56 | def __call__(self, frames): 57 | last = frames[12:] 58 | sec_last = frames[11:-1] 59 | return [self.resize_and_crop(img) for img in np.max([last, sec_last], axis=0)] 60 | 61 | def resize_and_crop(self, im): 62 | # Resize so smallest dim = 256, preserving aspect ratio 63 | if self.method == "resize": 64 | return skimage.transform.resize(im, (84, 84), preserve_range=True).astype(dtype=np.uint8) 65 | else: 66 | im = im[40:-10, :] 67 | h, w = im.shape 68 | if h < w: 69 | im = skimage.transform.resize(im, (84, w*84//h), preserve_range=True) 70 | else: 71 | im = skimage.transform.resize(im, (h*84//w, 84), preserve_range=True) 72 | 73 | # Central crop to 224x224 74 | h, w = im.shape 75 | return im[h//2-42:h//2+42, w//2-42:w//2+42].astype(dtype=np.uint8) 76 | 77 | 78 | class ALEGameVisualizer: 79 | def __init__(self, screen_size): 80 | import pygame 81 | self.screen_size = screen_size 82 | self.screen = pygame.display.set_mode((screen_size * 2, screen_size * 8)) 83 | self.mem = {} 84 | 85 | def show(self, prev_frames): 86 | import pygame 87 | 88 | def l(x): 89 | if x not in self.mem: 90 | self.mem[x] = (x, x, x) 91 | return self.mem[x] 92 | 93 | f_l = np.frompyfunc(l, 1, 3) 94 | rect = pygame.Surface((self.screen_size * 2, self.screen_size * 8)) 95 | image = np.reshape(zip(*list(f_l(np.concatenate(prev_frames).flatten()))), (self.screen_size * 4, self.screen_size, 3)) 96 | 97 | image = np.transpose(image, [1, 0, 2]) 98 | pygame.surfarray.blit_array(rect, np.repeat(np.repeat(image, 2, axis=0), 2, axis=1)) 99 | self.screen.blit(rect, (0, 0)) 100 | 101 | pygame.display.flip() 102 | 103 | def next_game(self): 104 | pass 105 | 106 | 107 | class ALEGame(object): 108 | 109 | def __init__(self, ale): 110 | self.ale = ale 111 | self.finished = True 112 | self.cum_reward = 0 113 | self.action_set = self.ale.getMinimalActionSet() 114 | 115 | self.h = 210 116 | self.w = 160 117 | 118 | self.prev_frames = [np.zeros((self.h, self.w), dtype=np.uint8), 119 | np.zeros((self.h, self.w), dtype=np.uint8), 120 | np.zeros((self.h, self.w), dtype=np.uint8), 121 | np.zeros((self.h, self.w), dtype=np.uint8), 122 | 123 | np.zeros((self.h, self.w), dtype=np.uint8), 124 | np.zeros((self.h, self.w), dtype=np.uint8), 125 | np.zeros((self.h, self.w), dtype=np.uint8), 126 | np.zeros((self.h, self.w), dtype=np.uint8), 127 | 128 | np.zeros((self.h, self.w), dtype=np.uint8), 129 | np.zeros((self.h, self.w), dtype=np.uint8), 130 | np.zeros((self.h, self.w), dtype=np.uint8), 131 | np.zeros((self.h, self.w), dtype=np.uint8), 132 | 133 | np.zeros((self.h, self.w), dtype=np.uint8), 134 | np.zeros((self.h, self.w), dtype=np.uint8), 135 | np.zeros((self.h, self.w), dtype=np.uint8), 136 | np.zeros((self.h, self.w), dtype=np.uint8), ] 137 | 138 | def reset_game(self): 139 | self.ale.reset_game() 140 | self.finished = False 141 | self.cum_reward = 0 142 | 143 | def n_actions(self): 144 | return len(self.action_set) 145 | 146 | def input(self, action): 147 | lives_before = self.ale.lives() 148 | action_reward = self.ale.act(self.action_set[action]) 149 | self.cum_reward += action_reward 150 | 151 | if self.ale.game_over(): 152 | self.finished = True 153 | 154 | self.prev_frames.append(np.dot(self.ale.getScreenRGB(), np.array([0.2126, 0.7152, 0.0722])).astype(np.int8)) 155 | self.prev_frames = self.prev_frames[1:] 156 | 157 | return action_reward, lives_before != self.ale.lives() 158 | 159 | def get_state(self): 160 | return self.prev_frames 161 | -------------------------------------------------------------------------------- /analysis/.layer_view.py.swp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/maciejjaskowski/deep-q-learning/88363786881830b8e410230d6ad72d964673eb20/analysis/.layer_view.py.swp -------------------------------------------------------------------------------- /analysis/Analaze DQN.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 112, 6 | "metadata": { 7 | "collapsed": false, 8 | "scrolled": true 9 | }, 10 | "outputs": [ 11 | { 12 | "name": "stdout", 13 | "output_type": "stream", 14 | "text": [ 15 | "Downloaded!\n" 16 | ] 17 | } 18 | ], 19 | "source": [ 20 | "%matplotlib inline\n", 21 | "import pandas as pd\n", 22 | "from subprocess import call\n", 23 | "import os\n", 24 | "\n", 25 | "client_id = \"dqn13\"\n", 26 | "\n", 27 | "logs_dir = \"../\" + client_id + \"/logs\"\n", 28 | "config_dir = \"../\" + client_id + \"/config\"\n", 29 | "ret = call([\"aws\", \"s3\", \"sync\", \"s3://\" + client_id + \"/logs\", logs_dir])\n", 30 | "ret = call([\"aws\", \"s3\", \"sync\", \"s3://\" + client_id + \"/config\", config_dir])\n", 31 | "\n", 32 | "\n", 33 | "if ret != 0:\n", 34 | " print(\"Problem \", ret)\n", 35 | "else:\n", 36 | " print(\"Downloaded!\")" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 94, 42 | "metadata": { 43 | "collapsed": false, 44 | "scrolled": true 45 | }, 46 | "outputs": [ 47 | { 48 | "name": "stdout", 49 | "output_type": "stream", 50 | "text": [ 51 | "794200\n" 52 | ] 53 | } 54 | ], 55 | "source": [ 56 | "import os\n", 57 | "import pandas as pd\n", 58 | "paths = [(logs_dir + \"/\" + file_name, file_name) for file_name in sorted(os.listdir(logs_dir)) if file_name.endswith(\".s\")]\n", 59 | "\n", 60 | "\n", 61 | "log=None\n", 62 | "log = pd.Series()\n", 63 | "try:\n", 64 | " for path, filename in paths:\n", 65 | " if filename <= '@4000000056d31eff151cea34.s':\n", 66 | " log = pd.concat([log, pd.read_csv(path, sep='|', header=None, engine='python')])\n", 67 | " else:\n", 68 | " log = pd.concat([log, pd.read_csv(path, sep='|', header=None, engine='python', compression='gzip')])\n", 69 | "except:\n", 70 | " print(filename)\n", 71 | "print(len(log))\n", 72 | "log = log.iloc[:,0]" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": 110, 78 | "metadata": { 79 | "collapsed": false 80 | }, 81 | "outputs": [], 82 | "source": [ 83 | "\n", 84 | "rewards = log[log.str.contains(\"Game reward: \")].str.extract('Game reward: (\\d*)').astype(float)\n", 85 | "rewards.index = range(0,len(rewards.index))\n", 86 | "loss = log[log.str.contains(\"loss:\")].str.extract('loss: (\\d+\\.\\d+)').astype(float)\n", 87 | "loss.index = range(0, len(loss.index))\n", 88 | "epsilon = log[log.str.contains('epsilon:')].str.extract('epsilon: (\\d+\\.\\d+)').astype(float)\n", 89 | "epsilon.index = range(0, len(epsilon.index))\n", 90 | "weights = log[log.str.contains(\"saved\")].str.extract('File saved: weights/weights_(\\d+).npz').astype(float)\n", 91 | "weights.index = range(0, len(weights.index))" 92 | ] 93 | }, 94 | { 95 | "cell_type": "code", 96 | "execution_count": 96, 97 | "metadata": { 98 | "collapsed": false 99 | }, 100 | "outputs": [ 101 | { 102 | "data": { 103 | "text/plain": [ 104 | "" 105 | ] 106 | }, 107 | "execution_count": 96, 108 | "metadata": {}, 109 | "output_type": "execute_result" 110 | }, 111 | { 112 | "data": { 113 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEACAYAAACj0I2EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYVNWZx/Hvi2wKgriBAQUNLuC4xAWMTmK7xGh00Gii\nhmiMRuO4RTGKEHUgSiROgokZxX0SdTRoHKNoIri2RhN3CIyAooIsCsgqi2zd7/xx6npvVVd1V3VX\nd3VX/T7P08+999zt3Orut0+fexZzd0REpO1rV+oMiIhIcSigi4iUCQV0EZEyoYAuIlImFNBFRMqE\nArqISJnIO6CbWTsze9vMJqa2e5jZ02b2rplNNrPuiWNHmtlsM5tpZsc0R8ZFRCRdISX0S4EZie0R\nwLPuvifwPDASwMwGAqcCA4DjgPFmZsXJroiI5JJXQDezPsC3gLsTyScC96bW7wVOSq0PASa4+2Z3\nnwvMBgYVJbciIpJTviX03wBXAslupT3dfTGAuy8Cdkyl9wbmJ45bmEoTEZFm1GBAN7PjgcXuPhWo\nr+pEYwiIiJRQ+zyOOQwYYmbfArYEtjaz+4FFZtbT3RebWS9gSer4hcDOifP7pNLSmJn+AIiINIK7\nZy1cN1hCd/efufsu7r4bcDrwvLufCTwB/DB12FnA46n1icDpZtbRzHYF+gOv57h2xX2NGjWq5HnQ\nM+uZW/qZwz/wzqZNpc9nW/8+16cp7dB/CXzDzN4Fjkpt4+4zgIcJLWL+ClzoDeVCRCpChw7w3//d\n+POHDwe1mcutoIDu7i+6+5DU+nJ3P9rd93T3Y9x9ZeK4se7e390HuPvTxc60iLQtAwbE6z/6UeOv\n86tfNT0v5Uw9RVtYVVVVqbPQ4vTMlaG+Z26f8bZu3rzG3SO6zutZK3FbXmv7PlupakPMTDUxImXu\n889hxQro3RsGDoQZia6Js2dD//75X2v5cthuu7B+111w7rnFzWtbYWZ4Y1+Kiog0Vq9eIZhDejAH\nuO8+ePHF/K/14x/H6+ed1/S8lSOV0EWk6Gpq6laz/PjHcOeddY/98EPo16/hl52Z+++4Iz3IVwqV\n0EWkRT32WN20G26A55+vm77bbnD11flfe+zYsDz//MblrZwpoItIg8wKay74ne+kb19+eaj/PuKI\n7Md/8kn911u+PCz79IFLL43TL7ww/zxVAgV0Ecnb/PkNHwNwxRXp2z17xuuffhq+Nm6M0w48sO41\n1q+H6dPD+nvvheX778OWW8bH3HZbfvmpFAroIlKvFSvi9YYC6ObNsGAB/PrXYTsqTX/5y/Ex228f\nvjp0iNMuuQTefDP9WqedBvvuC3/5C3zwAey/P3TqFPbp9Vt2CugiUq85c+L1sWPhySdzH3vkkbBz\naiSnESNgzBjo3BlOOSX78a+9Fq//9Kfxem0tTJwY1k84Ac44I1wn6Z13Cmv2WAkU0EWkXmvXwhZb\nxNs/+UnuY//2t3j9pJOga9fQFj2XQYmZEv7t3+L1xx+ve2xmQO/SBTZsyH3tSqSALlJBopebs2aF\n5VtvNXzOZ5/BMcfARx+F7TPPjPetXg1XXZX9vC99Kb88zZsXOh1FzRw3bgzXzVRdnb7dpUuo06+p\nye8+lUABXaQCHXtsWL70UsPHvvsudO8Ou+wCv/hF+svMyy6D//zP7OftvHP29GzHnXwyrFoVtjt1\ngt/+tuHzunQJy5tvzu8+lUABXaQCRaXt7bdPT//1r9ObJ86aFeq2H3oobK9aBZMnx/sXLAjL6CVl\nz56hbjtKL8QLL8TNF6dMCcvZs3MfH1XBJOveK50CukgLmj276S00pk2DpUuLk597743XH30Urrwy\nrEd5XLkyfXvSpDjYAjydGkt1zpzwInP58vCisneBk06OHx+GAVi0KD29f/9w7/33Dz1Kk8zC0ALR\nem1tYfcsRwroIi1ojz3g1Vcbd+6114YAu99+sMMOhZ8fBbzbb4/TnnsuXk+2RGnXLlS1RE0Wzzkn\nLH/3u/RrRb785fDitLYWOnYsPG/Dh9e9brJZ45QpsOuudc/be+94/bPPCr9vuVFAF2khmzeH5fr1\njTt/zJgwfkljRQHv6KPD8rTTQr14LvPnh3N22gluuimktUtFjLffDsvMF5+NfUF5xhnhPslSeLK5\nZC7J+vMePeCVVxp3/3KhgC7SQqL64ejlX2OMGBGvF/qHYcUK6Ns3lKbdw9gq8+bFpfCTTko//pJL\nQmuT444LL0Uhnqiipib8gfr008Y9R6bttoNly+KSunt+1TaZf5D+9V+Lk5+2SgFdpIVEwWrlyvqP\nyyZbybe+Dj7ZLF8O224bb0dVGNEfmqiqZOXKEBi/8pXQCiZ57+glam1teLG6aVNhecilc+fQbPGw\nw+D00/M/b+ut8x+OoDV68034+OPiXU8BXaSF7LRTWDamhL5uXd207343dIu/+eaw3pB580JHn0jU\nmiWa/aemJpSKt946lJD/+Ee4//70F6cQqmxWr4Y33gjb9bVEKcS228IDD8CaNYWd16dPce7f0iZM\ngIMPhuOPL941FdBFWkjUfvuyy+pO9tCQZJBLvrw84YRwvUceafgaJ5+c3pMzcu65ocT91FNhvPJ2\n7WDIkHj/tdemH9+tW6hb79IlBKP+/eMemx98kP8zZYr+e/jFLxp/jbbke98Ly6lTi3dNBXSRFnLr\nrfH6I4+ktzBpSDKgn39+9nHFM5v15aumJrRQWbcOliwJaVGrFoCvfjX9+O7dQ7XM6tWhNA+humb7\n7eP/QhojGllxt90KP3fePNh998bfu1wooIs0swceCNUXSaNGhaqLfMcYf+edsPzd78J5e+1V95h8\nOthEoyBGxo1L34466yQ7HGXOg9ytWwjmyYAO4QVpcmjbQkVt3ZPVQvnaeefwGWXOktQa1NTEzTHd\nw/e8piY0Py02BXSRZnbGGTB0aFjP1oMyW/14pmh2nksuCQFh223h619PP+aZZxq+zh57pG9ffHH6\n9re+VfeczCDtHiaseOqpuj1NS6l9+xA4o+ahrUX79vHEHtF/WitWhO9jsauXFNBFWlDv3qHpYFI+\nVSVRVUikU6fQs/Kaa+K0qB77zjvr9kaN6mn32Sc9vWPH9Hrvbt0azkvUuejxxxvXwSmXN9/MXsef\nL7Pwx6eh5pw1NcVrndOQqBPZSy+FfEWf75NPhqB+/PHhv5z27eNnf/758Cxr18Lo0YXdTwFdpAC1\ntbBwYdOukVlKf/HF+o+PAm5ykojIv/xLWF59dfgvYP36UJr/v/+Lj3EPrVUgdL7JlKvOevz47FU7\nyenlGtMEM5cDD2x6O/IuXbKP1Jj0gx+EP2QtMVRA8v3DccfF62efHZp97rJLyG9NTfwf17RpYfnm\nm/Dznxd2PwV0KWvLljW+Z2Y2F18cmslFL/A2bgxjoOQKDtn+/Z88GQ46KC5FZ1Z7ZHrhhbDM1hri\n1FNDi5MDDggBIpp7c99942OWLw89PQcOjDsIZVqyBJ54Ij3tggtg5sy6x/7pT/F6u1YWQXbYoeHO\nTg8+GJbJMd5vuCG901Yhrroq+/jtmTKH/4W634/f/AaGDQvr0fuVgsb+cfeSfIVbizSP9evd//d/\n3cOvQ/GuG12vb9+wfc45YXvYsLD9+uvuq1fHx995Z3xOtnwMH+4+dGh+96zP3XeHYy68MP1+w4fX\nf//G2nnncL0pU4p3zWI49FD3v/0t9/6//z375xFtb9hQ2P02bYrPXbs2vtbHH7t/+mn270nm/XPt\ni77WrUu/Zyp2Zo+ruXY095cCujSnK6+s+4uxeXPjrlVT475qlfvixenXmzvXfbfdwvrAge5PPhnW\nr7sunDd9enzs17/u3rVr3Wv/z//kDrYffxyW7ds3HIwXLUrP2wUX5A4gxfDKK8W9XrEcc4z7fffl\n3j95csNBtaYm//stXRqf16+f+5tvxttPPBGWtbX1B/R589yfey73MZ98kn7PJgV0oBPwGjAFmA6M\nSqWPAhYAb6e+jk2cMxKYDcwEjslx3fw/NZECbN6c/Rfj9tsbd71sfxzq+7r44nBeMm31avdly+pe\ne+zY7MF21aqQtnCh+3nnNZz3mpqG87Xffo17/rYk1x+uFSvcb7453n/MMfH6Xnulf06//W3+93vv\nvdyf9zPPxHm5/vqwftxx4b+AbPkcNy77dWbNynzG3AG9wRowd98AHOHuXwH2B44zs2gmwJvc/YDU\n16RQ72MDgFOBAcBxwHizfFvbijTNmjXpc1MmJet+C5E53O2YMfUff8stddO6dk0fRyUSDS6VbM8N\nYdxxCK1iXnkld913pF279KFksxk8uP795eCHP8yePnw4XHppvN2pU7w+a1b6sXffnf/96nv/MXdu\nPLLlNdeE8PzXv+YeXvjyy8MyuX///QsbKiKvVxruHrWU7QS0Bzy1nS1QnwhMcPfN7j6XUFIflOU4\nkaI788zQPjqbzHbb+Ur2frzuutDVPik5c30k2SzuwANzX/t73wuTLq9enf7yNvlLPWNGfmOMRy1b\nkq0pIu+9Fzc3LGdRe/2NG0MLk6hF0l13pR83YULdc087LSyTLYSmTw/jwufqBBRN8JHNeefBs89m\n37dyZTwEcVJtbfg5OOmkMLXfttsWOPZPrqJ78osQ+KcAnwFjPa5ymQNMBe4GuqfS/wsYmjj3buDk\nLNfM//8akTwdemj4N/Wii+J/Wb/zHfddd3W/7LLGXTPz32h39yuuCNunnJJ+TLIOFdz/+Ef3NWvq\nv35UN//229nvCe7vv194XpNftbWFP3dblFlfPWFCSI+2Bw50/8c/QtoHH9T9jKL16OVocv+CBen3\n+sY3QvpOO4V6cHDfe++65zXFEUe4n39++vePptShpx0M3YDngIHADoCl0scAd7sCupTY174WfqqP\nP9793HPjX6h77w3rM2eGlgn5mjMn/Zfz9dfjfcuXx+vRvdauLfyXORlI3EOrhsyAnC8In8HEie7V\n1e6XXuo+eHD+55eD5Od2003h5XK0/eqruY91D+86IHxumfuT34e//S1Ou+WWkJas6y5WQK+qCtf4\n5z+T184d0Asa+cDdPzOzasIL0JsSu+4ColasC4HkfN99Uml1jE50g6qqqqIqc9AIkTxF81xutVVY\nzpkDDz8M//EfYdtTlYQDBsBZZ8Ef/pD7WsuXhwkX3OG++0LaZ5/BkUemd+5JdtL5r/8K/8ZH9y9E\n8g3Thg31560hK1eG3ojRNQ8/vPHXKgeXXx7XTZ93Xnr7fAhjwX/wQRiHHeJxZG6+OXvfgoceCtUh\nX/tanBYN37vnnnHan/4UhjT+9reblv/Qdr2a88+vpmvXOJ855Yr00RewPXF1ypbAS8C3gF6JY4YB\nD6bWBxKqZzoCuwLvkyrJZ1y3aX+6RBKSJaIxY9yXLEnf//TT8f4jjqj/WlF1SnV140pZo0bF1S2F\n5n/sWPdJk8L6yy+H0vvnnxd2/0p3333Zq53ylavaCkJTxMsvz++669Y1vaor8/6hmWQTqlyAfQjN\nEqcC04CrU+n3pbanAo8BPRPnjEwFcjVblBaR/KGfObPhY+bPz94B5Zln4iZtUeeZQn9U16xx79mz\nsHOOPDK+1113uR91VGHnSyz6g9jYgL7DDvUH9ehrm23qFhyKbeXK9HtOnFh/QI/qwFucmXmp7i3l\nZ6+9QmsEiKtXMj38cNySIZJ5bLYGtldf3XBTxWLIvLd+PRrvrbdCC5OoO/9tt8G//3t+5y5blj6K\n5LhxoTt+5jAHtbX5D3/cFMl7jBsHP/2p4e5Z76yALm3ekiXQs2e8Xd+P1dNPhzr0RYvC9po1YUAn\nCMPYRutJLfVjOm4cXHFFy9+3XNXWhmamP/95aBZayDjrURDdsCFuMlqqP7gXX5w+OQrkDuitbGgd\nkcIlg3lDtt8+BPPol/vGG+N92QaiaknJCSqS+ZLGadcuTJ93222FT5qxYUPoX5Bs/x+1cYd4ELSW\ncMst+c/bqoAubV7UOGrw4HgigVyiUlU00cCyZfG+99+P1196qWjZa5TPPy/t/cvFFlvkX9WS1LEj\nDMroDnn99fF6Zs/e5ta/f36TYSugS4uYOzf8y9ocvRX794c77ghBePLk+o+NutpHxo8PyzVr4PTT\nw7p73CwtarbYUqIql969W/a+0rAddoCvfCWsl2Kqu/nzGz6mFc7AJ+Vmxox4jI0332z69Wprw4QN\nZ50VXlbefXeYUzKf7vHZZtiZOzf7H5pS1GGPGhXawJ91VsvfWxr21lulneJuzhzYddfc+/VSVIpi\n/fow/dfChfClL4WxQ6L5K5Mvk44+Osx9udtuYaAiCMfnM/VZ5JNPwjlPPRWPW/Lqq/kPPvXcc6GO\n+vHH63YEeuGFupMii7QmZmrlIs3sww9DL8oHH4QhQ8JLqOXL644w+OUvx1OqfetbcVB3D6Mk7rJL\n5hv9uqZPr9vjL5rOq1BqKihtTX0BXXXoUhTRqHZDh8YtCjKHqz3nnPQJiaNgDqGk/eSTcZ32nDlw\n4YXx/osugl/+MqxnTpgMhZXwc/njH5t+DZFSUgldimLwYHj99dz799wzvHTMZ9LbadPiEnj0IxKV\npL///TBT+tCh8fFvvBHm6GyMZAm9pqb1zZEpkkkldGl29QXzQYPCJALZ2lZnm+U9WZ3y6KPp+x54\nABYvTh+jPDlgVqHOOAN+9avwh0PBXNo6/QhLo40ZE1eD5PKDH8QTQGRrSdK3b/3nn3IKPPJIetqw\nYaFVS6Sh2Xzqc//96b0zRdoyVblIowwaFKo6IJRuTzwRzj47LJcsCS9JDz0UVqyAbbZJP9cM/vKX\nUMVx7LHQoUNIu+ii+IXoHnuEljJJ99wDP/pRWB8wAObNC80g99qreZ9VpDVRKxdplNra0Dvu5JNh\nn33S9yXrnt3hqKNg5Mh4DsXG+uyzMIfmscfWrQKpqQk9/yB001cgl0pUX0BXxyLJad994Z13Qok7\nKjn36pXeizGaa3HNmsLHy8imW7fsc2JCeoBvqKpGpBKpDl2yqq0NwRzipoSbN4cXksnJbbfbLizn\nzStOQE9atix0WLr++rqT7XbuXNx7iZQDBXTJat689O1HHw113Ul33RVKzUuXhhEMd9yxuHnYdlvo\n1AmuuSZU6SS1xDjUIm2Nqlwkq+iF56mnhhebp5xS95iBA8McltH4KMUO6NmsX9/89xBpq1RCl6yi\nMU7OPz8E9EhUrXL22WHslmIMtlWITp3Cl4jUpRK6ZLVsWeh0069fetDu3z8MUdutW1zHLiKtgwK6\nZLV0aXjhmeyRCfDd78ZVKwccEKf/4x8tlzcRyU4BXbJasQJ69AjVKpGamvSXkWZhUomDD1arE5HW\nQHXoktWaNfE0WzfcEJbt2tVtXfK1rymYi7QWCuiSVbKjUOYkECLSOqnKRbK6++64umXo0HhSZRFp\nvRTQJc1RR8UzzkczAO2wA1x9denyJCL50eBckiZZR/7556ofF2ltNMGF5CXz76uCuUjbooAuX1i7\nNl4fN650+RCRxlFAly8kJ3XOnJRCRFq/BgO6mXUys9fMbIqZTTezUan0Hmb2tJm9a2aTzax74pyR\nZjbbzGaa2THN+QDSdLW18OKLcUuWefPghz8saZZEpBHyeilqZlu5+zoz2wJ4BfgJcAqwzN3/08yu\nAnq4+wgzGwg8ABwM9AGeBXbPfAOql6Ktxy23wCWXxNv6toi0Xk1+Keru61KrnQhNHR04Ebg3lX4v\ncFJqfQgwwd03u/tcYDYwqHFZl5bw0Ufx+pgxpcuHiDRNXgHdzNqZ2RRgEfCMu78B9HT3xQDuvgiI\nRsPuDcxPnL4wlSat1PzEd+v990uXDxFpmrw6Frl7LfAVM+sG/NnM9iaU0tMOK/Tmo0eP/mK9qqqK\nqqqqQi8hTbRsGTz0ULz9zW+WLi8iUld1dTXV1dV5HVtwxyIzuxZYB5wLVLn7YjPrBbzg7gPMbATg\n7n5j6vhJwCh3fy3jOqpDbwWijkRVVdClC0ycmD4Zs4i0Lk2qQzez7aMWLGa2JfANYCYwEfhh6rCz\ngMdT6xOB082so5ntCvQHXm/SE0izGz8ennxSwVykLcvn13cn4AUzmwq8Bkx2978CNwLfMLN3gaOA\nXwK4+wzgYWAG8FfgQhXFm8cHH8CmTY07d9Kk9NYsO+9cnDyJSOloLJc2zAyuuw6uvTaMjDhnDvTq\n1fB5GzaEbv0ffQR9+8JvfgOXXdb8+RWRptNYLmUo6tU5a1Yopa9fD48+mt+5ixeHZd++YXnyycXP\nn4i0PJXQ26jkqIjvvAN77x3W8/lI33gDBiV6BmhURZG2o74SugJ6G5U5FVwkn4/0hBPgL38p7BwR\naR1U5VKGDjmkblrHjvDee/WfV12dHsx/+tOiZktESkgBvY367DOYMiU9beNG2HPP+s874oiwvP32\nsFy6tPh5E5HSUEBvo2bMgO23D+sdO4aSdyGGDAnLs88uarZEpIQU0NugjRvDcsfU6DkXXACHHx7v\nr62Fmpr0Qbcy7bgjXH45HHhg8+VTRFqWAnobtHQp9OwZSuYQV7O8+y5stRWsWwft20O/fiF98+YQ\n5KOXn9dfD1tsEWYl6tq1xbMvIs0kr8G5pHX59NO4dL56dRiDBWCPPUIwf/HF9OO/9KVwTvQC9Jpr\nWi6vItJyFNDboCVLYIcdwnq2EvYJJ6Rvf/ppWGqeUJHypiqXNujjj2GnnRo+bttt66apdC5SvhTQ\n25h168J8n7kmcb7//rAcNgxWrYrrzXv0CMutt272LIpIiSigtzFXXBGW99yTfX///mH5/e+H3qSr\nVoXtBQvCcv365s2fiJSOuv63IZs2xS1b+vaFuXOzH3fOOXDrraHFyx57hN6jtbVw7rkhXeO2iLRd\nGsulTAwbBr/9bVjftCk0TaxPcrwXfdQi5UFjuZSJKJgPH95wMBeRyqOA3opt2FBYeqZOnYqXFxFp\n/VTl0opFVSbu6dUn69fnF6xXr4Zu3eJriEjbV1+Vi/5xbwP+/Od4/amn8i95b701/OEPcSckESlv\nKqG3UsnSdZI+MpHKppeibVC2YB61MRcRyUYBvZX7wx/CctQomD27pFkRkVZOAb0VeumlsHz1VVi+\nPKwfdljp8iMibYMCeiuzaVPo6QmhR+exx4b1/fYrXZ5EpG1QK5dW5u9/hw8+COv77huWjz0Wj38u\nIpKLAnors3BhWP74x3Hb8xNPLF1+RKTtULPFEvr889AMcautwva118KYMWG9wj8aEclBg3O1Usme\noMntHj3il6EiIklNaoduZn3M7Hkze8fMppvZJan0UWa2wMzeTn0dmzhnpJnNNrOZZnZM8R6lPH3y\nCXz72/G2grmINEaDJXQz6wX0cvepZtYVeAs4ETgNWO3uN2UcPwB4EDgY6AM8C+yeWRxXCT19fJbI\n559rvHIRya1JJXR3X+TuU1Pra4CZQO/o2llOORGY4O6b3X0uMBsY1JiMVyIFcxFprILaoZtZP2B/\n4LVU0sVmNtXM7jaz7qm03sD8xGkLif8AiIhIM8m72WKquuUR4FJ3X2Nm44Hr3N3NbAwwDji3kJuP\nHj36i/WqqiqqqqoKOb1NW7s2fXubbeChh0qTFxFpvaqrq6murs7r2LxauZhZe+BJ4Cl3vznL/r7A\nE+6+r5mNANzdb0ztmwSMcvfXMs6pmDp0d3jtNfjqV8P2ggVw5pnwwgvw+99Dr15xj1ARkfo0udmi\nmd0HLHX3yxNpvdx9UWp9GHCwuw81s4HAA8BgQlXLM1T4S9FsLz8jFfIRiEiRNGmCCzM7DPg+MN3M\npgAO/AwYamb7A7XAXOB8AHefYWYPAzOATcCFFRO5C3TkkaXOgYiUE3UsagGDBsEbb4T1SZPi6pXa\n2vpL7yIimTTBRYn9859wxx1hSrhvfhPGjg0vRRXMRaSYVEJvAWbw8ssa01xEmk4l9BK6PPUa+aCD\nSpsPESl/KqE3s8wBuEREmkIldBGRCqCA3ow2bAjLCRNKmw8RqQyqcmlG8+fDLruEeULba24oESkC\nVbmUyOGHh6WCuYi0BJXQm5FeiIpIsamEXgIK4iLS0hTQm8lbb4XlhReWNh8iUjkU0JvJxo0weDDc\nemupcyIilUIBvZmsXQtdu5Y6FyJSSRTQm8mqVQroItKyFNCbSXW1RlMUkZalZovNRE0WRaQ5qNli\nCxs6NCyHDy9tPkSksqiE3gyi0rm6/ItIsTVpTlEp3KGHwiGHKJiLSMtSlUsz+Pvfw3RzIiItSVUu\nRfbWW2F2ovXroVOnUudGRMpNfVUuCuhFptYtItKc1MpFRKQCKKAX2XHHwZ//XOpciEglUkAvsqee\ngo4dS50LEalECuhFtHBhWA4aVNp8iEhlUkAvomjs8+23L20+RKQyKaAX0bJlsNVWpc6FiFSqBgO6\nmfUxs+fN7B0zm25mP0ml9zCzp83sXTObbGbdE+eMNLPZZjbTzI5pzgdoTbp3h+uuK3UuRKRS5VNC\n3wxc7u57A18FLjKzvYARwLPuvifwPDASwMwGAqcCA4DjgPFmlTGQ7Jo1cOCBpc6FiFSqBgO6uy9y\n96mp9TXATKAPcCJwb+qwe4GTUutDgAnuvtnd5wKzgbJ/TegOL70EnTuXOiciUqkKqkM3s37A/sCr\nQE93Xwwh6AM7pg7rDcxPnLYwlVa23OMXogcdVNq8iEjlyns8QDPrCjwCXOrua8wss3N7wZ3dR48e\n/cV6VVUVVVVVhV6iVbjySrj99rCuERZFpJiqq6uprq7O69i8xnIxs/bAk8BT7n5zKm0mUOXui82s\nF/CCuw8wsxGAu/uNqeMmAaPc/bWMa5bNWC633w4XXADnngt33VXq3IhIOWvy4Fxmdh+w1N0vT6Td\nCCx39xvN7Cqgh7uPSL0UfQAYTKhqeQbYPTN6l0tAX7gQ+vSBqip44YVS50ZEyl2TArqZHQa8BEwn\nVKs48DPgdeBhYGfgI+BUd1+ZOmck8CNgE6GK5uks123zAX3ePOjbN95u448jIm2Ahs8tsiVL4NFH\n4cMP4Ve/CmmHHgqvvFLafIlI+VNAL7KoVf3RR8Ozz4b1NvooItLGaDz0Jpo5E2bNqpv+7LOw++6w\neHHL50nqJrk7AAAJvklEQVREJJNK6A3YvBk6dAjrUXYPOQReS7TZaQOPISJlQiX0Jpg2LV7//HOo\nrQ3BfOjQkHbttaXJl4hIJpXQG5A5Cs2tt8JFF8HUqbDffqXJk4hULr0UbaQlS6Bnz+z7WnnWRaRM\nqcqlkaJgfs89MHt2afMiItIQldBzaNcuLoVHy6j65bHH4MQTS5MvEalsKqEX6JZb6q9SUTAXkdao\n4gP6xo2wYAEMGQJ33BHSLrkk3l9bm368eoOKSGtV8VUuma1YPvwQdtst3k5mccMG6NSpZfIlIpJN\nfVUuFTt69+LFcMYZddOjYP7rX9fdp2AuIq1ZRZbQ+/YNIyXWp6YmvBgVEWlNKvqlqDs8/nhcdTJn\nTnow32YbuO8+GDsWli2L0xXMRaStKfsS+j/+EYa2/fnP4b334IEH4n3HHx+C/RZbJPMVlq2gel9E\npI6KrkO/996wHDUqPX3yZDjqqPRgDvDVr8Knn7ZM3kREiqnsS+iZrVgiuerI160LpfMuXZo3XyIi\njVFxdehr1oTqFYADD4Ru3eJ9//xnKJ3nqiPfaisFcxFpm8qyymXrrcPSPQx5O3kyTJ8O550X0vfd\nt3R5ExFpLmVX5bJwIfTpE9bXrg2l7blz0ydzFhFpqyqmymX69DiYQ1x10r17afIjItKSyqqEnusF\nqJogiki5qIgSembQ3mef0uRDRKRUWv1L0dmzoWPHhuvAo7FXVq0KMw317x+aIGaOligiUq5adZXL\nW2/BQQeF9YayqR6eIlIJ2uyconvsEU/9Vt+h7nG7cgV0ESlnbbIOvaYmDuZHHVV3//r1IXi7h6Fw\nQdUrIlLZWnVABzjiCHj//fR9GzfCllvC1VdDv36w005hHPNcrVxERCpBg1UuZnYPcAKw2N33TaWN\nAs4DlqQO+5m7T0rtGwmcA2wGLnX3p3Nct8Eql/XrQ5Du3Dmsd+gQtnN121d1i4iUu6ZWufwe+GaW\n9Jvc/YDUVxTMBwCnAgOA44DxZo0vN3fuHM8S1LlzGBkx2XEoSc0URaTSNRjQ3f1lYEWWXdkC9YnA\nBHff7O5zgdnAoCblMMPHH4fl3Llx2vz5MG1aMe8iItL2NKUO/WIzm2pmd5tZ1Lm+NzA/cczCVFqT\nvPxy3bRku/SePZt6BxGRtq+xAX08sJu77w8sAsYVL0t1HXYYPPhg3fRrrgnLDh2a8+4iIm1Do3qK\nuntyTp+7gCdS6wuBnRP7+qTSsho9evQX61VVVVRVVeW85/e+F77GjoUbbghpV18N555bWN5FRNqS\n6upqqqur8zo2r45FZtYPeMLd90lt93L3Ran1YcDB7j7UzAYCDwCDCVUtzwC7Z2vO0tjBuTZtCs0W\nNQmFiFSiJs0pamYPAlXAdmY2DxgFHGFm+wO1wFzgfAB3n2FmDwMzgE3AhcUeUrFDB1WxiIhk06q7\n/ouISLo22fVfREQKo4AuIlImFNBFRMqEArqISJlQQBcRKRMK6CIiZUIBXUSkTCigi4iUCQV0EZEy\noYAuIlImFNBFRMqEArqISJlQQBcRKRMK6CIiZUIBXUSkTCigi4iUCQV0EZEyoYAuIlImFNBFRMqE\nArqISJlQQBcRKRMK6CIiZUIBXUSkTCigi4iUCQV0EZEyoYAuIlImFNBFRMqEArqISJloMKCb2T1m\nttjMpiXSepjZ02b2rplNNrPuiX0jzWy2mc00s2OaK+MiIpIunxL674FvZqSNAJ519z2B54GRAGY2\nEDgVGAAcB4w3Mytedtu+6urqUmehxemZK4OeufQaDOju/jKwIiP5RODe1Pq9wEmp9SHABHff7O5z\ngdnAoOJktTy0th+AlqBnrgx65tJrbB36ju6+GMDdFwE7ptJ7A/MTxy1MpYmISDMr1ktRL9J1RESk\nkcy94VhsZn2BJ9x939T2TKDK3RebWS/gBXcfYGYjAHf3G1PHTQJGuftrWa6pPwIiIo3g7lnfTbbP\n83xLfUUmAj8EbgTOAh5PpD9gZr8hVLX0B14vJEMiItI4DQZ0M3sQqAK2M7N5wCjgl8CfzOwc4CNC\nyxbcfYaZPQzMADYBF3o+/wKIiEiT5VXlIiIirV9Jeoqa2bFmNsvM3jOzq0qRh2Ixsz5m9ryZvWNm\n083sJ6n0gjtfmdkBZjYt9bn8thTPky8za2dmb5vZxNR2WT8vgJl1N7M/pZ7jHTMbXO7PbWbDzOz/\nUvl9wMw6luMzF6sDZa7nTH1uE1Ln/MPMdmmWB3H3Fv0i/BF5H+gLdACmAnu1dD6K+Dy9gP1T612B\nd4G9CO8XhqfSrwJ+mVofCEwhVHf1S30W0X9KrwEHp9b/Cnyz1M9Xz3MPA/4HmJjaLuvnTeXxD8DZ\nqfX2QPdyfm7gS8CHQMfU9kOEd2Zl98zAvwL7A9MSaUV7TuACYHxq/TRCf53iP0cJPrhDgKcS2yOA\nq0r9DS3i8z0GHA3MAnqm0noBs7I9L/AUMDh1zIxE+unAbaV+nhzP2Ad4hvBuJQroZfu8qfx1Az7I\nkl62z50K6B8BPVLBa2I5/2wTCpnJgF605wQmAYNT61sAnzbHM5SiyiWz89ECyqTzkZn1I/yVf5Xw\ng1BI56vehM8i0po/l98AV5Le/6CcnxdgV2Cpmf0+VdV0p5ltRRk/t7t/DIwD5hHyv8rdn6WMnzlD\noR0o63vOL85x9xpgpZltW+wMa7TFIjGzrsAjwKXuvoa6na3K4u2zmR0PLHb3qaQ3Zc1UFs+b0B44\nALjV3Q8A1hJKamX5fQYws20Iw3z0JZTWu5jZ9ynjZ25AMZ+zWZptlyKgLwSSLwT6pNLaLDNrTwjm\n97t71CZ/sZn1TO3vBSxJpS8Edk6cHj1/rvTW5jBgiJl9CPwRONLM7gcWlenzRhYA8939zdT2/xIC\nfLl+nyFUr3zo7stTpco/A4dS3s+cVMzn/GKfmW0BdHP35cXOcCkC+htAfzPra2YdCfVME0uQj2L6\nb0Ld2c2JtKjzFdTtfHV66q33rqQ6X6X+pVtlZoPMzIAfJM5pNdz9Z+6+i7vvRvjePe/uZwJPUIbP\nG0n96z3fzPZIJR0FvEOZfp9T5gGHmFnnVF6PIvQxKddnztWBEpr+nBNT1wD4LmGU2uIr0cuHYwmt\nQWYDI0r9MqSJz3IYUENorTMFeDv1fNsCz6ae82lgm8Q5IwlvxmcCxyTSDwSmpz6Xm0v9bHk8++HE\nL0Ur4Xn3IxRIpgKPElq5lPVzEzoSzgSmEUZW7VCOzww8CHwMbCD8ITub8DK4KM8JdAIeTqW/CvRr\njudQxyIRkTKhl6IiImVCAV1EpEwooIuIlAkFdBGRMqGALiJSJhTQRUTKhAK6iEiZUEAXESkT/w9U\n+Axi4Q23tAAAAABJRU5ErkJggg==\n", 114 | "text/plain": [ 115 | "" 116 | ] 117 | }, 118 | "metadata": {}, 119 | "output_type": "display_data" 120 | } 121 | ], 122 | "source": [ 123 | "pd.rolling_mean(rewards, 300).plot()" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": 90, 129 | "metadata": { 130 | "collapsed": false 131 | }, 132 | "outputs": [ 133 | { 134 | "data": { 135 | "text/plain": [ 136 | "" 137 | ] 138 | }, 139 | "execution_count": 90, 140 | "metadata": {}, 141 | "output_type": "execute_result" 142 | }, 143 | { 144 | "data": { 145 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXEAAAEACAYAAABF+UbAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXeYFdX5x78vu8suTZoUBUUR7KiIYonKiooYazQWjDUx\nkp8aE5VYYsPEKMYSu0FjN2IjUTSigroqMShRiqgURZHepAhL2bt7fn+89+ScmTtz79yZuXvb+3me\nfc7cuXPPnLm7+50z73kLKaUgCIIgFCct8j0AQRAEITwi4oIgCEWMiLggCEIRIyIuCIJQxIiIC4Ig\nFDEi4oIgCEVMRhEnokeJaBkRzfB5/0wimp78mURE/eIfpiAIguBFkJn44wCOTvP+PACHKaX2BnAz\ngEfiGJggCIKQmcpMByilJhFRrzTvT7ZeTgbQI46BCYIgCJmJ2yZ+AYDxMfcpCIIg+JBxJh4UIjoc\nwPkADomrT0EQBCE9sYg4Ee0F4GEAQ5VSq9McJ4laBEEQQqCUIq/9Qc0plPxJfYNoewBjAZytlPo6\nwEDK6ufGG2/M+xjkeuV65XqL+5rTkXEmTkTPAqgF0JmIvgNwI4CWrMfqYQDXA+gE4EEiIgANSqmB\nmfoVBEEQohPEO+XMDO//EsAvYxuRIAiCEBiJ2MwxtbW1+R5CsyLXW9qU2/UChX/NlMneEuvJiFRz\nnk8QBKEUICKoiAubgiAIQgEiIi4IglDEiIgLgiAUMSLigiAIRYyIuCAIQhEjIi4IEdmyBbjzznyP\nQihXxMVQECIybRrQvz8gf9pCrhAXQ0HIIS3kv0jII/LnJwgRqajI9wiEckZEXBAiUqozcSJg3rx8\nj0LIRIn++QlC81HKM/EBA/I9gtKCCFi5Mt4+RcQFISYaG/M9gvhZsybfIyg9li+Ptz8RcUGIyLRp\n3G7Zkt9x5IJjj833CEqPuG/2IuKCEJG2bbltaMjvOHJBmzb5HkHpISIuCAWGnq0uWJDfceSCRCLf\nIyg9RMQFoUDZc898jyAaSgFNTc59IuLxob/buJ/YRMQFQQAADBsG7Lefc9+4cfkZSymiRTzutZOM\nNTYFQSgPnn/ebLtn5EJ0tBklbhGXmbggRKRHD2633jq/44gT224rgh4PIuKCUKBs2ADccgtw8sm5\n6X/CBGDdutz07YdtCxe7eDzkypwiIi4IEdm8GWjXDvjwQ2DFivj7HzIEuO+++PtNR2MjuxdWV5dm\nEFM+kJm4IBQg//kPsHEj8MknwMyZwM9/npvzVFXlpl/N9OnO14kEpxOoqBARjwuZiQtCAXLwwdxu\n3sztd9/l5jxXXZWbfjXu8PpEAqisBDZtEpt4XMhMXBAKmK5duS3WMHX3bLuxkUW8qQl49938jKnU\n0N+x+IkLQoEwdarZvu46oGNHzlJXjLhFfMsWk6hJZuLxoL/HSy+N94lNRFwQQrJ6tdneemt+fcst\n+RtPFNwiftdd3O65p2QyjAvbnHLDDfH1KyIuCCFp3dr5+o9/BAYNivcczeXe5xZxfW25XKwtN+zv\n+Mkn4+tXRFwQQuIujHznncB77wHffx/fOewoykmT4uvXjS0wiQSw7bbARRfl7nzlyNq1uelXRFwQ\nQrJlC5sb6uud+y+7LL5znHWW2daCvngxMH++2X/UUZz3xH1TyYZNm8z25s38uqYmfH9CKnYMgU7x\nSwSsWhWtXxFxQQjJli1At25Aq1b8Wv9jak+VuNGietxxwA478HYiAUycCDz3HIt7WE4/3Ww3NhoR\nf/ddYJ99wvcrGGxf/w0bgNde4217bSUMIuKCEJItW4CWLc3rRx/ltnPn3Jxv2TJubVv8hg1mOw5z\nS4sWThFv1650C0E3N/bTDgAcfzy3ehIQFvn1CEJIjjsOGD/evN5+e25zJXpPPw1MmWJm4YDThPLi\ni9HP0bq1U8QrKiR3ShQeeIBdNZVKFXFNVBdOEXFBiInddgNOPTW3Yep/+APQs6d5bQtAv37Z9zd5\nsnO8NTUs2lrEKysl7D4Kl1zCJrd//pO/04qK1GO23z7aeoaIuCDESHU18Nlnuet//XrnTN+eJYcR\n8YMOMrZZgO22eiZeXc0iLjPx6CxaxN+pX81SbSoLQ0YRJ6JHiWgZEc1Ic8y9RDSXiKYRkSyDCGXL\nM88AY8bkrv8ffjDbTU1mlnzwwcC//x29fy3a2t4v5pR40OYUd2yB5o03wvcdZCb+OICj/d4komMA\n7KSU6gtgOIC/hh+OIBQXI0c6Xw8f7rRZR6W6mkO033uPXx98sDGhJBL807Mn8NVXJsoyW+yZvc5a\nmEjwrFxm4uFYswa4+27zev16znbpJ+JRFsMzirhSahKAdE4wJwJ4KnnsRwDaE1G38EMShMJmyhS2\nfwPAAQc43xs8GNh///jO1dAAbLMNcNhhwLnnAvvua1zSXnnFZBvUeU6icNhhLOiJBJ+3slJs4mF5\n6SVnvMDIkRzs06lT6rH77x/tO47DJt4DwALr9aLkPkEoSd55B5g1i7fdM6iaGn8vhGxpauIfvRim\nZ8UPP8yvTzvNiHgUdNKu8ePZpfD771nEZSYeHvfvpKGBf9wBVKeeCmy3XbTvuNkLJY+0nj9ra2tR\nW1vb3EMQhEjY/3DuWXd1tcktHsd5KiuNyM6endq3Lt4QBSL+qa7mTIz19ebclZXA0qVs0y3WDI3N\nycqVnAzNy20wkXDGFZxwAjBqFHD11akz8bq6OtTV1QU6ZxwivgjAdtbrnsl9nox0GxEFochYv57b\nbbdNfa+mBliwIHV/GPRsWDNpUmpAj877PW1a9pGVtlubUmxKadWKZ+Kvvca5U7p04fc3b5Yw/CB0\n6cLrF0FE/Oabgd69vasnuSe4N910k+85g5pTKPnjxTgA5wAAER0IYI1SKoLDjCAUNqNGcWv/Q2qm\nTQO+/DKe81x8MS+GpWPOHBZx7V6Yjb/xJ59wm0iY2Xjr1sblUN9AOnVyesUI6Vmxwinit98ODBjA\nQm3/zejfWdQSeEFcDJ8F8CGAnYnoOyI6n4iGE9GFAKCUeh3AN0T0FYDRACT3mVCy2P+cXiJ+3HHx\nnOeLL4KlKz3lFBbxFi1MyHxQvv6a2/HjjfiPHWvOq+26rVvHZ+cvByorndV7Bg82Hj9efzNR1x0y\nmlOUUmcGOOaS8EMQhOLB9gKZOzf1/Y4dgQ4dop8nm3zT2ibe1MQVhvSTQiZ0atRx47zf1zPxli3j\ns/OXA5WVznwottum/l0dfXTq+2GRiE1ByAI745yX6SIu75Sdd+b2vvvMvm++8T7W9oS47bbgNRzb\nt+e2m+UQ/LvfpfbbsiX36+UeVw68+Sbw4IPBj6+s5EVijRbpxkaOIejWzRncIyIuCM1IJnu3FvEo\nuTAAY7Z5+WWzzy+IyO3OFjSb4RlncDttmklFe+ih5n3tjVJdDfztb9FTphYrl1/O6xOZ0L/zqipn\nRXt7Jt63L3v72IiIC0IzMm8et336eL/fogXPXKPOxrWI2zNxP95/3/l68GDg22+zO59eQG3XLvW9\nli2BAw/Mrr9SImilJi3ERM6nIXsm7uXTHzWgSkRcELKga1eutpPOZ7pNm9RqP9mibac6MtSN103C\nLr57zjnZnU8//rdta/bpmWXLlub9TN4ypYh75uyHnn03NjpFfONGTovQ0ODt0x81P42IuCBkQWMj\n/9OlE7M2bZzFGsIwZgyw666p+92eI4CJGrX3BRGFtm1NYQK9yGmLuM6pUl1tbkrlGL3ZLWASEb34\n29TkFHGdZ37yZO+ZeFRzSrNHbApCMaNF/IEH/B+z4xDxd97x3v/jHwPHHONMWvWXv6Qe95//cMk2\nbff2gogLTXToYOzdtojvtRe3LVua6ynHPCpB1zd0kFdTk7NUnvZUqa/PjYjLTFwoe7p3B956K9ix\n2q55wgnAeed5HxNVxO3FTDdbbw28/rrTnKMFwC0Ef/5z+vPodLM//MCiDxgRf+klY0KprjZRqjvt\nFOwaSomgIq5/J/PncxbD007jYhBauP3MKWITF4SILFsWPBd3kFwlrVpFsx3/5CfcnnJKsOP9TB1T\np/p/ZvNmnjHW1LBw62vSIm67yNkz8aCLfKVEUBHXQvyrX/H2UUcBJ51kvttVq/xn4mITF4QQJBLG\ni+MPfwj2GW1OSUdcvuJnZgyzY/QNQ4uInQLVj2XLeGboXqCtrAR23NGZ2MsWcaD8An+CirgtxBs3\nGjOK/R2LOUUQYuShh1iwND/7WWaBCiLib78NjB4dflx7782tXwEBjY4e1f7jWkS23jrzOaZM8X9v\n3jznYl51tfOm9MEHmfsvJYIKrJ+I23gV0RYRF4QQNDUBM1wFB599lgNf0uHn6+tm7NjwY9Mza9uk\n4UWXLnwd2uyik1StW5f5HF27AoccEmw87ptC1PzlzUG7dpy6VzNrFuejCcOaNdzOmZP+OH2ja9/e\nrDf4HWPzxBPAn/4UbmyAiLhQprzyCkchurnqqvSfiyN/dyb23JPbTCIOOB/V9cxcL0KmI5tiEu5c\nMH7FfguJ9eudBav33RfYY49wfR15JLe77JLetHL44dyuXZuaRljTv3/qPi+xzwYRcaEs8VugyyRs\nQcwpUWnbFjj77OyjJI89lt0PtbD7RZUCHNWp63ZmQrsf3nYbt0Fzs+QbO+OkvdB8yy3+Sb8yEeQp\np1Wr1Jn4+PHA+ecbn3EbXSUqLCLiggBT/EAnhfKjOUS8oYGz3HnZT9Pxi1+w++H995t+0hF0we5X\nv+L21FM5t0oxB/w0NQHXXpta4Doddh6UO+7IfLxSbNqyZ+JDhwKPPRb8nNkgIi6UJW4hWrGC24ED\n039u6dLMInbEEZnPn25GpyvNh0WH3Mc1Y9bZC9u2LY6am999x63XTerqq7nNVGpOKfN5OzeNXxCW\npmNHtnt/+ml0M0lQRMSFskTbjR9/3PnPnsk18KGHMufrvv56znliL6xpLr+cK7q0b28CbGxeeAF4\n8cVoi4davDOJ+CLfIopOtJdMmzapBQ8KES26Xh4ft9/O7aefpu/jjDP4qUPPwnWfmf4+Hn/cbAe9\nEXfsGOw4P0TEhbJkq6241Ytdt97KbZAgHb+kVJqWLTll7a67Av/6l9mfSHCI/MyZ/Frb5W+/3RSY\n0Clho4h4z57cphPbzp2DzxSrqoAhQ9jWW1VV+DNxbRrL9B2mMye99BIHgJ13Hl+3TtHr54KqPVfs\nhcug329UM4uIuFCWdO3KofM6qEX7WuvFu3RkOsaeAdrl2saMcR6nTThXXskzOJ3mFogm4tqWnk7E\ns/GyIeLCCETFMRPX3jP6hgh4pwtIlx9dL4qOGeOcUfuZYXSUrV08O+hM/Igjonn8iIgLZclJJzmT\nFHlVJ/eipoY9O9Lh93jsPsfq1Rw5CfBNZcAA814UET/7bGD48PQz5qD+7m5sm3hjI7DNNtELYADc\np86kGBUvM0pjY6oZbMQI/z7sYCf9PR1yiH+KX/2d2N9pEBdRwGTF/L//C3a8GxFxoWz573/N9skn\nm223X/Qf/mBmYA0Nmf853f7IOlhmyRLn/oYGTr4FsAnAflS3swlmyx57cJZFvxmzUrwmEMbLxjan\nbNnCC70rV4Yfq+anP42nNingfbNMJFIjKNP5jZ9/vtnWwrz//v4eQ143RK+ITS8qKnjMf/1rsOPd\niIgLZYeeOdqeKDU1xr3QnhHecw9w4428/e67/M+WrfitWsUzrWuuce63Z4xLlzof1bXNPiwtWvBY\nvZ4wPvqI27AzcffCadSkWGvXcvBVOq67ztw8JkxIH9Dkvnl16gQsXMil0TSnncZrBy++6P0d2fu0\nWSRdePyPf8x9Asb/PhsRj4KIuFB2aDHo18+538vr5PXXzfZll/FiVSb3NCC1HqZXHhRbEEaMcFYD\nyuSvngkiFp909usw4mGbU3SrTUJhWbgw/ftNTRyWPnEivx4yhG+ufth+3Tb77We2X3iBPVBOO81E\nutrYaXy1F4+eMXuxdCn7ggNA797ciogLQg6orze5MPQ/m6amxmzrHCq9epl9iURwO+f06d777Yi9\ndHb4qDNxwCniDz1kZrv6hpJtMJHuU4u37jtKsi8gc7ZGnUHxmGPMvnTl72yzlH2j1GL5t785f49B\nk0+1aOF/bH29qU+qn3CCPukEmRSkHVe0jwtCcdGmjalY89vfOt+ziwTrYBx7lvT559GCcADgrrvM\ntjbTeOFVsDhbbMG96CJezNX7d9klnHjY5hTd949+FG2ctuh6LZJ6FdjQAT1e2DNxW0jtbbsAtZdb\nqf4bsUk3E7cjeTt2zBw05kcY900RcaHsWLqUZ7puE8fRR5ttLVTuxbZVq4Kdw89jw09UbK64Itws\n2etc114LbLed2bdwYbTUAV4z8aguhxdcYLbdIrZ0KYewa9PEvfdy+8wz/v1t3uxMMazR33fv3s4x\nuwN4nnuOM1y6E6S5q9jb2N9pdbVZd8iWqqrgnlIaEXGhLPEKe7eFU88OM+X09sPvc42NLPBPP218\n1D/+2HlMkPwcQVi1CnjwQafNefRoFqiwaVntmbie8UYVcVtEdV9EbGvfZhuOgNXmpd/8JnN/s2c7\nFzE1lZX83R9+uNO91G0iGTaMW7dN+49/5B8v4sypY49n06bMT0wi4oKQxP6n1dF27n9wrxmeF1VV\n3o/82quiooLD7nfZJfgCWBy8/DIv5Nm2/mywFzZ1zvQoIn722WZ27e5Lu18uXJhqXvKrbwrwArW7\nZmp9vTOC0r4Z+M183b8XbWLx82aJ4+kJcP7NBcmaKCIuCEmIzOLZKaewN4RboLLxZfaaQekoQf1o\nP3t27kT84INT982cybNanfs6W+wqP9qVLoyIr13L388zz3Dk6iWX8Pdw6aWpNz+3AAP+5iqd4+TU\nU5373d+xDs0Hgov45ZebsbuJcyZuj+fNNzMfLyIulA1BIgtff53Fb906Drl2C5SeHQZBz8zuv5/P\n/ac/mSAS2/Zri8ULLwTvPxNeCbYATlkbVnDatzezw169+PsII+LutYV+/XiB8amnUl0/p05NXVD2\n8xLRHji1tcHH4hZxncddB2lNmuR832vxMVfmlIsuyny8iLhQNgQVGz3r69DBfGb8eG6z8RrRM3E9\n6/79741t135MtmeZUd3NbB58MPPYsqVVK+PNsXQpL5qG8ahwi7Dta+6V1mDJElPxCPBf2NRmogMO\nCD4W9819m2243XZbDuLRfenZvV9YfxQRt6sQ2TcVP593GxFxoeQh4tmcrkGZCZ3EqGNHFvETTjCe\nK9n4b+uZuDvcHnCaDDp3NkFFcdlVAQ5u6dzZ1OC0CRtlaYv4J59wNOSoUdnNxseMSU3Ta6fF/eGH\nVH/85cvNd5MuWVTfvvy7svPQ+KHTxj7wgHO/FuNWrTgLpb4Jt24N9OjhfdOKahO3b1D2TUJEXBCS\nzJgRrAo8YBJYXX89i5Nd8szL68EP/Znnn099T4vU5s1OW3ycM/H99uO8Jocdlvre0qXh+rRF/Pvv\njc32wguD93HmmWwDt/nd78z2228b76Cf/czs1yI5fbpzgXniRBP841Xb0nYdtdGLo0895dyvRdwr\nJsCvKMakScEENwj2TPyoozIfLyIulAXuPNDffON/rG0ymTfP/DPPmZMqPunQonP99f7H2KaU/fZz\nhobHxaBBqfu+/TZcX9XVnCsdcJoh3NGvmXCbJNypYvfZh1tdxAEw32e7ds6nqqOOMiXpfvMb4LXX\nePuDD7h95JH0Y3Gvc+gbrJcff7rKRmF9w93o7+aTT9JHpmpExIWywF1TMd2s/LTTjNlk4kROfAXw\nLDwbv3E9q7YzJGq8cqNMmeIMzImLvffm9oorzL4gC2ZetGvHTzVbtrCYaTHVduSgKJX61PHmm8A/\n/sHbxx3HN0/9e+rTx5yjXTt+wrBvInqNwb45HXIIH5PuOz322FTf80QCuOkmZxoGTToRz+ZpJB16\nJr7fflyYIhMi4kJZYNully1Ln+q1f3+TbAnwrlAeBP3P6JVv5aKLgK++CtdvWLTLYceOHMkZhr32\nYve8115jMWvblnOXZ+uh0tTEN5crrzT7hgwx5qqaGr5R6KegFi2AZ59ln3EtrvPnm8/q0PwrrjCF\nnYOw++58M3nkEfbhX7WKz+Pn3eIn4i1bxpPvBgiey0UTSMSJaCgRzSKiOUR0lcf7WxHROCKaRkSf\nEdF52Q1DEJqPrl0zH2PPuLOpjG7TpQvPWr3s3FVV3tVmcok2D6SraBOkj0TCuViaKVuiF0uWsO3Z\n7dGhx2jf+G6/nbMWbrUVLyzq7/Ohh8wx990HHH88v+fOIJmOigq+WVx4Id+YdJCQXRTCfXwiwbno\nf/1r3tfUxNcfV2Hk2MPuiagFgPsBHA1gDwDDiGhX12EXA/hcKbUPgMMB3ElEEWqTCEJ+sUU8yj+n\n2+c5n1RUhE8jYPfhnimGEXGAbb5ul009854yxewbMcKkebWx08U2NrIIb9kSPNMkwDcNLdybNpmb\niN+Tmr6J3X+/scNv2sTnjGtResIE5+tMv7MgM/GBAOYqpeYrpRoAPAfgRNcxCoD+dbQDsEopVeDl\nVIVyIGxIeFSxK0TatQu2UJYOLWIHHWRcMaMUT77sMnb/1NjufekYPtx7EVgLalCamoyP+kcfsUsm\nkFnEdWQowN467iRaUXDb8O2Mi14EEfEeABZYrxcm99ncD2B3IloMYDqAAGlqBCH3hJ1Fx1UqrFD4\n+mtTsT0KlZU8691pJ1NE49//5kCmMNTUGE8UwNxkMhWjPv10b3/xhx82i6NBuOUWYO5c3h440Aim\nny+6vn7buyldSuEwuD2p7OLLXsS1sHk0gKlKqW0B9AfwABFFqBIoCN68/rp/OHmcVFez3TdT1Zli\noXdvfty3FxLDoG3CGzea2fIvfhHcjpsp4Kp3b86f4meT1rRqxblbZsxIfc9rXxAuvJAThAH+aYLb\ntuVcLzYDBrD3TFQmT+bK91OnOs+faZYfxG69CIC9Pt8zuc/mfAC3AoBS6msi+gbArgD+6zoOI61V\notraWtRmk+RAKHuOPZYXt5pDXF96KffnaG56uJ+hs0TbxDduNF4ie+3FHj1eXH8954PREZqZPC9q\natKXXtPoG4hd7UeTjdfPgAFsmweCmYQ6dwZ+8hPnPqXYnTEqBxzAEbBffQU0NtahsrIOiQTwxhvp\nPxdExKcA6ENEvQAsAXAGgGGuY+YDOBLAv4moG4CdAczz6mxk2KV+QUiyaBGXT7Mfw20mTeJcFHZ1\nlm++4ci8bFf+S43I9Rxb8Iy+vt6IuG1i2H575zkmT+YgqSVL2M+7sZFdHN97L5pPvBZcOy+4Jpv8\nNnffnWpmSle9ye07rlT2i6npqKzUs/BaJBK12LyZTYKjR9/k+5mM5hSlVCOASwC8BeBzAM8ppb4k\nouFEpN3bbwZwMBHNADABwJVKqYg1sAXBHz178uK669gP2w4t32EH4IYbwrsLlgpnnZUaZp4tSgF1\ndUbQtImld2/g5pudx+oZs7brNjXx8f36RVt32Hdfs62LPQD+N3Y/vMwmur6qF26Bf+894P/+L75o\nzaoqk9PFb3xuArkBKqXeALCLa99oa3sJ2C4ulDgLFvBCzmOP5XccdsIgm7o6k+c6SOrZcqN9ey7E\nEAda0CorTT4VXfRCYyeFIuIZeRxJvoi4Wv1zz7GHC8B9d+qUXT/uRcS+fTkAyA+3iN99N7czZ2Z3\nXj/coh3ku5KITSErHnjAOVPIF372S51PGjAJiaK61QneaEGrqDCLfe5CFDqFr0bPxONAn0v31717\n9t5I7myOmf5W7ChRwCQXyybPfDqCzLzdiIgLWaFdv+LK2JYN9gKPn/+3nZNk8mQOEmnO8mflhDZp\nVFSYGbjtPw2kRk/+8IN3at4waDt0lJn93ntzDnItzpkKYbtT6Op8NEGigINQWcmLxJWVJptmJkTE\nhVAEzc0dF+PGGU+EQYP8byL2LO+++0rHRbCQsWePd9/tjDh0PzH985/xnVeLeJSZfe/enDRL58fJ\n5M7nZ8fP5BIZlMpK/ts+99zgOd9FxIWs8KpMEyfvv288SJQy2ydaMcJt2njPxFes4MVLm7hslYI/\nbhFdvtxsu0PRu3cPliM7CFrE41r7CDKjd9v8d00mIAmaqz4TM2cCn3+eXQi/iLiQlm++cc5OfvlL\nbnMxE//qK55lL0jGB591FgvEypVOr4OqKudMvH9/9kf+b0pUgtPNUIiP3XYz2+60utpWPncu52O3\nOf/8+HKMaBG/4454+tttt8xFP2wR79TJpGeIGkSleecdboMUSNaIiAtp6d3bWXVFi6fblSwOzj2X\nW+0x8OKL3B54oHH7evFFXryyZ+LTpgGvvmoCSez83WEWioTM6Ax+gDPPyOmnm9/DwIE8S3Y/DemE\nU1HRIm5X/4nChx96TwRsHngAGDYMeOIJfjr89FPe7+ctFZYFCzIfoxERFzJipy7VIq4FNk6GDXOe\nQwv111+bY446yjkT1/bXTZuMiWfYMDPb84skFKLhDpqqqOCnoaoqDu4BgDVruI2zbqiNFvHTToun\nv622ypwT/PjjOd/4ueeaCcJdd8VzfiBcemKZpwgZsauL5NIrRbuH6XMceCB7mGhateJHd3smrgVj\n9mwzI2vf3thJR4+GkAPci256AVPfPG+80YS026YXALj66njGoP9e8vW0pZ8Yf/vb+PoMY2qSmbiQ\nEbvk1ebNPBsB4hd0LQS63169nO9rH157Jn7vvan92B4EcfkkC4ZTTvHOWeJm8GCT6RAwPtXZ1ClN\nR7pamM2BDnCKs7i17uuss4J/RmbiQmDWr+dHyVdeYRv0hg3xVTMBgLVrudUC7V681OiZ+OTJqXky\nTjwxN8WGBUO6xGDV1WaGmkiYRc4VK9h2fu+90ZNw2ecC8i/icbLvvmyqevrp4J8RERcyonNkfPEF\ntzvswP+IGzYED0gIgp7xX3MNcPHFThG3q55XVfEMb5E7lybYRTHOmZGQHfvsw3lE1q1jEdcCq13w\n4vLiAIyIp0tYlUtyIeJ//3sOyrMJ5Yu2K+s/qk2bTPKitm15pT7OiiYPP8zthx+yfdsWcTt73KpV\nRsCJgCcuPz3TAAAXzElEQVSfNO/Z/wDZZLMT4kFH1T7xhFPEc0G+Z+K5mCxUVGR/UxIRF3xJJNiz\noKGBfbU3buSk9UT8M2qUSTYVFa+bwYQJxjZop/rUvrQA/9EfdJB5fc013O68c2oeDyH36PWI7bfn\nPOKlLOJhS9LFjYi44Et9Pc+4q6u5cvv8+Sa4YdYsbv/613jOpe3hbuHVQT52/hO7JmMiwWM64gh+\nPWIEt9Onc6i+0PzstRcXTli1KremrXybUwolN72IuODLihVsy9TCvW6dEVO9T5ezisratVziyu1N\nohML2YWLt97a6bnSurVJQao/X1MT76KrEBy7PFouTVq677DFsKMiIi4UJEqxXXPePM6B0bWrCdr4\n3e+AMWN4e9kyM+uNg7Vr2b/b7baow+btR2YiXgTVlWFat07NCy0UBrmsvtiyJbsxRqkQFIXHH3cm\n+8oXIuKCgxkz2Ad4p51YxLt08Z5xtG0LnHlmfOedNYtt7u4KKZWVHFbv9Y+qQ5NbthQRLyT0ugQQ\nX55tP95+O1qFoCj06QMceWR+zm0jIi440PlLACPimn33NR4kgHEbc+dYDsM557ALo55x62ICRJzz\nOR1EbIONo1itEJ3LL8/3CMoLEXHBwamnmu3Vq7m6t04dWl8P9Oxp3tfbcSwgHn00V7K/9lp+PXQo\nnz9dqSyb4cOBDz6IPg4hOjoB1nff5Xcc5YKIuOCgvp5zXgAceNChA2ed23FHto3bC1Xa8yCOhaU3\n32RPlOuu40r1QObH5HXrUvOHC/lHi3i+bNXlhoi44GD2bE4nCrCY6so4y5dz9XgvbwM9e47KP/7B\n5pSgaT3btQNuuimecwvx0b078Oij+R5F+SBh94KDsWPZx1pz6KHcbtjAbS5dxvLlKibES4sWwM9/\nnu9RlA8yExdSuPlmYwfXtQc1ufQE0I/hgiAER0RccHDggSzcWsTtqi1AasIr7Ymgk2NlS1OTmd0/\n/ni4PgShnBERFxw0NLBdWptP3O597jDqW27hNqyf9vPPm7qFO+4Yrg9BKGdExAUHOgd0nz7Bjq+u\nZhuozn2SLWPHmm236UYQhMyIiBcA/foBH3+c71EwDQ0s4j/7mXOhccAA//qDTU3ZVSKxWb6cW3fF\ndEEQgiEiXgDMnMl+0vlg4UKTD6WpyYg4kTNfycSJzoLFNqeeGj78WAfoZFPJRBAEg4h4gZAv97q7\n7uIcKE1NnAFw7lzv/MwdOpgwezcHHGCyCIYlU2i9IAjeiIjnGe2Hna+0lvq8dk6UbPMzt2ljFkKz\nZaeduJXoPkEIh4h4npk0iVu7onxzkUgA99zD27ZwN6eIr1/PpdakLqYghENEvED4+9+dhQ+aA9st\nUIfXA9mXu2rThvOrhAnW+eEH/wVTQRAyIyKeJ6ZMSZ19btzImfuaC/0UAJhya0C4mfiKFbz4mQ2N\njXzNzX3zEoRSQkQ8DxABAwd6v3f22c03jqFDzfZzz5ntMCIOmLzijY3A739vTDV+1NezgLeQv0JB\nCI38+xQAugQZ4Ew+lUvsMmjz5zvfq6nJri+luP3kE26vvhq49Vbgt79l/+8LL/T+3Pr1qWH9giBk\nh4h4ATB9uqncHTQNaxQaGpxmG1tIlcp+ZqzdA/ffn9s77jDvrVsHPPKI9+c2bBARF4SoBPp3JaKh\nRDSLiOYQ0VU+x9QS0VQimklE78Y7zNJBz1rdbNoEjB7NRYr9jknHFVewm2BTE5c6228//2MHD+ZS\nawDwl7+kVpjPlq22Ai67jGf3fmP/6qvUfTITF4ToZPRDIKIWAO4HcASAxQCmENErSqlZ1jHtATwA\nYIhSahER+YSFCLoQ8IcfctuypXlvm224HTECuPPO4H02NHDQDgDsvHPm6Ed7QfPii42XSpSoSV2s\neNEi7/cHDEjNryIiLgjRCTITHwhgrlJqvlKqAcBzAE50HXMmgLFKqUUAoJRaGe8wSwe9+HfggcBB\nB5lSaABw/PHcakEOii5nBgCHH262P/4YeOed9J+tqjLmk223ze68No2NwHvvGY+b/v2d7/funfqZ\n9evNoqggCOEI4hHcA8AC6/VCsLDb7AygKmlGaQvgXqWUZMPw4LzzuI0ruKWuzincNkceyX7YfiaO\nK67gVj8N9O0bfhzaDp5IcDtlCvueX3wxcOKJ5gnERmbighCduBY2KwHsC+AYAEMBXE9EAZOZlg/u\nggpePPEEMGhQ8D7vu8//vR9+8N4/cCCwww5GeCsrWejjCH1PJDgveEUF0KsX8NprfJOwvWE0y5f7\n52MRBCEYQWbiiwDYmZ57JvfZLASwUim1CcAmInofwN4AUpazRo4c+b/t2tpa1NbWZjfiImbNGm6f\necb/mL59swthX7Ikdd/LLwMnnZS6v6kJePJJvplYv4ZYeOIJrqu4enVqBGbLlt4JvjZuFHOKIHhR\nV1eHurq6YAcrpdL+AKgAi3EvAC0BTAOwm+uYXQFMSB7bGsBnAHb36EuVM337KgUotXy5/zFTp/Ix\nixYF65Pn0PzTv79Sv/oV7999d7Nfs2gRvyZSauLE8NfhxUcfcb+XXKLUMcc43xszxjkOzXHHKXX+\n+fGOQxBKkaR2emo0qQD+bEQ0FMA9YPPLo0qpUUQ0PNnxw8ljRgA4H0AjgEeUUikP+kSkgpyvVNF2\n8MbG9L7Y+rggX5U+tn17M9MH2JyhMxTqflauBLp04e0PPgAOOST42DMxfTqwzz68vddezqClBQu4\nao/7erK5TkEoZ4gISinPlbRAqY6UUm8A2MW1b7Tr9R0A7oDgydtvm+04w8yrq9ll0L0ouXo18Oij\nppAxwMUn7M/FiR116jbldO9u7O5auL//ntunnop3HIJQbkjEZjMxYgS3f/1r5mOPOALo2jVYvx06\nAD16pJY322orDne3k0v9/OdmO2oRBze2t43Oka6pquIFTzvg58EHuU0XlCQIQmZExJsJ7fnh5zFi\n88IL3t4cNkoBn38OrFoFtGvnfUyrVrx4qM0VdrrZuEUcMDlX/BJo2eYefdPZYYf4xyEI5YSIeDPw\nzTfAq6/yDFv7ZqejY0cWvFGjvN9fuJBNMnvuyTNcP1/rFi1YwHWAke0hkovCxNrTxKtK0UEH8Y3p\no484u+G11/K+Vq3iH4cglBMi4s3Apk3c2hGN6dDH2HZ0m1/+Mrvzn3CC8/Vnn+Um/euUKdx6+cMn\nEmyTHzmSsxv27g385Cfxj0EQyo0sa7gIYdB+37feGvwzxxzDoflezJuX3fnnzjWRlN9/HyzoKAw7\n7shZC73MO1OmGJEH2Ly06665GYcglBMyE88xSpkUrdmE2g8enJowSjNnTvB+dG6Wl1/mNlcCrvGz\nz7vZsiX74hOCIKQiIp5jPv7YbNtueJmoqck+EZYXo5OOoKeeGr2vKDz7rPN1Q4Mzg6MgCOEQc0qO\n0SaRE08Ejjoq+Od00eHGxtR83507A6+8wt4pdrFjL3Te8HzTrZvz9ZIlMhMXhDgIFLEZ28nKMGLz\nxz/mIsSzZmU385w3D9hpJ7anuwsJ9+4NTJjA7wdBm3F22cVZELk5SSRSRXvyZOCAA/IzHkEoJiJH\nbArhmDEDGD/eZPLLht69OZBn82aniK9YwS6LnTtnP564ozSzodLjL03MKYIQHbGJe9C1K/t1R0Uv\nKgZd7HNTXW3cEzXaTh7Gz3vixHDjiAttWtLFJ8ScIgjRERH3YMWKVN/qMGi3vrAFIKqrU23ee+zB\nZpRs+tTeLPnO3f2f/7C3ziuv8GuZiQtCdETEfYgzHHz33cN9rqYmVcQbGrLPPti3L4ffx1VNKCr6\nCSUXAUeCUG6ITdyFFk13YYMwHHooh8aHsV8D3uaUjRvDharrvCaFgL6Z6CcVQRDCI3MhFzffzK1X\nxZygrFkDLF7MObtPOy18P599ZoJ07L7d3irFStBMjYIg+CMuhi5228244YUdqm22mDePw9Gj9KPH\n8d13XLfy5JOBsWPD9SkIQvGRzsVQZuIWiYTTjzpTOtgghBVwmyee4FansZWcI4IgaETELdavN9tt\n26bao7PFHWqeLddey+2oUWwLf/VVNqVok48gCIKIuMWqVdxWVJiCCtly001me9iwaOP59a+5XbsW\neOMN4JprgPr6wvEyEQQh/4iIW/Tpw+3KlezNEWYmPnIkt9mmi/WiWzfgkkvYFe/kk6P3JwhC6SEi\nnsQuItyhQ7CZ+NKlwN//ztv19dweeyy3cZU/GzKEPV0EQRC8EBFPMmQIt8cdx+2cOcBVV6X/zKOP\nAmedBXzxBZcmq60F/vUvfs+vZFq2uG8Gq1fH068gCKWBiHgSHdxz6aVm37hx3sc+9hjQr5+pXbnH\nHty+9x63X3yRmj42LLpupaZDh3j6FQShNBART6LNIXblGz/BnDjRaX5xE2fIfr9+3FZVcfZCQRAE\nGxFxAA8/DCxYwNu6eMHrr/vnuh4zhtvZs4FBg1Lfj7OCe0WFKQAR581BEITSQEQcwPDhZlunSe3e\nPXPo/ccfAzvvzNtDh+ZmbABnVOzbN3f9C4JQvJS9iH/2mdn+73+NLXubbfxFfM89zfY//sFJs8aP\nB+rqgKamnA1VEAQhhbIXcTu0Xi9QAkCXLuwJsmgRB9fMmGHes13+nn/e5MUeNEgCcQRBaF7KXsQ3\nbOD83Eo507VWVPBiYs+e/Prbb7ndtInD85ua+DNHHNHsQxYEQfgfZS/ikyf757W2g33efZfbJUvY\n1CIzbkEQCoGyEvEbbzQ5rL/5hoX4qqtYyDNx993cLl5sFj8FQRDyTVmJ+F13cf3MW24BPvzQ7L/h\nBu/jp0xJ3XfBBVwrUhAEoRAoq6IQtgnk8MOdJpLu3b0/c9ZZJj+KUtzHoEHsiSIIgtAcSFEID7SA\nA+nraT7zjHEb1DeBXPqEC4IgZENZzcTbtXMWfthvP/YNb2rKvFBpv79uHfclCILQHMhMPEmnTs5Z\n9+67BxNwALj1Vm6rquLLUCgIghCVQCJOREOJaBYRzSEi3wStRLQ/ETUQUcGUMCDiBcp33uFCw7vv\nzvtvuonLnAV1FbzgAm4bGsS9UBCEwiGjOYWIWgCYA+AIAIsBTAFwhlJqlsdxEwBsBPCYUuofHn01\nqznlzjuBESN40XLpUt63ahXPpsOYQ9zV5wVBEJqDqOaUgQDmKqXmK6UaADwH4ESP434N4CUAy0OP\nNGZ0oWIt4ACbVKLYsw89NNqYBEEQ4iSIiPcAsMB6vTC5738Q0bYATlJKPQSgIIwNb74JfPpp/P2e\ncUb8fQqCIISlMqZ+7gZg28p9hXykriQMoLa2FrW1tTENwYmXG+Dxx0frc+1a8UoRBCH31NXVoS5g\nMEoQm/iBAEYqpYYmX18NQCmlbrOO0bXdCcDWADYAuFApNc7VV7PYxJXiCvEAC/errwLTpgF7753z\nUwuCIMROOpt4kJn4FAB9iKgXgCUAzgAwzD5AKdXbOtnjAF51C3hzsm4dt3/+M5c3e/XV9AE9giAI\nxUpGEVdKNRLRJQDeAtvQH1VKfUlEw/lt9bD7IzkYZ2AWLuQFzT592DNFZyj0C6sXBEEoZkouYnPI\nEGDCBPYief993qdzngiCIBQj6cwpJSfitliLP7cgCKVAWYbdP+w28giCIJQgcbkYFgzbb89mlF69\n8j0SQRCE3JP3mfhbb8Vrr163TjxRBEEoH/Iu4v/8J7dbthhPkm7dWNiJ2NskKEoBP/wgIi4IQvmQ\ndxH/6U+53XFH4Nxzud7lciv7ynbbBe+rvp4r1ldUxDtGQRCEQiXv3ilNTelF98orgdtu83/fZvFi\nLvSweHGEQQqCIBQYBe2d0iLDCP785+B9iT1cEIRyI+8iDvh7knTpkl0/UjZNEIRyoyBEvLra+bqy\nErjhBq5/CTht5OlYswbo2DHesQmCIBQyBSHiVVXcjh/P7bhxXD5NL2oedliwflavBjp0iH98giAI\nhUreFzYBzjQ4cybQ2MiJqubMMWKcTUm0Ll2AlSsl3F4QhNKioBc2ASO6LVqw6cSeTffpE7yfpiZg\nt93iHZsgCEIhU1Ai7sU++3C7ZUv6Phobge+/B66/Pr5xCYIgFDoFIeJffOH/3q23crtggf8xiQQw\ndixvi01cEIRyoiBEPB19+gB77gm88YbZN2kS8NBD5nVVFXD66by9887NOz5BEIR8UhALm5kWL/X7\n334L9OhhvFmUctbTBNguLgUgBEEoJQp+YfPrr4G338583AknGAHXfPKJ2T7mGBFwQRDKi4IQ8d69\ngcGD078PAEuXmn2dOwOjRwP772/2XXBBbsYnCIJQqBSEOSXYZ9O//+mnQP/+oboWBEEoaEqixqYt\n4osXA9tu63xfAnwEQShVCt4mHoTDDzfb22zjfO+cc5p3LIIgCIVC0czE160D2rfnbaU4sOf224FR\no4D164E2bWIcqCAIQgFREuYUANi0CVi1it0MBUEQyoWSEXFBEIRypCRs4oIgCEIqIuKCIAhFjIi4\nIAhCESMiLgiCUMSIiAuCIBQxIuKCIAhFjIi4IAhCESMiLgiCUMSIiAuCIBQxIuKCIAhFTCARJ6Kh\nRDSLiOYQ0VUe759JRNOTP5OIqF/8QxUEQRDcZBRxImoB4H4ARwPYA8AwItrVddg8AIcppfYGcDOA\nR+IeaLFSV1eX7yE0K3K9pU25XS9Q+NccZCY+EMBcpdR8pVQDgOcAnGgfoJSarJRam3w5GYDkGUxS\n6H8AcSPXW9qU2/UChX/NQUS8B4AF1uuFSC/SFwAYH2VQgiAIQjAq4+yMiA4HcD6AQ+LsVxAEQfAm\nYz5xIjoQwEil1NDk66sBKKXUba7j9gIwFsBQpdTXPn1JMnFBEIQQ+OUTDzITnwKgDxH1ArAEwBkA\nhtkHENH2YAE/20/A0w1CEARBCEdGEVdKNRLRJQDeAtvQH1VKfUlEw/lt9TCA6wF0AvAgERGABqXU\nwFwOXBAEQWjm8myCIAhCvDRbxGamgKFigYgeJaJlRDTD2teRiN4iotlE9CYRtbfeu4aI5hLRl0Q0\nxNq/LxHNSH4fdzf3dQSFiHoS0TtE9DkRfUZElyb3l+Q1E1E1EX1ERFOT13tjcn9JXq+GiFoQ0adE\nNC75utSv99tkcOJUIvo4ua84r1kplfMf8M3iKwC9AFQBmAZg1+Y4dw6u5RAA+wCYYe27DcCVye2r\nAIxKbu8OYCrYbLVD8jvQTz8fAdg/uf06gKPzfW0+19sdwD7J7bYAZgPYtcSvuXWyrQDHPQws5etN\nju8yAM8AGFfqf9PJ8c0D0NG1ryivublm4hkDhooFpdQkAKtdu08E8GRy+0kAJyW3TwDwnFIqoZT6\nFsBcAAOJqDuAdkqpKcnjnrI+U1AopZYqpaYlt9cD+BJAT5T2NdcnN6vB/7gKJXy9RNQTwI8B/M3a\nXbLXm4SQaokoymtuLhHPNmCo2OiqlFoGsOgB6Jrc777uRcl9PcDfgaYovg8i2gH8FDIZQLdSveak\naWEqgKUAJiT/SUv2egH8BcDvwDcrTSlfL8DXOoGIphDRBcl9RXnNsQb7CP+j5FaLiagtgJcA/EYp\ntd7D579krlkp1QSgPxFtBeCfRLQHUq+vJK6XiI4FsEwpNY2IatMcWhLXa/EjpdQSIuoC4C0imo0i\n/R0310x8EYDtrdc9k/tKhWVE1A0Ako9Yy5P7FwHYzjpOX7ff/oKEiCrBAv60UuqV5O6SvmYAUEqt\nA1AHYChK93p/BOAEIpoHYAyAwUT0NIClJXq9AACl1JJkuwLAy2CTb1H+jptLxP8XMERELcEBQ+Oa\n6dy5gJI/mnEAzktunwvgFWv/GUTUkoh2BNAHwMfJR7W1RDQw6Vd/jvWZQuQxAF8ope6x9pXkNRPR\n1torgYhaATgKvA5QkterlPq9Ump7pVRv8P/lO0qpswG8ihK8XgAgotbJJ0sQURsAQwB8hmL9HTfj\navBQsGfDXABXN/cKbozX8SyAxQA2A/gOnCumI4CJyet7C0AH6/hrwKvZXwIYYu0fAP7DmQvgnnxf\nV5rr/RGARrBH0VQAnyZ/l51K8ZoB9Ete4zQAMwBcm9xfktfruvZBMN4pJXu9AHa0/p4/03pUrNcs\nwT6CIAhFjJRnEwRBKGJExAVBEIoYEXFBEIQiRkRcEAShiBERFwRBKGJExAVBEIoYEXFBEIQiRkRc\nEAShiPl/QII12VyNFfkAAAAASUVORK5CYII=\n", 146 | "text/plain": [ 147 | "" 148 | ] 149 | }, 150 | "metadata": {}, 151 | "output_type": "display_data" 152 | } 153 | ], 154 | "source": [ 155 | "pd.rolling_mean(loss, 100).plot()" 156 | ] 157 | }, 158 | { 159 | "cell_type": "code", 160 | "execution_count": 92, 161 | "metadata": { 162 | "collapsed": false, 163 | "scrolled": true 164 | }, 165 | "outputs": [ 166 | { 167 | "data": { 168 | "text/plain": [ 169 | "" 170 | ] 171 | }, 172 | "execution_count": 92, 173 | "metadata": {}, 174 | "output_type": "execute_result" 175 | }, 176 | { 177 | "data": { 178 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXEAAAEACAYAAABF+UbAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYVNW1/vHvaiYlgIgQIiAk0TSTASSKaBxKUWg0iNcR\nNEac4DFBjUZl8kobJ9BrHGMMzj+TyHWIilEvSLSiRhGQSaAbMBgERKI4okzS6/fHLkil7aGaru5z\nqvr9PE8/1HDq1JtGV457r322uTsiIpKbCqIOICIiu05FXEQkh6mIi4jkMBVxEZEcpiIuIpLDVMRF\nRHJYtUXczO43s/VmtqiKY+4wsxVmtsDM+mQ3ooiIVCaTK/EHgUGVvWlmg4F93f0HwCjgnixlExGR\nalRbxN39NeCTKg4ZCvy/1LFvAnuYWfvsxBMRkapkY0y8I7A67fna1GsiIlLHNLEpIpLDGmfhHGuB\nfdKed0q99g1mphu1iIjsAne3il7P9ErcUj8VmQb8DMDM+gOfuvv6yk60ebNz661Ou3bOeec5a9Y4\n7vH5mThxYuQZlFEZcyGfMtZfxqpk0mL4J+B1oNDM3jOzc8xslJmNBHD354F3zewd4PfAz6s6X7Nm\n8MtfwvLl0LYt9OoFV10Fn39eXRIRESkvk+6UM9y9g7s3c/fO7v6gu//e3aekHTPa3fdz997uPi+T\nL27dGiZNgvnzYc0aKCyEu+6CrVtr8z9HRKRhiXxis3NneOghmD4dnn0WevaEJ56Aav4Los4kEolo\nvrgGlDE74p4x7vlAGbOlNhmtuvGWbDIzr+77XnwRrrwyDLvcfDMcfng9hRMRiSkzwyuZ2IxdEQco\nK4M//QkmTIA+fcKwS/fu9RBQRCSGqirikQ+nVKSgAH76U1i2DI44IvyMGgXr1kWdTEQkXmJZxHfY\nbTf41a9CMW/VCvbfHyZOhC++iDqZiEg8xLqI79CmTRgff+stWLkydLLccw9s2xZ1MhGRaMVyTLw6\n8+eHyc/33gvj5SeeCFbZUiQRkRyXcxObmXCHGTNCMW/RIlypH3poVk4tIhIrOTexmQkzGDQI5s2D\nkSPh9NPh5JPDSlARkYYiZ4v4Do0awdlnh+Ldr1+4Gv/FL2B9pXdvERHJHzlfxHfYfXcYMwZKS6Fp\nU+jRA669Fr78MupkIiJ1J2+K+A5t28Ktt8KcObBkSehkufde+PrrqJOJiGRfzk5sZmrOnDD5uX59\n6GQZMkSdLCKSW/KyO6Um3OGFF0Ix39FzfvDB9R5DRGSX5GV3Sk2YwXHHwcKFMGJE6GI57TR4552o\nk4mI1E6DKOI7NGoE554bOln69IH+/eHii+HDD6NOJiKyazIq4mZWZGalZrbczMZU8H5rM/uzmS00\ns1lm1iP7UbOneXMYPx5KSsLz7t3hhhvgq6+izSUiUlOZbM9WANwFDAJ6AsPNrFu5w8YD8929N3A2\ncEe2g9aFdu3gjjtg1ixYsCB0sjzwAGzfHnUyEZHMZHIl3g9Y4e6r3H0bMBUYWu6YHsBLAO6+DPiu\nmbXLatI6tN9+8NhjYUehBx8MQy3PPx/d7kIiIpnKpIh3BFanPV+Tei3dQuAkADPrB3QGOmUjYH3q\n3x9eeQWuvz7cAvfoo2Hu3KhTiYhUrnGWzjMJuN3M5gFvA/OBCgcliouLdz5OJBKx2//ODE44IXSz\n3H9/eJxIhML+ve9FnU5EGoJkMkkymczo2Gr7xM2sP1Ds7kWp52MBd/fJVXzmXeCH7r6x3OuR9InX\nxsaN8JvfhLHzn/0sbBm3115RpxKRhqS2feJzgP3MrIuZNQWGAdPKfcEeZtYk9fgC4G/lC3iuatEC\nrr46LOHfvBm6dYPJk2HTpqiTiYhkUMTdfTswGpgBLAGmunuJmY0ys5Gpw7oDi82shNDFckldBY5K\n+/Zw993w2mswezZ07QoPP6xOFhGJVoNYdl8XXn8dLr883CXxpptg4EDdk0VE6kaDv3dKXXGHp56C\nceOgc+dQzA84IOpUIpJvGvy9U+qKGZx0EixeHP487jg46yxYtSrqZCLSUKiIZ0GTJnDhheGeLN/7\nHvTtC1dcAZ98EnUyEcl3KuJZ1LIl/PrX4cr888/DMv5bbgldLSIidUFFvA7svTf8/vfwt7+Fn27d\n4I9/hLKyqJOJSL7RxGY9eOWVMLyybVuY/DzmmKgTiUguUXdKDLiHG2yNGwc/+EFYMNSrV9SpRCQX\nqDslBszg1FNh6dLQxXLssXDOObB6dfWfFRGpjIp4PWvaFC66CFasgI4dw21vx42DTz+NOpmI5CIV\n8Yi0agXXXQeLFoXt4QoL4bbbYMuWqJOJSC5REY9Yx45w333w0kswc2bYKm7qVHWyiEhmNLEZMy+/\nHDpZzODmm8O9zEWkYVN3So4pKwvbxY0fDz16hE6Wnj2jTiUiUVF3So4pKIBhw6CkBAYMgKOOgvPP\nh7Vro04mInGjIh5jzZrBpZeGe7K0bRv6yq+6KizpFxEBFfGc0Lo1TJoE8+eHvvLCQrjrLti6Nepk\nIhK1jIq4mRWZWamZLTezMRW838rMppnZAjN728xGZD2p0Llz2E1o+nT4y1/COPkTT4TVoCLSMGWy\nUXIBsBwYALxP2HNzmLuXph0zDmjl7uPMrC2wDGjv7l+XO5cmNrNo5ky48sqwgOjmm+Hww6NOJCJ1\nobYTm/2AFe6+yt23AVOBoeWOcaBl6nFLYEP5Ai7Zd8wxMHduWAF61lkwdGiYDBWRhiOTIt4RSL/D\nx5rUa+nuAnqY2fvAQvJwo+S4KiiAM8+E0lI44ojwM2oUrFsXdTIRqQ+Ns3SeQcB8dz/azPYFXjSz\nXu6+sfyBxcXFOx8nEgkSWs2SFbvtBr/6Vbip1o03wv77w+jRYTPnli2r/7yIxEcymSSZTGZ0bCZj\n4v2BYncvSj0fC7i7T0475i/Aje7+99TzvwJj3H1uuXNpTLye/POf8N//HcbNr7469Jk3aRJ1KhHZ\nFbUdE58D7GdmXcysKTAMmFbumFXAMakvaw8UAit3PbLU1ne/C488As89B08+Ga7Mn3pKnSwi+Saj\nZfdmVgTcTij697v7JDMbRbgin2JmewMPAXunPnKjuz9awXl0JR4Bd5gxI9yTpWVL+J//gUMOiTqV\niGRK904RALZvD1fnV18NBx0Uxs4LC6NOJSLV0b1TBIBGjWDECFi2DPr1g0MPhV/8AtavjzqZiOwq\nFfEGaPfdYcyY0JbYtGlY+XnttfDll1EnE5GaUhFvwNq2hVtvhdmzw96fhYUwZQp8rWVaIjlDY+Ky\n09y5YfLzgw/CPcyHDAmbU4hItDSxKRlzh+efD8MtbdqEe7IcfHDUqUQaNk1sSsbM4PjjYeHCMAl6\n8slw2mnwzjtRJxORiqiIS4UaNYJzzw0bUvTuDf37w8UXw4cfRp1MRNKpiEuVmjeHCRP+fXfE7t3h\nhhvgq6+izSUigYq4ZKRdO7jjDpg1K+wwVFgIDzwQFhCJSHQ0sSm7ZNas0Mny6aehk2XwYHWyiNQV\ndadInXCHZ58NnSwdOsBNN8GPfhR1KpH8o+4UqRNmcMIJ8PbbcPrpoa/8jDPg3XejTibScKiIS601\nbgwjR4ZOlm7d4MAD4bLLYMOGqJOJ5D8VccmaFi3CHRKXLoXNm0NBnzwZNm2KOplI/lIRl6xr3x7u\nvhteey3cl6VrV3j4YXWyiNQFTWxKnfv73+HKK2HjxjD5OWhQ1IlEckutu1NSO/vcxr939plc7v3L\ngTMBB5oA3YG27v5pueNUxBsod3j6aRg7Fjp3DsX8gAOiTiWSG2pVxM2sAFgODADeJ+y5OczdSys5\n/ifAL939mAreUxFv4LZtg3vvDfcvP+YYuO466NIl6lQi8VbbFsN+wAp3X+Xu24CpwNAqjh8OfGN/\nTRGAJk3g5z8PnSzf/z707RsWDX3ySdTJRHJTJkW8I7A67fma1GvfYGa7A0XAk7WPJvmsZUu45hpY\nvBg++yws47/lltDVIiKZa5zl8w0BXis/Fp6uuLh45+NEIkEikchyBMkle+8ddhP65S9h3Di48064\n/noYPhwK1DslDVQymSSZTGZ0bCZj4v2BYncvSj0fC3j5yc3Ue38GHnP3qZWcS2PiUqVXXw3DK1u3\nhg0pBgyIOpFI9Go7sdkIWEaY2FwHzAaGu3tJueP2AFYCndy9wuUdKuKSCXd48snQyfKDH4QFQ716\nRZ1KJDq1mth09+3AaGAGsASY6u4lZjbKzEamHXoiML2yAi6SKTM45ZSw8vP442HgwLDL0OrV1X5U\npMHRYh+Jvc8/D33lv/sdXHBBuEJv3TrqVCL1R3cxlJzWqlXoJ1+0CD76KHSy3HYbbNkSdTKR6KmI\nS87o2BHuuw9eeglmzoQePeB//xfKyqJOJhIdDadIznr55dDJYhY6WdStKvlKO/tI3iorg8ceg/Hj\noWdPmDQp/CmSTzQmLnmroACGDYOSktBTftRRcP75sHZt1MlE6oeKuOSFZs3Cqs/ly6Ft29BXftVV\nobNFJJ+piEtead06DKnMnw9r1oROljvvDCtARfKRirjkpc6d4aGHYPp0eO650Mny+ONhNahIPtHE\npjQIM2eG3YWaNg2dLIcfHnUikcypO0WE0Mny6KMwYQL07h2GXbp3jzqVSPXUnSJC6GQ580woLQ1X\n4kccAaNGwbp1UScT2XUq4tLg7LYbXH45LFsWlvTvvz9MnAhffBF1MpGaUxGXBqtNmzA+/tZb8I9/\nhE6We+4J+4CK5AqNiYukzJsXJj/XrIEbb4QTTwxL+kWipolNkQy5w4wZoZi3bBmu1A85JOpU0tDV\nemLTzIrMrNTMlpvZmEqOSZjZfDNbbGYv1yawSFTMYNCgcFV+wQVw+ulw8slhJahIHGWyPVsBsJyw\nPdv7wBxgmLuXph2zB/A6MNDd15pZW3f/qIJz6UpccsqmTXDHHWFTir/+Ffr0iTqRNES1vRLvB6xw\n91Xuvg2YCgwtd8wZwJPuvhagogIukot23x3GjIGiIli8OOo0It+USRHvCKTvbrgm9Vq6QqCNmb1s\nZnPM7KxsBRSJgw4d4P33o04h8k2Ns3ievsDRwLeAN8zsDXd/J0vnF4lUhw6walXUKUS+KZMivhbo\nnPa8U+q1dGuAj9x9M7DZzF4BegPfKOLFxcU7HycSCRLajkVyQIcO8MYbUaeQhiKZTJJMJjM6NpOJ\nzUbAMsLE5jpgNjDc3UvSjukG3AkUAc2AN4HT3X1puXNpYlNy0quvwrhx8NprUSeRhqiqic1qr8Td\nfbuZjQZmEMbQ73f3EjMbFd72Ke5eambTgUXAdmBK+QIukss0Ji5xpcU+Ihn46quwTH/TJq3ilPqn\nuxiK1FLz5qHd8JNPok4i8p9UxEUypCEViSMVcZEMqYhLHKmIi2RIRVziSEVcJEMq4hJHKuIiGVIR\nlzhSERfJkIq4xJGKuEiGVMQljlTERTKkIi5xpBWbIhnasiVs2bZ5MxTo8kfqkVZsimRBs2awxx7w\nkbY8kRhRERepAQ2pSNyoiIvUgIq4xI2KuEgNqIhL3KiIi9SAirjEjYq4SA2oiEvcZFTEzazIzErN\nbLmZjang/SPN7FMzm5f6uSr7UUWipyIucVPt9mxmVgDcRdhj831gjpk94+6l5Q59xd1PqIOMIrGh\nIi5xk8mVeD9ghbuvcvdtwFRgaAXHadMqyXsq4hI3mRTxjsDqtOdrUq+Vd4iZLTCz58ysR1bSicRM\n+/Zhsc/XX0edRCTI1sTmW0Bnd+9DGHp5OkvnFYmVxo1hr73gX/+KOolIUO2YOLAW6Jz2vFPqtZ3c\nfWPa4xfM7G4za+PuH5c/WXFx8c7HiUSCRCJRw8gi0doxpNKhQ9RJJF8lk0mSyWRGx1Z7AywzawQs\nI0xsrgNmA8PdvSTtmPbuvj71uB/wmLt/t4Jz6QZYkvOGDIHzz4ehFc0MidSBqm6AVe2VuLtvN7PR\nwAzC8Mv97l5iZqPC2z4FOMXMLgS2AZuA07MXXyReNLkpcaJb0YrU0DXXhInNa6+NOok0FLoVrUgW\n6Upc4kRFXKSGVMQlTlTERWpIRVziREVcpIZUxCVONLEpUkNlZbDbbvDFF2HLNpG6polNkSwqKAjL\n7z/4IOokIiriIrtEQyoSFyriIrugUyd47DHYvDnqJNLQqYiL7IKbboJ334Vu3eAPfwjj5CJR0MSm\nSC28+ipccQVs2RIK+7HHRp1I8lFVE5sq4iK15A5PPAHjx8O++8LkydC7d9SpJJ+oO0WkDpnBqafC\nkiXwk5/AoEEwYgSsXl3tR0VqTUVcJEuaNoXRo2H58jDx2acPjB0Ln34adTLJZyriIlnWqhVcdx0s\nWgQbNkBhIdx2Wxg3F8k2FXGROtKxI9x7L7z8MsycCd27w9Sp6mSR7NLEpkg9SSZDJwuETpajjoo0\njuSQWk9smlmRmZWa2XIzG1PFcQeZ2TYzO2lXw4rkq0QC3nwTLr8czjsPjj8eFi+OOpXkumqLuJkV\nEHawHwT0BIabWbdKjpsETM92SJF8UVAAp58OJSUwcCAMGBAK+tq11X9WpCKZXIn3A1a4+yp33wZM\nBSraIvYi4AngX1nMJ5KXmjWDSy6BZcvg29+GXr1gwgT47LOok0muyaSIdwTSO17XpF7bycw6ACe6\n+++ACsdtROSbWreGG2+EBQvC1XhhIdx5J2zdGnUyyRXV7nafoduA9LHySgt5cXHxzseJRIJEIpGl\nCCK5a5994KGHQlvimDFw++1www1hEZHpsqjBSSaTJJPJjI6ttjvFzPoDxe5elHo+FnB3n5x2zMod\nD4G2wJfASHefVu5c6k4RycDMmXDlldCkCdx8MxxxRNSJJEq1uneKmTUClgEDgHXAbGC4u5dUcvyD\nwLPu/ucK3lMRF8lQWRk8+mgYK+/dGyZNCr3m0vDUqsXQ3bcDo4EZwBJgqruXmNkoMxtZ0UdqlVZE\ngNDJcuaZUFoarsSPPBJGjYJ166JOJnGixT4iOeKTT8Ik6P33h3u0XH45tGwZdSqpD7qLoUge2HPP\nsNJz3jxYuTJ0stx9N2zbFnUyiZKKuEiO6dIFHnkEnn8ennoK9t8//Kn/yG2YNJwikuNmzAj3ZGnR\nInSyHHpo1Ikk2zScIpLHBg4MQywjR8KwYXDSSWElqDQMKuIieaBRIzj77FC8+/eHww6Dn/8c1q+P\nOpnUNRVxkTyy++5hkVBpabg/S48e8Otfw8aNUSeTuqIiLpKH9toLbr0V5swJBb2wEKZMga+/jjqZ\nZJsmNkUagLlzw+TnBx/A5MkwZIjuyZJLarXsPstBVMRFIuIOL7wQhlvatAmdLAcfHHUqyYS6U0QE\nMzjuOFi4EM45B045BU47Dd55J+pkUhsq4iINTKNGoYgvWwYHHBC6WS6+GD78MOpksitUxEUaqObN\nYdy4sFWcWbhD4vXXw1dfRZ1MakJFXKSBa9cubEIxa1bYlKKwEB54ALZvjzqZZEITmyLyH958M3Sy\nfPxxuOHW4MHqZImaulNEpEbc4dlnYexY+M53QjE/8MCoUzVc6k4RkRoxgxNOCMMrw4fD0KHhz5Ur\nq/+s1K+MiriZFZlZqZktN7MxFbx/gpktNLP5ZjbbzH6c/agiUt8aN4YLLoDly8PE50EHwaWXwoYN\nUSeTHaot4mZWANwFDAJ6AsPNrFu5w2a6e293PwA4D7gv60lFJDLf+hZcfTUsXQpbt0LXrmHl56ZN\nUSeTTK7E+wEr3H2Vu28DpgJD0w9w9/SmpBZAWfYiikhctG8Pv/0t/P3vMHt2KOYPP6xOlihlUsQ7\nAqvTnq9JvfYfzOxEMysBngXOzU48EYmjrl3hySdh6tRwY62+fWH6dO0uFIXG2TqRuz8NPG1mhwHX\nAcdWdFxxcfHOx4lEgkQika0IIlLPDj0UXnsNnnkmrPrcZ5/QydK3b9TJclsymSSZTGZ0bLUthmbW\nHyh296LU87GAu/vkKj7zD+Agd/+43OtqMRTJU9u2wX33hfuXDxgQVn926RJ1qvxQ2xbDOcB+ZtbF\nzJoCw4Bp5b5g37THfYGm5Qu4iOS3Jk3gwgtDJ8t++4Wr8csvD4uGpO5UW8TdfTswGpgBLAGmunuJ\nmY0ys5Gpw042s8VmNg+4EzitzhKLSKy1bAnFxbB4MXzxRRg/v+UW2Lw56mT5SSs2RaROlZSEG20t\nWBCGWIYPhwItM6wRLbsXkci9+mq4J8vWrWFDigEDok6UO1TERSQW3ENr4tixYdx88mTo3TvqVPGn\ne6eISCyYhR2Fli6Fn/wEBg6EESNg9epqPyqVUBEXkXrXtCmMHg0rVkCnTtCnD4wZA59+GnWy3KMi\nLiKRadUKrrsu3C1xw4awIcWtt8KWLVEnyx0q4iISuY4dw0Khl1+Gl14Kd0x89FEo012YqqWJTRGJ\nnWQydLJAWMZ/1FGRxomculNEJOeUlcFjj8H48dCjB0yaBPvvH3WqaKg7RURyTkEBDBsWFgsdeywc\nfTScdx6sXRt1snhREReRWGvWDC65JNyTpV076NULJkyAzz6LOlk8qIiLSE5o3ToMqcyfH67GCwvh\nzjvDCtCGTEVcRHJK587w0EPw4ovw3HNhvPzxxxvuhhSa2BSRnDZzJlx5ZVhAdPPNcPjhUSfKPnWn\niEheKysLW8VNmBDGzCdNCr3m+ULdKSKS1woK4IwzoLQUjjwy/IwaBevWRZ2s7qmIi0jeaNYMLrsM\nli2DPfYIfeVXXx02p8hXGRVxMysys1IzW25mYyp4/wwzW5j6ec3Mfpj9qCIimdlzz7DSc948ePfd\n0Mly991hH9B8k8lGyQXAcmAA8D5hz81h7l6adkx/oMTdPzOzIsLGyv0rOJfGxEWk3s2fHyY/33sv\njJefeGK4LW6uqNXEZqpAT3T3wannVe52b2atgbfdfZ8K3lMRF5HITJ8einmLFqGT5dBDo06UmdpO\nbHYE0m/Zvib1WmXOB17IPJ6ISP0YNCgMsYwcGZb0n3RSGD/PZY2zeTIzOwo4BzissmOKi4t3Pk4k\nEiQSiWxGEBGpUqNGcPbZcNppcMcdcNhhcOqpMHEitG8fdbogmUySTCYzOjbT4ZRidy9KPa9wOMXM\negFPAkXu/o9KzqXhFBGJlQ0b4Prr4eGHwz1aLrssDLfESW2HU+YA+5lZFzNrCgwDppX7gs6EAn5W\nZQVcRCSO9toLfvMbmDs39JkXFsKUKfD111Eny0xGKzZTHSe3E4r+/e4+ycxGEa7Ip5jZvcBJwCrA\ngG3u3q+C8+hKXERibe7cMPm5bh1MngxDhkTfyaJl9yIiNeAO//d/oZjvuWfoZDn44OjyaNm9iEgN\nmMHgwbBgAYwYASefHCZC33kn6mTfpCIuIlKJRo3g3HPDhhR9+kD//nDRRfDhh1En+zcVcRGRajRv\nHvb6LCkJV+ndu8MNN8BXX0WdTEVcRCRj7dqF3vJZs2DhwtDJ8sADsH17dJk0sSkisovefBOuuAI+\n/jjccGvw4LrpZFF3iohIHXGHv/wFxowJKz5vugkOOii736HuFBGROmIWeskXLQobUwwdCsOHw8qV\n9fP9KuIiIlnQuDFccEHoZOnePVyNX3ppWNZfl1TERUSyqEWLsJvQ0qWwdSt07RpWfm7aVDffpyIu\nIlIH2reH3/4WXn8d5swJxfzhh7PfyaKJTRGRevD666GTZePGMPk5cGDmnSzqThERiQF3ePppGDsW\nOncOxfyAA6r/nLpTRERiwAz+679g8eKwq9Bxx8FPfwr//Oeun1NFXESknjVpAhdeGDpZ9t0XfvSj\nMNTyySc1P5eKuIhIRFq2hGuuCVfmX3wRJj9vuQU2b878HBkVcTMrMrNSM1tuZmMqeL+rmb1uZpvN\n7LLMv15ERPbeG+65B/72N3jlFejWDf7wBygrq/6z1RZxMysA7gIGAT2B4WbWrdxhG4CLgJtrnD5m\nMt2cNErKmB1xzxj3fKCM2bIjY/fu8Mwz8MgjcNddcOCBMHNm1Z/N5Eq8H7DC3Ve5+zZgKjA0/QB3\n/8jd3wJyZFe6yuXSX3icKWPtxT0fKGO2lM94+OHwxhvh9rcXXlj1ZzMp4h2B1WnP16ReExGROmIG\np5wCS5ZUfZwmNkVEYqxp06rfr3axj5n1B4rdvSj1fCxhl/vJFRw7EfjC3X9Tybm00kdEZBdUttin\ncQafnQPsZ2ZdgHXAMGB4FcdXupC0shAiIrJrMlp2b2ZFwO2E4Zf73X2SmY0iXJFPMbP2wFygJVAG\nbAR6uPvGuosuIiL1eu8UERHJrnqb2KxuwVAUzOx+M1tvZovSXtvTzGaY2TIzm25me0SYr5OZvWRm\nS8zsbTO7OIYZm5nZm2Y2P5VxYtwypmUtMLN5ZjYtjhnN7J9mtjD1u5wd04x7mNnjZlaS+ufy4Lhk\nNLPC1O9uXurPz8zs4rjkS8t5qZktNrNFZvZHM2tam4z1UsQzXDAUhQcJmdKNBWa6e1fgJWBcvaf6\nt6+By9y9J3AI8IvU7y02Gd19C3CUux8A9AEGm1m/OGVMcwmwNO153DKWAQl3P8Dd+6Vei1vG24Hn\n3b070BsoJSYZ3X156nfXF/gR8CXwVFzyAZhZB8LCyL7u3oswLzm8Vhndvc5/gP7AC2nPxwJj6uO7\nM8jWBViU9rwUaJ96/B2gNOqMadmeBo6Ja0agOWFu5KC4ZQQ6AS8CCWBaHP+ugXeBvcq9FpuMQCvg\nHxW8HpuMaZkGAq/GLR/QAVgF7Jkq4NNq++90fQ2n5NKCoW+7+3oAd/8A+HbEeQAws+8SrnRnEf6y\nY5MxNUwxH/gAeNHd58QtI3ArcAWQPgkUt4wOvGhmc8zs/NRrccr4PeAjM3swNWQxxcyaxyzjDqcD\nf0o9jk0+d38fuAV4D1gLfObuM2uTUYt9qhf5zK+ZtQCeAC7x0PFTPlOkGd29zMNwSiegn5n1rCBT\nZBnN7HhgvbsvoIoWWKL/u/6xh6GA4whDZ4dXkCnKjI2BvsBvUzm/JPxXdZwyYmZNgBOAx1MvxSaf\nmbUm3LakC+Gq/FtmdmYFmTLOWF9FfC3QOe15p9RrcbQ+1TKJmX0H+FeUYcysMaGAP+Luz6RejlXG\nHdz9cyBvpA5YAAABiUlEQVQJFBGvjD8GTjCzlcCjwNFm9gjwQYwy4u7rUn9+SBg660e8fo9rgNXu\nPjf1/ElCUY9TRoDBwFvu/lHqeZzyHQOsdPeP3X07Ycz+0NpkrK8ivnPBkJk1JSwYmlZP310d4z+v\nzqYBI1KPzwaeKf+BevYAsNTdb097LTYZzaztjpl0M9sdOBYoIUYZ3X28u3d29+8T/tl7yd3PAp4l\nJhnNrHnqv7gws28RxnTfJl6/x/XAajMrTL00AFhCjDKmDCf8n/UOccr3HtDfzHYzMyP8DpdSm4z1\nOKBfBCwDVgBjo5pYKJfpT8D7wJbUL/ccwoTDzFTWGUDrCPP9GNgOLADmA/NSv8c2Mcr4w1SuBcAi\nYELq9dhkLJf3SP49sRmbjITx5h1/z2/v+HckThlTeXoTLsoWAH8G9ohTRsLk+odAy7TXYpMvlWci\n4UJnEfAw0KQ2GbXYR0Qkh2liU0Qkh6mIi4jkMBVxEZEcpiIuIpLDVMRFRHKYiriISA5TERcRyWEq\n4iIiOez/A5GYpgxZnKJ1AAAAAElFTkSuQmCC\n", 179 | "text/plain": [ 180 | "" 181 | ] 182 | }, 183 | "metadata": {}, 184 | "output_type": "display_data" 185 | } 186 | ], 187 | "source": [ 188 | "epsilon.plot()" 189 | ] 190 | }, 191 | { 192 | "cell_type": "code", 193 | "execution_count": 111, 194 | "metadata": { 195 | "collapsed": false 196 | }, 197 | "outputs": [ 198 | { 199 | "data": { 200 | "text/plain": [ 201 | "" 202 | ] 203 | }, 204 | "execution_count": 111, 205 | "metadata": {}, 206 | "output_type": "execute_result" 207 | }, 208 | { 209 | "data": { 210 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEACAYAAABPiSrXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X2QleV9//H3B/EhD8qsJmAFrTSKgpMm6IixTtoDCqhp\nfKjKD1PDkpBmrNKQJ1uJrexOzERpbcjMT21/8QlsZEVsInX8hQ2FnfbXwQCJiiIC01QLGNYG2JWY\nmQTk+/vjvlZuD7vsWfbhPH1eMwz3fs913+e+BPme63td93UUEZiZmZViWLlvwMzMqoeThpmZlcxJ\nw8zMSuakYWZmJXPSMDOzkjlpmJlZyXpNGpLGSXpe0s/S752SviSpQVKrpC2SVkoakTtnvqRtkjZL\nmpaLny9po6Stkhbl4sdJaknnrJV0Ru61xtR+i6RZufiZkp5Lry2VNHxg/pOYmVlPek0aEbE1IiZG\nxPnABcDbwA+A24FVEXEOsBqYDyBpAjADGA9cAdwvSelyDwBzImIcME7S9BSfA+yJiLOBRcDCdK0G\n4E7gQuAiYEEuOd0D3Juu1ZGuYWZmg6iv5anLgP+MiO3A1cDiFF8MXJOOrwJaIuJARLwGbAMmSToV\nODEi1qd2S3Ln5K+1HJiSjqcDrRHRGREdQCtweXptCvBU7v2v7WNfzMysj/qaNP4X8Hg6HhUR7QAR\nsQsYmeKjge25c3am2GhgRy6+I8Xec05EvAN0Sjq5p2tJOgXYGxEHc9c6rY99MTOzPio5aUg6lmwU\n8WQKFe8/MpD7kaj3JiW1MTOzAdSXyeMrgJ9GxC/Tz+2SRkVEeyo9vZniO4HTc+eNSbGe4vlz3pB0\nDHBSROyRtBMoFJ2zJiJ2SxohaVgabeSv9R6SvLmWmdlRiIjDPpz3pTx1I7A09/MKYHY6bgSezsVn\nphVRY4GzgHWphNUpaVKaGJ9VdE5jOr6BbGIdYCUwNSWIBmBqigGsSW2L3/8wEVFTvxYsWFD2e3Cf\n3Kda+lWL/epvn3pS0khD0vvJJsG/mAvfAyyT9HngdbIVU0TEK5KWAa8A+4Fb4tAd3Ao8CpwAPBsR\nP0rxh4DHJG0DdgMz07X2SvomsIGs/NUc2YQ4ZKu3WtLrz6drmJnZICopaUTEr4EPF8X2kCWS7tp/\nG/h2N/GfAh/tJv4bUtLp5rVHyRJNcfy/yJbhmpnZEPET4VWoUCiU+xYGnPtUHWqxT1Cb/RqsPulI\ntataIClqvY9mZgNNEtHPiXAzM6tzThpmZlYyb/JnZmbvioBnnun5dScNMzMDYNMmuPVW6OzsuY3L\nU2Zmde6tt+BrX4NCAW64ATZs6Lmtk4aZWZ2KgMcfh/HjYe/eQyONY47p+RyXp8zM6tDLL8PcuVkp\n6skn4Q/+oLTzPNIwM6sjXaWoyZMPlaJKTRjgpGFmVhfypag9e0orRXXH5SkzsxqXXxXVl1JUdzzS\nMDOrUd2tiupPwgAnDTOzmhMBS5f2bVVUqVyeMjOrIZs2ZauiOjpg+XK4+OKBvb5HGmZmNWDfPvj6\n17NS1HXXZaWogU4Y4KRhZlbVukpR5557aFXU3LkDU4rqjstTZmZVarBLUd3xSMPMrMrkV0UNZimq\nO04aZmZVortVUYNZiuqOy1NmZlVgIB/Q6w+PNMzMKli+FHX99bB+ffkSBjhpmJlVpO72ipo7F4aX\nuT5UUtKQNELSk5I2S9ok6SJJDZJaJW2RtFLSiFz7+ZK2pfbTcvHzJW2UtFXSolz8OEkt6Zy1ks7I\nvdaY2m+RNCsXP1PSc+m1pZJcajOzmrBpU7YL7d/+bVaKeuQRGDmy3HeVKXWk8V3g2YgYD3wMeBW4\nHVgVEecAq4H5AJImADOA8cAVwP2SlK7zADAnIsYB4yRNT/E5wJ6IOBtYBCxM12oA7gQuBC4CFuSS\n0z3AvelaHekaZmZVazD2ihpovSYNSScBn4yIRwAi4kBEdAJXA4tTs8XANen4KqAltXsN2AZMknQq\ncGJErE/tluTOyV9rOTAlHU8HWiOiMyI6gFbg8vTaFOCp3PtfW3KvzcwqyNF8g165lFLSGQv8UtIj\nZKOMDcCXgVER0Q4QEbskdQ2eRgNrc+fvTLEDwI5cfEeKd52zPV3rHUmdkk7Ox/PXknQKsDciDuau\ndVoJfTEzqyiVsiqqVKUkjeHA+cCtEbFB0nfISlNR1K745/5Q701KagNAU1PTu8eFQoFCodD3OzIz\nG0D79kFzMyxeDE1NcPPN5R1ZtLW10dbW1mu7UpLGDmB7RGxIPz9FljTaJY2KiPZUenozvb4TOD13\n/pgU6ymeP+cNSccAJ0XEHkk7gULROWsiYneanB+WRhv5ax0mnzTMzMopAlpass0Fp0/PRhqVMMld\n/IG6ubm523a9zmmkEtR2SeNS6FJgE7ACmJ1ijcDT6XgFMDOtiBoLnAWsi4hdQKekSWlifFbROY3p\n+AayiXWAlcDUlCAagKkpBrAmtS1+fzOzitS1KmrhwqwU9fDDlZEw+kIRvVeVJH0MeBA4Fvg58Dng\nGGAZ2QjhdWBGmqxG0nyy1Uz7gXkR0ZriFwCPAieQrcaal+LHA48BE4HdwMw0iY6k2cAdZOWvuyJi\nSYqPBVqABuB54KaI2N/NvUcpfTQzGyxvvZWVopYsqYxSVCkkERGHTQOUlDSqmZOGmZVL115Rt90G\n06bB3XfDqFHlvqvS9JQ0/ECcmdkgqLZVUaXyNiJmZgOo0vaKGmhOGmZmA6B4r6iXX66MvaIGWo11\nx8xs6NVqKao7HmmYmR2lffuy5y0qea+ogeakYWbWR/lv0Nu9u7L3ihpoLk+ZmfVB1/dadHRkpaih\n+m7uSuGRhplZCfKlqOuvz0pR9ZYwwEnDzOyIukpR555bf6Wo7rg8ZWbWg/yqqOXL63NkUcwjDTOz\nIt2tinLCyDhpmJklLkX1zuUpMzPq6wG9/vBIw8zqWn6vqHp5QK8/nDTMrC4V7xXlUlRpXJ4ys7rj\nUtTR80jDzOqGS1H956RhZjWva1XUhAmwd69LUf3h8pSZ1bT8XlHLlnlk0V8eaZhZTep6QG/y5EN7\nRTlh9J+ThpnVlPwDel3foOdS1MBxecrMaoZXRQ0+jzTMrOp1rYqaPNmrogZbSUlD0muSXpT0vKR1\nKdYgqVXSFkkrJY3ItZ8vaZukzZKm5eLnS9ooaaukRbn4cZJa0jlrJZ2Re60xtd8iaVYufqak59Jr\nSyV51GRWZ/LfoLd3r0tRQ6HUkcZBoBAREyNiUordDqyKiHOA1cB8AEkTgBnAeOAK4H5JSuc8AMyJ\niHHAOEnTU3wOsCcizgYWAQvTtRqAO4ELgYuABbnkdA9wb7pWR7qGmdWJTZuykcU992SlqIcfhpEj\ny31Xta/UpKFu2l4NLE7Hi4Fr0vFVQEtEHIiI14BtwCRJpwInRsT61G5J7pz8tZYDU9LxdKA1Ijoj\nogNoBS5Pr00Bnsq9/7Ul9sXMqlj+AT2vihp6pSaNAH4sab2kL6TYqIhoB4iIXUBXjh8NbM+duzPF\nRgM7cvEdKfaecyLiHaBT0sk9XUvSKcDeiDiYu9ZpJfbFzKpQvhTVtVfU3Lkw3IXpIVXqf+5LIuIX\nkj4MtEraQpZI8op/7g/13qSkNgA0NTW9e1woFCgUCn2/IzMrm/wDel4VNTja2tpoa2vrtV1JSSMi\nfpF+/x9JPwQmAe2SRkVEeyo9vZma7wROz50+JsV6iufPeUPSMcBJEbFH0k6gUHTOmojYLWmEpGFp\ntJG/1mHyScPMqse+fdDcDIsXQ1MT3HyzJ7kHS/EH6ubm5m7b9VqekvR+SR9Mxx8ApgEvASuA2alZ\nI/B0Ol4BzEwrosYCZwHrUgmrU9KkNDE+q+icxnR8A9nEOsBKYGpKEA3A1BQDWJPaFr+/mVW54gf0\nvFdU5VDEkatK6R/+H5CVn4YD34+Iu9OcwzKyEcLrwIw0WY2k+WSrmfYD8yKiNcUvAB4FTgCejYh5\nKX488BgwEdgNzEyT6EiaDdyR3v+uiFiSu68WoAF4HrgpIvZ3c//RWx/NrHLkS1H33edSVLlIIiIO\nmwboNWlUOycNs+rgUlRl6Slp+IlwMysrl6KqixermVnZeFVU9fFIw8yGXP4Bveuug/XrnTCqhZOG\nmQ2ZCHj8cT+gV838R2VmQ8KlqNrgkYaZDSqXomqLk4aZDQrvFVWb/MdnZgPOpaja5ZGGmQ2Yffvg\n618/VIrytuW1x0nDzPqtuwf05s71A3q1yOUpM+uXrie4Ozth+XK4+OJy35ENJo80zOyo5EtRN9yQ\nlaKcMGqfk4aZ9Yn3iqpvLk+ZWclcijKPNMysVy5FWRcnDTPrUb4UtXu3S1Hm8pSZ9SBfivIDetbF\nIw0ze4/8XlFdpSgnDOvipGFmQPfblrsUZcVcnjIzl6KsZB5pmNWx7lZFOWHYkThpmNUhP6BnR8vl\nKbM64wf0rD9KHmlIGibpZ5JWpJ8bJLVK2iJppaQRubbzJW2TtFnStFz8fEkbJW2VtCgXP05SSzpn\nraQzcq81pvZbJM3Kxc+U9Fx6bakkJ0CzI+gqRU2e7Af07Oj1pTw1D3gl9/PtwKqIOAdYDcwHkDQB\nmAGMB64A7pekdM4DwJyIGAeMkzQ9xecAeyLibGARsDBdqwG4E7gQuAhYkEtO9wD3pmt1pGuYWZHi\nB/ReftmlKDt6JSUNSWOAK4EHc+GrgcXpeDFwTTq+CmiJiAMR8RqwDZgk6VTgxIhYn9otyZ2Tv9Zy\nYEo6ng60RkRnRHQArcDl6bUpwFO597+2lL6Y1ZNNm7KRxcKFWSnqkUdg5Mhy35VVs1JHGt8BbgMi\nFxsVEe0AEbEL6PqrOBrYnmu3M8VGAzty8R0p9p5zIuIdoFPSyT1dS9IpwN6IOJi71mkl9sWs5u3b\nd/gDei5F2UDodR5A0qeA9oh4QVLhCE3jCK/1lXpvUlIbAJqamt49LhQKFAqFvt+RWRWIgJaWbO5i\n2rRspOGRhZWira2Ntra2XtuVMnl8CXCVpCuB9wEnSnoM2CVpVES0p9LTm6n9TuD03PljUqyneP6c\nNyQdA5wUEXsk7QQKReesiYjdkkZIGpZGG/lrHSafNMxqlR/Qs/4o/kDd3Nzcbbtey1MR8Y2IOCMi\nfg+YCayOiM8C/wLMTs0agafT8QpgZloRNRY4C1iXSlidkialifFZRec0puMbyCbWAVYCU1OCaACm\nphjAmtS2+P3N6oof0LOh1J+H++4m+wd9C3Bp+pmIeAVYRrbS6lnglojoKl3dCjwEbAW2RcSPUvwh\n4EOStgFfJluZRUTsBb4JbAB+AjSnCXFSm69K2gqcnK5hVje6VkWNH+9ty23o6NC/57VJUtR6H63+\nbNoEc+dCRwfcd59HFjbwJBERh80dexsRsyqSL0Vdf71LUTb0nDTMqkC+FOW9oqycvPWGWYXLr4pa\ntswjCysvjzTMKpS/Qc8qkZOGWYVxKcoqmctTZhXED+hZpfNIw6wC5EtR118P69c7YVhlctIwK6Pu\nSlFz58Jw1wCsQvmvplmZuBRl1cgjDbMh5r2irJo5aZgNkfw36HlVlFUrl6fMhkB+r6jly/2FSFa9\nPNIwG0Td7RXlhGHVzEnDbBC4FGW1yuUpswGWXxXlUpTVGo80zAZId6uinDCs1jhpmPWTS1FWT1ye\nMuuH/KooP6Bn9cAjDbOjkN8r6rrrvFeU1Q8nDbM+iIDHH/deUVa//FfdrETeK8rMIw2zXhVvW+69\noqyeOWmY9SC/bfnevYdKUV4VZfWs16Qh6XhJP5H0vKSXJC1I8QZJrZK2SFopaUTunPmStknaLGla\nLn6+pI2StkpalIsfJ6klnbNW0hm51xpT+y2SZuXiZ0p6Lr22VJJLbTZgNm2CKVNg4cLsAb2HH4aR\nI8t9V2bl12vSiIjfAJMjYiLwceAKSZOA24FVEXEOsBqYDyBpAjADGA9cAdwvSelyDwBzImIcME7S\n9BSfA+yJiLOBRcDCdK0G4E7gQuAiYEEuOd0D3Juu1ZGuYdYv+Qf0rrvOD+iZFSupPBURv06Hx5NN\nngdwNbA4xRcD16Tjq4CWiDgQEa8B24BJkk4FToyI9andktw5+WstB6ak4+lAa0R0RkQH0Apcnl6b\nAjyVe/9rS+mLWXe6e0DPpSizw5VU0pE0DPgp8BHgvohYL2lURLQDRMQuSV2D99HA2tzpO1PsALAj\nF9+R4l3nbE/XekdSp6ST8/H8tSSdAuyNiIO5a51WSl/MinlVlFnpSkoa6R/niZJOAn4g6Tyy0cZ7\nmg3gfan3JiW1AaCpqend40KhQKFQ6PsdWc156y1oboYlS2DBArj5Zj9vYfWrra2Ntra2Xtv16X+R\niHhLUhtZiai9a7SRSk9vpmY7gdNzp41JsZ7i+XPekHQMcFJE7JG0EygUnbMmInZLGiFpWEpo+Wsd\nJp80zCKgpSWbu5g2LRtpeJLb6l3xB+rm5uZu25WyeupDXZPPkt4HTAU2AyuA2alZI/B0Ol4BzEwr\nosYCZwHrImIX0ClpUpoYn1V0TmM6voFsYh1gJTA1JYiG9N4r02trUtvi9zfr0aZNMHlytirqySfh\nkUecMMz6QhFHripJ+ijZRPOw9OuJiPhWmnNYRjZCeB2YkSarkTSfbDXTfmBeRLSm+AXAo8AJwLMR\nMS/FjwceAyYCu4GZaRIdSbOBO8jKX3dFxJIUHwu0AA3A88BNEbG/m/uP3vpotS9fimpqykpRnuQ2\n65kkIuKwaYBek0a1c9Kob/lS1PTpcPfdHlmYlaKnpOFpP6tZXhVlNvC8jYjVnO6+Qc8Jw2xgOGlY\nzfA36JkNPpenrCb4G/TMhoZHGlbV8qUob1tuNvicNKwq5UtRu3e7FGU2VFyesqrjVVFm5eORhlWN\n/DfoeVWUWXk4aVjFi4DHH8++Qc+roszKy+Upq2guRZlVFo80rCLt3+8H9MwqkUcaVpG+8hV49VVv\nW25WaZw0rOLcfz+sXg1r18KIEb23N7Oh411uraKsWgU33QT/8R/wkY+U+27M6pd3ubWKt3Ur/Omf\nwhNPOGGYVSpPhFtF2LsXPv1puOuubPLbzCqTy1NWdvv3w5VXwnnnwaJF5b4bM4Oey1MeaVhZbdwI\nU6bAccfB3/1due/GzHrjpGFl0dkJX/4yXHZZNo+xYgUM9wybWcVz0rAhFQH/9E/ZliBvvw2vvAI3\n3+wtQcyqhT/b2ZDZuDH7oqS334Z//mf4xCfKfUdm1lceadigy5eibrwR1q1zwjCrVk4aNmjypahf\n/SrbEuTP/9ylKLNq1mvSkDRG0mpJmyS9JOlLKd4gqVXSFkkrJY3InTNf0jZJmyVNy8XPl7RR0lZJ\ni3Lx4yS1pHPWSjoj91pjar9F0qxc/ExJz6XXlkpyqa2CbNwIf/RH8J3vZKWoBx+ED3+43HdlZv1V\nykjjAPDViDgPuBi4VdK5wO3Aqog4B1gNzAeQNAGYAYwHrgDul9S11vcBYE5EjAPGSZqe4nOAPRFx\nNrAIWJiu1QDcCVwIXAQsyCWne4B707U60jWszFyKMqttvSaNiNgVES+k418Bm4ExwNXA4tRsMXBN\nOr4KaImIAxHxGrANmCTpVODEiFif2i3JnZO/1nJgSjqeDrRGRGdEdACtwOXptSnAU7n3v7bUTtvA\n625VlEtRZrWnTyUdSWcCHweeA0ZFRDtkiUVS1wbWo4G1udN2ptgBYEcuviPFu87Znq71jqROSSfn\n4/lrSToF2BsRB3PXOq0vfbGB89JL2RcleVWUWe0rOWlI+iDZKGBeRPxKUvHeHAO5V8dhj64fZRsA\nmpqa3j0uFAoUvLnRgOjshKYm+P73obkZvvhFjyzMqlVbWxttbW29tispaaRJ5uXAYxHxdAq3SxoV\nEe2p9PRmiu8ETs+dPibFeornz3lD0jHASRGxR9JOoFB0zpqI2C1phKRhabSRv9Zh8knD+i8iSxR/\n+ZfwqU9lpagPfajcd2Vm/VH8gbq5ubnbdqUuuX0YeCUivpuLrQBmp+NG4OlcfGZaETUWOAtYFxG7\ngE5Jk9LE+KyicxrT8Q1kE+sAK4GpKUE0AFNTDGBNalv8/jaIXnrpvauivvc9JwyzetLrLreSLgH+\nDXiJrAQVwDeAdcAyshHC68CMNFmNpPlkq5n2k5WzWlP8AuBR4ATg2YiYl+LHA48BE4HdwMw0iY6k\n2cAd6X3vioglKT4WaAEagOeBmyJifzf3711uB4BLUWb1paddbr01uh1RvhR15ZXw7W/7eQuzeuBv\n7rM+815RZlbM24jYYfyAnpn1xEnD3uW9osysNy5PGeAH9MysNB5p1LnOTvjKV+DSS+Ezn3EpysyO\nzEmjTnWVoiZMOFSK8jfomVlvXJ6qQ12lqF//OitFXXRRue/IzKqFRxp1ZuHCQ6Won/zECcPM+sYj\njTqydCncfz+8+CL8zu+U+27MrBr5ifA6sW5dtrngv/4r/P7vl/tuzKzS9fREuMtTdWDHDviTP4GH\nHnLCMLP+cdKocW+/DVddBV/6Uva7mVl/uDxVww4ehBkz4AMfgEcfBZX8tVVmVu+8YWGdOXgQ7rgD\ndu3K5jGcMMxsIDhp1KAXX8yew/jtb+GZZ+D448t9R2ZWKzynUUM6OrK5i2nTYNYsWLsWRo4s912Z\nWS1x0qgBBw/C4sXZ7rS/+U32nd3+Zj0zGwwuT1W5fClqxQq48MJy35GZ1TKPNKpUvhTV2AjPPeeE\nYWaDz0mjyuRLUb/9bVaK+rM/g2H+kzSzIeDyVBVxKcrMys2fT6uAS1FmVimcNCqYS1FmVml6/edH\n0kOS2iVtzMUaJLVK2iJppaQRudfmS9omabOkabn4+ZI2StoqaVEufpyklnTOWkln5F5rTO23SJqV\ni58p6bn02lJJNVdme+EF+OQn4b77slLUP/wDnHJKue/KzOpdKZ9ZHwGmF8VuB1ZFxDnAamA+gKQJ\nwAxgPHAFcL/07gYWDwBzImIcME5S1zXnAHsi4mxgEbAwXasBuBO4ELgIWJBLTvcA96ZrdaRr1ISO\nDviLv4Dp012KMrPK02vSiIj/B+wtCl8NLE7Hi4Fr0vFVQEtEHIiI14BtwCRJpwInRsT61G5J7pz8\ntZYDU9LxdKA1IjojogNoBS5Pr00Bnsq9/7W99aPSHTyYbSqYL0V98YsuRZlZZTnass7IiGgHiIhd\nkro2qxgNrM2125liB4AdufiOFO86Z3u61juSOiWdnI/nryXpFGBvRBzMXeu0o+xHRXjhBa+KMrPq\nMFBzAQO593gp+7H2ac/Wpqamd48LhQKFQqFvdzRIOjrgb/4Gli2Db34T5szx1h9mVh5tbW20tbX1\n2u5ok0a7pFER0Z5KT2+m+E7g9Fy7MSnWUzx/zhuSjgFOiog9knYChaJz1kTEbkkjJA1Lo438tbqV\nTxqV4OBBeOwxuP327IuRXnnFk9xmVl7FH6ibm5u7bVdqxVy899P9CmB2Om4Ens7FZ6YVUWOBs4B1\nEbEL6JQ0KU2Mzyo6pzEd30A2sQ6wEpiaEkQDMDXFANaktsXvX/FefBH+8A8PrYr6x390wjCz6tHr\nN/dJepzsE/8pQDuwAPgh8CTZCOF1YEaarEbSfLLVTPuBeRHRmuIXAI8CJwDPRsS8FD8eeAyYCOwG\nZqZJdCTNBu4gK3/dFRFLUnws0AI0AM8DN0XE/h7uvyK+ua+jA+68E554IitFfeELnuQ2s8rV0zf3\n+eteB1m+FHX11fCtb3lkYWaVz1/3WgZeFWVmtcYFkkHgvaLMrFY5aQyg4m/Q27zZ36BnZrXF5akB\n0lWK2r/fpSgzq10eaQyAn/0MLrsMZs92KcrMaptXT/VTBEyZAjfemJWizMxqQU+rpzzS6KdnnoE3\n34TPf77cd2JmNvg8p9EP+/fDbbfB3/89DPd/STOrAx5p9MODD8Lpp8MVV5T7TszMhobnNI7SW2/B\nuHHwox/Bxz8+4Jc3Mysrz2kMsLvvzkYYThhmVk880jgK//3fMHFitmPtmDEDemkzs4rgkcYA+uu/\nhltuccIws/rjNT99tGYNrFoFW7aU+07MzIaeRxol2r0bbr4ZPvMZeOghOPHEct+RmdnQc9LoxcGD\n8L3vwYQJcOyx2SaEXmJrZvXK5akj2LAhm7s49lhYudIrpczMPNLoRlcp6tOfzpLGv/+7E4aZGThp\nvEd3pajZs/1d3mZmXVyeSjZsyL4PY/hwl6LMzHpS95+hXYoyMytd3SaN7kpRjY0uRZmZHUlVl6ck\nXQ4sIkt+D0XEPaWc51KUmdnRqdrP1ZKGAf8bmA6cB9wo6dwjndNVivrjP85+r9ZSVFtbW7lvYcC5\nT9WhFvsEtdmvwepT1SYNYBKwLSJej4j9QAtwdXcNi0tRr74Kn/tc9Zai/Be8OrhP1aMW+zVYfarm\n8tRoYHvu5x1kieQwF1/sUpSZ2UCo5qRRsltugc9+tnpHFmZmlaJqv09D0ieApoi4PP18OxDFk+GS\nqrODZmZl1t33aVRz0jgG2AJcCvwCWAfcGBGby3pjZmY1rGrLUxHxjqS5QCuHltw6YZiZDaKqHWmY\nmdnQq9mpYUmXS3pV0lZJf1Xu+zlakh6S1C5pYy7WIKlV0hZJKyWNKOc99oWkMZJWS9ok6SVJX0rx\nqu0TgKTjJf1E0vOpXwtSvNr7NUzSzyStSD9XdX8AJL0m6cX0Z7Uuxaq6X5JGSHpS0ub0/9ZFg9Wn\nmkwaR/PgXwV7hKwfebcDqyLiHGA1MH/I7+roHQC+GhHnARcDt6Y/m2ruExHxG2ByREwEPg5cIWkS\nVd4vYB7wSu7nau8PwEGgEBETI6JrmX619+u7wLMRMR74GPAqg9WniKi5X8AngP+b+/l24K/KfV/9\n6M/vAhtzP78KjErHpwKvlvse+9G3HwKX1Vif3g9sAC6s5n4BY4AfAwVgRYpVbX9y/fov4JSiWNX2\nCzgJ+M9u4oPSp5ocadD9g3+jy3Qvg2FkRLQDRMQuYGSZ7+eoSDqT7FP5c2R/uau6T6mU8zywC/hx\nRKynuvv1HeA2ID/xWc396RLAjyWtl/SFFKvmfo0FfinpkVRK/D+S3s8g9alWk0a9qbrVDJI+CCwH\n5kXErzj0XI8JAAABz0lEQVS8D1XXp4g4GFl5agwwSdJ5VGm/JH0KaI+IF4DD1urnVEV/ilwSEecD\nV5KVRz9Jlf45JcOB84H7Ur/eJquuDEqfajVp7ATOyP08JsVqRbukUQCSTgXeLPP99Imk4WQJ47GI\neDqFq7pPeRHxFtAGXE719usS4CpJPweWAlMkPQbsqtL+vCsifpF+/x+y8ugkqvfPCbJKyvaI2JB+\nfoosiQxKn2o1aawHzpL0u5KOA2YCK8p8T/0h3vtpbwUwOx03Ak8Xn1DhHgZeiYjv5mJV3SdJH+pa\nnSLpfcBUYDNV2q+I+EZEnBERv0f2/8/qiPgs8C9UYX+6SHp/GuUi6QPANOAlqvTPCSCVoLZLGpdC\nlwKbGKQ+1exzGum7Nr7LoQf/7i7zLR0VSY+TTUSeArQDC8g+HT0JnA68DsyIiI5y3WNfSLoE+Dey\n/1Ej/foG2RP9y6jCPgFI+iiwmOzv2zDgiYj4lqSTqeJ+AUj6I+BrEXFVtfdH0ljgB2R/74YD34+I\nu2ugXx8DHgSOBX4OfA44hkHoU80mDTMzG3i1Wp4yM7NB4KRhZmYlc9IwM7OSOWmYmVnJnDTMzKxk\nThpmZlYyJw0zMyuZk4aZmZXs/wOgO+OohCg0ywAAAABJRU5ErkJggg==\n", 211 | "text/plain": [ 212 | "" 213 | ] 214 | }, 215 | "metadata": {}, 216 | "output_type": "display_data" 217 | } 218 | ], 219 | "source": [ 220 | "weights.plot()" 221 | ] 222 | }, 223 | { 224 | "cell_type": "code", 225 | "execution_count": 114, 226 | "metadata": { 227 | "collapsed": true 228 | }, 229 | "outputs": [], 230 | "source": [ 231 | "ret = call([\"./fetch_weights.sh\"])\n", 232 | "if ret != 0:\n", 233 | " print(\"Problem \", ret)" 234 | ] 235 | } 236 | ], 237 | "metadata": { 238 | "kernelspec": { 239 | "display_name": "Python 2", 240 | "language": "python", 241 | "name": "python2" 242 | }, 243 | "language_info": { 244 | "codemirror_mode": { 245 | "name": "ipython", 246 | "version": 2 247 | }, 248 | "file_extension": ".py", 249 | "mimetype": "text/x-python", 250 | "name": "python", 251 | "nbconvert_exporter": "python", 252 | "pygments_lexer": "ipython2", 253 | "version": "2.7.11" 254 | } 255 | }, 256 | "nbformat": 4, 257 | "nbformat_minor": 0 258 | } 259 | -------------------------------------------------------------------------------- /analysis/fetch_from_s3.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e 3 | set -u 4 | echo "Syncing s3://$1/logs with $1/logs." 5 | 6 | aws s3 sync s3://$1/logs $1/logs 7 | 8 | echo "Syncing s3://$1/config with $1/config." 9 | aws s3 sync s3://$1/config $1/config 10 | 11 | echo "Syncing s3://$1/analysis with $1/analysis." 12 | aws s3 sync s3://$1/analysis $1/analysis 13 | 14 | -------------------------------------------------------------------------------- /analysis/layer_view.py: -------------------------------------------------------------------------------- 1 | """ Utility to plot the indicated layer of convolutions learned by 2 | the Deep q-network, with the first layer corresponding to 0. 3 | (Assumes dnn convolutions) 4 | Usage: 5 | python layer_view.py weight.npz layer_integer 6 | """ 7 | 8 | import sys 9 | import matplotlib.pyplot as plt 10 | import numpy 11 | import matplotlib.animation as animation 12 | 13 | net_file = numpy.load(sys.argv[1]) 14 | array_index = (int(sys.argv[2])*2) 15 | w = net_file['arr_'+str(array_index)] 16 | 17 | fig, _ = plt.subplots() 18 | #for c in range(w.shape[1]): # channels/time-steps 19 | 20 | [w / numpy.linalg.norm(v) for v in w] 21 | #w = w / numpy.linalg.norm(w, axis=1) 22 | #w = w / numpy.lina 23 | 24 | ims = [] 25 | def init(): 26 | for f in range(w.shape[0]): # filters 27 | sqrt = numpy.ceil(numpy.sqrt(w.shape[0])) 28 | print(sqrt) 29 | plt.subplot(sqrt, sqrt, f+1) 30 | img = w[f, 0, :, :] 31 | ims.append(plt.imshow(img, vmin=img.min(), vmax=img.max(), 32 | interpolation='none', cmap='gray')) 33 | plt.xticks(()) 34 | plt.yticks(()) 35 | return ims 36 | 37 | def run(c): 38 | for f in range(w.shape[0]): # filters 39 | ims[f].set_array(w[f, c%w.shape[1], :, :]) 40 | return ims 41 | 42 | def inf(): 43 | cnt = 0 44 | while True: 45 | cnt -= 1 46 | if cnt < 0: 47 | cnt += w.shape[1] 48 | yield cnt 49 | 50 | #run() 51 | ani = animation.FuncAnimation(fig, run, inf, blit=False, interval=500, 52 | repeat=True, init_func=init) 53 | plt.show() 54 | -------------------------------------------------------------------------------- /analysis/sync.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e 3 | set -u 4 | name=`basename \`pwd\`` 5 | echo "Syncing s3://$name/logs with logs." 6 | 7 | aws s3 sync s3://"$name"/logs logs 8 | 9 | echo "Syncing s3://$name/config with config." 10 | aws s3 sync s3://"$name"/config config 11 | 12 | echo "Syncing s3://$name/analysis with analysis." 13 | aws s3 sync s3://"$name"/analysis analysis 14 | 15 | -------------------------------------------------------------------------------- /analysis/translate_weights.py: -------------------------------------------------------------------------------- 1 | """ Converts a .npz file to something human-readable 2 | Usage: 3 | python translate_weights.py weight.npz readable.txt 4 | """ 5 | 6 | import numpy 7 | import sys 8 | 9 | npzfile = numpy.load(sys.argv[1]) 10 | 11 | f = open(sys.argv[2], 'w') 12 | for key, item in npzfile.iteritems(): 13 | f.write(key + " - " + str(item.shape)) 14 | f.write(str(list(item))) 15 | f.write("\r\n") 16 | 17 | #numpy.savetxt(key+".txt", item) 18 | f.close() 19 | -------------------------------------------------------------------------------- /analyze_layers.py: -------------------------------------------------------------------------------- 1 | 2 | import sys 3 | import os 4 | from PIL import Image 5 | import numpy as np 6 | import ale_game as ag 7 | import network 8 | import theano.tensor as T 9 | import theano 10 | import lasagne 11 | 12 | directory = sys.argv[1] 13 | frame = int(sys.argv[2]) 14 | 15 | directory = "dqn16/record/" 16 | 17 | weights_file = "dqn39/weights_8000100.npz" 18 | 19 | 20 | 21 | 22 | 23 | s0_var = T.tensor4("s0", dtype=theano.config.floatX) 24 | input_var = s0_var 25 | n = network.build_nature_with_pad(6, input_var, 84) 26 | out = lasagne.layers.get_output(n, s0_var) 27 | 28 | outs = [lasagne.layers.get_output(layer, s0_var) for layer in lasagne.layers.get_all_layers(n)] 29 | 30 | with np.load(weights_file) as initial_weights: 31 | param_values = [initial_weights['arr_%d' % i] for i in range(len(initial_weights.files))] 32 | lasagne.layers.set_all_param_values(n, param_values) 33 | 34 | ff = theano.function([s0_var], outs) 35 | 36 | import matplotlib.pyplot as plt 37 | inputs = [] 38 | for frame in range(1200, 1400, 20): 39 | files = [os.path.join(directory, "%06d.png" % i) for i in range(frame, frame+16)] 40 | frames = [np.array(Image.open(f)) for f in files] 41 | gray_frames = [(np.dot(f, np.array([0.2126, 0.7152, 0.0722]))).astype(np.float32) for f in frames] 42 | 43 | phi = ag.Phi(method="resize") 44 | inputs.append(np.stack(phi(gray_frames), axis=0).reshape(1, 4, 84, 84)) 45 | 46 | 47 | 48 | output = plt.imshow(ff(inputs[0])[1][0][0], cmap='Greys_r') 49 | print(output) 50 | 51 | -------------------------------------------------------------------------------- /dqn.py: -------------------------------------------------------------------------------- 1 | # Alt + Shift + E 2 | 3 | from __future__ import print_function 4 | from __future__ import division 5 | 6 | import numpy as np 7 | import theano 8 | import theano.tensor as T 9 | 10 | import lasagne 11 | 12 | 13 | class ReplayMemory(object): 14 | def __init__(self, size=1000000, grace=10000): 15 | self.max_size = size 16 | self.grace = grace 17 | self.list = [] 18 | 19 | def init_state(self, s0): 20 | if len(self.list) > 0: 21 | self.list[-1] = s0 22 | else: 23 | self.list.append(s0) 24 | 25 | def append(self, a0, r0, fri, s1): 26 | el = [a0, r0, fri, s1] 27 | 28 | self.list.extend(el) 29 | if len(self) > self.max_size + self.grace * 4: 30 | self.list = self.list[self.grace * 4:] 31 | 32 | def sample(self, sample_size): 33 | import random 34 | indices = random.sample(xrange(len(self)), sample_size) 35 | return [self[i] for i in indices] 36 | 37 | def __len__(self): 38 | return int((len(self.list) - 1) / 4) 39 | 40 | def __getitem__(self, idx): 41 | return tuple(self.list[idx * 4:idx * 4 + 5]) 42 | 43 | 44 | class DQNAlgo: 45 | def __init__(self, n_actions, replay_memory, build_network, updates, screen_size, initial_weights_file=None): 46 | self.screen_width, self.screen_height = screen_size 47 | self.mood_q = None 48 | self.last_q = 0 49 | self.n_parameter_updates = 0 50 | self.alpha = 0.00025 51 | # update frequency ? 52 | # gradient momentum ? 0.95 53 | # squared gradient momentum ? 0.95 54 | # min squared gradient ? 0.01 55 | self.save_every_n_frames = 100000 # ~ once per hour 56 | 57 | self.final_exploration_frame = 1000000 58 | self.replay_start_size = 50000 59 | self.i_action = 0 60 | 61 | self.state = None 62 | self.initial_epsilon = 1 63 | self.final_epsilon = 0.1 64 | self.epsilon = self.initial_epsilon 65 | self.gamma = 0.99 66 | self.replay_memory = replay_memory 67 | 68 | self.log_frequency = 1 69 | 70 | self.minibatch_size = 32 71 | # self.replay_memory_size = 1000000 72 | 73 | self.target_network_update_frequency = 10000 74 | 75 | s0_var = T.tensor4("s0", dtype=theano.config.floatX) 76 | a0_var = T.bmatrix("a0") 77 | r0_var = T.wcol("r0") 78 | s1_var = T.tensor4("s1", dtype=theano.config.floatX) 79 | future_reward_indicator_var = T.bcol("future_reward_indicator") 80 | 81 | self.n_actions = n_actions 82 | self.a_lookup = np.eye(self.n_actions, dtype=np.int8) 83 | 84 | self.network = build_network(n_actions=self.n_actions, input_var=T.cast(s0_var, 'float32') / np.float32(256), 85 | screen_size=(self.screen_height, self.screen_width)) 86 | print("Compiling forward.") 87 | self.forward = theano.function([s0_var], lasagne.layers.get_output(self.network, deterministic=True)) 88 | 89 | self.network_stale = build_network(n_actions=self.n_actions, input_var=T.cast(s1_var, 'float32') / np.float32(256), 90 | screen_size=(self.screen_height, self.screen_width)) 91 | print("Compiling forward_stale.") 92 | self.forward_stale = theano.function([s1_var], 93 | lasagne.layers.get_output(self.network_stale, deterministic=True)) 94 | 95 | self._update_network_stale() 96 | 97 | out = lasagne.layers.get_output(self.network) 98 | out_stale = lasagne.layers.get_output(self.network_stale) 99 | self.loss, self.err, __y, __q = build_loss(out=out, 100 | out_stale=out_stale, 101 | a0_var=a0_var, 102 | r0_var=r0_var, 103 | future_reward_indicator_var=future_reward_indicator_var, 104 | gamma=self.gamma) 105 | 106 | params = lasagne.layers.get_all_params(self.network, trainable=True) 107 | 108 | print("Compiling train_fn.") 109 | self.train_fn = theano.function([s0_var, a0_var, r0_var, s1_var, future_reward_indicator_var], 110 | [self.loss, self.err, T.transpose(__y), T.transpose(__q), out, out_stale], 111 | updates=updates(self.loss, params)) 112 | print("Compiling loss_fn.") 113 | self.loss_fn = theano.function([s0_var, a0_var, r0_var, s1_var, future_reward_indicator_var], 114 | self.loss) 115 | 116 | 117 | def log(self, *args): 118 | if self.i_action % 10000 < self.log_frequency: 119 | print(self.i_action, *args) 120 | 121 | def init_state(self, state): 122 | self.state = self._prep_state(state) 123 | if self.replay_memory is not None: 124 | self.replay_memory.init_state(self.state) 125 | 126 | def _update_network_stale(self): 127 | lasagne.layers.set_all_param_values(self.network_stale, lasagne.layers.get_all_param_values(self.network)) 128 | 129 | def _prep_state(self, state): 130 | return np.reshape(np.stack(state, axis=0), (1, 4, self.screen_width, self.screen_height)) 131 | 132 | def action(self, _state): 133 | import random 134 | if self.i_action < self.final_exploration_frame: 135 | self.epsilon = (self.final_epsilon - self.initial_epsilon) * ( 136 | self.i_action / self.final_exploration_frame) + self.initial_epsilon 137 | else: 138 | self.epsilon = self.final_epsilon 139 | #self.log("{i_frame} | epsilon: {epsilon}".format(i_frame=self.i_action, epsilon=self.epsilon)) 140 | 141 | if random.random() < self.epsilon: 142 | action = random.randint(0, self.n_actions - 1) 143 | #print("{i_frame} | random action: {action}".format(i_frame=self.i_frames, action=action)) 144 | return action 145 | else: 146 | return self._best_action(self._prep_state(_state)) 147 | 148 | def _best_action(self, state): 149 | q = self.forward(state) 150 | self.last_q = np.max(q) 151 | action = np.argmax(q) 152 | # print("{i_frame} | q: {q} | action: {action}".format(i_frame=self.i_frames, q=q, action=action)) 153 | return action 154 | 155 | def feedback(self, exp): 156 | # exp -> s0 a0 r0 s1 game_over 157 | self.i_action += 1 158 | self.state = self._prep_state(exp.s1) 159 | 160 | r0_clipped = min(1, max(-1, exp.r0)) 161 | fri = 1 - int(exp.game_over) 162 | 163 | if self.mood_q: 164 | expectation = np.max(self.forward(self.state)) 165 | surprise = (r0_clipped + self.gamma * expectation) - self.last_q 166 | self.mood_q.put({'i_frame': self.i_action, 'surprise': surprise, "expectations": expectation}) 167 | 168 | if self.replay_memory is None: 169 | return 170 | 171 | # if exp.r0 != 0 or r0_clipped != 0: 172 | # print("{i_frame} | reward: {reward} | creward: {creward}" 173 | # .format(i_frame=self.i_action, reward=exp.r0, creward=r0_clipped)) 174 | 175 | self.replay_memory.append(self.a_lookup[exp.a0], r0_clipped, fri, self.state) 176 | 177 | if len(self.replay_memory) > self.replay_start_size and self.i_action % 4 == 0: 178 | sample = zip(*self.replay_memory.sample(self.minibatch_size)) 179 | 180 | s0 = np.array(sample[0], dtype=theano.config.floatX).reshape(self.minibatch_size, 4, self.screen_width, self.screen_height) 181 | 182 | a0 = np.array(sample[1], dtype=np.int8).reshape(self.minibatch_size, self.n_actions) 183 | 184 | r0 = np.array(sample[2], dtype=np.int16).reshape(self.minibatch_size, 1) 185 | 186 | future_reward_indicators = np.array(sample[3], dtype=np.int8).reshape(self.minibatch_size, 1) 187 | 188 | s1 = np.array(sample[4], dtype=theano.config.floatX).reshape(self.minibatch_size, 4, self.screen_width, self.screen_height) 189 | 190 | t = self.train_fn(s0, a0, r0, s1, future_reward_indicators) 191 | 192 | self.n_parameter_updates += 1 193 | 194 | self.log('{i_frame} | loss_elems: '.format(i_frame=self.i_action), t[1]) 195 | self.log('{i_frame} | y, q: '.format(i_frame=self.i_action), t[2], t[3]) 196 | self.log('{i_frame} | out: '.format(i_frame=self.i_action), t[4]) 197 | self.log('{i_frame} | out_stale: '.format(i_frame=self.i_action), t[5]) 198 | 199 | if self.n_parameter_updates % self.target_network_update_frequency == 0: 200 | self._update_network_stale() 201 | 202 | def get_state(self): 203 | return lasagne.layers.get_all_param_values(self.network) 204 | 205 | def set_state(self, param_values): 206 | lasagne.layers.set_all_param_values(self.network, param_values) 207 | lasagne.layers.set_all_param_values(self.network_stale, param_values) 208 | 209 | def __str__(self): 210 | return """ 211 | self.mood_q = {self.mood_q} 212 | self.last_q = {self.last_q} 213 | self.n_parameter_updates = {self.n_parameter_updates} 214 | self.replay_memory = {self.replay_memory} 215 | self.alpha = {self.alpha} 216 | self.save_every_n_frames = {self.save_every_n_frames} 217 | self.final_exploration_frame = {self.final_exploration_frame} 218 | self.replay_start_size = {self.replay_start_size} 219 | self.i_action = {self.i_action} 220 | self.state = {self.state} 221 | self.initial_epsilon = {self.initial_epsilon} 222 | self.final_epsilon = {self.final_epsilon} 223 | self.epsilon = {self.epsilon} 224 | self.gamma = {self.gamma} 225 | self.log_frequency = {self.log_frequency} 226 | self.minibatch_size = {self.minibatch_size} 227 | self.target_network_update_frequency = {self.target_network_update_frequency} 228 | """.format(**{'self': self}) 229 | 230 | 231 | def build_loss(out, out_stale, a0_var, r0_var, future_reward_indicator_var, gamma): 232 | # s0_var mini_batch x 4 x,80 x 80 233 | # a0_var mini_batch x 1, 234 | # r0_var mini_batch x 1, 235 | # s1_mini_batch x 4 x 80 x 80 236 | future_reward_indicator_var.tag.test_value = np.random.rand(32, 1).astype(dtype=np.int8) 237 | r0_var.tag.test_value = np.random.rand(32, 1).astype(dtype=np.int16) 238 | a0_var.tag.test_value = np.random.rand(32, 6).astype(dtype=np.int8) 239 | out.tag.test_value = np.random.rand(1, 6).astype(dtype=theano.config.floatX) 240 | out_stale.tag.test_value = np.random.rand(32, 6).astype(dtype=theano.config.floatX) 241 | 242 | y = r0_var + gamma * future_reward_indicator_var * T.max(out_stale, axis=1, keepdims=True) # 32 x 1 243 | q = T.sum(a0_var * out, axis=1, keepdims=True) # 32 x 1 244 | err = y - q 245 | theano.shared 246 | 247 | quadratic_part = T.minimum(abs(err), 1) 248 | linear_part = abs(err) - quadratic_part 249 | loss = 0.5 * quadratic_part ** 2 + linear_part 250 | return T.sum(loss), loss, y, q 251 | -------------------------------------------------------------------------------- /network.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | from __future__ import division 3 | 4 | import lasagne 5 | 6 | 7 | def build_nature_with_pad(n_actions, input_var, screen_size): 8 | network = lasagne.layers.InputLayer(shape=(None, 4, screen_size[0], screen_size[1]), 9 | input_var=input_var) 10 | 11 | network = lasagne.layers.Conv2DLayer( 12 | network, num_filters=32, filter_size=(8, 8), stride=4, pad=2, 13 | nonlinearity=lasagne.nonlinearities.rectify, 14 | W=lasagne.init.GlorotUniform(), 15 | b=lasagne.init.Constant(.1)) 16 | 17 | print(network.output_shape) 18 | 19 | network = lasagne.layers.Conv2DLayer( 20 | network, num_filters=64, filter_size=(4, 4), stride=2, 21 | nonlinearity=lasagne.nonlinearities.rectify, 22 | b=lasagne.init.Constant(.1)) 23 | 24 | network = lasagne.layers.Conv2DLayer( 25 | network, num_filters=64, filter_size=(3, 3), stride=1, 26 | nonlinearity=lasagne.nonlinearities.rectify, 27 | b=lasagne.init.Constant(.1)) 28 | 29 | network = lasagne.layers.DenseLayer( 30 | network, 31 | num_units=512, 32 | nonlinearity=lasagne.nonlinearities.rectify, 33 | b=lasagne.init.Constant(.1)) 34 | 35 | network = lasagne.layers.DenseLayer( 36 | network, 37 | num_units=n_actions, 38 | nonlinearity=None, 39 | b=lasagne.init.Constant(.1)) 40 | 41 | return network 42 | 43 | 44 | def build_nature_with_pad_he(n_actions, input_var, screen_size): 45 | network = lasagne.layers.InputLayer(shape=(None, 4, screen_size[0], screen_size[1]), 46 | input_var=input_var) 47 | 48 | network = lasagne.layers.Conv2DLayer( 49 | network, num_filters=32, filter_size=(8, 8), stride=4, pad=2, 50 | nonlinearity=lasagne.nonlinearities.rectify, 51 | W=lasagne.init.HeNormal(gain='relu'), 52 | b=lasagne.init.Constant(.1)) 53 | 54 | print(network.output_shape) 55 | 56 | network = lasagne.layers.Conv2DLayer( 57 | network, num_filters=64, filter_size=(4, 4), stride=2, 58 | nonlinearity=lasagne.nonlinearities.rectify, 59 | W=lasagne.init.HeNormal(gain='relu'), 60 | b=lasagne.init.Constant(.1)) 61 | 62 | network = lasagne.layers.Conv2DLayer( 63 | network, num_filters=64, filter_size=(3, 3), stride=1, 64 | nonlinearity=lasagne.nonlinearities.rectify, 65 | W=lasagne.init.HeNormal(gain='relu'), 66 | b=lasagne.init.Constant(.1)) 67 | 68 | network = lasagne.layers.DenseLayer( 69 | network, 70 | num_units=512, 71 | nonlinearity=lasagne.nonlinearities.rectify, 72 | W=lasagne.init.HeNormal(gain='relu'), 73 | b=lasagne.init.Constant(.1)) 74 | 75 | network = lasagne.layers.DenseLayer( 76 | network, 77 | num_units=n_actions, 78 | nonlinearity=None, 79 | b=lasagne.init.Constant(.1)) 80 | 81 | return network 82 | 83 | 84 | def build_nature(n_actions, input_var, screen_size): 85 | network = lasagne.layers.InputLayer(shape=(None, 4, screen_size[0], screen_size[1]), 86 | input_var=input_var) 87 | 88 | network = lasagne.layers.Conv2DLayer( 89 | network, num_filters=32, filter_size=(8, 8), stride=4, 90 | nonlinearity=lasagne.nonlinearities.rectify, 91 | W=lasagne.init.GlorotUniform(), 92 | b=lasagne.init.Constant(.1)) 93 | 94 | print(network.output_shape) 95 | 96 | network = lasagne.layers.Conv2DLayer( 97 | network, num_filters=64, filter_size=(4, 4), stride=2, 98 | nonlinearity=lasagne.nonlinearities.rectify, 99 | b=lasagne.init.Constant(.1)) 100 | 101 | network = lasagne.layers.Conv2DLayer( 102 | network, num_filters=64, filter_size=(3, 3), stride=1, 103 | nonlinearity=lasagne.nonlinearities.rectify, 104 | b=lasagne.init.Constant(.1)) 105 | 106 | network = lasagne.layers.DenseLayer( 107 | network, 108 | num_units=512, 109 | nonlinearity=lasagne.nonlinearities.rectify, 110 | b=lasagne.init.Constant(.1)) 111 | 112 | network = lasagne.layers.DenseLayer( 113 | network, 114 | num_units=n_actions, 115 | nonlinearity=None, 116 | b=lasagne.init.Constant(.1)) 117 | 118 | return network 119 | 120 | 121 | def build_nips(n_actions, input_var, screen_size): 122 | network = lasagne.layers.InputLayer(shape=(32, 4, screen_size[0], screen_size[1]), 123 | input_var=input_var) 124 | 125 | network = lasagne.layers.Conv2DLayer( 126 | network, num_filters=16, filter_size=(8, 8), stride=4, 127 | nonlinearity=lasagne.nonlinearities.rectify, 128 | W=lasagne.init.GlorotUniform()) 129 | 130 | network = lasagne.layers.Conv2DLayer( 131 | network, num_filters=32, filter_size=(4, 4), stride=2, 132 | nonlinearity=lasagne.nonlinearities.rectify) 133 | 134 | network = lasagne.layers.DenseLayer( 135 | network, 136 | num_units=256, 137 | nonlinearity=lasagne.nonlinearities.rectify) 138 | 139 | network = lasagne.layers.DenseLayer( 140 | network, 141 | num_units=n_actions, 142 | nonlinearity=None, 143 | W=lasagne.init.HeUniform(), 144 | b=lasagne.init.Constant(.1)) 145 | 146 | return network 147 | 148 | 149 | def build_simple_breakout_W_caffe_normal(n_actions, input_var, screen_size): 150 | 151 | network = lasagne.layers.InputLayer(shape=(None, 4, screen_size[0], screen_size[1]), 152 | input_var=input_var) 153 | 154 | network = lasagne.layers.Conv2DLayer( 155 | network, num_filters=4, filter_size=(3, 3), stride=1, pad=1, 156 | nonlinearity=lasagne.nonlinearities.rectify, 157 | W=lasagne.init.HeNormal(gain='relu')) 158 | 159 | network = lasagne.layers.DenseLayer( 160 | network, 161 | num_units=128, 162 | nonlinearity=lasagne.nonlinearities.rectify, 163 | W=lasagne.init.HeNormal(gain='relu')) 164 | 165 | network = lasagne.layers.DenseLayer( 166 | network, 167 | num_units=n_actions, 168 | nonlinearity=None, 169 | b=lasagne.init.Constant(.1)) 170 | 171 | return network 172 | -------------------------------------------------------------------------------- /provision/aws_installation.sh: -------------------------------------------------------------------------------- 1 | sudo apt-get install -y gcc g++ gfortran build-essential git wget linux-image-generic libopenblas-dev python-dev python-pip python-nose python-numpy python-scipy 2 | cuda-install-samples-7.5.sh ~/ 3 | cd NVIDIA_CUDA-7.5_Samples/1_Utilities/deviceQuery 4 | make 5 | ./deviceQuery 6 | echo -e "\n[global]\nfloatX=float32\ndevice=gpu\n[mode]=FAST_RUN\n\n[nvcc]\nfastmath=True\n\n[cuda]\nroot=/usr/local/cuda" >> ~/.theanorc 7 | cd ~ 8 | nano theano_gpu_test.py # copied example theano gpu test from tutorial 9 | python theano_gpu_test.py 10 | # Copied cudnn file from Maciej's computer 11 | tar xvfz cudnn-7.0-linux-x64-v4.0-rc.tgz 12 | echo $CUDA_ROOT 13 | 14 | cd /user/local/cuda 15 | sudo cp ~/cuda/include/cudnn.h ./include/ 16 | sudo cp ~/cuda/lib64/libcudnn.so ./lib64/ 17 | sudo cp ~/cuda/lib64/libcudnn.so.4 ./lib64/ 18 | sudo cp ~/cuda/lib64/libcudnn.so.4.0.4 ./lib64/ 19 | sudo cp ~/cuda/lib64/libcudnn_static.a ./lib64/ 20 | 21 | cd ~ 22 | git clone https://github.com/Lasagne/Lasagne.git 23 | cd ~/Lasagne/ 24 | sudo pip install . 25 | python -c "from theano.sandbox.cuda.dnn import dnn_available as d; print(d() or d.msg)" 26 | cd ~ 27 | wget http://www.arcadelearningenvironment.org/wp-content/uploads/2015/10/Arcade-Learning-Environment-0.5.1.zip 28 | sudo apt-get install unzip 29 | unzip Arcade-Learning-Environment-0.5.1.zip 30 | sudo apt-get install cmake libsdl1.2-dev 31 | cd Arcade-Learning-Environment-0.5.1/ 32 | cmake -DUSE_SDL=ON -DUSE_RLGLUE=OFF . 33 | make -j 4 34 | sudo pip install . 35 | 36 | python mnist.py cnn 37 | 38 | 39 | 40 | cd ~ 41 | git clone https://github.com/NVIDIA/cnmem.git cnmem 42 | cd cnmem/ 43 | mkdir build 44 | cd build 45 | cmake .. 46 | make 47 | cd /usr/local/cuda 48 | sudo cp ~/cnmem/include/cnmem.h ./include 49 | sudo mv ~/cnmem/build/libcnmem.so ./lib64 50 | -------------------------------------------------------------------------------- /provision/gen_run.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | from subprocess import call 3 | import botocore 4 | 5 | 6 | def upload_user_data(**kargs): 7 | 8 | s3 = boto3.resource('s3') 9 | s3.create_bucket(ACL='private', Bucket=kargs['exp_name']) 10 | k = s3.Object(kargs['exp_name'], 'config/run.sh') 11 | 12 | try: 13 | script = k.get()['Body'].read() 14 | except botocore.exceptions.ClientError as e: 15 | 16 | print("No run.sh not found. Uploading.") 17 | script = """#!/bin/bash 18 | cd /usr/local/cuda/samples/1_Utilities/deviceQuery && make && ./deviceQuery 19 | 20 | cd /home/{user_name} 21 | 22 | sudo su {user_name} -c "mkdir -p /home/{user_name}/.aws" 23 | 24 | sudo su {user_name} -c "aws s3 sync s3://dqn-setup /home/{user_name}/dqn-setup" 25 | 26 | 27 | sudo su {user_name} -c "git clone https://github.com/maciejjaskowski/{project_name}.git" 28 | sudo su {user_name} -c "cd {project_name} && git reset --hard {sha1}" 29 | sudo su {user_name} -c "mkdir -p /home/{user_name}/{project_name}/{exp_name}/weights" 30 | sudo su {user_name} -c "mkdir -p /home/{user_name}/{project_name}/logs" 31 | sudo su {user_name} -c "mkdir -p /home/{user_name}/{project_name}/record0.1" 32 | sudo su {user_name} -c "cp /home/{user_name}/dqn-setup/*.bin /home/{user_name}/{project_name}/" 33 | 34 | sudo su ubuntu -c 'aws s3 sync s3://{exp_name}/weights /home/{user_name}/{project_name}/{exp_name}/weights' 35 | 36 | export PATH=/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:; 37 | export LD_LIBRARY_PATH=/usr/local/cuda/lib64; 38 | echo $PATH > /home/{user_name}/path.log; 39 | echo $LD_LIBRARY_PATH /home/{user_name}/ld.log; 40 | cd /home/{user_name}/{project_name} 41 | cp runs/run_{exp_name}.py . 42 | THEANO_FLAGS='floatX=float32,mode=FAST_RUN,allow_gc=False,device=gpu,lib.cnmem=0.9' python run_{exp_name}.py 2>&1 | multilog t s100000 '!tai64nlocal|gzip' ./logs & 43 | 44 | watch -n 60 "sudo su {user_name} -c 'aws s3 sync /home/{user_name}/{project_name}/{exp_name}/weights s3://{exp_name}/weights' && sudo su {user_name} -c 'aws s3 sync /home/{user_name}/{project_name}/logs s3://{exp_name}/logs' && echo \`date\` >> /home/{user_name}/last_sync" & 45 | """.format(**kargs) 46 | 47 | k.put(Body=script) 48 | call(["aws", "s3", "sync", "../analysis", "s3://" + kargs['exp_name'] + "/analysis"]) 49 | call(["mkdir", "-p", "../" + kargs['exp_name']]) 50 | call(["cp", "../analysis/sync.sh", "../" + kargs['exp_name']]) 51 | else: 52 | print("run.sh found on S3. Reusing.") 53 | 54 | return script 55 | -------------------------------------------------------------------------------- /provision/mount.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e 3 | set -u 4 | INSTANCE=`cat instance.dns` 5 | ssh -oStrictHostKeyChecking=no -i ~/.ssh/gpu-east.pem ubuntu@$INSTANCE 'bash -s' < mount_remote.sh 6 | -------------------------------------------------------------------------------- /provision/mount_remote.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | sudo mkdir -p /mnt/dqn && sudo mount /dev/xvdf /mnt/dqn 3 | 4 | cd /mnt/dqn/deep-q-learning && sudo git pull origin aws 5 | 6 | sudo docker pull mjaskowski/dqn 7 | 8 | cd /usr/local/cuda/samples/1_Utilities/deviceQuery && make && ./deviceQuery 9 | -------------------------------------------------------------------------------- /provision/provision.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | import base64 3 | import time 4 | from datetime import datetime 5 | from gen_run import upload_user_data 6 | 7 | ec2 = boto3.client('ec2') 8 | 9 | 10 | def _prices(az): 11 | next_token = '' 12 | a = [] 13 | for i in range(1): 14 | res = ec2.describe_spot_price_history(StartTime=datetime(2016, 1, 1), InstanceTypes=['g2.2xlarge'], 15 | AvailabilityZone=az, ProductDescriptions=['Linux/UNIX'], MaxResults=1000, NextToken=next_token) 16 | a = a + res['SpotPriceHistory'] 17 | next_token = res['NextToken'] 18 | 19 | return sorted(a, key=lambda s: s['Timestamp']) 20 | 21 | 22 | def prices(): 23 | return zip(_prices('us-east-1a'), _prices('us-east-1b'), _prices('us-east-1c'), _prices('us-east-1e')) 24 | 25 | 26 | 27 | def provision(client_token, availability_zone, spot_price): 28 | 29 | user_data = """#!/bin/bash 30 | aws s3 sync s3://{exp_name}/config /home/ubuntu/ 31 | chmod a+x /home/ubuntu/run.sh 32 | /home/ubuntu/run.sh > /home/ubuntu/run_log.out 2> /home/ubuntu/run_log.err 33 | """.format(exp_name=client_token) 34 | 35 | print(user_data) 36 | 37 | result = ec2.request_spot_instances(DryRun=False, 38 | ClientToken=client_token, 39 | SpotPrice=spot_price, 40 | InstanceCount=1, 41 | AvailabilityZoneGroup=availability_zone, 42 | Type='persistent', 43 | LaunchSpecification={ 44 | 'ImageId': 'ami-bdd2efd7', 45 | 'KeyName': 'gpu-east', 46 | 'InstanceType': 'g2.2xlarge', 47 | 'Placement': { 48 | 'AvailabilityZone': availability_zone 49 | }, 50 | 'BlockDeviceMappings': [{ 51 | 'DeviceName': '/dev/sda1', 52 | 'Ebs': { 53 | 'VolumeSize': 25, 54 | 'DeleteOnTermination': True, 55 | 'VolumeType': 'standard', 56 | 'Encrypted': True 57 | } 58 | }], 59 | 'IamInstanceProfile': { 60 | 'Name': 's3' 61 | }, 62 | 'EbsOptimized': False, 63 | 'Monitoring': { 64 | 'Enabled': True 65 | }, 66 | 'UserData': base64.b64encode(user_data.encode("ascii")).decode('ascii'), 67 | 'SecurityGroupIds': ['sg-ab1236d2'], 68 | 69 | }) 70 | 71 | req_id = result['SpotInstanceRequests'][0]['SpotInstanceRequestId'] 72 | 73 | print("") 74 | instance_description = None 75 | public_dns_name = '' 76 | 77 | while True: 78 | instance = ec2.describe_spot_instance_requests(SpotInstanceRequestIds=[req_id])['SpotInstanceRequests'][0] 79 | sys.stdout.write(str(datetime.now().time()) + " " + instance['Status']['Message'] + '\r') 80 | sys.stdout.flush() 81 | 82 | if 'Fault' in instance.keys(): 83 | return { 84 | 'status': instance['Status'], 85 | 'instance': instance 86 | } 87 | 88 | if 'InstanceId' in instance.keys(): 89 | instance_description = ec2.describe_instances(InstanceIds=[instance['InstanceId']]) 90 | public_dns_name = instance_description['Reservations'][0]['Instances'][0]['PublicDnsName'] 91 | 92 | if public_dns_name != '': 93 | break 94 | 95 | time.sleep(1) 96 | 97 | return { 98 | 'status': instance['Status'], 99 | 'instance': instance, 100 | 'instance_description': instance_description, 101 | 'public_dns_name': public_dns_name 102 | } 103 | 104 | 105 | # def attach_volume(instance): 106 | # while True: 107 | # try: 108 | # ec2.attach_volume( 109 | # DryRun=False, 110 | # VolumeId='vol-32b4d7ed', 111 | # InstanceId=instance['InstanceId'], 112 | # Device='/dev/xvdf') 113 | # time.sleep(1) 114 | # except: 115 | # import traceback 116 | # traceback.print_exc() 117 | # print(datetime.now().time(), "Not ready yet.") 118 | # import sys 119 | # print(sys.exc_info()[2]) 120 | # else: 121 | # break 122 | 123 | 124 | def main(availability_zone, spot_price, client_token): 125 | 126 | project_name = "deep-q-learning" 127 | from subprocess import Popen, PIPE 128 | process = Popen(["git", "rev-parse", "HEAD"], shell=False, stdout=PIPE) 129 | sha1, _ = process.communicate(str.encode("utf-8")) 130 | sha1 = sha1[:-1] 131 | 132 | user_script = upload_user_data(exp_name=client_token, sha1=sha1, user_name="ubuntu", project_name=project_name) 133 | 134 | print(""" 135 | project_name: {project_name} 136 | sha1: {sha1} 137 | 138 | availability_zone: {availability_zone} 139 | spot_price: {spot_price} 140 | 141 | client_token: {client_token} 142 | user_script: 143 | 144 | {user_script} 145 | """.format(project_name=project_name, sha1=sha1, 146 | client_token=client_token, user_script=user_script, 147 | spot_price=spot_price, availability_zone=availability_zone)) 148 | 149 | instance = provision(client_token=client_token, availability_zone=availability_zone, 150 | spot_price=spot_price) 151 | 152 | print(instance) 153 | print("public_dns_name: ", instance['public_dns_name']) 154 | 155 | with open('instance.dns', 'w') as f: 156 | f.write(str(instance['public_dns_name'])) 157 | 158 | if __name__ == "__main__": 159 | import sys 160 | import getopt 161 | import os 162 | 163 | print("Running ") 164 | print(' '.join(sys.argv)) 165 | 166 | optlist, args = getopt.getopt(sys.argv[1:], '', [ 167 | 'price=', 168 | 'client_token=' 169 | ]) 170 | 171 | d = {'availability_zone': 'us-east-1a'} 172 | for o, a in optlist: 173 | if o in ("--price",): 174 | d['spot_price'] = a 175 | elif o in ("--client_token",): 176 | d['client_token'] = a 177 | else: 178 | assert False, "unhandled option" 179 | 180 | if not os.path.exists("../" + d['client_token']): 181 | os.makedirs("../" + d['client_token']) 182 | with open("../" + d['client_token'] + '/provision.sh', 'w') as f: 183 | f.write("#!/bin/bash\n") 184 | f.write(' '.join(sys.argv)) 185 | 186 | main(**d) 187 | 188 | 189 | def plot(): 190 | import matplotlib.pyplot as plt 191 | import numpy as np 192 | import pandas as pd 193 | plt.ion() 194 | fig, ax1 = plt.subplots() 195 | 196 | x, y = zip(*[(p['Timestamp'], p['SpotPrice']) for p in _prices('us-east-1a')]) 197 | print(min(x)) 198 | print(max(x)) 199 | print(len(x)) 200 | sx = pd.Series(x) 201 | delta = (sx - sx.shift(1))[1:] 202 | conc = pd.concat([delta, pd.Series(np.array(y, dtype=np.float32))], axis=1) 203 | conc.iloc[:, 1] = conc.iloc[:, 1].shift(1) 204 | 205 | df = pd.DataFrame(np.array(y, dtype=np.float32)) 206 | bins = 25 207 | 208 | def calc(c): 209 | eligible = (pd.concat([df, df.shift(1)], axis=1) <= c)[1:] 210 | raised = eligible.apply(lambda x: not x.iloc[0] and x.iloc[1], axis=1) 211 | return len(np.where(raised)[0]) 212 | 213 | def calc_time(c): 214 | return conc[conc.iloc[:, 1] < c].iloc[:, 0].sum().total_seconds() / 60 / 60 215 | 216 | lin = np.linspace(df.min()[0], df.max()[0], bins) 217 | 218 | print([calc_time(c) for c in lin]) 219 | ax1.plot(lin, [calc_time(c) for c in lin]) 220 | 221 | ax2 = ax1.twinx() 222 | 223 | ax2.plot(lin, [calc(c) for c in lin], color='r') 224 | ax2.set_ylabel('Liczba przerwan', color='r') 225 | 226 | fig2 = plt.figure() 227 | npx = np.array(x) 228 | npy = np.array(y) 229 | npx = np.apply_along_axis(lambda x: x[0].total_seconds() / 60 / 60, 1, np.reshape((npx - max(npx)), (len(npx), 1))) 230 | 231 | plt.plot(npx, npy) 232 | 233 | 234 | -------------------------------------------------------------------------------- /provision/ssh.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | INSTANCE=`cat instance.dns` 3 | echo "sshing to $INSTANCE" 4 | ssh -i ~/.ssh/gpu-east.pem ubuntu@$INSTANCE $1 5 | -------------------------------------------------------------------------------- /run.py: -------------------------------------------------------------------------------- 1 | import teacher as q 2 | import ale_game as ag 3 | import dqn 4 | import theano 5 | import lasagne 6 | import network 7 | import simple_breakout 8 | import updates as u 9 | import pprint 10 | 11 | 12 | def main(game_name, network_type, updates_method, 13 | target_network_update_frequency, 14 | initial_epsilon, final_epsilon, test_epsilon, final_exploration_frame, replay_start_size, 15 | deepmind_rmsprop_epsilon, deepmind_rmsprop_learning_rate, deepmind_rmsprop_rho, 16 | rmsprop_epsilon, rmsprop_learning_rate, rmsprop_rho, 17 | phi_type, phi_method, 18 | epoch_size, n_training_epochs, n_test_epochs, 19 | visualize, record_dir, show_mood, 20 | replay_memory_size, no_replay, 21 | repeat_action, skip_n_frames_after_lol, max_actions_per_game, 22 | weights_dir, algo_initial_state_file, 23 | log_frequency, theano_verbose): 24 | args = locals() 25 | 26 | if theano_verbose: 27 | theano.config.compute_test_value = 'warn' 28 | theano.config.exception_verbosity = 'high' 29 | theano.config.optimizer = 'fast_compile' 30 | 31 | if game_name == 'simple_breakout': 32 | game = simple_breakout.SimpleBreakout() 33 | class P(object): 34 | def __init__(self): 35 | self.screen_size = (12, 12) 36 | 37 | def __call__(self, frames): 38 | return frames 39 | phi = P() 40 | else: 41 | ale = ag.init(game=game_name, display_screen=(visualize == 'ale'), record_dir=record_dir) 42 | game = ag.ALEGame(ale) 43 | if phi_type == '4': 44 | phi = ag.Phi4(method=phi_method) 45 | elif phi_type == '1': 46 | phi = ag.Phi(method=phi_method) 47 | else: 48 | raise RuntimeError("Unknown phi: {phi}".format(phi=phi_type)) 49 | 50 | if network_type == 'nature': 51 | build_network = network.build_nature 52 | elif network_type == 'nature_with_pad': 53 | build_network = network.build_nature_with_pad 54 | elif network_type == 'nips': 55 | build_network = network.build_nips 56 | elif network_type == 'nature_with_pad_he': 57 | build_network = network.build_nature_with_pad_he 58 | elif hasattr(network_type, '__call__'): 59 | build_network = network_type 60 | else: 61 | raise RuntimeError("Unknown network: {network}".format(network=network_type)) 62 | 63 | 64 | if updates_method == 'deepmind_rmsprop': 65 | updates = \ 66 | lambda loss, params: u.deepmind_rmsprop(loss, params, 67 | learning_rate=deepmind_rmsprop_learning_rate, 68 | rho=deepmind_rmsprop_rho, 69 | epsilon=deepmind_rmsprop_epsilon) 70 | elif updates_method == 'rmsprop': 71 | updates = \ 72 | lambda loss, params: lasagne.updates.rmsprop(loss, params, 73 | learning_rate=rmsprop_learning_rate, 74 | rho=rmsprop_rho, 75 | epsilon=rmsprop_epsilon) 76 | else: 77 | raise RuntimeError("Unknown updates: {updates}".format(updates=updates_method)) 78 | 79 | replay_memory = dqn.ReplayMemory(size=replay_memory_size) if not no_replay else None 80 | 81 | def create_algo(): 82 | algo = dqn.DQNAlgo(game.n_actions(), 83 | replay_memory=replay_memory, 84 | build_network=build_network, 85 | updates=updates, 86 | screen_size=phi.screen_size) 87 | 88 | algo.replay_start_size = replay_start_size 89 | algo.final_epsilon = final_epsilon 90 | algo.initial_epsilon = initial_epsilon 91 | 92 | algo.log_frequency = log_frequency 93 | algo.target_network_update_frequency = target_network_update_frequency 94 | algo.final_exploration_frame = final_exploration_frame 95 | return algo 96 | 97 | algo_train = create_algo() 98 | algo_test = create_algo() 99 | algo_test.final_epsilon = test_epsilon 100 | algo_test.initial_epsilon = test_epsilon 101 | algo_test.epsilon = test_epsilon 102 | 103 | 104 | import Queue 105 | algo_train.mood_q = Queue.Queue() if show_mood else None 106 | 107 | if show_mood is not None: 108 | import Queue 109 | algo_train.mood_q = Queue.Queue() 110 | if show_mood == 'plot': 111 | plot = Plot() 112 | elif show_mood == "log": 113 | plot = Log() 114 | 115 | def worker(): 116 | while True: 117 | item = algo_train.mood_q.get() 118 | plot.show(item) 119 | algo_train.mood_q.task_done() 120 | 121 | import threading 122 | t = threading.Thread(target=worker) 123 | t.daemon = True 124 | t.start() 125 | 126 | print(str(algo_train)) 127 | 128 | if visualize != 'q': 129 | visualizer = q.GameNoVisualizer() 130 | else: 131 | if game_name == 'simple_breakout': 132 | visualizer = simple_breakout.SimpleBreakoutVisualizer(algo_train) 133 | else: 134 | visualizer = ag.ALEGameVisualizer(phi.screen_size) 135 | 136 | teacher = q.Teacher(game=game, 137 | algo=algo_train, 138 | game_visualizer=visualizer, 139 | phi=phi, 140 | repeat_action=repeat_action, 141 | max_actions_per_game=max_actions_per_game, 142 | skip_n_frames_after_lol=skip_n_frames_after_lol, 143 | tester=False) 144 | 145 | tester = q.Teacher(game=game, 146 | algo=algo_test, 147 | game_visualizer=visualizer, 148 | phi=phi, 149 | repeat_action=repeat_action, 150 | max_actions_per_game=max_actions_per_game, 151 | skip_n_frames_after_lol=skip_n_frames_after_lol, 152 | tester=True) 153 | 154 | q.teach_and_test(teacher, tester, n_epochs=n_training_epochs, 155 | frames_to_test_on=n_test_epochs * epoch_size, 156 | epoch_size=epoch_size, 157 | state_dir=weights_dir, 158 | algo_initial_state_file=algo_initial_state_file) 159 | 160 | 161 | class Log(object): 162 | def __init__(self): 163 | pass 164 | 165 | def show(self, info): 166 | print(str(info['i_frame']) + " | Expectations: " + str(info['expectations'])) 167 | print(str(info['i_frame']) + " | Surprise: " + str(info['surprise'])) 168 | 169 | 170 | class Plot(object): 171 | 172 | def __init__(self): 173 | import matplotlib.pyplot as plt 174 | plt.ion() 175 | self.fig = plt.figure() 176 | plt.title('Surprise') 177 | plt.ylabel('Surprise (red), Expectation (blue)') 178 | plt.xlabel('frame') 179 | 180 | self.expectation = self.fig.add_subplot(2, 1, 1) 181 | self.expectations_l, = self.expectation.plot([], [], color='b', linestyle='-', lw=2) 182 | self.expectation.set_xlim([0, 105]) 183 | self.expectation.set_ylim([-5, 10]) 184 | 185 | self.surprise = self.fig.add_subplot(2, 1, 2) 186 | self.surprise_l, = self.surprise.plot([], [], color='r', linestyle='-', lw=2) 187 | self.surprise.set_xlim([0, 105]) 188 | self.surprise.set_ylim([-5, 5]) 189 | 190 | self.expectations_y = [] 191 | self.surprise_y = [] 192 | self.i = 0 193 | self.print_every_n = 1 194 | 195 | def show(self, info): 196 | self.i += 1 197 | 198 | self.expectations_y.append(info['expectations']) 199 | self.surprise_y.append(info['surprise']) 200 | if len(self.expectations_y) > 100: 201 | self.expectations_y = self.expectations_y[1:] 202 | self.surprise_y = self.surprise_y[1:] 203 | 204 | print(info) 205 | if self.i % self.print_every_n == 0: 206 | self.expectations_l.set_xdata(list(range(len(self.expectations_y)))) 207 | self.expectations_l.set_ydata(self.expectations_y) 208 | self.surprise_l.set_xdata(list(range(len(self.surprise_y)))) 209 | self.surprise_l.set_ydata(self.surprise_y) 210 | 211 | self.fig.canvas.draw() 212 | self.fig.canvas.flush_events() 213 | 214 | 215 | 216 | defaults = { 217 | 'game_name': 'space_invaders', 218 | 'visualize': False, 219 | 'record_dir': None, 220 | 'weights_dir': 'weights', 221 | 'theano_verbose': False, 222 | 'show_mood': None, 223 | 'replay_start_size': 50000, 224 | 'initial_epsilon': 1, 225 | 'final_epsilon': 0.1, 226 | 'test_epsilon': 0.05, 227 | 'log_frequency': 1, 228 | 'replay_memory_size': 400000, 229 | 'no_replay': False, 230 | 'network_type': 'nature', 231 | 'updates_method': 'rmsprop', 232 | 'repeat_action': 4, 233 | 'skip_n_frames_after_lol': 30, 234 | 'target_network_update_frequency': 10000, 235 | 'final_exploration_frame': 1000000, 236 | 'algo_initial_state_file': None, 237 | 'max_actions_per_game': 10000, 238 | 'phi_type': '1', 239 | 'phi_method': 'resize', 240 | 241 | 'deepmind_rmsprop_learning_rate': 0.00025, 242 | 'deepmind_rmsprop_rho': .95, 243 | 'deepmind_rmsprop_epsilon': 0.01, 244 | 'rmsprop_learning_rate': 0.0002, 245 | 'rmsprop_rho': .95, 246 | 'rmsprop_epsilon': 1e-6, 247 | 248 | 'n_training_epochs': 50, 249 | 'n_test_epochs': 1, 250 | 'epoch_size': 50000 251 | } 252 | 253 | 254 | -------------------------------------------------------------------------------- /run_dev.py: -------------------------------------------------------------------------------- 1 | import run 2 | import network 3 | 4 | d = run.defaults 5 | d['game'] = 'space_invaders' 6 | d['reshape'] = 'mean' 7 | #d['no_replay'] = True 8 | d['visualize'] = None #'q' 9 | d['replay_start_size'] = 100 10 | d['show_mood'] = run.Log 11 | d['network'] = network.build_nature 12 | d['weights_dir'] = None 13 | #d['weights_dir'] = 'weights' 14 | run.main(**d) 15 | -------------------------------------------------------------------------------- /run_play.py: -------------------------------------------------------------------------------- 1 | import run 2 | import network 3 | import updates 4 | 5 | d = run.defaults 6 | d['game'] = 'space_invaders' 7 | d['reshape'] = 'mean' 8 | d['phi'] = 'phi2' 9 | d['screen_size'] = 84 10 | d['no_replay'] = True 11 | 12 | d['visualize'] = 'ale' 13 | d['replay_start_size'] = 100 14 | d['log_frequency'] = 10 15 | d['final_epsilon'] = 0.1 16 | d['initial_epsilon'] = 0.1 17 | d['weights_dir'] = 'dqn20/weights' 18 | d['network'] = network.build_nature_with_pad 19 | #d['updates'] = updates.deepmind_rmsprop 20 | d['show_mood'] = run.Plot 21 | run.main(**d) 22 | -------------------------------------------------------------------------------- /run_simple_breakout.py: -------------------------------------------------------------------------------- 1 | import run 2 | import network 3 | 4 | d = run.defaults 5 | d['game_name'] = 'simple_breakout' 6 | #d['replay'] = 'uniform' 7 | d['visualize'] = None#'q' 8 | d['replay_start_size'] = 50000 9 | d['show_mood'] = None 10 | d['network_type'] = network.build_simple_breakout_W_caffe_normal 11 | d['updates_method'] = 'deepmind_rmsprop' 12 | d['weights_dir'] = 'weights_breakout' 13 | 14 | d['repeat_action'] = 1 15 | d['skip_n_frames_after_lol'] = 0 16 | d['target_network_update_frequency'] = 10000 17 | d['final_exploration_frame'] = 100000 18 | d['replay_memory_size'] = 400000 19 | import pprint 20 | pp = pprint.PrettyPrinter(indent=4) 21 | pp.pprint(d) 22 | run.main(**d) 23 | -------------------------------------------------------------------------------- /runs/run_dqn42.py: -------------------------------------------------------------------------------- 1 | import run 2 | import network 3 | import updates 4 | 5 | 6 | d = run.defaults 7 | d['game_name'] = 'space_invaders' 8 | d['phi'] = '4' 9 | d['network_type'] = 'nature_with_pad' 10 | d['updates_method'] = 'deepmind_rmsprop' 11 | d['weights_dir'] = 'dqn42/weights' 12 | d['epoch_size'] = 5000 13 | d['n_test_epochs'] = 0 14 | #d['dqn.replay_start_size'] = 100 15 | 16 | 17 | run.main(**d) 18 | -------------------------------------------------------------------------------- /runs/run_dqn43.py: -------------------------------------------------------------------------------- 1 | import run 2 | import network 3 | import updates 4 | 5 | 6 | d = run.defaults 7 | d['game_name'] = 'space_invaders' 8 | d['phi_type'] = '4' 9 | d['network_type'] = 'nature_with_pad' 10 | d['updates_method'] = 'deepmind_rmsprop' 11 | d['weights_dir'] = 'dqn43/weights' 12 | d['n_test_epochs'] = 0 13 | d['n_training_epochs'] = 500 14 | #d['dqn.replay_start_size'] = 100 15 | 16 | run.main(**d) 17 | -------------------------------------------------------------------------------- /simple_breakout.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import random 3 | 4 | class SimpleBreakoutVisualizer: 5 | def __init__(self, algo): 6 | import pygame 7 | self.screen_size = 12 8 | self.screen = pygame.display.set_mode((self.screen_size * 10, self.screen_size * 10 * 4)) 9 | self.mem = {} 10 | 11 | self.algo = algo 12 | 13 | def show(self, prev_frames): 14 | import pygame 15 | 16 | def l(x): 17 | if x not in self.mem: 18 | self.mem[x] = (x, x, x) 19 | return self.mem[x] 20 | 21 | f_l = np.frompyfunc(l, 1, 3) 22 | rect = pygame.Surface((self.screen_size * 10, self.screen_size * 10 * 4)) 23 | image = np.reshape(zip(*list(f_l(np.concatenate(prev_frames).flatten()))), (self.screen_size * 4, self.screen_size, 3)) 24 | 25 | image = np.transpose(image, [1, 0, 2]) 26 | pygame.surfarray.blit_array(rect, np.repeat(np.repeat(image, 10, axis=0), 10, axis=1)) 27 | self.screen.blit(rect, (0, 0)) 28 | 29 | pygame.display.flip() 30 | import time 31 | time.sleep(0.05) 32 | 33 | def next_game(self): 34 | pass 35 | 36 | 37 | class SimpleBreakout(object): 38 | 39 | def __init__(self): 40 | self.action_set = [4, 7, 10] 41 | 42 | self.finished = True 43 | self.bar = [random.randint(0, 11), 10] 44 | self.h = 12 45 | self.w = 12 46 | 47 | self.prev_frames = [np.zeros((self.h, self.w), dtype=np.uint8), 48 | np.zeros((self.h, self.w), dtype=np.uint8), 49 | np.zeros((self.h, self.w), dtype=np.uint8), 50 | np.zeros((self.h, self.w), dtype=np.uint8)] 51 | 52 | def reset_game(self): 53 | self.finished = False 54 | self.ball = [random.randint(0, 11), 0] 55 | self.cum_reward = 0 56 | self.bar = [random.randint(0, 11), 11] 57 | 58 | def n_actions(self): 59 | return len(self.action_set) 60 | 61 | def input(self, action): 62 | if action == 0: 63 | self.bar[0] = max(self.bar[0] - 1, 0) 64 | if action == 1: 65 | self.bar[0] = min(self.bar[0] + 1, 11) 66 | 67 | self.ball[1] += 1 68 | # print(self.bar, action) 69 | action_reward = 0 70 | game_over = False 71 | lol = False 72 | if self.ball[1] == 11: 73 | if abs(self.ball[0] - self.bar[0]) <= 1: 74 | action_reward = 1 75 | self.ball = [random.randint(0, 11), 0] 76 | else: 77 | game_over = True 78 | lol = True 79 | 80 | self.cum_reward += action_reward 81 | self.prev_frames = self.prev_frames[1:] 82 | self.prev_frames.append(self._as_frame()) 83 | self.finished = game_over 84 | 85 | return action_reward, lol 86 | 87 | def _as_frame(self): 88 | frame = np.zeros((12, 12), dtype=np.uint8) 89 | frame[self.ball[0], max(0, self.ball[1] - 1)] = 125 90 | frame[self.ball[0], min(11, self.ball[1] + 1)] = 125 91 | frame[self.ball[0], self.ball[1]] = 230 92 | frame[self.bar[0], self.bar[1]] = 200 93 | frame[max(0, self.bar[0]-1), self.bar[1]] = 180 94 | frame[min(11, self.bar[0]+1), self.bar[1]] = 180 95 | return frame 96 | 97 | def get_state(self): 98 | return self.prev_frames 99 | -------------------------------------------------------------------------------- /teacher.py: -------------------------------------------------------------------------------- 1 | from __future__ import division 2 | 3 | from collections import namedtuple 4 | import random 5 | import numpy as np 6 | 7 | 8 | Point = namedtuple('Point', 'x y') 9 | Experience = namedtuple('Experience', 's0 a0 r0 s1 game_over') 10 | 11 | 12 | class GameNoVisualizer: 13 | def show(self, game): 14 | pass 15 | 16 | def next_game(self): 17 | pass 18 | 19 | 20 | class RandomAlgo: 21 | def __init__(self, n_actions): 22 | self.n_actions = n_actions 23 | 24 | def init_state(self, state): 25 | pass 26 | 27 | def action(self): 28 | return random.randint(0, self.n_actions - 1) 29 | 30 | def feedback(self, x): 31 | pass 32 | 33 | 34 | class ConstAlgo: 35 | def __init__(self, const_actions): 36 | self.const_actions = const_actions 37 | self.i = 0 38 | 39 | def init_state(self, state): 40 | pass 41 | 42 | def action(self): 43 | self.i += 1 44 | return self.const_actions[self.i % len(self.const_actions)] 45 | 46 | def feedback(self, x): 47 | pass 48 | 49 | 50 | class Teacher: 51 | def __init__(self, game, algo, game_visualizer, phi, repeat_action, 52 | max_actions_per_game, 53 | skip_n_frames_after_lol, 54 | tester): 55 | self.game = game 56 | self.algo = algo 57 | self.game_visualizer = game_visualizer 58 | self.repeat_action = repeat_action 59 | self.phi = phi 60 | self.skip_n_frames_after_lol = skip_n_frames_after_lol 61 | self.max_actions_per_game = max_actions_per_game 62 | self.run_ave = 0 63 | self.tester = tester 64 | 65 | def teach(self, i_total_action, total_n_actions): 66 | game_rewards = [] 67 | while i_total_action < total_n_actions: 68 | game_reward, i_total_action = self.single_episode(i_total_action, total_n_actions) 69 | game_rewards.append(game_reward) 70 | return game_rewards 71 | 72 | def single_episode(self, i_total_action, total_n_actions): 73 | 74 | if self.game.finished: 75 | self.game.reset_game() 76 | self.algo.init_state(self.phi(self.game.get_state())) 77 | 78 | i_action = 0 79 | while not self.game.finished and i_action < self.max_actions_per_game and i_total_action < total_n_actions: 80 | i_action += 1 81 | i_total_action += 1 82 | 83 | self.single_action() 84 | 85 | # if not self.game.finished: 86 | # print "Failure." 87 | 88 | print "Game reward: " + str(self.game.cum_reward) 89 | print "" 90 | 91 | self.run_ave = self.run_ave * 0.99 + 0.01 * self.game.cum_reward 92 | print "Running average: " + str(self.run_ave) 93 | 94 | return self.game.cum_reward, i_total_action 95 | 96 | def single_action(self): 97 | old_state = self.phi(self.game.get_state()) 98 | action = self.algo.action(old_state) 99 | 100 | import numpy as np 101 | 102 | rewards, lols = zip(*[self.game.input(action) for _ in range(self.repeat_action)]) 103 | rep_reward = np.sum(rewards) 104 | lol = np.any(lols) 105 | new_state = self.phi(self.game.get_state()) 106 | 107 | if not self.tester: 108 | exp = Experience(old_state, action, rep_reward, new_state, lol) 109 | self.algo.feedback(exp) 110 | 111 | self.game_visualizer.show(new_state) 112 | 113 | if lol: 114 | for _ in range(self.skip_n_frames_after_lol): 115 | self.game.input(action) 116 | 117 | self.game_visualizer.show(new_state) 118 | 119 | 120 | def teach_and_test(teacher, tester, n_epochs, frames_to_test_on, epoch_size, state_dir, algo_initial_state_file=None): 121 | import numpy as np 122 | import os, re 123 | 124 | def extract_epoch(f): 125 | return int(re.match(r".*epoch_([0-9]*).npz", f).groups()[0]) 126 | 127 | if algo_initial_state_file is None: 128 | state_files = [(extract_epoch(f), os.path.join(state_dir,f)) for f in os.listdir(state_dir) if f.startswith("epoch_")] 129 | if len(state_files) > 0: 130 | algo_initial_state_file = max(state_files)[1] 131 | print("Latest file with state found: {state_file}".format(state_file=algo_initial_state_file)) 132 | 133 | if algo_initial_state_file is None: 134 | start_epoch = 0 135 | else: 136 | print("Using file with state: {state_file}".format(state_file=algo_initial_state_file)) 137 | import re 138 | start_epoch = extract_epoch(algo_initial_state_file) + 1 139 | teacher.algo.set_state(np.load(algo_initial_state_file)['arr_0']) 140 | teacher.algo.i_action = start_epoch * epoch_size 141 | print("Continuing from epoch: {start_epoch} and action: {i_action}".format( 142 | start_epoch=start_epoch, i_action=teacher.algo.i_action)) 143 | 144 | for i_epoch in range(start_epoch, n_epochs): 145 | import time 146 | start = time.time() 147 | print("Epsilon | {epsilon}".format(epsilon=teacher.algo.epsilon)) 148 | rewards = teacher.teach(i_epoch*epoch_size, (i_epoch+1) * epoch_size) 149 | end = time.time() 150 | print("Epoch {i_epoch} mean training result {result} on {n} games and {epoch_size} actions in {t} seconds.".format( 151 | i_epoch=i_epoch, result=np.mean(rewards), n=len(rewards), t=end-start, epoch_size=epoch_size)) 152 | 153 | algo_state = teacher.algo.get_state() 154 | filename = '{state_dir}/epoch_{i_epoch}.npz'.format(state_dir=state_dir, i_epoch=i_epoch) 155 | np.savez(filename, algo_state) 156 | print("Algo state saved: {filename}".format(filename=filename)) 157 | tester.algo.set_state(algo_state) 158 | 159 | start = time.time() 160 | rewards = tester.teach(0, frames_to_test_on) 161 | end = time.time() 162 | print(tester.algo.epsilon) 163 | print("Epoch {i_epoch} mean validation result: {result} on {n} games in {t} seconds.".format( 164 | i_epoch=i_epoch, result=np.mean(rewards), n=len(rewards), t=end-start)) 165 | 166 | 167 | -------------------------------------------------------------------------------- /test/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/maciejjaskowski/deep-q-learning/88363786881830b8e410230d6ad72d964673eb20/test/__init__.py -------------------------------------------------------------------------------- /test/test_loss.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | import theano.tensor as T 3 | import theano 4 | import numpy as np 5 | import dqn 6 | 7 | 8 | class MyTestCase(unittest.TestCase): 9 | def setUp(self): 10 | # data which will be used in various test methods 11 | self.avals = np.array([[1, 5, 3], [2, 4, 1]]) 12 | self.bvals = np.array([[2, 3, 1, 8], [4, 2, 1, 1], [1, 4, 8, 5]]) 13 | 14 | def test_validity(self): 15 | theano.config.on_unused_input = 'warn' 16 | a0_var = T.dmatrix('a0') 17 | r0_var = T.dmatrix('r0') 18 | fri_var = T.dmatrix("fri") 19 | out = T.dmatrix("out") 20 | out_stale = T.dmatrix("out_stale") 21 | 22 | f = theano.function([a0_var, r0_var, fri_var, out, out_stale], 23 | dqn.build_loss(out, out_stale, a0_var, r0_var, fri_var, gamma=0.5)) 24 | 25 | loss, not_loss, y, q = f(np.array([[1, 0, 0, 0, 0, 0], 26 | [0, 1, 0, 0, 0, 0]]), 27 | np.array([[1], 28 | [0]]), 29 | np.array([[1], 30 | [1]]), 31 | np.array([[-5, 1, 2, 3, 4, 7], 32 | [1, 4, 3, 4, 5, 9]]), 33 | np.array([[-5, 1, 2, 3, 4, 5], 34 | [1, 2, 3, 4, 5, 6]])) 35 | 36 | self.assertTrue(np.all(y == [[3.5], [3]])) 37 | self.assertTrue(np.all(q == [[-5], [4]])) 38 | print(loss) 39 | print(not_loss) 40 | self.assertTrue(loss == 8.5) 41 | 42 | def test_validity2(self): 43 | theano.config.on_unused_input = 'warn' 44 | a0_var = T.dmatrix('a0') 45 | r0_var = T.dmatrix('r0') 46 | fri_var = T.dmatrix("fri") 47 | out = T.dmatrix("out") 48 | out_stale = T.dmatrix("out_stale") 49 | 50 | f = theano.function([a0_var, r0_var, fri_var, out, out_stale], 51 | dqn.build_loss(out, out_stale, a0_var, r0_var, fri_var, gamma=0.5)) 52 | 53 | sqr_mean, mean, y, q = f(np.array([[1, 0, 0, 0, 0, 0], 54 | [0, 1, 0, 0, 0, 0], 55 | [0, 0, 0, 0, 0, 1]]), 56 | np.array([[1], 57 | [0], 58 | [5]]), 59 | np.array([[1], 60 | [1], 61 | [0]]), 62 | np.array([[-5, 1, 2, 3, 4, 7], 63 | [1, 4, 3, 4, 5, 9], 64 | [0, 9, 0, 3, 2, 1]]), 65 | np.array([[-5, 1, 2, 3, 4, 5], 66 | [1, 2, 3, 4, 5, 6], 67 | [8, 0, -1, -1, 2, 3]])) 68 | 69 | print(y, q) 70 | 71 | -------------------------------------------------------------------------------- /updates.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copy pasted from https://github.com/spragunr/deep_q_rl/ 3 | 4 | Gradient update rules for the deep_q_rl package. 5 | Some code here is modified from the Lasagne package: 6 | 7 | https://github.com/Lasagne/Lasagne/blob/master/LICENSE 8 | """ 9 | 10 | import theano 11 | import theano.tensor as T 12 | from lasagne.updates import get_or_compute_grads 13 | from collections import OrderedDict 14 | import numpy as np 15 | 16 | # The MIT License (MIT) 17 | 18 | # Copyright (c) 2014 Sander Dieleman 19 | 20 | # Permission is hereby granted, free of charge, to any person obtaining a copy 21 | # of this software and associated documentation files (the "Software"), to deal 22 | # in the Software without restriction, including without limitation the rights 23 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 24 | # copies of the Software, and to permit persons to whom the Software is 25 | # furnished to do so, subject to the following conditions: 26 | 27 | # The above copyright notice and this permission notice shall be included in all 28 | # copies or substantial portions of the Software. 29 | 30 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 31 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 32 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 33 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 34 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 35 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 36 | # SOFTWARE. 37 | # The MIT License (MIT) 38 | 39 | # Copyright (c) 2014 Sander Dieleman 40 | 41 | # Permission is hereby granted, free of charge, to any person obtaining a copy 42 | # of this software and associated documentation files (the "Software"), to deal 43 | # in the Software without restriction, including without limitation the rights 44 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 45 | # copies of the Software, and to permit persons to whom the Software is 46 | # furnished to do so, subject to the following conditions: 47 | 48 | # The above copyright notice and this permission notice shall be included in all 49 | # copies or substantial portions of the Software. 50 | 51 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 52 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 53 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 54 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 55 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 56 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 57 | # SOFTWARE. 58 | 59 | def deepmind_rmsprop(loss_or_grads, params, learning_rate, 60 | rho, epsilon): 61 | """RMSProp updates [1]_. 62 | Scale learning rates by dividing with the moving average of the root mean 63 | squared (RMS) gradients. 64 | Parameters 65 | ---------- 66 | loss_or_grads : symbolic expression or list of expressions 67 | A scalar loss expression, or a list of gradient expressions 68 | params : list of shared variables 69 | The variables to generate update expressions for 70 | learning_rate : float or symbolic scalar 71 | The learning rate controlling the size of update steps 72 | rho : float or symbolic scalar 73 | Gradient moving average decay factor 74 | epsilon : float or symbolic scalar 75 | Small value added for numerical stability 76 | Returns 77 | ------- 78 | OrderedDict 79 | A dictionary mapping each parameter to its update expression 80 | Notes 81 | ----- 82 | `rho` should be between 0 and 1. A value of `rho` close to 1 will decay the 83 | moving average slowly and a value close to 0 will decay the moving average 84 | fast. 85 | Using the step size :math:`\\eta` and a decay factor :math:`\\rho` the 86 | learning rate :math:`\\eta_t` is calculated as: 87 | .. math:: 88 | r_t &= \\rho r_{t-1} + (1-\\rho)*g^2\\\\ 89 | \\eta_t &= \\frac{\\eta}{\\sqrt{r_t + \\epsilon}} 90 | References 91 | ---------- 92 | .. [1] Tieleman, T. and Hinton, G. (2012): 93 | Neural Networks for Machine Learning, Lecture 6.5 - rmsprop. 94 | Coursera. http://www.youtube.com/watch?v=O3sxAc4hxZU (formula @5:20) 95 | """ 96 | 97 | grads = get_or_compute_grads(loss_or_grads, params) 98 | updates = OrderedDict() 99 | 100 | for param, grad in zip(params, grads): 101 | value = param.get_value(borrow=True) 102 | 103 | acc_grad = theano.shared(np.zeros(value.shape, dtype=value.dtype), 104 | broadcastable=param.broadcastable) 105 | acc_grad_new = rho * acc_grad + (1 - rho) * grad 106 | 107 | acc_rms = theano.shared(np.zeros(value.shape, dtype=value.dtype), 108 | broadcastable=param.broadcastable) 109 | acc_rms_new = rho * acc_rms + (1 - rho) * grad ** 2 110 | 111 | 112 | updates[acc_grad] = acc_grad_new 113 | updates[acc_rms] = acc_rms_new 114 | 115 | updates[param] = (param - learning_rate * 116 | (grad / 117 | T.sqrt(acc_rms_new - acc_grad_new **2 + epsilon))) 118 | 119 | return updates -------------------------------------------------------------------------------- /video/compile-video.sh: -------------------------------------------------------------------------------- 1 | ffmpeg -r 60 -i record/%06d.png video.webm 2 | -------------------------------------------------------------------------------- /video/video.webm: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/maciejjaskowski/deep-q-learning/88363786881830b8e410230d6ad72d964673eb20/video/video.webm --------------------------------------------------------------------------------