├── .gitignore ├── README.md ├── Simulation1.ipynb ├── Simulation1_1 - Analysis.ipynb ├── Simulation1_1 - Training.ipynb ├── Simulation2_batch32 - Analysis.ipynb ├── Simulation2_batch32 - Training.ipynb ├── Simulation2_batch51 - Analysis.ipynb ├── Simulation2_batch51 - Training.ipynb ├── agent.py ├── const.py ├── evaluate.py ├── evaluate_extended.py ├── generate_actions.py ├── mockSQLenv.py └── utilities.py /.gitignore: -------------------------------------------------------------------------------- 1 | .ipynb_checkpoints/ 2 | __pycache__/ 3 | ignore_* 4 | models/ 5 | *.zip 6 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # CTF-SQL 2 | Modelling SQL Injection Using Reinforcement Learning 3 | 4 | ### Requirements 5 | The following code requires *numpy, scipy, matplotlib* and [OpenAI gym](https://github.com/openai/gym); [stable-baselines3](https://stable-baselines3.readthedocs.io/en/master/) (together with [pytorch](https://pytorch.org/)) is used to train reinforcement learning agents. 6 | 7 | **Warning:** Simulation1 and Simulation2 rely at the moment on synthetic SQL server simulators. Simulation1 uses the module *mockSQLenv.py*. Simulation2 uses the OpenAI gym environment [gym-CTF-SQL](https://github.com/avalds/gym-CTF-SQL) (check the gym repository for installing and running the environment) 8 | 9 | ### Content 10 | The project *CTF-SQL* contains the simulations running reinforcement agent on a CTF challenge containing a simple SQL injection vulnerability. Every *SimulationX* file contains a simulation, including training and analysis. 11 | - *Simulation1* runs a tabular Q-learning agent; 12 | - *Simulation2* runs a deep Q-learning agent (with different batch settings). 13 | Details about the setup and the interpretation may be found in [1]. 14 | 15 | ### References 16 | 17 | \[1\] Erdodi, L., Sommervoll, A.A. and Zennaro, F.M., 2020. Simulating SQL Injection Vulnerability Exploitation Using Q-Learning Reinforcement Learning Agents. arXiv preprint. -------------------------------------------------------------------------------- /Simulation1_1 - Analysis.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Simulation 1.1 - Analyzing the agents\n", 8 | "\n", 9 | "\n", 10 | "## Importing libraries" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 1, 16 | "metadata": {}, 17 | "outputs": [], 18 | "source": [ 19 | "import numpy as np\n", 20 | "import matplotlib.pyplot as plt\n", 21 | "from scipy.ndimage import uniform_filter1d\n", 22 | "from IPython.display import clear_output\n", 23 | "import datetime\n", 24 | "import joblib\n", 25 | "from tqdm import tqdm\n", 26 | "\n", 27 | "import const\n", 28 | "import utilities as ut\n", 29 | "import mockSQLenv as SQLenv\n", 30 | "import agent as agn" 31 | ] 32 | }, 33 | { 34 | "cell_type": "markdown", 35 | "metadata": {}, 36 | "source": [ 37 | "## Defining the parameters of the simulations" 38 | ] 39 | }, 40 | { 41 | "cell_type": "code", 42 | "execution_count": 2, 43 | "metadata": {}, 44 | "outputs": [], 45 | "source": [ 46 | "n_simulations = 10\n", 47 | "n_episodes_training = 10**6\n", 48 | "n_episodes_test = 10**2\n", 49 | "\n", 50 | "exploration_train = 0.1\n", 51 | "exploration_test = 0\n", 52 | "learningrate = 0.1\n", 53 | "discount = 0.9\n", 54 | "max_steps = 1000\n", 55 | "\n", 56 | "flag_reward = 10\n", 57 | "query_reward = -1" 58 | ] 59 | }, 60 | { 61 | "cell_type": "markdown", 62 | "metadata": {}, 63 | "source": [ 64 | "## Loading the statistics" 65 | ] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": 3, 70 | "metadata": {}, 71 | "outputs": [], 72 | "source": [ 73 | "train_data = joblib.load('ignore_simul1_traindata_20210217232217657942.pkl')\n", 74 | "test_data = joblib.load('ignore_simul1_testdata_20210217232217657942.pkl')" 75 | ] 76 | }, 77 | { 78 | "cell_type": "markdown", 79 | "metadata": {}, 80 | "source": [ 81 | "## Analyzing the agent\n", 82 | "\n", 83 | "### Training: number of states\n", 84 | "\n", 85 | "Plotting the evolution in the number of states (average and standard deviation over 10 repetitions)." 86 | ] 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": 11, 91 | "metadata": {}, 92 | "outputs": [ 93 | { 94 | "data": { 95 | "text/plain": [ 96 | "array([2000000. , 2000000.1, 2000000.2, ..., 2099999.7, 2099999.8,\n", 97 | " 2099999.9])" 98 | ] 99 | }, 100 | "execution_count": 11, 101 | "metadata": {}, 102 | "output_type": "execute_result" 103 | } 104 | ], 105 | "source": [ 106 | "0.1*np.arange(0,n_episodes_training)+2*10**6" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": 13, 112 | "metadata": {}, 113 | "outputs": [ 114 | { 115 | "data": { 116 | "text/plain": [ 117 | "array([4.76759389e+00, 1.54385359e+06])" 118 | ] 119 | }, 120 | "execution_count": 13, 121 | "metadata": {}, 122 | "output_type": "execute_result" 123 | } 124 | ], 125 | "source": [ 126 | "params = np.polyfit(range(200000, n_episodes_training), np.mean(train_data[:,2,200000:],axis=0),1)" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": 14, 132 | "metadata": {}, 133 | "outputs": [ 134 | { 135 | "name": "stdout", 136 | "output_type": "stream", 137 | "text": [ 138 | "Linear regression params: slope = 4.767593893285314 ; intercept = 1543853.5924806432\n" 139 | ] 140 | }, 141 | { 142 | "name": "stderr", 143 | "output_type": "stream", 144 | "text": [ 145 | "/home/fmzennaro/miniconda2_1/envs/quantumgymstable/lib/python3.7/site-packages/IPython/core/events.py:88: UserWarning: Creating legend with loc=\"best\" can be slow with large amounts of data.\n", 146 | " func(*args, **kwargs)\n", 147 | "/home/fmzennaro/miniconda2_1/envs/quantumgymstable/lib/python3.7/site-packages/IPython/core/pylabtools.py:128: UserWarning: Creating legend with loc=\"best\" can be slow with large amounts of data.\n", 148 | " fig.canvas.print_figure(bytes_io, **kw)\n" 149 | ] 150 | }, 151 | { 152 | "data": { 153 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaQAAAEGCAYAAAAqmOHQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3hU1dr38e+dQhISkgAJRYqAglKkhiZFbIhYUAQFC1hRPCrn+D7YjuUc7L0fkceC2AU9ymNDRIr0Il3pNbQkhISEENLu94+9gwFSZkYmmST357rmysyavddak0nyy957zVqiqhhjjDEVLaiiO2CMMcaABZIxxpgAYYFkjDEmIFggGWOMCQgWSMYYYwJCSEV3INDFxcVps2bNKrobxhhTqSxbtixFVeO92ccCqQzNmjVj6dKlFd0NY4ypVERku7f72Ck7Y4wxAcECyRhjTECwQDLGGBMQ7BqSD3Jzc0lMTCQ7O7uiu2JOsvDwcBo3bkxoaGhFd8WYascCyQeJiYnUqlWLZs2aISIV3R1zkqgq+/fvJzExkebNm1d0d4ypduyUnQ+ys7OpW7euhVEVIyLUrVvXjnyNqSAWSD6yMKqa7H01puJYIBljjAkIFkiVVFRUFAC7d+9myJAhFdybwHD22WdXdBeMqRKkyK08WSBVcqeccgpTpkzxaxt5eXk+Peep/Pz8v1wHwPz5809KPcZUVxURQkVZIFVy27Zto127dgBMnDiRwYMHM2DAAFq2bMl99913dLuffvqJnj170rlzZ4YOHUpmZiYA48aNo2vXrrRr145Ro0ZRuIJwv379eOihhzjnnHN49dVXj2nzX//6F6NGjaJ///6MGDGC/Px8xo4dS9euXWnfvj1vv/02AAUFBdx55520bduWSy+9lIEDBx4Nz2bNmjFu3Dh69+7N5MmT2bx5MwMGDKBLly706dOHdevWATB58mTatWtHhw4d6Nu3LwBr166lW7dudOzYkfbt27Nx40bgz6NGVWXs2LG0a9eOs846i88//xyAWbNm0a9fP4YMGcKZZ57Jddddh62YbEzFB1EhG/Z9MvTrd2LZ1VfDnXdCVhYMHHji8zfe6NxSUuD4U26zZvnclRUrVrB8+XLCwsI444wzuPvuu4mIiOCJJ57g559/JjIykmeffZaXXnqJRx99lLvuuotHH30UgBtuuIFvv/2Wyy67DIC0tDRmz55dbDvLli1j7ty5REREMGHCBGJiYliyZAlHjhyhV69e9O/fn2XLlrFt2zZWr15NUlISrVu35uabbz5aR3h4OHPnzgXg/PPPZ/z48bRs2ZJFixZx55138ssvvzBu3DimTZtGo0aNSEtLA2D8+PGMGTOG6667jpycnBOOsL766itWrFjBypUrSUlJoWvXrkfDbPny5axdu5ZTTjmFXr16MW/ePHr37u3z99uYyioQAuh4FkhVzPnnn09MTAwAbdq0Yfv27aSlpfH777/Tq1cvAHJycujZsycAM2fO5LnnniMrK4vU1FTatm17NJCuueaaEtu5/PLLiYiIAJyjr1WrVh09+klPT2fjxo3MnTuXoUOHEhQURIMGDTj33HOPqaOw/szMTObPn8/QoUOPPnfkyBEAevXqxY033sjVV1/N4MGDAejZsydPPvkkiYmJDB48mJYtWx5T79y5cxk+fDjBwcHUr1+fc845hyVLlhAdHU23bt1o3LgxAB07dmTbtm0WSKZa8SiIVKmZlUVWZKS/u3MMC6STobQjmpo1S38+Lu4vHREdLyws7Oj94OBg8vLyUFUuvPBCPv3002O2zc7O5s4772Tp0qU0adKEf/3rX8d8BieylB/Gos+pKq+//joXXXTRMdt89913pfa1sI6CggJiY2NZsWLFCduMHz+eRYsW8d1339GxY0dWrFjBtddeS/fu3fnuu++46KKLeOeddzjvvPOO6U9Jivv+GFMdeBJEUlDApd9+y0NPPcW2Zs0Y/tlnfu9XUXYNqRro0aMH8+bNY9OmTQBkZWWxYcOGo+ETFxdHZmamz4MjLrroIt566y1yc3MB2LBhA4cOHaJ37958+eWXFBQUsG/fPmaVELzR0dE0b96cyZMnA06grFy5EoDNmzfTvXt3xo0bR1xcHDt37mTLli20aNGCe+65h8svv5xVq1YdU1/fvn35/PPPyc/PJzk5mTlz5tCtWzefXpsxlZ0314cef+QRpg4aRL2kJL7ufRGSnevPrp3AjpCqgfj4eCZOnMjw4cOPngp74oknaNWqFbfddhtnnXUWzZo1o2vXrj7Vf+utt7Jt2zY6d+6MqhIfH8/XX3/NVVddxYwZM2jXrh2tWrWie/fuR08nHu/jjz9m9OjRPPHEE+Tm5jJs2DA6dOjA2LFj2bhxI6rK+eefT4cOHXjmmWf46KOPCA0NpUGDBkevgRW68sorWbBgAR06dEBEeO6552jQoMHRgRLGVAeehFCNI0cYMWkSi7t1Y1WHDkwcMYK5jdqzPLgJ4VsPELVkJ/Rp4fe+FhIbZVS6hIQEPX6Bvj/++IPWrVtXUI8ql8zMTKKioti/fz/dunVj3rx5NGjQoKK7VSp7f01l5emRUM1Dhxg1YQL/88ILNNq9myfH3s8zV95B9LxthKYcIi8mnIxezcjo2pSCCN8mGhaRZaqa4M0+doRk/OrSSy8lLS2NnJwcHnnkkYAPI2MqI29GzI155RUefuIJ4vbv58v+l/LEw++QkhRM3W/WcqRJLMnDO5HVrgEEl/8VHQsk41clXTcyxvx1ngZRXHIyKXFxIEJ8cjKfn3s5b/QdRtaePNhRQHa7ehzs3YKcU2v7tb9lsUDykaraRJxVkJ3CNpWBp395muzYwdjnn+fWd95hyBeTmdm8C683voTw/AMUpCiZZzfj4NnNyK9T06/99ZRfj8lEJFZEpojIOhH5Q0R6ikgdEZkuIhvdr7XdbUVEXhORTSKySkQ6F6lnpLv9RhEZWaS8i4isdvd5TdyE8KUNb4SHh7N//37741XFFK6HFB4eXtFdMaZYno6Ya7V+Pe/efDObTzuN69+dyJhbHmHtqhrU++g3Qg5mk3ppGxIfPI8Dl7YJmDAC/x8hvQr8qKpDRKQGUBN4CJihqs+IyAPAA8D9wMVAS/fWHXgL6C4idYDHgARAgWUiMlVVD7jbjAIWAt8DA4Af3Do9bsPbF9W4cWMSExNJTk725XtiAljhirHGBApvz8NIQQE/DhhAXlY+19zzCssiWyA5BWTXiSLj8uZktW0AQYF5dsdvgSQi0UBf4EYAVc0BckRkENDP3ewDYBZOWAwCJqlz2LHQPbpq6G47XVVT3XqnAwNEZBYQraoL3PJJwBU4geRVG6q6x5vXFhoaaiuKGmP8ypvIOHvePO4YP55b33kH9mZx8T8mcHhPDoiQ1boBB3s3J6dJrN/6erL48wipBZAMvC8iHYBlwBigfmEAqOoeEannbt8I2Flk/0S3rLTyxGLK8aGNYwJJREbhHHnRtGlT7161Mcb8BR4HkSr9f/qJh556il6/zuWLhIto/Nps8vfncig8hMy+Lcjo2Yz82Aif+lERFyT8GUghQGfgblVdJCKv4pw6K0lx74P6UF4aj/ZR1QnABHA+h1RGncYY85d5c0RUNyWFHwcMoNXqP3i771DuePAfHM4PJZdQMi5vRWaXxmiYb3/eK/IPnj8DKRFIVNVF7uMpOIG0r/A0mXtKLqnI9k2K7N8Y2O2W9zuufJZb3riY7fGhDWOMqRCeBlFIbi7tV63ity5dSAuK4JWu17DgwjPJ1yCym9bhYJ8WHD6zns/XhwLhP2+/jbJT1b3AThE5wy06H/gdmAoUjpQbCXzj3p8KjHBHwvUA0t3TbtOA/iJS2x0t1x+Y5j6XISI93NF1I46ry5s2jDGm3HizImtYdjZ3vPUWG1q14o2hN3PK+wtp9Pwsfq3dlvSOTdhzd2/23d6Tw23qex1GWuQWCPw9yu5u4GN3hN0W4CacEPxCRG4BdgCFaw58DwwENgFZ7raoaqqIPA4scbcbVzjAARgNTAQicAYz/OCWP+NNG8YYUx68iYuojAzuGD+eMS+/zJqoZtwx4D7WxzQleFs6B/u2IOPsZuTHVJ7rQ56wuezKUNxcdsYY4w1fTqKdtXQ5997+KG/2HUZyWCx5tSM42Ks5mQmN0XDf5pcrz7/2NpedMcYEEG+CqMGePfy/F1+k4FAeT192O2mLk/jXhXeQfWptMno3J6tNfZ/ml6tMhxwWSMYYcxJ5ezTUbOtW7nvuORK+m8OkjpfwXevziZ67lax27ueHmvo2v1xlCqJCFkjGGHMS+HJabtjHn3DD4//h3a6DePbaSyE0iPQepzrXh2r7NqVPZQyiQhZIxhjzF3gbRAlLlpCrQWzJrcP87fEsuOJBNLoGB/qeTmbXJj59fqgyh1BRFkjGGOMDr4JIlXNmz+aO519ja1YskxIupU7wXuf60GXtfJ5frqoEUSELJGOM8YK3sXHejBmMfO4tloY1559tb6QgKIjs1vGkntuqWl0f8oQFkjHGeMCbIArKz4cCJWx9CukzU3m0002ESAEHe5zKgb6nV8vrQ56wQDLGmBJ4ezQUmpPDsEkf027KXN7rOojs3GD2xtYh8+xTOdi9WbW+PuQJCyRjjDmOt0EUkZXFtW9PJH7mer5tcTZzOg4mvKaQfGknuz7kBQskY4xx+TJ0u8bONC4b9w4za7dCWzchsl4we4b0JOdUuz7kLQskY0y1520Qxe1L4rLXP2Fagy6EJh5kcf3WRDWtwR9X965U6w8FGgskY0y15MvRUKMt27ngzSlszorll5iWRCans/fSttX+80MniwWSMaZa8em0XGomg//9v6yRRswKP5MmESmEnlufjRd2setDJ5EFkjGmWvAliBqu3cKRdZlE/baLheGn0ypvH+uvbMfc7pf41AcLotJZIBljqjSvg0iVTt/P5ZTvV7GmVjNCg4XMhCYc7NOcbXFRXrdvIeQ5CyRjTJXkdRDl5dP18xlELtjB1qiG5IfWpX3oHubceQXpDeO8bt+CyHsWSMaYKsOX03JyOJdai3cQ++tmkjJzaVIQRIda+5hx31WsqhPjdX0WRL6zQDLGVHo+DVTYd5BOk35mf0owuRLC4dPq0qxlBosvv5aciHCv6rIQOjkskIwxlZYvQRS5OYk2n8whOTOM/QVB9N29gin/vp2Dpzcgycu6LIhOLgskY0yl43UQFSgR65Jo9N0KcvfnkZ2dz6C989lwVQ/ev+oRNMi7pcEtiPzDAskYU2n4MlAhft5GIuftQA/mciQ6jKF7ZrF8+Hm8etFTIN7VaEHkX34NJBHZBmQA+UCeqiaISB3gc6AZsA24WlUPiIgArwIDgSzgRlX9za1nJPCwW+0TqvqBW94FmAhEAN8DY1RVfWnDGBOYfBqokJ1Lw5/XEjN3C1mE0ezgbhbfNpCs9g15KfgCr+qyECo/3h2n+uZcVe2oqgnu4weAGaraEpjhPga4GGjp3kYBbwG44fIY0B3oBjwmIoWzFr7lblu43wBf2jDGBB7B+zAKTs+m6WeLOP2x7wmdu4vO237nln0/k3J7N7I6NYJgz//kKRZG5a0iTtkNAvq59z8AZgH3u+WTVFWBhSISKyIN3W2nq2oqgIhMBwaIyCwgWlUXuOWTgCuAH7xtQ1X3+PH1GmO84MsRUUhSBjFzthC5fBeSX0D/9fOp3yCfSY/fzsctW3pVl4VQxfF3ICnwk4go8LaqTgDqFwaAqu4RkXruto2AnUX2TXTLSitPLKYcH9o4JpBEZBTOERRNmzb19jUbY3zgy4wKYVv20+S7FeTuPkKQFJDWozmHuzfmm9wOJDZp4l113rZvTjp/B1IvVd3tBsJ0EVlXyrbF/TyqD+Wl8WgfNzgnACQkJNjPqTF+5HUQ5RdQc81eTvl+Fbnp+dTKSmfo7z+z8+KOvDXoMgASifWoKvvlDix+DSRV3e1+TRKR/+JcA9pXeJrMPSVXOPQ/ESj6L01jYLdb3u+48llueeNitseHNowx5czbIJKcPKKWJhL96xZCDhymyf6dXL3uF7Zc3JlnHn+ZgzGez6pgQRSY/DaoQUQiRaRW4X2gP7AGmAqMdDcbCXzj3p8KjBBHDyDdPe02DegvIrXdwQz9gWnucxki0sMdPTfiuLq8acMYU068HawQlHmE2J/W03zcj9SZupa8WmHEdwqjzVm53DttIs8/9IDHYWQDFQKbP4+Q6gP/dbKCEOATVf1RRJYAX4jILcAOYKi7/fc4w7E34QzJvglAVVNF5HFgibvduMIBDsBo/hz2/YN7A3jGmzaMMf7l00CF/VnEzNlE9OIdFKhw7sZF0DGOd+8cwz5gKZ4P37YQqhzEGXBmSpKQkKBLly6t6G4YUyn5NMfcrnSiZ28mctVuQvLzGbz2F/qk/86HY27jy6uuoiA42OO67K9bxRGRZUU+7uMRm6nBGHPS+TJiLnxTCrEzNxG2JZWCsBD6pa1h6Lpf+M/YMTw/8BWvZlWwIKqcLJCMMSeNTyPmVu+hzi8bCE7KIu7QAdL7tGDz5V35oqAPH0Tcb0FUjVggGWP+Mq9HzOXmE7VkJ7VnbUQO5tA8dRejF06BltE80uFJNDyUw4R6VJeFUNVhgWSM8Zm3QRSUeYRaC3dQa8E2gg/l0GHvBkYvmMy+bqfx709f5ve2bT2uy4Ko6rFAMsZ4zdsgCknOJHr2Fmr9thMKIOvMehzs24Kuy7K489GP2Nqihcd1WRBVXRZIxhiP+DRiLjGN6NlbiFy9h9CCPIaunM41v0/nytk/kNyiLlNaDC27EpcFUdVngWSMKZVPI+Y2JBMzewvhW/YTnn+Emxd/w+CNs/lo1I2c98l8UuvW9bw6b9s3lZYFkjGmWL6MmItctYfo2ZupsTeDvOhwMi48nU/uG8F/hw2h0x1vkFmrlkdVWQhVT2UGkogMBX5U1QwReRjojLNIni1sZ0wV5PWIuSN5RC3ZSfTcLYSkZdMkYx/D/5jOPd9/QH54DXotXejx0G0LourNkyOkR1R1soj0Bi4CXsBZ2K67X3tmjClXPo2Ym7+N6AXbCDqcx1nJm/n77I9oEJrJsw8+ACHuVJkehJEFkQHPAinf/XoJ8JaqfiMi//Jfl4wx5cWnOeZSDhH96xYilyUi+QXE1FbenzAWGtTkqWcf4r9XXokGeTZvswWRKcqTQNolIm8DFwDPikgY5bP0uTHGT3waMbczjeg5m6m5ei/BFNAkIoP5Yy5nZ92ajE2oycxzz7VTc+Yv8SSQrgYGAC+oapq7vtBY/3bLGOMPXgdRgVJz7V5qzd1K+PYD1CCP63/7jjvmT+GnKwYyJ/5aAGaed16ZVVkImbKUGUiqmiUiSUBvYCOQ5341xlQSPg1UWLqTWvO2EZqaRXhoHmPmfMS1v33PtMsv4aJfZ7CiUyeP6rIgMp7yZJTdY0ACcAbwPhAKfAT08m/XjDF/hS+n5YIPZlNr/jaiFm4nODsPbVCT5Os60/bQdsITo+j+3jLWn3mmR3VZEBlveXLK7kqgE/AbOMuSF64Ea4wJPL4EUejeg0TP2Urkyl1IgdIuczuPff0Gyy7py+i/j2cJDVnSo4dHdVkQGV95Ekg5qqoionB0OXJjTIDxdUaF6LlbidiYggRDn6TVPD7lVWrnH+I/d97Jy//4h+fVedu+McfxJJC+cEfZxYrIbcDNwDv+7ZYxxlNeB1FuPlErdlHr163USMokr1YYB/q34t9fvMr1//2MV/7f33nzb38jrXZtj6qzIDIni0dLmIvIhUB/nJ/9aao63d8dCxS2hLkJVD4v/bBwG8GZOYTXVG5d+jVf3ncTC3v3om5KCtnh4RyKiiqzLgshUxa/LGEuIs+q6v3A9GLKjDHlzOulH1IOET1nC5G/JRKUV0DtqFzGzn+P4b/+H1tatGD6IWfG7f1xcWXWZUFk/MmTU3YXAseHz8XFlBlj/MjbIDr6QdY1eyE4iMxOjXjntfu5cs6PrGnblus+/pgvrr6a/JCy/wxYEJnyUOJPooiMBu4EWojIqiJP1QLm+btjxhiHV0GkSvj6ZGJmbyZ8ayoF4SG0iMlg7ujLyYupyTcZ1zDx3jv4v8su82h6HwsiU55K+9foE+AH4GnggSLlGaqa6mkDIhIMLAV2qeqlItIc+AyogzOU/AZVzXGnJJoEdAH2A9eo6ja3jgeBW3Dm1btHVae55QOAV4Fg4B1VfcYt97oNYwKNV0FUoNRcs5eYmZuosecgBbVqkBC6k5fGP07Tfbs5r9sMZp53Hh/ceGOZVVkImYpS4r9IqpquqttUdbiqbgcO4/ysRolIUy/aGAP8UeTxs8DLqtoSOIATNLhfD6jq6cDL7naISBtgGNAWZwqj/4hIsBt0b+KcPmwDDHe39boNYwKJ4EUY5RdQc9UeGr4+l/hPfiMoJ4/zCtax4rmhTHliNMu7J9BjwQKPp/axMDIVqcxjdhG5TEQ2AluB2cA2nCOnMolIY5xZwt9xHwtwHjDF3eQD4Ar3/iD3Me7z57vbDwI+U9UjqroV2AR0c2+bVHWLqubgHBEN8rENYyqcV0GUm0/Uou2c8tJsJ4iO5JJyTQf2junN/d9M4MdLBtJ+5Uqu+OYbFpXxgVYLIhMoPBnU8ATQA/hZVTuJyLnAcA/rfwW4D+e6E0BdIE1V89zHiUAj934jYCeAquaJSLq7fSNgYZE6i+6z87jy7j62kVK00yIyChgF0LSpNweDxnjPm/+IJDuXWot2UGvuVkIyjiB1w7j6wDz+9tEkzrxrA/k1Qum4YgXZERFl1mUhZAKNJ4GUq6r7RSRIRIJUdaaIlHmqS0QuBZJUdZmI9CssLmZTLeO5ksqLO7orbfuy2v+zQHUCMAGczyEVs48xf5k3QVS4GF6t+dsIzs4jtF4Nbl73I/c99yZHwsN5+/bbqZGTw+GaNUsNI/thNoHMk0BKE5EoYA7wsTvzd14Z+4Az+erlIjIQCAeicY6YYkUkxD2CaQzsdrdPBJoAiSISAsQAqUXKCxXdp7jyFB/aMKbceBNEwalZRP+6hailOwnKLSCrbX3qnVLAmgu6kxYTw9MPPcSrY8aQEh9faj0WRKYy8CSQBuEMaPgHcB3OH/F/l7WTqj4IPAjgHiH9j6peJyKTgSE413xGAt+4u0x1Hy9wn//FnUNvKvCJiLwEnAK0BBbj/F63dEfU7cIZ+HCtu89Mb9rw4HtgzF/iy4dZY2ZtIvK3XSAQG690ZBeTbriEZFVueu89vho8mIMxMaXWYz/cpjIpc+qg4mZl8HamhiKBdKmItODPIdnLgetV9YiIhAMf4swsngoMU9Ut7v7/xJlDLw/4u6r+4JYPxDnqCgbeU9Un3XKv2yiJTR1k/ipvwih0XwbRszYTuXI3BAn16+TwxOSX6b9wNuvOOIN2a9bYB1lNpeDL1EGeBNJvqtr5uLJVqtrehz5WOhZIxldeBdHeg8RM30jk2r0UhAYT3TiYCf95iLNXLWV9q1Y888ADfHzddeTWqFFqPRZEJlCc1LnsiszUcJrN1GCM57wKot3pxMzY5ARRWDB53Rqw56KziNyxmfC387n688/58qqrKAgOLrUeCyJTFfh9pgZjqgtvgihseyrRMzdTc10SBWHBtAlP4pV3H2dljy5cPXgy61q3psuyZVDGx+QsiExVUmIgqWo6kC4iDwN73Wsw/YD2IjJJVdPKq5PGBDKPg0iV8I0pxMzcRPjWVDQihM5BO3ll/OM0TdrNtP79eeOuu4pUXHLNFkSmKip7dkX4EsgXkdOBd4HmOEdPxlRrHs+s4K7MWv+t+dR/bzEhqVmkXtKam47M5aunR7Okdw8SlixhwLRpzDnnnJKrwWZVMFWbJ8O+C9xZDQYDr6jq6yKy3N8dMyZQeXVEtCmFmJ83Er79AESGMODQSrb27cRPfVrwRusxfHH9cP5o06b0av5yj42pHDyaqUFEhgMjgMvcslD/dcmYwOT1qbmfNxC+Iw2pGcxlqYt59sVnCAqC+097lp8GXsz+uLhSF8WzIDLVjSeBdBNwB/Ckqm51P4j6kX+7ZUzg8CqINiQTO2MjYTvSyIsJ54LsNbz5wsPkhofxxt/v4aV772Vvw4alV/OXe2xM5VTm55CqO/scUvXl7RFR7PQNhO1MIzhc2H/hmaR3b8aNH35A0x07eP3uuzlQp07p1fzlHhsTOE7q55CMqa68CqLN+51Tc9sOEFpDGbn+O+6b+g63nzqBib1aMPGmm8qu5i/11piqwwLJGJfXp+Z+3kjYzjRCQwq4Zc3/8Y8f32d3k8aMeeM1Ph1e9gotFkTGHKu0mRo+VNUbRGSMqr5anp0ypjx59YHWzSnE/rSB8O0HyKsdwf5BbfnpnmuIOZTBre+/x2fDhpEXWvqYHwsiY4pX2hFSFxE5FbhZRCZx3O+tzdZgqgJPw6jG9gPE/rSeiM37CQ/K45aV3/D0pOfIjI9j8NSv2Ve/PhpU+sf6LIiMKV1pgTQe+BFoASzj2N9ddcuNqZQ8DaKwbanEzNhIxMYUwiWX0Qsnc/vcySzsfTZxB1I5EB9no+aMOUlKmzroNeA1EXlLVUeXY5+M8RuPj4gS04idtp6IjSlozRDGLPyUUfO/ZNaF53H+7JnM79WrzDosiIzxTpmDGlR1tIh0APq4RXNUdVVp+xgTaDwNopCUQ8T+uI7INXsJCS4gaWBbMnucyt6wFfR65X9Y2bFjmXVYEBnjmzIDSUTuAUYBX7lFH4vIBFV93a89M+Yk8SSMgg7lEDNjI9ELthGan8cdC6cwYuV3nPH/NqE1gnnu/rLXo7QgMuav8WTY961Ad1U9BM5qsThLgFsgmYDmSRDJkTyif91K7JxNyJE8hq38iTuWTOGb64bS5aMVZX6YFSyIjDlZPAkkAfKLPM7Hu5GyxpQrj3448wuIWrKT2j+tJygrl4IW0Ux+/i5+GTKQbh+sIKl+/TKrsCAy5uTyJJDeBxaJyH/dx1fgLENhTEDx9L+k8HVJNJm8hJxD0DIjkTkPXENOk1j6jlxMTlhYmftbEBnjH54ManhJRGYBvXF+529SVVt+wgQUT8Koxu50Wn0wh4z0IBoe2M2oVd+w+OoLmNEoGjFpoMUAAB2QSURBVKDMMLIgMsa/PJo6SFV/A37zc1+M8ZpHAxYyjhA7fQO1Fm9HsrO4Y/10Ngzuwc2vfWlHRMYEEL/NZSci4cAcIMxtZ4qqPuYuX/EZUAcn5G5Q1RwRCQMmAV2A/cA1qrrNretB4Bac61f3qOo0t3wA8CoQDLyjqs+45V63YSqfssIoLOMQfd78ms0HY8gniMNdG9Hl8HoefmE8+SFl/+hbEBlTvjxZwtxXR4DzVLUD0BEYICI9gGeBl1W1JXAAJ2hwvx5Q1dOBl93tEJE2wDCgLTAA+I+IBItIMPAmcDHQBhjubou3bZjKpaylwyMzMrjmybfp+OAUNqbFcsahXez+R1+SrurEV9cPKzOMbJlwYypGqYHk/uH/2ZeK1ZHpPgx1bwqcB0xxyz/AGSQBMMh9jPv8+SIibvlnqnpEVbcCm4Bu7m2Tqm5R1RycI6JB7j7etmEqgbKCCGDo6xO54O7/ZWFGY0KClIYJNfjxlVHkxUeVWb8FkTEVq9R/FVU1X0SyRCRGVdO9rdw9ilkGnI5zNLMZSFPVPHeTRKCRe78RsNNtN09E0oG6bvnCItUW3WfnceXd3X28bSPluH6PwvkwME2bNvX2ZRs/KC2IGuzZwyGpQejcXSzeFUdUXCS1OkUxb+hACCr7/w0LIWMCgyfXkLKB1SIyHThUWKiq95S1o6rmAx1FJBb4L9C6uM3cr8X95dBSyos7uitt+9LaOLZAdQIwAZwVY4vZx5ST0uKk2dat3Pv8C9RYvpcX+o0kR0JI73saO847HQ0vfQkIsCAyJtB4EkjfuTefqWqaO3S8BxArIiHuEUxjYLe7WSLQBEgUkRAgBkgtUl6o6D7Flaf40IYJQCWF0Zl//MGDTz9Ns7lrGHf+bWw6ZyA0imT31Qnk1fPs1JwxJvB48jmkD0QkAmiqqus9rVhE4oFcN4wigAtwBhHMBIbgXPMZCXzj7jLVfbzAff4XVVURmQp8IiIvAacALYHFOH+vWroj6nbhDHy41t3HqzY8fU2mfJR1km3cvQ/wS0RbHr36GjS6BslXtOdw63rgweVAe7ONCVyeTK56GfACUANoLiIdgXGqenkZuzYEPnCvIwUBX6jqtyLyO/CZiDwBLOfPWR/eBT4UkU04Ry3DAFR1rYh8AfwO5AF/c08FIiJ3AdNwhn2/p6pr3bru96YNEzhOiBRV+s6Zw33PPcftb40nfVc+YxPuQAuU9HNbkn5OCwgNLrNeCyJjAp+UdYAgIstwRq3NUtVObtlqVT2rHPpX4RISEnTp0qUV3Y0qr7gguviHH3joqafoPW8ev57ZhVtvGMeRg8rh0+NIvaIdeXGRZdZrQWRMxRCRZaqa4M0+nlxDylPV9ONGR9vvuTlpjg+jkNxc5p99Nl2XLmVd89M59/GP2XIohoKCUFKHtSGrwyllnp6zH1BjKh9PAmmNiFwLBItIS+AeYL5/u2Wqg6KREpqTw3m//MK0AQPICw3lxwEDGDfyHyzPqE9IejaZ3ZuSdtGZFNQsffScBZExlZcnMzXcjTNLwhHgU+Ag8Hd/dspUfYVhFH74MH974w02nX46P158MW3WriXoUA6vth3M6sQYCsJD2Du6J6lXnmVhZEwV58kouyzgn+7CfKqqGf7vlqmqCoMoMjOTv735Jve+9BL1k5KY26sXo8aPZ1tebU55aTZBh3NJO78l6eeeDiGl/99kQWRM1VDmEZKIdBWR1cAqnA/IrhSRLv7vmqlqBMAdRBOam8s/n3yS5Z060WfOHPp9/zO/pdQj/tPl5MVGsOfu3qRf2KrUMLKpfoypWjy5hvQucKeq/gogIr1xFu1r78+Omaql0a5dvPjii3RdsoS+c+aQVrs2rTZsYF/9+tRcuZuGr8whKCefAxefycHezSHYjoqMqW48CaSMwjACUNW5ImKn7YxHTt+0ifuee46tEycSVFDAx9ddR+ShQxyKiiI5sjbxHy6j5u/7ONI0lpQhHcqcacGCyJiqq8RAEpHO7t3FIvI2zoAGBa4BZvm/a6ayO++XX1h/4YXkhobyv7fdxvNjx7K9WTNQpebK3dT5Zs2fR0V9WpQ6EaoFkTFVX2lHSC8e9/ixIvft74Mp3qJFkJKCXHIJob178/gjj/D27bezt2FDAIKycqjz9RoiV+3hSJNYUoa2J69erVKrtB82Y6qHMmdqqO5spgYPqMLMmfDUUzBjBis6dKDT8uUnfHg1fGMydSevJDgzh7QLWnLwnNNKvVZkP5nGVF5+manBXTpiBNCs6PaeLD9hqoG5c+F//gcWLWJ3w4a8+MILvH377ceEkeTmE/vDOqLnbyOnXhRJI7uS2yim1GotjIypfjwZ1PA9zgJ5q4EC/3bHVAp5eXDkCERGwsGDkJTE7ePH88HIkRwJDz9m09A9B4n7bDk19mVysFcz0gaciZYyGaoFkTHVlyeBFK6q9/q9JybwHTkCkybBs8/CkCHwzDPIxRcTvGED+SHH/SgVKLXmb6P2D+vIrxnKvpu7kd0qvsSqLYiMMZ4E0ocichvwLc70QQCoqi1sV10cOgT/+7/wwguwaxckJECfPs4HXUVOCKOgjCPETV5JxIZkslrXZ/+Q9hRE1iixegsjYwx4Fkg5wPPAP/nzb4cCLfzVKRNg7roLJk6Efv3g/ffhgguQEmbbDt+YTNznK5HsXPZf0Y7M7k1LnJnbgsgYU5QngXQvcLqqpvi7MyZA7NsHr7wCI0ZA69Zw//1w221w9tklr+aaX0DszxuInrWZ3PgoUm7tTm6DkodzWxgZY47nSSCtBbL83RETAHbsgOefh3feca4XNWniBNKZZwIlLy0enHaYuM+WE77tABkJTThweVu0RvEDFyyIjDEl8SSQ8oEVIjKTY68h2bDvquSuu+Dtt53TayNGwH33QatWR58uKYwi/thH3ckrkbwCkod1JKtjoxKbsDAyxpTGk0D62r2ZqmbdOjjjDCeEatWCO+90PlPUpMnRTUo7RVf7h3VEz91KTsNokq/rXOKS4hZExhhPeLIe0gfl0RFTjubNc2ZV+P57+OUXOPdcePrpEzYr7RRd/Ce/EbYjjYM9T+XAwNZQwmeLLIyMMZ7yZKaGrRTzd0VVbZRdZaIK06fDk0/CnDkQF+fc79Sp2M1LCqPw9UnEfb4CyVeSr+1EVvtTSm7yJHTbGFN9eHLKruhcROHAUKCOf7pj/CY7G264AWrUgFdfhVtvhZo1i9202DAqUGJ+2UjMjI3k1q9F8vVd7BSdMeakKnPFWFXdX+S2S1VfAc4raz8RaSIiM0XkDxFZKyJj3PI6IjJdRDa6X2u75SIir4nIJhFZVWT5C0RkpLv9RhEZWaS8i4isdvd5TdwPx/jSRpWTmwsffgiXXOJM9RMR4Rwhbd4M99xTbBgJxYdR0KEc6k1cQuzPGznUqRF77+xlYWSMOek8WcK8c5FbgojcAZS+XoAjD/h/qtoa6AH8TUTaAA8AM1S1JTDDfQxwMdDSvY0C3nLbr4Oz9EV3oBvwWGHAuNuMKrLfALfcqzaqlOxseOstZ4TciBGwc6czuwJA+/bOEVIxSjpFV2NnGg1fn0v45v3sv/Is9g/tYEO6jTF+4ckpu6LrIuUB24Cry9pJVfcAe9z7GSLyB9AIGAT0czf7AGexv/vd8knqrIexUERiRaShu+30wqmKRGQ6MEBEZgHRqrrALZ8EXAH84G0bbl8rv02boE8f2LsXevSA1193jpBKmCmhULHPqhK1eAd1pv5Ofq0w9o7uSU7j2GL3tyAyxpwMnoyyO/evNiIizYBOwCKgfmEAqOoeEannbtYI2Flkt0S3rLTyxGLK8aGNYwJJREbhHEHRtGlT715sedu/H1avdqb1adHCCaDrrnMelxFEUHwYSW4+db5eQ9SyRA63iiflmo4lzkVnYWSMOVk8GWUXBlzFieshjfOkARGJAr4E/q6qB0uaA43i/zaqD+WldseTfVR1AjABnAX6yqizYuzZAy++COPHQ3i4c1ouLMyZZcFDxX0zgtMOE//hUsJ2HSTt/Jakn9+y2KXFA/ObYoypzMq8hgR8g3OqKw84VORWJhEJxQmjj1X1K7d4n3sqDvdrklueCDQpsntjYHcZ5Y2LKfeljcpjxw4YPRqaNYOXX4YrroDZs50w8lBJgxfCtqbS8PW5hKZkkTQigfQLW1kYGWPKjSfXkBqr6oCyNzuWO+LtXeAPVX2pyFNTgZHAM+7Xb4qU3yUin+EMYEh3T7dNA54qMpChP/CgqqaKSIaI9MA5FTgCeN2XNrx9bRWioACCgpwjoffeg5tugrFj4bTTvKqmpOPTqIXbqTN1LXl1apI0IoG8elHFbmdhZIzxF08Cab6InKWqq72suxdwA7BaRFa4ZQ/hhMQXInILsAPnc03grEw7ENiEM5nrTeCsuyQijwNL3O3GFVmLaTQwEYjAGczwg1vuVRsBbelSZ1aF+vWd0XM9e0JiIsSXvNhdSYoNo7wC6kxdS63FO8g6I56UYZ3QiNATNrMgMsb4mzgDzkrZQOR34HRgK87kqgKoqrb3f/cqXkJCgi5durR8G1V1ZlN46in46SeIjXXmmPvnP32ustjPF2UcIf6jZYRvP0B6v9NI63+GnaIzxpwUIrJMVRPK3vJPnhwhXexjf4yvnn0WHnzQOSp69lm44w6Ijva5uuLCqEZiOvEfLiUoK4fk4Z3I6lD8FEAWRsaY8uLJsO/t5dGRai0/H778Elq2dOaWGzrUmX375pudGRZ8VNL1opord1N38koKosLYO/psck+JKXY7CyNjTHnyZJSd8ZecHGeAQuvWcM01MGGCU37aafC3v538MCpQYqetI/7T5eQ0jmXPXb2KDSPFwsgYU/48OWVn/OH99+Gxx5ypfTp1gilTnCHcJ0GxH3bNyafu5yuIXLuXjK5NSB3UDkJO/H/EgsgYU1EskMpTerpzKi4oCLZvh1NPdY6KLrrIo1kVPFHsh10PZhP/wVJq7E4n9ZLWZPRuXmx7FkbGmIpkp+zKQ3IyPPywE0Bfu4vvPvww/PorDBjg1zAK3Z1OgzfnEZqcSfINCWT0aWFhZIwJSHaE5E+JifDCC85RUHY2XHWVMws3QMjJ/dYXF0YRv+8j7rPlFESEsveOniVeLzLGmEBggeQvqtC/P2zYANdfD/ff7wxe8IMTwkiVWr9upfYPf5DTKIbkEQnkR4ef2EW/9MYYY3xjgeQvIs6RUePGzrxz/miiuMJ8d+aFRTs41K4B+6/uWOz6RRZGxphAY4HkT717+63qYkfSZecS/8lyIjYk28wLxphKxwKpEip2JF16NvUmLiF0Xwb7B59FZrfi13GyMDLGBCoLpEqm2JF0ezOo9/5igrLzSLqxK9mtTpx41YLIGBPoLJAqkeLCKHxTCvEfLqOgRjB7b+9hI+mMMZWWBVIlUVwYRS7fRd0pK8mNiyTppm7kx5441ZCFkTGmsrBAqgSKG9YdPWsztaetJ7tFHZJuSLA1jIwxlZ4FUmVToNT+v7VEL9hOZsdT2D+kPYTYsG5jTOVngRTgjjk6ys0n7vMVRK7ZS3qf5qRd3PqEYd0WRMaYysoCKYAVjRo5kke9iUsI35rqTJDap8UJ21sYGWMqMwukSkBy8qk3cQlh2w+QPKwjWR0bnbCNhZExprKz2b4D1NGjo/wC4j5eRti2VFKu7mBhZIypsiyQAtDRMFKl7lerqbk+mdQrzrIwMsZUaX4LJBF5T0SSRGRNkbI6IjJdRDa6X2u75SIir4nIJhFZJSKdi+wz0t1+o4iMLFLeRURWu/u8JuIs8uNLG4Eq9qf1RC1LJO2ClmR2P3EqIAsjY0xV4s8jpInAgOPKHgBmqGpLYIb7GOBioKV7GwW8BU64AI8B3YFuwGOFAeNuM6rIfgN8aSPQFB4dRS3YRszMzWR0a0L6+S1P2M7CyBhT1fgtkFR1DpB6XPEg4AP3/gfAFUXKJ6ljIRArIg2Bi4DpqpqqqgeA6cAA97loVV2gqgpMOq4ub9oIOBFr91Jn6lqyWtcndVC7E1Z4tTAyxlRF5X0Nqb6q7gFwv9ZzyxsBO4tsl+iWlVaeWEy5L22cQERGichSEVmanJzs1Qv8KwQISc4k7vMV5DSOJWV4Jwj+8y1SLIyMMVVXoAxqKG6qNvWh3Jc2TixUnaCqCaqaEB9/4szZ/iAAufnEf7IcDQki+frOxyysZ0FkjKnqyjuQ9hWeJnO/JrnliUCTIts1BnaXUd64mHJf2ggYtaetp8aeg+wf2oH8mD8nSrUwMsZUB+UdSFOBwpFyI4FvipSPcEfC9QDS3dNt04D+IlLbHczQH5jmPpchIj3c0XUjjqvLmzYqnABhm1OInruVjB6ncrh1fcBO0Rljqhe/zdQgIp8C/YA4EUnEGS33DPCFiNwC7ACGupt/DwwENgFZwE0AqpoqIo8DS9ztxqlq4UCJ0Tgj+SKAH9wb3rYRCCQ7l7jJq8iNi+TAwDMBCyJjTPUjziA1U5KEhARdunSpX9uo8+3v1Jq3lb2jzyanaW0LI2NMpSciy1Q1wZt9AmVQQ7UVmpRBrfnbyOzaxMLIGFOtWSBVIFWlzrd/oKHBpPU/w8LIGFOtWSBVoJqb9hOxIZm0C1qSHxVW0d0xxpgKZYFUgaJ/3UJ+VBjJPU+t6K4YY0yFs0CqIKFJGURsSCaj56mEFbMEuTHGVDcWSBUkeu42NCSIzcXM4m2MMdWRBVIFSD2UQ+RviWR2akRdu3ZkjDGABVKFaL5oO0F5BWT0bl7RXTHGmIBhgVTOcvIKqLVgO4dbxZNTv1ZFd8cYYwKGBVI5q7NqNyEZRzhoR0fGGHMMC6RyVmvBdnLjI8lqGVfRXTHGmIBigVSOwhLTCNuZRkaPUxEpbnkmY4ypviyQylGtBdspqBFMYpfGZW9sjDHVjAVSOUnPyqXmqt0c6tiI6PDQiu6OMcYEHAukctLkt0SCcguY08M+CGuMMcWxQCoHqkqtRds50iSWtqfEVHR3jDEmIFkglYOIramEJh8iw6YJMsaYElkglYOoxTsoCA8huf0pFd0VY4wJWBZIfnbgUA6Ra/aS2akRETVsVm9jjCmJBZKfnfpbIpJXwHw7XWeMMaWyQPIjVSVqyU6ONI3lzAbRFd0dY4wJaBZIfrR8Zxo1kjLJTGhS0V0xxpiAV+0CSUQGiMh6EdkkIg/4s60LlyZSEBrE7vYN/dmMMcZUCdUqkEQkGHgTuBhoAwwXkTb+aOtwTj6RK3eT1a4htWxmBmOMKVO1CiSgG7BJVbeoag7wGTDIHw3Fr91L0JE8pibYvHXGGOOJ6hZIjYCdRR4numXHEJFRIrJURJYmJyf71NDXYSFc2KY+PZrX9a2nxhhTzYRUdAfKWXFrPugJBaoTgAkACQkJJzzviQva1OeCNvV92dUYY6ql6naElAgUHfLWGNhdQX0xxhhTRHULpCVASxFpLiI1gGHA1ArukzHGGKrZKTtVzRORu4BpQDDwnqqureBuGWOMoZoFEoCqfg98X9H9MMYYc6zqdsrOGGNMgLJAMsYYExAskIwxxgQECyRjjDEBQVR9+txntSEiycB2H3ePA1JOYncqA3vN1YO95urhr7zmU1U13psdLJD8SESWqmpCRfejPNlrrh7sNVcP5f2a7ZSdMcaYgGCBZIwxJiBYIPnXhIruQAWw11w92GuuHsr1Nds1JGOMMQHBjpCMMcYEBAskY4wxAcECyU9EZICIrBeRTSLyQEX3pywi0kREZorIHyKyVkTGuOV1RGS6iGx0v9Z2y0VEXnNf3yoR6VykrpHu9htFZGSR8i4istrd5zURkdLaKMfXHiwiy0XkW/dxcxFZ5Pbnc3epEkQkzH28yX2+WZE6HnTL14vIRUXKi/05KKmNcnq9sSIyRUTWue93z6r+PovIP9yf6zUi8qmIhFe191lE3hORJBFZU6Sswt7X0tookara7STfcJa22Ay0AGoAK4E2Fd2vMvrcEOjs3q8FbADaAM8BD7jlDwDPuvcHAj/grMLbA1jkltcBtrhfa7v3a7vPLQZ6uvv8AFzslhfbRjm+9nuBT4Bv3cdfAMPc++OB0e79O4Hx7v1hwOfu/TbuexwGNHff++DSfg5KaqOcXu8HwK3u/RpAbFV+n4FGwFYgosj3/saq9j4DfYHOwJoiZRX2vpbURqmvobx+CarTzX3TphV5/CDwYEX3y8vX8A1wIbAeaOiWNQTWu/ffBoYX2X69+/xw4O0i5W+7ZQ2BdUXKj25XUhvl9DobAzOA84Bv3V+eFCDk+PcSZx2tnu79EHc7Of79LdyupJ+D0tooh9cbjfPHWY4rr7LvM04g7XT/yIa47/NFVfF9BppxbCBV2PtaUhul9d9O2flH4S9AoUS3rFJwT1F0AhYB9VV1D4D7tZ67WUmvsbTyxGLKKaWN8vAKcB9Q4D6uC6Spal4x/Tz62tzn093tvf1elNaGv7UAkoH3xTlN+Y6IRFKF32dV3QW8AOwA9uC8b8uo2u9zoYp8X73+O2iB5B9STFmlGF8vIlHAl8DfVfVgaZsWU6Y+lFcYEbkUSFLVZUWLi9lUy3iuMn0vQnBO67ylqp2AQzinWUpSmV5bsdxrGoNwTrOdAkQCFxezaVV6n8tSHq/F630skPwjEWhS5HFjYHcF9cVjIhKKE0Yfq+pXbvE+EWnoPt8QSHLLS3qNpZU3Lqa8tDb8rRdwuYhsAz7DOW33ChArIoWrKRft59HX5j4fA6Ti/fcipZQ2/C0RSFTVRe7jKTgBVZXf5wuAraqarKq5wFfA2VTt97lQRb6vXv8dtEDyjyVAS3eETQ2cC6NTK7hPpXJHzLwL/KGqLxV5aipQONJmJM61pcLyEe5Imh5Aunu4Pg3oLyK13f9M++OcN98DZIhID7etEcfVVVwbfqWqD6pqY1VthvMe/aKq1wEzgSHF9KdoP4e426tbPswdndUcaIlzAbjYnwN3n5La8CtV3QvsFJEz3KLzgd+pwu8zzqm6HiJS0+1T4Wuusu9zERX5vpbURsnK46JidbzhjDDZgDP65p8V3R8P+tsb53B6FbDCvQ3EOQ8+A9jofq3jbi/Am+7rWw0kFKnrZmCTe7upSHkCsMbd5w3+nCmk2DbK+fX3489Rdi1w/tBsAiYDYW55uPt4k/t8iyL7/9N9XetxRx+V9nNQUhvl9Fo7Akvd9/prnNFUVfp9Bv4NrHP79SHOSLkq9T4Dn+JcI8vFOTq5pSLf19LaKOlmUwcZY4wJCHbKzhhjTECwQDLGGBMQLJCMMcYEBAskY4wxAcECyRhjTECwQDImAInIOBG54CTUk3ky+mNMebBh38ZUYSKSqapRFd0PYzxhR0jGlBMRuV5EFovIChF5W5x1mDJF5EUR+U1EZohIvLvtRBEZ4t5/RkR+d9eUecEtO9XdfpX7talb3lxEFojIEhF5/Lj2x7rlq0Tk325ZpIh8JyIrxVkr6Jry/a4Y8ycLJGPKgYi0Bq4BeqlqRyAfuA5nos/fVLUzMBt47Lj96gBXAm1VtT3whPvUG8Akt+xj4DW3/FWciVO7AnuL1NMfZ6qbbjgzNXQRkb7AAGC3qnZQ1XbAjyf9xRvjIQskY8rH+UAXYImIrHAft8BZ9uJzd5uPcKZwKuogkA28IyKDgSy3vCfOooLgTIVTuF8vnClkCssL9Xdvy4HfgDNxAmo1cIGIPCsifVQ1/S++TmN8ZoFkTPkQ4ANV7ejezlDVfxWz3TEXddVZS6cbzizsV1DyEYyWcL9o+08Xaf90VX1XVTfgBOVq4GkRedS7l2XMyWOBZEz5mAEMEZF64JyKE5FTcX4HC2eDvhaYW3Qnd32qGFX9Hvg7zuk2gPk4s0qDc+qvcL95x5UXmgbc7NaHiDQSkXoicgqQpaof4Sxi1/lkvFhjfBFS9ibGmL9KVX8XkYeBn0QkCGdG5r/hLJDXVkSW4axMevygglrANyISjnOU8w+3/B7gPREZi7MC7E1u+RjgExEZg3NUVdj+T+51rAXO6gFkAtcDpwPPi0iB26fRJ/eVG+M5G/ZtTAWyYdnG/MlO2RljjAkIdoRkjDEmINgRkjHGmIBggWSMMSYgWCAZY4wJCBZIxhhjAoIFkjHGmIDw/wGWuBSQE6gfTQAAAABJRU5ErkJggg==\n", 154 | "text/plain": [ 155 | "
" 156 | ] 157 | }, 158 | "metadata": { 159 | "needs_background": "light" 160 | }, 161 | "output_type": "display_data" 162 | } 163 | ], 164 | "source": [ 165 | "plt.errorbar(range(n_episodes_training),np.mean(train_data[:,2,:],axis=0),yerr=np.std(train_data[:,2,:],axis=0),ecolor='cyan')\n", 166 | "plt.xlabel('episodes')\n", 167 | "plt.ylabel('number of states')\n", 168 | "\n", 169 | "params = np.polyfit(range(200000, n_episodes_training), np.mean(train_data[:,2,200000:],axis=0),1)\n", 170 | "plt.plot(range(n_episodes_training), np.arange(n_episodes_training)*params[0]+params[1], c='red',ls='--',label='linear regression')\n", 171 | "plt.legend()\n", 172 | "\n", 173 | "print('Linear regression params: slope = {0} ; intercept = {1}'.format(params[0],params[1]))" 174 | ] 175 | }, 176 | { 177 | "cell_type": "markdown", 178 | "metadata": {}, 179 | "source": [ 180 | "### Training: number of steps\n", 181 | "\n", 182 | "We first plot the number of steps (average and standard deviation over 10 repetitions)" 183 | ] 184 | }, 185 | { 186 | "cell_type": "markdown", 187 | "metadata": {}, 188 | "source": [ 189 | "### Training: number of steps\n", 190 | "\n", 191 | "We first plot the number of steps (average and standard deviation over 10 repetitions)" 192 | ] 193 | }, 194 | { 195 | "cell_type": "code", 196 | "execution_count": 5, 197 | "metadata": {}, 198 | "outputs": [ 199 | { 200 | "data": { 201 | "text/plain": [ 202 | "Text(0, 0.5, 'number of steps per episode')" 203 | ] 204 | }, 205 | "execution_count": 5, 206 | "metadata": {}, 207 | "output_type": "execute_result" 208 | }, 209 | { 210 | "data": { 211 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEGCAYAAACUzrmNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZwc5Xng8d/TPacOZnRL6ISYOHGyNgYZQ0xM29iO7Rw4jp04G6/xkbBrJ7Eddh2DnYXYidfgE1/hsAGDjfGBiSHmttCAZUAgCSGBDpAEQkJCSIBGGl0z0/3sH/XWTHVNVVf1TF8z/Xw/M5/ueruOt7q66+n3qLdEVTHGGGNKydQ7A8YYYxqfBQtjjDGJLFgYY4xJZMHCGGNMIgsWxhhjErXUOwNjMXPmTF2yZEm9s2GMMePK6tWr96nqrHKWGdfBYsmSJaxatare2TDGmHFFRLaXu4xVQxljjElU1WAhIv8kIk+IyOMicqOIdIjICSKyUkSeEpGfiEibm7fdTW9xry+pZt6MMcakV7VgISLzgY8DS1X194Es8D7gUuDrqnoS8DLwEbfIR4CXVfUVwNfdfMYYYxpAtauhWoBOEWkBJgG7gTcDN7nXrwPe5Z6f46Zxr58tIlLl/BljjEmhasFCVZ8DvgI8ixckeoHVwH5VHXSz7QTmu+fzgR1u2UE3/4zwekXkPBFZJSKr9u7dW63sG2OMCahmNdQ0vNLCCcDxwGTgHRGz+iMZRpUiRoxyqKpXqepSVV06a1ZZPb+MMcaMUjWrod4CPK2qe1V1ALgZ+AOg21VLASwAdrnnO4GFAO71LuClKubPGGNMStUMFs8Cp4vIJNf2cDawAVgOvMfNcy5wi3t+q5vGvX6v2vjpxhjTEKrZZrESr6F6DbDebesq4NPA+SKyBa9N4mq3yNXADJd+PnBBtfKWc//GGGPSqeoV3Kp6MXBxKHkbcFrEvEeB91YzP8YYY0bHruA2xhiTyIKFMcaYRBYsjDHGJLJgYYwxJpEFC2OMMYksWBhjjElkwcIYY0wiCxbGGGMSWbAwxhiTyIKFMcaYRE0bLNZi40MZY0xaTRss+vAChjHGmGRNGyyMMcak17TBIo9Xushh1VHGGJOkaYMFeAHDGGNMsqrez6JRrY15bowxJlpTlyzAq4ry5bAqKWOMidL0wcJvuzDGGBOvKauh4oKDVUkZY0y0pi9ZGGOMSdaUwSLcCyqPtVUYY0wpTVkNFWUt0Atk650RY4xpQE1Zsojit2PYtRfGGDOSBQtjjDGJLFg4VqIwxph4Fiwi5OqdAWOMaTAWLCLYvS6MMaaYBYsIvdgFesYYE5QYLERkkoj8XxH5rps+SUT+pPpZM8YY0yjSlCyuBY4BZ7jpncC/Vy1HDaIX6K53JowxpkGkCRa/papfAgYAVPUIIFXNlTHGmIaSJlj0i0gnoAAi8lt4JQ1jjDFNIs1wHxcDdwILReQG4A3AB6uZqUZhQ5cbY4wnMVio6j0isgY4Ha/66ROquq/qOWsA4QEGe8awLn89Y1mHMcbUS2w1lIic4v8Di4HdwC5gkUtLJCLdInKTiGwSkY0icoaITBeRe0TkKfc4zc0rIvJNEdkiIuvSbqPa0nahzWHXZhhjJq5SJYuvuscOYCnwGF7J4tXASuDMFOv/BnCnqr5HRNqAScBngGWqeomIXABcAHwaeAdwkvt/PXC5e6wrvypqLV7vqJNprNJBzj321DEPxpiJL7ZkoapvUtU3AduBU1R1qaqeCrwW2JK0YhE5DngjcLVbX7+q7gfOAa5zs10HvMs9Pwe4Xj0PAd0iMm+U+1UxebxAEdd+kcNKFMaYiS9Nb6jfUdX1/oSqPo73AzvJicBe4FoReVREvicik4E5qrrbrWs3MNvNPx/YEVh+p0srIiLnicgqEVm1d+/eFNkYu168oGFXdhtjmlWaYLHRnehzInKWu5J7Y4rlWoBTgMtV9bXAIbwqpzhR127oiATVq1wpZ+msWbNSZKM6cowsUdiYUsaYiSpNsPgQ8ATwCeCTwAaXlmQnsFNVV7rpm/CCxx6/esk9vhCYf2Fg+QV4DeoNJ4eVMIwxzSVN19mjIvId4Fd4v/Q3q+pAiuWeF5EdIvJKVd0MnI0XaDYA5wKXuMdb3CK3Av8gIj/Ga9ju9aurmlnOPfaMk/UaYyamxGAhIjm8huhn8KqKForIuap6f4r1/yNwg+sJtQ2vRJIBfioiHwGeBd7r5r0deCde4/lh0pVeaq6P5FJFzj32VDUn9Zdzjz11zIMxpjbSXMH9VeBtrnSAiPw2cCNwatKCqroWr9tt2NkR8yrw9yny07Ry7rEnML2W4t4G4XniNHo1Ws499tQxD8aYYWnaLFr9QAGgqk8CrdXLUmPLU3ztRZpSRndgvhyVawQv1aW30tsyxjS3NCWLVSJyNfADN/1+YHX1sjR+BING+KTtX8Tnp08Zw3aCvaz85z1jWJ8xxpQrTbD4KF710Mfx2izuB/6jmplqdPkS6X4Joo/iANEHrHBpJxNdzRKVNhHl3GNPHfNgjClPmt5Qx4CvAV8TkenAApdmGL5YL+um46qF8hGvR5US4q7ViKru6mNk4FoRm9PGYiUkY8aXNLdV7RGR41ygWIt3RfbXqp+1xhZ1oq7Vdhu9cbrScljbizH1lqaBu0tVDwDvBq5140O9pbrZqr2WF/qYc9WDSP9gqvlLBYo8ww3h+YR5YfhkmNRgnZSHXCgtTQO8McakkSZYtLgrrf8S+GWV81MTrXsO0n3HRtDh0USm3b6Rjm0v0bH1xVGts5xSRtxJ3A8uK9x/1HzB3lhBucC8wSqvcm/glMNrmM9FpIfTxqscE2dfailHbd63Wm3HlCdNsPg8cBewRVUfEZETgaeqm63qmvPdlXTdt43Mof6qbicqgPgne/8/6Zd/H16bSG/CeuO2X2rePuxLWU057P01E0eaBu6fAT8LTG8D/qKamao6HTE+YeC16m8+eAL3R7JNM4xvsJSQCzz3q6/yMa/7z3tC64qqxhoP1VY599hTxzyY6siR/vtgais2WIjIP6vql0TkW0SP/vrxquasFqoQGEbT6N2LV+3kL5sPpMetO3hSD88Xft2Xi1hX+BqOuCouv9tveD3GmOZQqmThD0O+qhYZqamowdDrrNwgExUggsJddGH411qjlx5y2K9LU3k599hTxzyMZ7HBQlX/yz1eB0N3vlNVPVijvJkxCFd1ZQPTUVebh4NP0nUQ4QAUlHOPPTHTcXIJr6c12u2b2sq5x5465qHR5NxjTx3zECfNdRZLRWQ9sA54XEQeE5HEQQTHh0A9VAOWNqphLeWXYvxl+ige5ypKLuH1NNtq9JLPWOSobFCs1Lrqsf5S7EZijSdNb6hrgI+p6hJVXYw39Me11c1WtZWIDKUavyeAcKki3NMqyP/Cltv91l822AU37sufi0lPWnejB5Qc9ctjDjvRTlQ56nds04wNdVBVf+1PqOoKEZmAVVETu2jhX78xmgb4cMO7z+/6mwukhU+QYz1h5hhuv/Cfh8fdqpace+ypwbZKybnHnojXJuKwKaP5cRInF3g+ntrBcu6xp455CEtTsnhYRK4M3IP7P4AeETlFRE6pdgarSSZ2IWKEcgJFL9Ff2mBa3AWCwYsBgyUXv7SRJoBMhGFNytmHHCO7Owenm6VaZjTVpHFyjP/PUCNJU7Lwg/HFofQ/wKv0f3NFc1QD6goRM29YAyL0vum36puhBlXqosJgaSM4mm5wuWAgKVUaCJ8IKzXuVi5xjur+gvPfq2A+4n7dNuNJrRKlzqCeMa4vuM5KrKtScu6xp455gHQX5b2pFhmph47tL3uP177E4VfNqXNuGtN9KeYpNWR78HlcwAgO3x4MFP4v87hf6LnQdE+KvI6Wv63RbiPuxOivt1JVL7nAtnJU7j3x13tyYJoKrb+S66rGttPMU6ltNbI0vaHmiMjVInKHm36Vu3/2uNVy0EZYrzT/wsIkoxmvKsloG7y7GZnnboYb5kud4HNlbCd8seMKt43RLp923kpVXeVI9/76713SunJjy06kpH1N+x6OtfNErkQ+wusuNa8/f/j1enbuSNNm8X28saGOd9NPAp+sVobqrsnaMUZjtFVE+cB/VHq4+imqQb2X4qvPy/nyBaU9eazAK34Hf62Hv6zlbNdXqXG5Rtu2k4vZflR6OcEq7X7lKA5qK1LkJypvaUTlP7yuUusey+csSZog1yhVlGmCxUxV/SlQAFDVQepzK4eaaOk9Uu8sjGvlnFTCX4Sxfqiivlhr8arS/BJEzj2Py6cfkML56g3Ncx9e/7moX9K50PJRQdC/q+LaUFrUunoD8+di8h0WtR+5wPJjOQlFlcgqLXjXSUhXuvHfH/8/LX+U51rzL4Yt9UOkkaRp4D4kIjNwv7lF5HSSR5sYd9qe83Zp+n9t4OAbTqhzbsavck/4ScHFP9FGXR8C3pXpUwLTXRRfXR4efDHtl9FfLmo8rnxoPj+9heGG/qjly5XmF31cj7Rg/sLrzEXMl4uYJ832/fmCnRuiSl5xDfvhYWnCnSLK7e4aPPZJ3az91/MUj3CQdhswsh0i6fPll7x6Qum5FMsmraPa0gSL84Fbgd8Skd8As4D3VDVXddDSe7TeWRj3yjkxVqto2osXMKC4G2YerzQQd1LIVTlfvnLba+JKG5X+BRo8yfrTpUpf+Yh5kn5BhrdRbv6iAl94fXFduf33MW17US60fDjg+KMZhLfvvwfdwP6YdUf1/gs+9wOkn16La4rSSNMbao2InAW8Eq/kvVlVB6qeszrruudJ+hd2ceR3rJdUtZR7V8CoE324iqicdQfbPuJeK2edSb/yo8S1OUSdHEv90u4mujSTx/uSB+//6Fd/+O9njpHjhwWXJ/RaVNfocD5z7nlPKD1q3WkFSy05hk/i4RJEVOkmuI64fAfXE1c96m83STm//oPdzxtZmpKF307xRJXz0lC6l3n3d9p+yR/XOScTV5qTRakG79GsO+rkl7ZONelGUv485VZ1BeXwTjBR15rEneiiOgeEXw/+mk5TRRbVzlLqJBk+uZZbfRRcNrzNcPtP8KQVrI4ML1/u9tLwS6tRgTVX5rqijOYHR62kChbNZvIjO+qdBTMG5ZQqxqrcL3fSdqPaRWolzTbDQaFUXX9c24dfskkrrkQS3nba41Aq3+HA68/XS3Qw8K8RyhEdMMNBOvh5CY+GEHxPSgWleikZLEREgAWq2lRnz+5fPVnvLJgxqPWJNu320lZfjGaUsqQ8xNV/5xjZEyiulFKqZBM++fvr8NuJ0rxHUQEkLjDEzReVhzRK7Vt4nVHrjwuA4YAwFt0x26iVkl1nVVWBX9QoLw3DGrvNWJQ62cbNn9ZYrnGJcl/Ea6W2kXbeagXstNWSSYEirqty0nrLnS/u2qK1EWnBZZICT9x1KdWU5jqLh0TkdVXPiTFNJu4CxTRKDS1fanvBixprpRLbq0Wey2lDG0t+yqm69OcNjxhdTttYpaRps3gT8L9E5BngEF4pWVX11dXMmDHjVbVPbGnG62rEdftq0YBb64CYNEJzWCN0My9XmmDxjqrnwhjTNBrl5FeuUld5l1PtOF4lVkOp6nZgIfBm9/xwmuWMMelMtJPKRNXsxynNqLMXA58GLnRJrcAPq5mpRrL4gtto2xl3LaYxxjSHNCWEPwf+DK+9AlXdBUxNuwERyYrIoyLySzd9goisFJGnROQnItLm0tvd9Bb3+pJyd6Za5n37N/XOgjHGFKl1N9o0waLfdaH1BxKcXOY2PgFsDExfCnxdVU8CXgb8e2N8BHhZVV8BfN3NZ4wxpgGkCRY/FZErgW4R+TvgV8B306xcRBYAfwx8z00L3m1Yb3KzXAe8yz0/x03jXj/bzW+MMabO0gwk+BUReStwAPht4CJVvSfl+i8D/pnhaqsZwH431hTATmC+ez4f2OG2OSgivW7+fSm3ZYwxpkrSjg21HujEq4pan2YBEfkT4AVVXS0iOT85YlZN8VpwvecB5wEsWrQoTVaMMcaMUZreUH8LPAy8G+8+Fg+JyIdTrPsNwJ+5i/l+jFf9dBledZYfpBYAu9zznXhddHGvdwEvhVeqqlep6lJVXTpr1qwU2agMOTrA3G+vYMHn767ZNo0xplGkabP4FPBaVf2gqp4LnIrXlbYkVb1QVReo6hLgfcC9qvo3wHKGb550LnCLe36rm8a9fq9rWG8IUoD2nb1kD0/4W3kYY8wIaYLFTuBgYPogrm1hlD4NnC8iW/DaJK526VcDM1z6+cAFY9hGw5H+PO3bXqx3NowxZlTStFk8B6wUkVvw2hDOAR4WkfMBVPVrSStQ1R7cTaNUdRtwWsQ8R4H3ps14zRUKY1p8+s3rmLJ2F8996k0MzphUoUwZY0xtpAkWW92/z682Sn1h3kRw/NfvH9Pybc97hTM5NpgwpzHGNJ40XWc/V4uMNLrsof4RaW3bX6Zzyz56zz6pDjkyxpjasQEBx2De5Q/QfU/l7qo3/eb1LL7gtrKWadl3iKkrnq5YHowxJordg7uBTH342bKXmXvFA2T7+ul7/SK0tZHu2GuMmUisZFEJ+QLTb1pHy75DNd905qjXBtIwfYyNMRNSmovyviQix4lIq4gsE5F9IvL+WmSuUbXuPlA03bazl6mrdjDzp96NDqV/kO67NsFgno4n99Ky7xAZv82jcS4dMcaY1NKULN6mqgeAP8G75uK38S7Ua1rHf+PXTHps19B0uPG7694tdC3fytSVzzLnmoeZ/5UeWg4eA6AjxbUW3XdtrmyGjTFmjNIEi1b3+E7gRlUdMQRHM2rbNVy6mH39Ku+JKzTIoLsmozCyFCGuZJHtPRp7U6Wu5VtS5WHKQ9uHt2WMMVWUJlj8l4hsApYCy0RkFnC0utlqfF33bY1/0cWITP/IGzGqGy9x/peWj+mmSpmDx5jxi8dHvbwxxpQjzT24LwDOAJaq6gDeHfPOqXbGJoJS3Wol75UIpqx8dlQX6km47cON2Sv9g5ElmiL5Aos+cztTRtH7yhjTnNI0cHcAHwJ+JiI/B/4nYDeljtC+Y39ijygJ9Vua8Z/rmX7LExXZvhwZYNFFd9G17KnS8w3kkYIy7baNJedrFJm+Y0x+ZCzDkRljxipNNdT1wO8B3wK+Dfwu8INqZmo8m33twxz3mxIXyUX86M8eOjYirXXP8NiNLS8egnyBlhf6mLRud8yKZaihffLa58rJ8ugVlOyB6tdIzvrhGmb+fB0tLx6u+raMMdHSXJT3SlV9TWB6uYg8Vq0MjXetSSe0fMH7TzDj5+t4/mNvYNL63cy6YQ39c6bS5gLI9lf/caq8ZA71Q0EpTG2PnmGM3Xi779xE1/3b2PHZt8RvowKyfS6YusEc27fuI3NkkCO/P7dq2zTGFEtTsnhURE73J0Tk9cDoW2ab3LS7n2Tu5Q8UJ5Y4Z8+6YQ3AUKAAktsknIX/dg8Lv/CrkS9E3Np80vrdLL7gNjJl3K+jc9MLAGQPjxw3qyRVb7j2UQarud9dyewfrh7VssaY0UkTLF4PPCAiz7i73j0InCUi60VkXVVzN0G17+wd0/KLP3N7ZPpx92+LTJdjg7Q9V3qbflBq2dc3prylMWXls8y96iE6n3i+vAXtekZj6iZNNdTbq56LJpc9cHREzyQ5WrqHlIYKBx3PvBQ7ttTcyx+g7fmDbP+3t0PE+FFFbQGK10Cu0PuW0qPpjuiRlVLrXq8TQMvLR9ItEHV3djPudW7aw5GTZkF2+DerHBskc6if/HS750ujSdN1djvevbHf7J4fAjKqut1NmzFqe/4gM25eX5z2Qh9Tfx1dUohcR6C00vri4aK78g3dSyPi5N7+9EvM//LyoenMsUG673mS7l8Vd/st1b03HLhMnRWU7MuN3RmgY/MLzP7+KrruLb4Adc5VD7LgS8tjljL1lKbr7MV4t0K90CW1AT+sZqaMZ3oZXVunPvhM0fSkDXtGzNPx1D7at73I/EvvBbyLBo8LBaTwNEDnhj0suviuoiFOgtr29NHyQoWrr/IFJq/akbp9piyDBRgcecHkRHHcfVtZcOlyWvZWv0pxtLJ9XjtXy0vFQa39uQNRs5sGkKbN4s+BP8MrUaCqu2iyu+SNBy29xV1YJ68aeV3C7B+sZu5VD5ENNGKHg0pUA7c/nEnnk3sjtz3rhjXM/9p96TPrSiLTb9sYW2I57v5tzLxpHZMfDXYDrkzgWHDJMhb/y50VWVcj6ti6D4CW/RN7oIX2Z14qqzt158Y9ZZXWTbE0waJfVRX3TRWRydXNkklj8troX/m+bEKbR5z2QEP4tFseL/pl3/pCH3LUBZPBPNlQgJIjA2QOlneCCg6sKP15Wp/rZcpD24eCX+ZI+t5ZkQpK192bi/Ll/6o1deSPODCGkuPcKx4sqkJNMvu6VWWV1k2xNMHipyJyJdAtIn8H/Ar4XnWzZZLU4kN/3IPbaQsMx96+Yz9zr3wIgJk3rh0x9tWCS+5l4ReWeROqMDD8eqbvGJ0bR1aNBc34yVqO/9YKZvzicaY+VJnmsPZnXqL73i3M/Nk47binyqT1u72qs4nEBYvJMVWbpvGkuQf3V0TkrcAB4JXARap6T9VzZhrCvG+tKJr2g8ekiBN/xlUpTX5kBzN/7p2cd376TSy4dPjX37Of+6PYbXVsLz2gsYziR6j/y7Weo/NOemwXMpDn0NKFZS/bsXnvULfm7ZekuxizUjJ9x9CWDNrRSqbvGHMvf4AXPvg6BmdNqcDaS/eKmHX9Kva99zVoZ2vJ+XzZA0eZvOY5Dpx1YuR1RGbs0jRwX6qq96jqp1T1/6jqPSJyaS0yZxpT0vURfqAARo6sG+qR1bl5L1NLDY8CqPvyT1q3u+x7lJfStv1lJq3bRevzB0aMPTX/knuZ8x9ju/a0/ZmXkIE8s258lJk3ee9Jx1P76NxUuoQVlHTBY+bgUe9GWwmjAsixQTrcRZRpLfz3X7HgEq8zxKT1u2l98fCIoWymPPws7a6NZLQmrds1osPBpA17WPS5u4vS2re+SNv2lyPXMfPGR5l25yZanz8Y+fp4Nv+Se5kTvpC3DtJUQ701Iu0dlc6IGT9m/2B16rrm8I2hwqY+tJ3p/7XBGwsrYZXdCQMk+jIHj444MUZ17513+QPM+tGjHH/Zr4sCHEDL/iN0PLuflr19rs1j5PhdpbS8dJi5VzzI9FCX6DlXr2T291eVta440p9n4ReWeTfaerB0td3Mn6xlzvcfGdH7KEkmoe1rxs3rmfvdlWWtE4qPx6wfPcq0O5Jv+DX3uw8xL+Kk2bK3j46nvVLpWNpAKiXbe4TOiN6Io9Wy/wgdMUGylmKroUTko8DHgBNDV2pPxYb7MKOl0Sfu479+P/nJbWNct3eimHvFg0NjdB15xcyyVtF95yaOLegemp593Spa9x2i7ble9n7otNTr8TsCTAn25ipxEeP8S+5lYPYUXvhwaBslzn2LLhru0ZVJGObeHw1ZBqrbZbhj0wscO3EG2jby4s9SsgdSXqAZYd43fj08UUasyPYeYf4l9/L8P5xJ//yuUW8/bO53fkPLgWMVqTacvHpnccJgnqkPbufgHywpupixFkpt7UfAnwK3ukf//1RVbep7cJvRm7pyO1JG88H0X26grcQ1HNkDR+lwXXq77t3C4gtvLxrMsXOLqyJJWY3d1bO1eNwpN3jhpM176Xx899A2J6/aMbI6qaBDDdFRvdWmPvBM7HZb9h+h88m9w6MNpxxwMtIYf1zLQL5EdV/8G9n6/EHmfP8Rpv9ifeTr0p+vynUzmYj2qJa9fSy+4LbYa4MAOjftRRSmrIwolRV0qGosc6ifrrs3p8q7HygqZebPisds7Vq+lem3bWTKqp0xS1RPbLBQ1V5VfQb4F+B5d7X2CcD7RaQ7bjljSpl25+bSQ7iXae63VjDnmocBb8ypOJ1bXkSODQ53/U2p9aXhX7yzf+g1NM/57kPMvGkds7+/qujK+dnXPMzif7mDxRfcRlfEOF3BnmWtz0dffCauh9niz97B8V+/P3Ke9m0v0rG5uJrNPzl2bBluP0h1v/c7N7HowuLAENVduW1PccDO9h4d8V5m3LR/7UPm4NGiE+yii+5k5k/XehPhRuiKxRBvRf6+z7rx0bSLADD119toff4A0295fOhanOm/WE/3vVvoeCr6OqOg9h2VudVPywsHmfX9R0ak+9WC0j+6rvFjkaYc83MgLyKvAK7GCxg/qmquTFNKat8YIV+gJdiWkFB6mHvFgyz617tjX0/bLuGPbeU97yPb6wWUzi3pG3qPv8yrOpl546OxdyxsjbmR1tyrHmLOtcUnEr/OPni/E/9e7pm+Y0MN5R1bXyy6V0pXz9aiXmadG/YMBawgvyuzDORp27GfBV9cNrQPvvZAvXrrnoMs/MIyuu8pbotIuj6oYgL7lN1/hNnffajo5dZdvUNduac+soO2nftp3dXL9Ns2Mu8bv2Zq4IeHDHglF8krFJQpjzwL+QLt215MvNFYKe3bXiwu+ajSfdcmWvb2Med7DzMpokNCJX9olSvNQIIFVR0UkXcDl6nqt0QkRbg2prqKxhVK0TU2+Ms+SuRw7mGhqqGZP/F+Ke/4v1H9QIqNGBxysMDkx3YVX2sQatdoDVTBdW7YE1siie0uOpBn4b8P79f0W4fvynh08bSiWdue62X29as48tuzitLDQ7nM+47XZNmyv7idYdodmwDo2P4yc7/tdbnuWr6V/W97ZclSH8Dkx5+n9bLoklTY3G+v4NBrjo9+UYF8YfgeKHjHyA+mvuO/WdwlPNhrL66LdrbvGJPX7GTGz9fTuqeP41Z4J+7es70BNzN9I39sTPvlBo6+YgZHfmfOyP24ygtgz75qDtqaJfvyEbqWb2Xy2l20RNxUbNL6uBuf1UaaksWAiPw18AHgly4tXednY6qoq2fr0PPsof6a9K9f/Nk7ItNnXZ/cwylcRSGF5ADXdd/wPs6+fhXT7o6/rzuAhLqgzv9yT+y8RT1sVIeqn8I9poJDucT1pgrfMTEzMLxv7U+/xIxfPD782qH+yOqhtohur113bwZVFnx+uETYvrO35EWpiy66k+5fxQlKk1YAABU8SURBVP/iL3eUAd+Mm9cP5dEPFEHHf7V4yJvjv7yc41Y8XdT7LXvgKLO/t5JMoEv0rB+sRo4NDg2gGDca8+Q1tW+nCEoTLD4EnAF8QVWfFpETsIEETQOQwK/8BV9cNuKXbi1Vqmtj+xjXM2XNc3RuHW6riPqFGmX6fw43SsdVfwG07You2Sz4f8tilwnfZ2Xmj9NXTHTfu4WZP1xTNJ5ZEskXFw3CpYo55Xb1DfwGiQoSWRdAs6G2nqi7Zh533zY6t+xj4eeHr2vufHJvbPtUkcBuTb9tY81vM5xmiPINqvpxVb3RTT+tqpdUP2vGTDCh6o1FF901Ypbpt20cMeZWGmMNlFMf3sFxy7ckzpfURTdKuO49W+Y1K5PLuElW3A3Agkr1rgubsvJZJm0sfTFjOYEsrs0hzfELd1go1buuGtK0WRhjKiDtr/wFX4z/lV5NwRJJNUVVN1XK5ArX68/4z+huwGFJVVuLL7iNIyeVd83PiG1EdDyopdpe1WGMMRPM9JvXkU1xbUXnU2MbFqXeYoOFiPzAPX5iNCsWkYUislxENorIE/56RGS6iNwjIk+5x2kuXUTkmyKyRUTWicgpo9muMcbUUvuuAxwfGnCzFlpejG9bqoZSJYtTRWQx8GERmeZO8kP/KdY9CPxvVf1d4HTg70XkVcAFwDJVPQlY5qbBG2/qJPd/HnD5KPfJGGMmPBntFf6jVKrN4grgTuBEYDXFlzypS4+lqruB3e75QRHZCMwHzgFybrbrgB6827aeA1zvbrT0kIh0i8g8tx5jjDEBte79V2q4j2+6UsE1qnqiqp4Q+C8ZKMJEZAnwWmAlMMcPAO5xtpttPhAcJ3qnSwuv6zwRWSUiq/buTb783hhjJqLgSAK1kObmRx8VkdcAf+iS7lfV1LcdE5EpeEOGfFJVD0j8hVNRL4y4llJVrwKuAli6dGn9xyM2xpgmkObmRx8HbsArAcwGbhCRf0yzchFpxQsUN6jqzS55j4jMc6/PA/xOzDuB4K3EFgB2z0VjjGkAabrO/i3welW9SFUvwmus/rukhcQrQlwNbFTVrwVeuhU41z0/F7glkP4B1yvqdKDX2iuMMaYxpLkoT4Dg1SB50t0d4A3A/wDWi4gbl5jPAJcAPxWRjwDPAu91r90OvBPYAhzGG2bEGGNMA0gTLK4FVorIf7rpd+GVGEpS1RXEB5WzI+ZX4O9T5McYY0yNpWng/pqI9ABn4p38P6SqNkS5McY0kVRjQ6nqGmBNlfNijDGmQdnYUMYYYxJZsDDGGJOoZLAQkayIpLjXpDHGmImsZLBQ1TxwWES6apQfY4wxDShNA/dRvGsl7gGGBiNR1Y9XLVfGGGMaSppgcZv7N8YY06TSXGdxnYh0AotUdXMN8mSMMabBpBlI8E+BtXj3tkBEThaRW6udMWOMMY0jTdfZfwVOA/YDqOpa4IQq5skYY0yDSRMsBlW1N5Rm95EwxpgmkqaB+3ER+e9AVkROAj4OPFDdbBljjGkkaUoW/wj8HnAMuBE4AHyympkyxhjTWNL0hjoMfFZELvUm9WD1s2WMMaaRpOkN9ToRWQ+sw7s47zERObX6WTPGGNMo0lRDXQ18TFWXqOoSvBsUXVvVXFXZi+/+b/XOgjHGjCtpgsVBVf21P+HugDeuq6L6Xrew3lkwxphxJbbNQkROcU8fFpEr8Rq3FfgroKf6WasiSXMLcWOMMb5SDdxfDU1fHHhu11kYY0wTiQ0WqvqmWmak1nb/w5nM+/aKemfDGGPGhcSusyLSDXwAWBKcf7wPUd6/wG7RYYwxaaW5gvt24CFgPVCobnaMMcY0ojTBokNVz696Turs2IIu2neGh8AyxhgD6brO/kBE/k5E5onIdP+/6jmroZ2fOZsDZ9pAusYYEydNyaIf+DLwWYZ7QSlwYrUyVWv54zrqnQVjjGloaYLF+cArVHVftTNjjDGmMaWphnoCOFztjNRb/3zrHWWMMXHSlCzywFoRWY43TDkw/rvOAuy46G2gXs3a4Kwp9L7xRLru31bnXBljTONJEyx+4f4nnMKk1qLp/e/8XQsWxhgTIc39LK6rRUYazZ4Pn8acax6udzaMMaYhpLmC+2kixoJS1QnTGypo1z+9EYCBOVPrnBNjjGkcaaqhlgaedwDvBSbUdRZBFiSMMWakxN5Qqvpi4P85Vb0MeHMN8maMMaZBpKmGOiUwmcEraVTl57eIvB34BpAFvqeql1RjO8YYY8qTphoqeF+LQeAZ4C8rnRERyQLfAd4K7AQeEZFbVXVDpbeV1q5P/iGFjlbmXv4ALb1H65UNY4ypuzS9oWp1X4vTgC2qug1ARH4MnAPULVgMzD0OgD3nncGcKx6g5eCxhCWMMaY2+udMqen20lRDtQN/wcj7WXy+wnmZD+wITO8EXh+Rn/OA8wAWLVpU4SxEG5wxiec++5biRFVQkGODaFsWKSiaEWQgjxSUQnsLiCD9g5DNeN3Jsl4TkQzk0awA4v0N5NGWjNfnrCXj3fZVFRkswGABbcmQGShQaM9CRpB+N38wL2790p8nc3QQBLQlS6E1g+QLkBFUBMkXyBwdJD+l3VtXvgB5bxtkXP4GC2SO9JOf3Eb28ACazaAt3npUBG1v8fI3kEdbs8hA3ltWlczRQQqdrd76BCh4ecse6kczAiJoVtC2FrJ9x7w8trl9UcgeGd4egLZmyRwZAFUKk9ug4L0v2YPHQCA/pZ3sgaPkuzqR/kHvvWlvgUKBwqS2of3Rloz3/g8WkME8mSMDFCa1oRkh05/31q3qHc9shuyRAaQ/z2B3p/ceqaLZDBn3/uantHnvaUa8Y++OGRkZ3p7bX+nPe+9xewuZw/3ePh0bpNDegra3IEcHvPkGC14+3DEqdLQU3QJY8oWhY585Ooi2ZNBW73Oj7nMjg3lvHrcPg1Pbvfz5xy/rHSey7vNWUG+/Fe8zmfWPhft8D3rroqCIe3/IZtGMd2z8fQSG9lMKOvR+klcyRwfQtha0LQuDBe9zAcP76J4DFDpbh/PnZHuPUmjPesdVBPIF7zEjXv4LBW9+/zX3PdBWt46CInkdPibuM0m+MPx9OK7D+0z4n7H+Qe/4trh9dPnO9nnfCzIy9F1GZPjzLn6e1Psuu+0M7W9bFlqzw/Nkhr/rmhXIZLzvlf85ct+17MFj5Ls7vc9na3bos1ZLolr6DqkicifQC6zGu5obAFUN33Z1bBkReS/wR6r6t276fwCnqeo/xi2zdOlSXbVqVfnbGnUujTGmcYz2/tYislpVlybPOSxNm8UCVX37KPNUjp3AwuB2gV012K4xxpgEaQYSfEBE/lvVcwKPACeJyAki0ga8D7i1Bts1xhiTIE3J4kzgg+5K7mN4tTiqqq+uZEZUdVBE/gG4C6/r7DWq+kQlt2GMMWZ00gSLd1Q9F46q3o53z29jjDENJE3X2e21yIgxxpjGlabNwhhjTJNrymBh98QzxpjyNGWwOLneGTDGmHGmKYNFT70zYIwx40xTBgtjjDHlsWBhjDEmkQULY4wxiZo2WGTrnQFjjBlHmjZY1HYkeGOMGd+aNlgEZbFrL4wxppSmDRb7gbNCaV1Y9ZQxxkRJM5DghNUDdLvn+91jU78hxhgTo2lLFnHOxKqkjDEmzIJFDAsYxhgzrOlrXaLGifLT7qtlRowxpoFZySKkBxs7yhhjwixYxOgBFOsdZYxpTLU+NzV9NVRPwutnYtVRxpjGc2aNt2cliwQ9eNdj+NdgZBl5fYYx44VdSzSSvR/pWLBIoQev0XsKw8OEKMVBJErShzDth3QsH+bR9urKhp4Hg2WSs2LmjUsrlQc/OPt50EBewo9x6+yi+FhFvSfZmH8oXv9ZDB/7uH3x89wV2pa/7bNC84bfl2zo0c9D8Hl4+Sj+ts4KzLsfGAysL7yN8H4F16ER+Ql/Nvz5/df84+Ufu2DVbvD4ht+n8PPg8uFjGV72rNB03GfZN4WR72Fwv4Of53Aegq8nbS84f1fo/6zQusP5CL5vfloPtdX01VBp9aRIy1FcZTUF6A1M+we426VPwfvy5gLrywXmX8twz6wVQJ7hL0ZfaP3+B2lK4HXc8msD6/Tn7wo8jyoprQ2sIziO1pTAOvuILwqfSfG+ZvFOUr4cw/vXE0rz983f9x6KBd+zsOC8uYi08Gvgvbfh7fl5IZAefB/9eYLp4WPpbyd4HOPyE86TP5//BfXf8/D7FZXfKOGTS3B9/nvurzu4rpNDy/mfs/0U85cp1bvQN0j08esKrLc79Fpwe34eg89XuGn/c1cuf33+evaH0oOfXX+7QeGLe7sZ/n74eSTwPOo4BtftLx/kfw/D732tWLCooB6iPyThL3H4ZNcTs644ucA8wQ9pcFvBL40/T9QJJ46/jH8STZufsLgTe09Metpb3pbKT5p5gq91R6T1UPx+Rq3LTwuf2MLz5Eq8HpcnX/gkFTe//zwX8Vrc9uOOa1SaL009eXj5crcB5Z0Qe4g+huXwl8vFpJcrnP+emOdx2wgHq0ZgwaLCoj7kUV/K8HQ5gsuW+sUV3H65Kvlh7UmZHjdfvfSknC/tiS3t+hpZT5nptTKaz3iUnlEuV61f+z012EZaoqrJczWopUuX6qpVq+qdDWPGnZx77KljHiop5x576piH8UREVqvq0nKWsZKFMU2op94ZqLCeemegCVhvKGOMMYksWBhjjElkwcIYY0wiCxbGGGMSWbAwxhiTyIKFMcaYRBYsjDHGJLJgYYwxJpEFC2OMMYnG9XAfIrIX2D7KxWcC+yqYnfHA9rk52D43h7Hs82JVnVXOAuM6WIyFiKwqd2yU8c72uTnYPjeHWu+zVUMZY4xJZMHCGGNMomYOFlfVOwN1YPvcHGyfm0NN97lp2yyMMcak18wlC2OMMSlZsDDGGJOoKYOFiLxdRDaLyBYRuaDe+UkiIgtFZLmIbBSRJ0TkEy59uojcIyJPucdpLl1E5Jtu/9aJyCmBdZ3r5n9KRM4NpJ8qIuvdMt8UESm1jRrue1ZEHhWRX7rpE0RkpcvPT0SkzaW3u+kt7vUlgXVc6NI3i8gfBdIjPwdx26jR/naLyE0isskd7zMm+nEWkX9yn+vHReRGEemYaMdZRK4RkRdE5PFAWt2Oa6ltxFLVpvoHssBW4ESgDXgMeFW985WQ53nAKe75VOBJ4FXAl4ALXPoFwKXu+TuBOwABTgdWuvTpwDb3OM09n+Zeexg4wy1zB/AOlx65jRru+/nAj4BfuumfAu9zz68APuqefwy4wj1/H/AT9/xV7hi3Aye4Y58t9TmI20aN9vc64G/d8zageyIfZ2A+8DTQGXjvPzjRjjPwRuAU4PFAWt2Oa9w2Su5Drb4EjfLv3tC7AtMXAhfWO19l7sMtwFuBzcA8lzYP2OyeXwn8dWD+ze71vwauDKRf6dLmAZsC6UPzxW2jRvu5AFgGvBn4pftg7wNawscSuAs4wz1vcfNJ+Pj688V9Dkptowb7exzeiVNC6RP2OOMFix3uBNjijvMfTcTjDCyhOFjU7bjGbaNU/puxGsr/cPp2urRxwRW7XwusBOao6m4A9zjbzRa3j6XSd0akU2IbtXAZ8M9AwU3PAPar6mBEPof2zb3e6+Yv970otY1qOxHYC1wrXtXb90RkMhP4OKvqc8BXgGeB3XjHbTUT+zj76nlcyz4PNmOwkIi0cdF/WESmAD8HPqmqB0rNGpGmo0ivGxH5E+AFVV0dTI6YVRNeG0/vRQteVcXlqvpa4BBe1UGc8bRvkVwd+jl4VUfHA5OBd0TMOpGOc5Ja7EvZyzRjsNgJLAxMLwB21SkvqYlIK16guEFVb3bJe0Rknnt9HvCCS4/bx1LpCyLSS22j2t4A/JmIPAP8GK8q6jKgW0RaIvI5tG/u9S7gJcp/L/aV2Ea17QR2qupKN30TXvCYyMf5LcDTqrpXVQeAm4E/YGIfZ189j2vZ58FmDBaPACe5nhBteI1kt9Y5TyW5ng1XAxtV9WuBl24F/B4R5+K1ZfjpH3A9Hk4Hel0R9C7gbSIyzf2iextePe1u4KCInO629YHQuqK2UVWqeqGqLlDVJXjH6F5V/RtgOfCeiPwE8/keN7+69Pe5XjQnACfhNQZGfg7cMnHbqCpVfR7YISKvdElnAxuYwMcZr/rpdBGZ5PLk7/OEPc4B9TyucduIV4tGrEb7x+sJ8CReL4nP1js/KfJ7Jl4RcR2w1v2/E6/edRnwlHuc7uYX4Dtu/9YDSwPr+jCwxf1/KJC+FHjcLfNthq/uj9xGjfc/x3BvqBPxTgJbgJ8B7S69w01vca+fGFj+s26/NuN6iZT6HMRto0b7ejKwyh3rX+D1epnQxxn4HLDJ5esHeD2aJtRxBm7Ea5MZwPtV/5F6HtdS24j7t+E+jDHGJGrGaihjjDFlsmBhjDEmkQULY4wxiSxYGGOMSWTBwhhjTCILFsaUSUQ+LyJvqcB6+iqRH2NqwbrOGlMnItKnqlPqnQ9j0rCShTGAiLxfRB4WkbUicqV499HoE5GvisgaEVkmIrPcvN8Xkfe455eIyAZ3T4CvuLTFbv517nGRSz9BRB4UkUdE5N9C2/+US18nIp9zaZNF5DYReUy8ez38VW3fFWOGWbAwTU9Efhf4K+ANqnoykAf+Bm9QuzWqegpwH3BxaLnpwJ8Dv6eqrwb+3b30beB6l3YD8E2X/g28QQJfBzwfWM/b8IanOA3vCu5TReSNwNuBXar6GlX9feDOiu+8MSlZsDDGG4/oVOAREVnrpk/EGxr9J26eH+INuxJ0ADgKfE9E3g0cduln4N2wCbzhK/zl3oA37IOf7nub+38UWAP8Dl7wWA+8RUQuFZE/VNXeMe6nMaNmwcIYb5yc61T1ZPf/SlX914j5ihr41LsXwml4owG/i/hf/hrzPLj9Lwa2/wpVvVpVn8QLYuuBL4rIReXtljGVY8HCGG+AtfeIyGwYum/xYrzvhz8q6X8HVgQXcvcX6VLV24FP4lUhATyAN7opeNVZ/nK/CaX77gI+7NaHiMwXkdkicjxwWFV/iHeDoOT7JBtTJS3JsxgzsanqBhH5F+BuEcngjQz693g3H/o9EVmNd0e2cAPzVOAWEenAKx38k0v/OHCNiHwK7853H3LpnwB+JCKfwCuN+Nu/27WbPOiNME0f8H7gFcCXRaTg8vTRyu65MelZ11ljYljXVmOGWTWUMcaYRFayMMYYk8hKFsYYYxJZsDDGGJPIgoUxxphEFiyMMcYksmBhjDEm0f8HOBPPYKxHSVYAAAAASUVORK5CYII=\n", 212 | "text/plain": [ 213 | "
" 214 | ] 215 | }, 216 | "metadata": { 217 | "needs_background": "light" 218 | }, 219 | "output_type": "display_data" 220 | } 221 | ], 222 | "source": [ 223 | "plt.errorbar(range(n_episodes_training),np.mean(train_data[:,0,:],axis=0),yerr=np.std(train_data[:,0,:],axis=0),ecolor='cyan')\n", 224 | "plt.xlabel('episodes')\n", 225 | "plt.ylabel('number of steps per episode')" 226 | ] 227 | }, 228 | { 229 | "cell_type": "markdown", 230 | "metadata": {}, 231 | "source": [ 232 | "We then smooth with a $1000$-step window each repetition, then we compute mean and standard deviation in number of steps." 233 | ] 234 | }, 235 | { 236 | "cell_type": "code", 237 | "execution_count": 6, 238 | "metadata": {}, 239 | "outputs": [ 240 | { 241 | "data": { 242 | "text/plain": [ 243 | "Text(0, 0.5, 'number of steps per episode')" 244 | ] 245 | }, 246 | "execution_count": 6, 247 | "metadata": {}, 248 | "output_type": "execute_result" 249 | }, 250 | { 251 | "data": { 252 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEGCAYAAACUzrmNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZwdVZn/8c/Ta/Z0ls5COgshQfa1ZRGUVWSZEVEYUFFElN8g4zLMb0YcHXCbIe7+0BmEYRFFUcdlQJRdAiIQSCAQIAbCErLv6XSn0/vz+6NOd25336Vup6vv7e7v+/W6r1t1qurWU7eS+3SdOnWOuTsiIiLZlBQ6ABERKX5KFiIikpOShYiI5KRkISIiOSlZiIhITmWFDmBvTJ482efMmVPoMEREBpUlS5ZscffqfLYZ1Mlizpw5LF68uNBhiIgMKma2Kt9tVA0lIiI5KVmIiEhOShYiIpKTkoWIiOSkZCEiIjkpWYiISE5KFiIikpOShYiI5DQsk0UZg/xpRBGRATYsfzPbCx2AiMggMyyvLEREJD9KFiIikpOShYiI5KRkISIiOSlZiIhITkoWIiKSk5KFiIjkpGQhIiI5KVmIiEhOShYiIpKTkoWIiOSkZCEiIjkpWYiISE5KFiIikpOShYiI5KRkISIiOSlZiIhITkoWIiKSk5KFiIjklHiyMLNSM3vOzO4J8/ua2SIze9XMfmlmFaG8MsyvDMvnJB2biIjEMxBXFp8FlqfMfwP4nrvPB7YDl4Xyy4Dt7j4P+F5YT0REikCiycLMaoBzgJvDvAGnAr8Oq9wOvC9MnxvmCctPC+uLiEiBJX1l8X3gX4COMD8J2OHubWF+DTAjTM8AVgOE5XVh/W7M7HIzW2xmizdv3pxk7CIiEiSWLMzsb4BN7r4ktTjNqh5j2Z4C95vcvdbda6urq/shUhERyaUswc8+AXivmZ0NjADGEV1pVJlZWbh6qAHWhfXXADOBNWZWBowHtiUYn4iIxJTYlYW7f8Hda9x9DnAR8Cd3/zDwCHB+WO0S4K4wfXeYJyz/k7v3urIQEZGBV4jnLD4PXGVmK4nuSdwSym8BJoXyq4CrCxCbiIikkWQ1VBd3XwgsDNOvA8ekWacJuGAg4hERkfzkvLIws1Fm9m9m9t9hfn64eS0iIsNEnGqo24Bm4Pgwvwb4emIRiYhI0YmTLPZz928CrQDuvpv0zVxFRGSIipMsWsxsJOGZBzPbj+hKQ0REhok4N7ivBe4DZprZz4ien/hYkkGJiEhxyZks3P1BM3sWOI6o+umz7r4l8chERKRoZEwWZnZUj6L14X2Wmc1y92eTC0tERIpJtiuL74T3EUAt8DzRlcVhwCLgxGRDExGRYpHxBre7n+LupwCrgKNC531HA0cCKwcqQBERKbw4raEOcPdlnTPu/iJwRHIhiYhIsYnTGmq5md0M3EHUfPZiuo98JyIiQ1ycZHEpcAXR8KgAjwE3JBaRiIgUnThNZ5vM7D+Bh4iuLFa4e2vikYmISNHImSzM7GSisbHfJGoNNdPMLnH3x5INTUREikWcaqjvAGe4+woAM9sfuBM4OsnARESkeMRpDVXemSgA3P0VoDy5kEREpNjEubJYbGa3AD8N8xcDS5ILSUREik2cZHEFcCXwGaJ7Fo8B/5VkUCIiUlzitIZqBr4LfNfMJgI1oUxERIaJOMOqLjSzcSFRLAVuM7PvJh+aiIgUizg3uMe7+07g/cBtoX+o05MNS0REikmcZFFmZtOBvwPuSTgeEREpQnGSxVeB+4GV7v6Mmc0FXk02LBERKSZxbnD/D/A/KfOvAx9IMigRESku2UbK+xd3/6aZ/YCoT6hu3P0ziUYmIiJFI9uVRWc35IsHIhARESleGZOFu/8+vN8OYGbjolmvH6DYRESkSMR5zqLWzJYBLwAvmtnzZqZOBEVEhpE43X3cCnzK3f8MYGYnArcBhyUZmIiIFI84TWfrOxMFgLs/DqgqSkRkGIlzZfG0md1INIaFAxcCC83sKAB3fzbB+EREpAjESRZHhPdre5S/gyh5nNqvEYmISNGJ81DeKQMRiIiIFK84raGmmtktZnZvmD/IzC5LPjQRESkWcW5w/5iob6h9wvwrwOeSCkhERIpPnGQx2d1/BXQAuHsb0J5oVCIiUlTiJItdZjaJ0D+UmR0H1CUalYiIFJU4yeIq4G5gPzP7C/AT4NO5NjKzEWb2dHji+yUz+0oo39fMFpnZq2b2SzOrCOWVYX5lWD6nz0clIiL9KmeyCM9RnETUVPb/AAe7+wsxPrsZONXdDydqfntmuCr5BvA9d58PbAc6b5ZfBmx393nA98J6iShfv5MxT79FW3tHUrsQERlS4lxZ4O5t7v6Su7/o7q0xt3F3bwiz5eHV+VzGr0P57cD7wvS5YZ6w/DQzszj7ytfIVzYz6bfLaG3v1fO6iIikEStZ9JWZlZrZUmAT8CDwGrAj3CQHWAPMCNMzgNXQdRO9DpiU5jMvN7PFZrZ48+bNfQwsemt3JQsRkTiyJguLzOzrh7t7u7sfAdQAxwAHplutc3dZlqV+5k3uXuvutdXV1X2LK1ywtHcoWYiIxJE1Wbi7A/+7tztx9x3AQuA4oMrMOp8crwHWhek1wEyAsHw8sG1v951WSZQsOpQsRERiiVMN9ZSZvT3fDzazajOrCtMjgdOJRt97BDg/rHYJcFeYvjvME5b/KSSrfuchWagaSkQknjgdCZ4C/L2ZvQnsIqoucnfPNZ7FdOB2MyslSkq/cvd7zOxl4Bdm9nXgOeCWsP4twE/NbCXRFcVFeR9NXKYrCxGRfMRJFmf15YND89oj05S/TnT/omd5E3BBX/aVN11ZiIjkJc5zFquI7iWcGqYb42xXzHSDW0QkP3F6nb0W+DzwhVBUDtyRZFCJ67rBXeA4REQGiThXCOcB7yW6X4G7rwPGJhlU4sJRd6gaSkQkljjJoiW0SursSHB0siElr6saSslCRCSWOMniV2EM7ioz+yTwEPDfyYaVMD1nISKSlzjDqn7bzN4N7AT2B65x9wcTjyxJurIQEclLnKazAMuAkURVUcuSC2dgdD2UpysLEZFY4rSG+gTwNPB+oiernzKzjycdWKJMraFERPIR58rin4Ej3X0rQBg17wng1iQDS5K6+xARyU+cG9xrgPqU+XpCV+KDVkgWx6oaSkQkljhXFmuBRWZ2F9E9i3OBp83sKgB3/26C8SUjVEOZkoWISCxxrixeI+qmvPOX9S5gPdGDeYPy4bxRL20AoOqBFWkH0RARke7iNJ39ykAEMpBaq6PnCke8EQ2XYXQfZakzgei6Q0QkMqg7BOyrXUfO6FVmpB+qT0REhmmy6KhIuaBq795+VglDRKS3YZksKNtz2LO+dG8BAxERGRziPJT3TTMbZ2blZvawmW0xs4sHIriBYA7W1FroMEREilqcK4sz3H0n8DdEz1zsT/Sg3pAx68sPUNLYO2GoSkpEJBInWZSH97OBO919W4LxDJhVC87pNj/zqw8UKBIRkeIXJ1n83sz+CtQCD5tZNdCUbFiFMWrp2kKHICJSlOKMwX01cDxQ6+6tRCPmnZt0YANh29kHdpuv/sVStY4SEUkj50N5ZjYCuBQ40cwceBy4IenABkLbpFG9ysq2NdJWPaYA0YiIFK841VA/AQ4GfgD8EDgQ+GmSQQ2UpvmTe5XN+M6jlDQ0U9LQXICIRESKk3mObrrN7Hl3PzxXWSHU1tb64sWL896uZ9WSNbcx69r7e63XeRNc3X6IyFBiZkvcvTafbeJcWTxnZsel7ORY4C/5BlfMvDJ7bZzuW4jIcBcnWRwLPGFmb5rZm8CTwElmtszMXkg0ugG06rqzexdqcCQRESDeeBZnJh5FMTDDS6zbGBfW1oGXlxYwKBGR4hCn6ewqYCZwapjeBZS4+6owP2S0j63sNm8t7QWKRESkuMTpG+pa4PPAF0JRBXBHkkENlJ6VTF7a/esorVeLKBERiHfP4jzgvURXFLj7OgbpCHnpOHuSRs9xLib+dsjckhER2StxkkWLR+1rHcDMRicbUuHUnTafhqNruuZHvLWjgNGIiBSPOMniV2Z2I1BlZp8EHgJuTjasAikxtl5wOKuvOaPXIjWfFZHhLM4Y3N82s3cDO4G3Ade4+4OJR1ZAHaPKc68kIjKMxOkb6hvu/nngwTRlIiIyDMSphnp3mrKz+juQYrPz+NmFDkFEpGhkvLIwsyuATwFzezypPZYh1t0HRHfvu92XMN2lEBHplO3K4ufA3wJ3h/fO19HunnMMbjObaWaPmNlyM3vJzD4byiea2YNm9mp4nxDKzcyuN7OVZvaCmR2110e3F8Y98SYA1tJWyDBERIpCxmTh7nXu/ibwJWBDeFp7X+BiM6uK8dltwD+5+4HAccCVZnYQcDXwsLvPBx4O8xBVbc0Pr8spwJgZ6XqCKt2xu2u6KrxERIabOPcsfgO0m9k84BaihPHzXBu5+3p3fzZM1wPLgRlEo+zdHla7HXhfmD4X+IlHniJqqjs9n4NJwsTfv9w1XRdeIiLDTZxk0eHubcD7ge+7+z8Cef2Im9kc4EhgETDV3ddDlFCAKWG1GcDqlM3WhLKen3W5mS02s8WbN2/OJ4w+Gfnqll5lupshIsNNnGTRamYfBD4K3BPKYj+IYGZjiK5OPufuO7OtmqasV82Qu9/k7rXuXltdXR03jLyt/b8nJ/bZIiKDTZxkcSlwPPDv7v6Gme1LzI4EzaycKFH8zN1/G4o3dlYvhfdNoXwNUe+2nWqAdXH2k4S2CSMLtWsRkaITp4vyl939M+5+Z5h/w90X5NrOzIzoHsdyd/9uyqK7gUvC9CXAXSnlHw2too4D6jqrqwZS16VMaZw8KiIyPMQZ/KivTgA+Aiwzs6Wh7F+BBUT9TV0GvAVcEJb9ETgbWAk0El3RiIhIEUgsWbj742S+F3xamvUduDKpePLR6wE9EZFhLmNdi5n9NLx/duDCERGRYpStYv5oM5sNfNzMJoQnr7teAxWgiIgUXrZqqB8B9wFzgSV0r5nxUC4iIsNAtu4+rg9dddzq7nPdfd+UlxKFiMgwEmfwoyvM7HDgnaHoMXfX4NQiIsNIzocJzOwzwM+IuuWYAvzMzD6ddGDFxJpaCx2CiEhBxXny7BPAse5+jbtfQ9SD7CeTDau4TP/B44UOQUSkoOIkCwPaU+bbGSaPIew6fB8AGg+eVuBIREQKK85DebcBi8zsd2H+fUTdeAx5O969P6OfX0frtLGFDkVEpKDi3OD+rpktBE4kuqK41N2fSzqwYuDlpQBUrN3JroKO2yciUlixuvsIgxg9m3AsRcfLo1q6sq27ei0z0o+sJyIyFCXZkeCg1zEiGrajrK4p7fKeTymKiAxV6oc7AwcoidJBxfpsYzaJiAx9WZOFmZWa2UMDFYyIiBSnrMnC3duBRjMbP0DxFK3ZV/+h0CGIiBRMnHsWTUQDGD0IdN3pdffPJBaViIgUlTjJ4g/hJSIiw1Sc5yxuN7ORwCx3XzEAMSUun5ZL9cfNYuxTb+VcT01pRWQoi9OR4N8CS4nGtsDMjjCzu5MOrFhse+8hXdMlu1oKGImISOHEaTr7ZeAYYAeAuy8F9k0wpuJSsudpiplfe5ApNz9VwGBERAojTrJoc/e6HmXDtsZl5MqthQ5BRGTAxUkWL5rZh4BSM5tvZj8Ankg4rqKy6rqzu82X7kz/RLeIyFAVJ1l8GjgYaAbuBHYCn0syqKJj3Xtkr/mPhwsUiIhIYcRpDdUIfNHMvhHNen3yYYmISDGJ0xrq7Wa2DHiB6OG8583s6ORDKy5tVSO7pptnVRUwEhGRgRenGuoW4FPuPsfd5wBXEg2INKys/ZdTuqYr39pRwEhERAZenGRR7+5/7pxx98eB4VcVVWKsWnBO1lWGxVizIjIsZbxnYWadY8M9bWY3Et3cduBCYGHyoYmISLHIdoP7Oz3mr02ZHrbPWYiIDEcZk4W7n5JpmYiIDC85m86aWRXwUWBO6vrqohxK63bTProSyjTgoIgMbXG6KP8j8BSwDOhINpzBI3UwpK3nHULDsbMLGI2ISLLiJIsR7n5V4pEMYpN+96KShYgMaXHqT35qZp80s+lmNrHzlXhkRWr72QcUOgQRkQEXJ1m0AN8CngSWhNfiJIMqFumafLVNGJV23dHPrE42GBGRAopTDXUVMM/dtyQdzGBgbe1pyyf/5gUaD5sOlXG+UhGRwSXOlcVLQGO+H2xmt5rZJjN7MaVsopk9aGavhvcJodzM7HozW2lmL6Q8EFh0dh1ZQ+PBU6k/ZlavZbOuvb8AEYmIJC/On8HtwFIze4Som3IgVtPZHwM/BH6SUnY18LC7LzCzq8P854GzgPnhdSxwQ3gvSps/UgvAtvMOoWxbIzO+tbCwAYmIJCxOsvjf8MqLuz9mZnN6FJ8LnBymbyfqNuTzofwn7u7AU2ZWZWbT3X19vvsdUGa0TRrdvSi86xF3ERlK4oxncXs/7m9qZwJw9/VmNiWUzwBS7xCvCWW9koWZXQ5cDjBrVu+qIBER6X9xxrN4w8xe7/nq5zjSddia9o9zd7/J3Wvdvba6urqfw+ibVQvOoX1EGTvfMQc6nNIduzHo9aLHNGmWi4gUozjVULUp0yOAC4C+PmexsbN6ycymA5tC+RpgZsp6NcC6Pu6jIMxh3BNvMu6JNwHY/p63sfOdc/d0BdLegZXuyc3pkkO6KqyTw/vCfo1WRCQ/caqhtvYo+r6ZPQ5c04f93Q1cAiwI73ellP+Dmf2C6MZ2XdHfr+ihpLmt2/yE+1cw4f4Vvdbb8Mnj6BhdzojXtkKHU3/ivr3G+NZVhogUmzgdCaY2Yy0hutIYG2O7O4n+MJ5sZmuIujhfAPzKzC4D3iK6SoGo/6mzgZVEzXQvjX8IyXL698d72n8/1W1+4h+WA7D5g0fSOm0srVPHUtLYQuWq7ew+cGrXeqkx6Oa5iAy0ONVQqeNatAFvAn+XayN3/2CGRaelWdeJhmsdtHYdMo3RL27o8/bVdz6XtryjrITdB05l6wWHMfG3y9h23qFYjAf/lFBEpD9Z9Ds9ONXW1vrixcn3PBL3yqJkVwteaky8+yXGPLs20Zg2fuJYmuZNBnfocCjt3VZh8J5ZEUmSmS1x99rca6ZskytZmFkl8AF6j2fx1T7E2K8GKllAH6ui2jqgxKIXULqziQl3v9R1BVJfO5Oxi/u3T6mWaWOp2FDP5g8dye4DpjJi5RZ2HzQ16zZKKiLDS1+SRZxqqLuAOqIOBJtzrDvsdbvHkTIokgM2bgRbLj6a0Vf/gdbq0Ww7/zB2vmsuZdsb8Yoypt345F7vv2JDPQDVP+9erdV4wBTqj59N26TRtE1O/yBhp5PC+8K9jkZEhoo4VxYvuvshAxRPXgbyygJyX13k+gs959VJewezvnQv2885iPoT96V8bR37/ODxrsX9eSWy9bxDKGluo3nWBEqaWtk9rzrtiH89j0k32kUGv6SuLJ4ws0PdfVkf4xoy9rZlVOePa7rnKQygtIS3rjunq6x1xnhWLTiHVD2TxaZLaplye/4Jc9LvXkxbvuPUeew8aT/KNzXQMrNKzXhFBIh3ZfEyMA94g6gayogaMB2WfHjZDfSVBWRPFv3xl3bGRELmZDX2z68z8Q/L2X7WAUy496/9EEWk8eCpdFSWsfUDh4FDaWML7eNGUNLQTMfIcry0JG28VeF9R79FIiL9Kakri7P6GI/0QbqE4z2mTwYeTZm3d86l/p1zAdg9bzIjX91Mw9E1jF30FuWbd1GyuzV6dmNNXV6xjHppI0DGll2WctWTmjQ691KFEoYMfuocNBLnCe5VAxGIxLcwQ7kDNmM8rTPGA1B3+v4ZVnTGPrmKiXe/tFdxjHlqFa3Tx1K6fTeNR8wAul/51NG7u5KeV0apV0vjUXIRKVZ6ziJPSVdD7a2efwXFuecw6VdLGfPsWjZ9LLoqnfLjvftON32slpbp42gfPzLvbXPe16F7gkm3vOdnpSpEFdlw+Mv05PC+sIAxJGUonr9EnrMoZoVIFieH94XhfTC2DoqTQKb98PG8q60y2fjxY2javxpraWPEK5uxtg5aZoynrXrMXn92nGSR655PnPOW6TvLZ9ti/veR7TvK9xgHw/Fmk+kPrsF6POkoWRTAUPiHlPVqpMMp2d3KzK892FW09QOHMuk3/dM4buNlxzD62bWMXL6RrRccTtO8yXhZCRUb6mmbMJKOURVd65bsamHcY6+z46wDusr6o++ubD9we9tcuudnDMS/kzj7y5Qcsn2fff1uMiXozvLS8N6WZr1ckvhuk0wWxfJ7kdQNbsmi0Ce9P/Q8hm4/GCVGx+iKPU143bHWDib9ZhmNB01l6/sPxTqcksYWJt71EiPe2JbXvqfe8nTX9JSfLum1fNWCcyjb3MCM7zzaVdZWNYKG4+cAe58oMn3G3nxutm3768ft5PC+MMa6/fUD1ZfPyfRdpJa39yjrvHd1cphfmGb/mc7Z3v7Axz136T47V3yZ1k03H2fbuOv3F11ZSFbZbkinU7FqOy37jIPyUmZc9zBldU2JxbZ73mQ2feJYZn7pXkraOlj9xdPpGFPBuEdfo2m/ybTUjGfcwtdoqJ1Jx9jKxOLIpXzdTsY98UbUBDmlO/p0f9V3yidZ5XpwMtd9tmJ/lqa/nm+C9Ek215WUhT+QvKI0w5p7J1vVX2oS6fzLvj3NevlSNZQUTKb/cCWNLZSvr2f082sZ+3T/9oMF0Pi2akat2Jx22aaLj2LKHc92zTccXcOYJWsA2HLhEUz+5VIaamtomT6Oib9/GaDXQ5Al9c2U1jezz/V/ZtMltew+cCpV9/0VLyuhuaaKpgOmkE3pjt3ULPhT13zPz89LewcljS10jB3Ra1G3Hxx3Rr68MeriviT3z+zIFzdQ0tLGrqNq+h5bobR3MPXmRWz62NvxGL0x95Sr2s2A2Vf/AYD1V7yDltkT+hTmiJVbKNnVQuPh+/RaVkr3BJCPgUwWqoaSfpHuH60BHaMqaN5vEs37TWLb+w+jfF0d+1z/OBs+eRyt08ZSfccSvLyUnSfMoWneZGZ/8V4A1v7TSd2qnjLJlCiAbokC6EoUAJN/uTQqW7ym2zqjlq1n9/zJlG1t7NbVCpD2SflVC86hfG0drdPGUvXQK+w4bT5TbnuGka9FY4Zt/Pgx3T9/6VrGLFlD07zJ7DxpP6yplVlffoDV15xBx6jyrMfa9d1cdRJtU7o3Dkj9wau6fwXjF75G3Snz2PGet1Gyq4WJd7/E7vmTaTx0Ol5ZxoiVW2itHk37+JFMuSOq/murGknz3EmMXL6R0c+uoeGYWV3VhKuuO5vSuiYoMdpHV0BpCSNf3sjYJ9+kfMsuyrbv7vo+ar7+EPXHzwaHutPmdV1Nle5somL1DnYfPI2Z/3YvJa0dNM2ewM6T9mPCH5ez7dxDaJs0iraJo9Ief8Vb2+kYWd6tYcSkX7/AiDe2Meva+7sl4s4f+KzJ2R3rOfDY7lZmfeWBaLrHtqOWb8yaLPb59kK2n3PgnnFo2juY+PuX2fa3BzH15kVRPGmSRXuIBTPG/elVJjzwyt79UZEQXVnIgOhLNUJJQzMzv/4QAKuvOQMvgVlffqB/AyugNf96GjX/8XDX/KoF51C6rZGy+iam3fAkDbU1bD3vUKbc+nT04/xKlBiba8az6ePHMPOrD9I8YxybPnYMpbua6agoo7Shmen/9UTXZzYeOIVRyzd122/jgVMZtXxj2pi2fOAwJv/mhZyx7zh9PlUPvdqXw2bn8bMZ92Tmx7d2nDqP8Y+9jrV10Dyzig1XngDtHV3JMh/rPvNOWvcZB0DZpgam3vo0ZTuixNY0ZyIb//54bHcr4xe+xvhHX+u2becPdmfi2XXodEYvW09HRSklLe2suu5spt70FCPe2Mb2Mw9gwn1R7wmbP3wU4x98hYpNDWlj6qgso23CSCo21He1FOzcR6e1/3RSV1IctWw9jYdMY/TStZRvbKBsx25GL13H6n97d5S4+0DVUDIo5XNjMFox+ius53+wXYdOY/Syvg9AJcPP5g8d2auH5k6rvnYmU29exIhV2xONYf0V72D6DU/0Kl+14Jxe/8ZTNdTWsOX8w/u0TyULGTJ6Jo3U5pWdyypWbWf6DU/QcOQMtl54BACVq7Yx7YY9Xb23Vo9m06XHMOaZ1TQeOIWWmipm/+sfu5a/9dUzo+qRNTuo/sXStLG0TRjZVc0iUix2nDqPHWe8rU/bKlnIkJKtiWASLXgqVu+g+ufPsv7KExj32Os0zZ9M0/xqAGZ840+Ubd/N+k+fSOWb26hcvYMtFx7B7C/sSTw9q3ya9xlH5bqdAKz79IlMve1p2kdXULExffVEf1r36RN73XPJpO6kuYx/9PW0y9Zf8Q7aJo/u9pxNqpZpY9n00VpqvvlIn2MdSnYdNp3RL6wfkH1t+sjRNB48rU/bKlnIsJN3FVZftbZTsamBltDvVlZtHZRtb6Stegwlja2Ubd1Fy8yqPcvdu5JMrqoGgIajZqTtzHHHu/en6sFXuuY3/P3xTLl5Eau/duaeJrrtHYxd9Ba7Dt8HOjoorW/GWjtomT2BsY+/gZeV0HDMrG5XW2uuPpX2qt5dtXTG2VBb09UwoLNef+TyjV0NAN768hnd7i2t/b8n0zZxFCW7mpn0uxcZ9fJG1l95Ah2jKqhYV0fjodO71i3fUE/bxFF4RWmv8VxSbfrI0VSu2s74x16nZeoYtp13KNN+lHvwsLe+8h5mXXt/5uVfPoPS+uZujSt2nrgv4x5/o2s+Wwu8VQvOoaS+idL6Zlqnj4uaj+9sjnWeN118FKNe2siY5zIPybz2qncx8tUtTPz9y2z+8FHsSvnu8qFkIcNerm4rMj2DMODPG3R4dO+ltATcKduyK233JyUNzXSMqey6wbt73iQqNjRQd8p+1J+wLyUNzVSu2UHT3Ml7/RxAVwuifz8r7ZjuXdwpX1/fdeO4U9W9f2X3gVNonjMRgMo3t1F13wo2Xn5crCa86Yxatp7GA6cw+0v3AVGi2pqhnn7U0rXdqhKb5k5iy0VH0D62kqr7VlB3+ny8vJTRi1dTdYOhP40AAAoHSURBVP8Ktp13KFN+Ev1+bLj8OLy0pKu107T//AuVq3fQcNQMtv7dEVS+vpVpNz3V7YZ5p/KN9ezzvceA3K2vSnc2U3Nd1Khhy4WHs3v/KVhbB+3j9zSHrlhTx/Qf7kmSde+ay/jHXu/6/M6HVDdfdAS7Qgee+VKyENlLuR6OgugJY4ieMs7nAbh0n1dMD8VZcxulO5v6pc+ugmjvYMrti9l6/mFUvrmdxkOm5UxSJQ3NlG3f3e3KL4lzkvqZ5et3Mv7R19hy0ZGxt09tCly2tZEZ33qELRccTsPRfXs2Rs9ZiOylTH86ZSuv6jEfp+uGnmOUkGa7viSevWGVZV2JIlcHjfk+2R93nb58Jp3rlZawKTzX0njY9G7rZErOHWMqaRnTf0/3Z7u3Np6o2/7W6ePyShSdGg/qfH4jeoRv5IpN0Mdk0RdKFiJ7qWd353390c6WkLLd7O/vuoFMn5+rT6J0LdayrZ+p+/ls+851rJ0/yD3XzTSd+tmpy/LtoynTWCypn9O5PFMPvdmOsfMBQQeaJ45iTs146t8xJ0aU/UfJQqRIZfuBS3p/fVme2mtsvp+1t/vem++nv5N7Ptv15RxXlpWy4R9O7OPe+07JQkQGzOC9Q1pYxfC9ZWnyICIiElGyEBGRnJQsREQkJyULERHJSclCRERyUrIQEZGc1HRWRGSQKURTWl1ZiIhITkoWIiKSk5KFiIjkpGQhIiI5FVWyMLMzzWyFma00s6sLHY+IiESKJlmYWSnwn8BZwEHAB83soMJGJSIiUETJAjgGWOnur7t7C/AL4NwCxyQiIhRXspgBrE6ZXxPKujGzy81ssZkt3rw5/aDpIiLSv4opWaQbQbHXsyfufpO717p7bXV19QCEJSIixfQE9xpgZsp8DbAu2wZLlizZYmar+ri/ycCWPm47WOmYhwcd8/CwN8c8O98NzL0YxmACMysDXgFOA9YCzwAfcveXEtrfYnevTeKzi5WOeXjQMQ8PA33MRXNl4e5tZvYPwP1EY7/fmlSiEBGR/BRNsgBw9z8Cfyx0HCIi0l0x3eAeaDcVOoAC0DEPDzrm4WFAj7lo7lmIiEjxGs5XFiIiEpOShYiI5DQsk8Vg67DQzGaa2SNmttzMXjKzz4byiWb2oJm9Gt4nhHIzs+vD8b1gZkelfNYlYf1XzeySlPKjzWxZ2OZ6M7Ns+xjAYy81s+fM7J4wv6+ZLQrx/NLMKkJ5ZZhfGZbPSfmML4TyFWb2npTytP8OMu1jgI63ysx+bWZ/Def7+KF+ns3sH8O/6xfN7E4zGzHUzrOZ3Wpmm8zsxZSygp3XbPvIyN2H1YuoWe5rwFygAngeOKjQceWIeTpwVJgeS/Q8ykHAN4GrQ/nVwDfC9NnAvURPxR8HLArlE4HXw/uEMD0hLHsaOD5scy9wVihPu48BPPargJ8D94T5XwEXhekfAVeE6U8BPwrTFwG/DNMHhXNcCewbzn1ptn8HmfYxQMd7O/CJMF0BVA3l80zUpc8bwMiU7/5jQ+08A+8CjgJeTCkr2HnNtI+sxzBQ/wmK5RW+0PtT5r8AfKHQceV5DHcB7wZWANND2XRgRZi+EfhgyvorwvIPAjemlN8YyqYDf00p71ov0z4G6DhrgIeBU4F7wj/sLUBZz3NJ9HzO8WG6LKxnPc9v53qZ/h1k28cAHO84oh9O61E+ZM8ze/qEmxjO2z3Ae4bieQbm0D1ZFOy8ZtpHtviHYzVUrA4Li1W47D4SWARMdff1AOF9Slgt0zFmK1+Tppws+xgI3wf+BegI85OAHe7elibOrmMLy+vC+vl+F9n2kbS5wGbgNouq3m42s9EM4fPs7muBbwNvAeuJztsShvZ57lTI85r37+BwTBaxOiwsRmY2BvgN8Dl335lt1TRl3ofygjGzvwE2ufuS1OI0q3qOZYPpuygjqqq4wd2PBHYRVR1kMpiOLa1Qh34uUdXRPsBoojFtehpK5zmXgTiWvLcZjski7w4Li4GZlRMlip+5+29D8UYzmx6WTwc2hfJMx5itvCZNebZ9JO0E4L1m9ibR2CanEl1pVFnUj1jPOLuOLSwfD2wj/+9iS5Z9JG0NsMbdF4X5XxMlj6F8nk8H3nD3ze7eCvwWeAdD+zx3KuR5zft3cDgmi2eA+aElRAXRTbK7CxxTVqFlwy3Acnf/bsqiu4HOFhGXEN3L6Cz/aGjxcBxQFy5B7wfOMLMJ4S+6M4jqadcD9WZ2XNjXR3t8Vrp9JMrdv+DuNe4+h+gc/cndPww8ApyfJp7UOM8P63sovyi0otkXmE90MzDtv4OwTaZ9JMrdNwCrzextoeg04GWG8Hkmqn46zsxGhZg6j3nInucUhTyvmfaR2UDcxCq2F1FLgFeIWkl8sdDxxIj3RKJLxBeApeF1NlG968PAq+F9YljfiIaofQ1YBtSmfNbHgZXhdWlKeS3wYtjmh+x5uj/tPgb4+E9mT2uouUQ/AiuB/wEqQ/mIML8yLJ+bsv0Xw3GtILQSyfbvINM+BuhYjwAWh3P9v0StXob0eQa+Avw1xPVTohZNQ+o8A3cS3ZNpJfqr/rJCntds+8j0UncfIiKS03CshhIRkTwpWYiISE5KFiIikpOShYiI5KRkISIiOSlZiOTJzL5qZqf3w+c09Ec8IgNBTWdFCsTMGtx9TKHjEIlDVxYigJldbGZPm9lSM7vRonE0GszsO2b2rJk9bGbVYd0fm9n5YXqBmb0cxgT4diibHdZ/IbzPCuX7mtmTZvaMmX2tx/7/OZS/YGZfCWWjzewPZva8RWM9XDiw34rIHkoWMuyZ2YHAhcAJ7n4E0A58mKhTu2fd/SjgUeDaHttNBM4DDnb3w4Cvh0U/BH4Syn4GXB/K/x9RJ4FvBzakfM4ZRN1THEP0BPfRZvYu4Exgnbsf7u6HAPf1+8GLxKRkIRL1R3Q08IyZLQ3zc4m6Rv9lWOcOom5XUu0EmoCbzez9QGMoP55owCaIuq/o3O4Eom4fOss7nRFezwHPAgcQJY9lwOlm9g0ze6e71+3lcYr0mZKFSNRPzu3ufkR4vc3dv5xmvW43+DwaC+EYot6A30fmv/w9w3Tq/q9L2f88d7/F3V8hSmLLgOvM7Jr8Dkuk/yhZiEQdrJ1vZlOga9zi2UT/Pzp7Jf0Q8HjqRmF8kfHu/kfgc0RVSABPEPVuClF1Vud2f+lR3ul+4OPh8zCzGWY2xcz2ARrd/Q6iAYJyj5MskpCy3KuIDG3u/rKZfQl4wMxKiHoGvZJo8KGDzWwJ0YhsPW8wjwXuMrMRRFcH/xjKPwPcamb/TDTy3aWh/LPAz83ss0RXI537fyDcN3ky6mGaBuBiYB7wLTPrCDFd0b9HLhKfms6KZKCmrSJ7qBpKRERy0pWFiIjkpCsLERHJSclCRERyUrIQEZGclCxERCQnJQsREcnp/wNBa6TBDD1nFAAAAABJRU5ErkJggg==\n", 253 | "text/plain": [ 254 | "
" 255 | ] 256 | }, 257 | "metadata": { 258 | "needs_background": "light" 259 | }, 260 | "output_type": "display_data" 261 | } 262 | ], 263 | "source": [ 264 | "mean_smmothed = np.mean(uniform_filter1d(train_data[:,0,:],axis=1,size=1000,mode='nearest'),axis=0)\n", 265 | "std_smmothed = np.std(uniform_filter1d(train_data[:,0,:],axis=1,size=1000,mode='nearest'),axis=0)\n", 266 | "\n", 267 | "plt.errorbar(range(n_episodes_training),mean_smmothed,yerr=std_smmothed,ecolor='cyan')\n", 268 | "plt.xlabel('episodes')\n", 269 | "plt.ylabel('number of steps per episode')" 270 | ] 271 | }, 272 | { 273 | "cell_type": "markdown", 274 | "metadata": {}, 275 | "source": [ 276 | "### Training: total number of steps\n", 277 | "\n", 278 | "We compute the total number of steps taken by each agent." 279 | ] 280 | }, 281 | { 282 | "cell_type": "code", 283 | "execution_count": 7, 284 | "metadata": { 285 | "scrolled": true 286 | }, 287 | "outputs": [ 288 | { 289 | "name": "stdout", 290 | "output_type": "stream", 291 | "text": [ 292 | "Average total number of steps: 65984107.0\n" 293 | ] 294 | }, 295 | { 296 | "data": { 297 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAERCAYAAABxZrw0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAYrElEQVR4nO3df5hWdZ3/8eeLHzpIUAkjq44EtAYSGrADiZjrCvkrV/uBJWGXrtpUij/arS51s7Vve5ntamuXpd+dyjRNTVESUTGx0BUUAkFFyR9LaGPugpiKGgr43j/OGR2YYebM3HPu+54zr8d1zTX3fX593jcMrzl8zud8jiICMzMrnj6VLsDMzPLhgDczKygHvJlZQTngzcwKygFvZlZQDngzs4KquoCXdJWk9ZJWZ9j2PyStSr+ekvRyOWo0M+sJVG3j4CUdArwG/DwixnVivzOBCRFxSm7FmZn1IFV3Bh8R9wMvtVwm6YOSFkhaIem/JI1pY9eZwA1lKdLMrAfoV+kCMmoEvhwRT0v6KHAFcFjzSkkfAEYCv6lQfWZmVafqA17Se4CDgJslNS/edYfNTgDmRMS2ctZmZlbNqj7gSbqRXo6I8e1scwJwRpnqMTPrEaquD35HEfEq8AdJxwMo8ZHm9ZJGA+8HHqxQiWZmVanqAl7SDSRhPVpSk6RTgVnAqZIeAR4Hjmuxy0zgxqi24UBmZhVWdcMkzcyse1TdGbyZmXWPqrrIOnTo0BgxYkSlyzAz6zFWrFjxYkTUtrWuqgJ+xIgRLF++vNJlmJn1GJKe3dk6d9GYmRWUA97MrKAc8GZmBVVVffBmZu3ZsmULTU1NbN68udKllF1NTQ11dXX0798/8z4OeDPrMZqamhg0aBAjRoygxdxUhRcRbNy4kaamJkaOHJl5P3fRmFmPsXnzZoYMGdKrwh1AEkOGDOn0/1wc8GbWo/S2cG/Wlc/tgDczKygHvJlZQRXmIuuIc+/IvY11F38i9zbMzLqLz+DNzDph3bp1jBkzhtNOO41x48Yxa9YsFi5cyNSpU9l3331ZtmwZr7/+OqeccgqTJk1iwoQJ3Hbbbe/s+7GPfYyJEycyceJElixZAsCiRYs49NBDmTFjBmPGjGHWrFl0x0y/uZ3Bpw/i+GWLRaOAb0XEZXm1aWa9zKGHtl722c/C6afDG2/A0Ue3Xn/yycnXiy/CjBnbr1u0KFOzzzzzDDfffDONjY1MmjSJ66+/ngceeIB58+Zx0UUXMXbsWA477DCuuuoqXn75ZSZPnsz06dPZY489uOeee6ipqeHpp59m5syZ78y/tXLlSh5//HH22msvpk6dyuLFizn44IM786fRSm4BHxFPAuMBJPUFngfm5tWemVm5jBw5kv333x+AD3/4w0ybNg1J7L///qxbt46mpibmzZvHJZdcAiTDO5977jn22msvZs+ezapVq+jbty9PPfXUO8ecPHkydXV1AIwfP55169ZVb8DvYBrw3xGx01nPzMw6rb0z7t12a3/90KGZz9h3tOuuu77zuk+fPu+879OnD1u3bqVv377ccsstjB49erv9LrzwQoYNG8YjjzzC22+/TU1NTZvH7Nu3L1u3bu1SbS2Vqw/+BOCGtlZIapC0XNLyDRs2lKkcM7P8HHHEEVx++eXv9KOvXLkSgFdeeYU999yTPn36cO2117Jt27Zc68g94CXtAhwL3NzW+ohojIj6iKivrW1zznozsx7lggsuYMuWLRxwwAGMGzeOCy64AIDTTz+da665hgMPPJCnnnqKgQMH5lpH7s9klXQccEZEHN7RtvX19dHVB354mKRZ8a1Zs4b99tuv0mVUTFufX9KKiKhva/tydNHMZCfdM2Zmlp9cA17SbsDHgVvzbMfMzFrLdRRNRLwBDMmzDTPrXSKiV0441pXudN/JamY9Rk1NDRs3buyWuzx7kub54FsOq8yiMHPRmFm+qmEgQ11dHU1NTfTGIdXNT3TqDAe8mVW9avjl0hM54M16EAeddUZ1BfyTT7aePCjjxEHvf+MVrvzVd1utvm7C0czf7xD2fHUD/zH/0lbrfzz5U9z71x9l1MYmLrr7h63WX37QCSweMZ6x/7u27YmNLroIDjoIliyB889vvf6yy2D8eFi4EP71X1uv/8//hNGj4fbb4dLW9XHttbDPPvDLX8KVV7ZeP2dOcsv11VcnXzu6887klu0rroCbbmq9vvlW7Usugfnzt183YADcdVfy+jvfgXvv3X79kCFwyy3J6/POgwcf3H59XR1cd13y+pxzYNWq7dd/6EPQ2Ji8bmiAFvNyAMmf22Xp3HQnnghNTduvnzIFvpv+nX/mM7Bx4/brp02D9AYTjjoK/vKX7dcfcwx87WvJ605MWvXQ2qSdOftPZ87+03P52Ttw1BD45jdh+vTkz+2ccwC4ce27n/HfDjmJh+v2Y2LTGr5x/zWtjv//pjXwxLBRTF23ijOX3Nhq/flHzGbtkDqmPbOULy5rMU3UQ/+efN/hZ69l2wBf+eR5/Hm39zLjsYXMeGxhq+OffPyFbO5fw4kP38Exv/+vVutP+PzFAHxx6a1M++9l27e9w8/ejddv/7P75wGD+cqnkn9v37jvaiY+//vt1r8waChf/fvk7/ZbCxsZu37tduvX7r435x95JgAXLbicUS89/27bULU/e+9oOWFaO3yR1cysoHK/k7UzfCer9QSV/Flz272r7Szau5O1urporNOq/YfPzCrHXTRmZgXlM3jrMv/vway6OeCtR/IvF7OOuYvGzKygHPBmZgXlgDczKygHvJlZQTngzcwKygFvZlZQDngzs4JywJuZFVTeD91+n6Q5kn4vaY2kKXm2Z2Zm78r7TtYfAAsiYoakXYDdcm7PzMxSuQW8pMHAIcDJABHxFvBWXu1Vkm+bN7NqlGcXzShgA/AzSSsl/UTSwBzbMzOzFvIM+H7ARODKiJgAvA6cu+NGkhokLZe0vDc+Kd3MLC95BnwT0BQRS9P3c0gCfzsR0RgR9RFRX1tbm2M5Zma9S24BHxH/A/xR0uh00TTgibzaMzOz7eU9iuZM4BfpCJq1wD/k3J6ZmaVyDfiIWAW0+TBYMzPLl+9kNTMrKAe8mVlBOeDNzArKAW9mVlAOeDOzgnLAm5kVlAPezKygHPBmZgXVYcBLOlvSYCV+KulhSYeXozgzM+u6LGfwp0TEq8DhQC3JdAMX51qVmZmVLEvAK/1+NPCziHikxTIzM6tSWQJ+haRfkwT83ZIGAW/nW5aZmZUqy2RjpwLjgbUR8YakIXhWSDOzqtdhwEfE25JGACdKCuCBiJibd2FmZlaaLKNorgC+DDwGrAa+JOlHeRdmZmalydJF87fAuIgIAEnXkIS9mZlVsSwXWZ8Ehrd4vw/waD7lmJlZd8lyBj8EWCNpWfp+EvCgpHkAEXFsXsWZmVnXZQn4b+VehZmZdbsso2juk/QBYN+IWChpANAvIjblX56ZmXVVhwEv6YtAA7A78EGgDvj/wLQM+64DNgHbgK0R4Qdwm5mVSZYumjOAycBSgIh4WtIenWjj7yLixa4UZ2ZmXZdlFM2bEfFW8xtJ/YDIryQzM+sOWQL+PknnAwMkfRy4Gbg94/ED+LWkFZIa2tpAUoOk5ZKWb9iwIeNhzcysI1kC/lxgA8nNTV8C7oyIf854/KkRMRE4CjhD0iE7bhARjRFRHxH1tbW1Wes2M7MOZAn4MyPixxFxfETMiIgfSzo7y8Ej4k/p9/XAXJK+fDMzK4MsAX9SG8tO7mgnSQPTqYWRNJDkgSGrO1WdmZl12U5H0UiaCXweGNl812pqMLAxw7GHAXMlNbdzfUQsKKFWMzPrhPaGSS4BXgCGApe2WL6JDHPRRMRa4CMlVWdmZl2204CPiGeBZyVNB/6Szgv/IWAMnk3SzKzqZemDvx+okbQ3cC/J05yuzrMoMzMrXaaHbkfEG8Cngcsj4lPA2HzLMjOzUmUKeElTgFnAHemyLFMcmJlZBWUJ+LOB84C5EfG4pFHAb/Mty8zMSpVluuD7Sfrhm9+vBc7KsygzMytdljN4MzPrgRzwZmYFtdOAl/S99Pvx5SvHzMy6S3tn8EdL6k9ygdXMzHqY9i6yLgBeBAZKehUQyfzuAiIiBpehPjMz66KdnsFHxNcj4r3AHRExOCIGtfxexhrNzKwLsgyTPE7SMGBSumhpRPjRS2ZmVa7DUTTpRdZlwPHAZ4FlkmbkXZiZmZUmy5QD3wQmpU9lQlItsBCYk2dhZmZWmizj4Ps0h3tqY8b9zMysgrKcwS+QdDdwQ/r+c8Cd+ZVkZmbdIctF1q9L+jRwMMkQycaImJt7ZWZmVpJM0/5GxK3ArTnXYmZm3ch96WZmBZV7wEvqK2mlpPl5t2VmZu9qN+DTcL6uxDbOBtaUeAwzM+ukdgM+IrYBtZJ26crBJdUBnwB+0pX9zcys67JcZF0HLJY0D3i9eWFEfD/DvpcB3wAG7WwDSQ1AA8Dw4cMzHNLMzLLI0gf/J2B+uu2gFl/tknQMsD4iVrS3XUQ0RkR9RNTX1tZmKMfMzLLIMg7+2wCSBkbE6x1t38JU4FhJRwM1wGBJ10XEiV0r1czMOiPLZGNTJD1BeqFU0kckXdHRfhFxXkTURcQI4ATgNw53M7PyydJFcxlwBMkcNETEI8AheRZlZmaly3on6x8ltVy0rTONRMQiYFFn9jEzs9JkCfg/SjoIiHS45Fl4XLuZWdXL0kXzZeAMYG/geWB8+t7MzKpYllE0LwKzylCLmZl1oyyjaEZJul3SBknrJd0maVQ5ijMzs67L0kVzPXATsCewF3Az7z78w8zMqlSWgFdEXBsRW9Ov64DIuzAzMyvNTvvgJe2evvytpHOBG0mC/XPAHWWozczMStDeRdYVJIHePAD+Sy3WBfCdvIoyM7PS7TTgI2JkOQsxM7Pu1eEwSUl9SeZ0H9Fy+4zTBZuZWYVkuZP1dmAz8Bjwdr7lmJlZd8kS8HURcUDulZiZWbfKMkzyLkmH516JmZl1qyxn8A8BcyX1AbaQjKqJiBica2VmZlaSLAF/KTAFeCwifIOTmVkPkaWL5mlgtcPdzKxnyXIG/wKwSNJdwJvNCz1M0sysumUJ+D+kX7ukX2Zm1gNkmQ/+2+UoxMzMuleWO1l/SxuzR0bEYblUZGZm3SJLF83XWryuAT4DbO1oJ0k1wP3Armk7cyLiX7pSpJmZdV6WLpoVOyxaLOm+DMd+EzgsIl6T1B94QNJdEfFQVwo1M7POydJFs3uLt32AvwH+qqP90mGVr6Vv+6dfHmppZlYmWbpoWs4Lv5VkRM2pWQ6ezkS5Avhr4EcRsbSNbRqABoDhw4dnq9rMzDqUpYumy/PCR8Q2YLyk95FMdzAuIlbvsE0j0AhQX1/vM3wzs26S5QweSQfRej74n2dtJCJelrQIOBJY3cHmZmbWDbL0wV8LfBBYBWxLFwfQbsBLqgW2pOE+AJgOfK+0cs3MLKssZ/D1wNguzEWzJ3BN2g/fB7gpIuZ3tkAzM+uaLAG/mmTUzAudOXBEPApM6EpRZmZWuiwBPxR4QtIytp9s7NjcqjIzs5JlCfgL8y7CzMy6X5ZhklnuWjUzsyqT5YEfZmbWAzngzcwKygFvZlZQO+2Dl/QYbU8OJpK5xA7IrSozMytZexdZjylbFWZm1u12GvAR8Ww5CzEzs+7VYR+8pAMl/U7Sa5LekrRN0qvlKM7MzLouy0XWHwIzgaeBAcBpwOV5FmVmZqXLNF1wRDwjqW86v/vPJC3JuS4zMytRloB/Q9IuwCpJ/0Yy6djAfMsyM7NSZemi+UK63WzgdWAf4NN5FmVmZqXLEvCfjIjNEfFqRHw7Iv4RD6E0M6t6WQL+pDaWndzNdZiZWTdr707WmcDngZGS5rVYNRjYmHdhZmZWmvYusi4huaA6FLi0xfJNwKN5FmVmZqXr6E7WZ4EpkoYBk9JVayJiazmKMzOzrstyJ+vxwDLgeOCzwFJJM/IuzMzMSpNlHPw3gUkRsR5AUi2wEJjT3k6S9gF+TvLA7reBxoj4QWnlmplZVlkCvk9zuKc2km30zVbgnyLiYUmDgBWS7omIJ7pSqJmZdU6WgF8g6W7ghvT954C7OtopIl4guUhLRGyStAbYG3DAm5mVQZaHbn9d0qeBg0ke9tEYEXM704ikEcAEYGkb6xqABoDhw4d35rBmZtaOLBdZvxcRt0bEP0bEVyNirqTvZW1A0nuAW4BzIqLVNMMR0RgR9RFRX1tb27nqzcxsp7L0pX+8jWVHZTm4pP4k4f6LiLi1M4WZmVlp2ruT9SvA6cAoSS1vbBoELO7owJIE/JRk3Pz3Sy3UzMw6p70++OtJLqZ+Fzi3xfJNEfFShmNPJZmJ8jFJq9Jl50fEnV2q1MzMOqW9O1lfAV4heZpTp0XEAyQXZc3MrAKy9MGbmVkP5IA3MysoB7yZWUE54M3MCsoBb2ZWUA54M7OCcsCbmRWUA97MrKAc8GZmBeWANzMrKAe8mVlBOeDNzArKAW9mVlAOeDOzgnLAm5kVlAPezKygHPBmZgXlgDczKygHvJlZQTngzcwKKreAl3SVpPWSVufVhpmZ7VyeZ/BXA0fmeHwzM2tHbgEfEfcDL+V1fDMza1/F++AlNUhaLmn5hg0bKl2OmVlhVDzgI6IxIuojor62trbS5ZiZFUbFA97MzPLhgDczK6g8h0neADwIjJbUJOnUvNoyM7PW+uV14IiYmdexzcysY+6iMTMrKAe8mVlBOeDNzArKAW9mVlAOeDOzgnLAm5kVlAPezKygHPBmZgXlgDczKygHvJlZQTngzcwKygFvZlZQDngzs4JywJuZFZQD3sysoBzwZmYF5YA3MysoB7yZWUE54M3MCsoBb2ZWULkGvKQjJT0p6RlJ5+bZlpmZbS+3gJfUF/gRcBQwFpgpaWxe7ZmZ2fbyPIOfDDwTEWsj4i3gRuC4HNszM7MWFBH5HFiaARwZEael778AfDQiZu+wXQPQkL4dDTyZS0HVZSjwYqWLqIDe+rmh9352f+78fSAiatta0S/HRtXGsla/TSKiEWjMsY6qI2l5RNRXuo5y662fG3rvZ/fnrqw8u2iagH1avK8D/pRje2Zm1kKeAf87YF9JIyXtApwAzMuxPTMzayG3LpqI2CppNnA30Be4KiIez6u9HqZXdUm10Fs/N/Tez+7PXUG5XWQ1M7PK8p2sZmYF5YA3MysoB3wZSdpH0m8lrZH0uKSzK11TOUnqK2mlpPmVrqVcJL1P0hxJv0//3qdUuqZykPTV9Gd8taQbJNVUuqa8SLpK0npJq1ss213SPZKeTr+/vxK1OeDLayvwTxGxH3AgcEYvm77hbGBNpYsosx8ACyJiDPAResHnl7Q3cBZQHxHjSAZZnFDZqnJ1NXDkDsvOBe6NiH2Be9P3ZeeAL6OIeCEiHk5fbyL5x753ZasqD0l1wCeAn1S6lnKRNBg4BPgpQES8FREvV7aqsukHDJDUD9iNAt8DExH3Ay/tsPg44Jr09TXAJ8taVMoBXyGSRgATgKWVraRsLgO+Abxd6ULKaBSwAfhZ2jX1E0kDK11U3iLieeAS4DngBeCViPh1Zasqu2ER8QIkJ3bAHpUowgFfAZLeA9wCnBMRr1a6nrxJOgZYHxErKl1LmfUDJgJXRsQE4HUq9F/1ckr7m48DRgJ7AQMlnVjZqnonB3yZSepPEu6/iIhbK11PmUwFjpW0jmRW0cMkXVfZksqiCWiKiOb/pc0hCfyimw78ISI2RMQW4FbgoArXVG7/K2lPgPT7+koU4YAvI0ki6Y9dExHfr3Q95RIR50VEXUSMILnY9puIKPwZXUT8D/BHSaPTRdOAJypYUrk8Bxwoabf0Z34aveDi8g7mASelr08CbqtEEXnOJmmtTQW+ADwmaVW67PyIuLOCNVm+zgR+kc7HtBb4hwrXk7uIWCppDvAwycixlVTJrft5kHQDcCgwVFIT8C/AxcBNkk4l+YV3fEVq81QFZmbF5C4aM7OCcsCbmRWUA97MrKAc8GZmBeWANzMrKAe8WTdKZ488vdJ1mIED3qy7vQ9wwFtVcMBbryPpV5JWpPOVN6TLTpX0lKRFkn4s6Yfp8lpJt0j6Xfo1NV1+YToP+CJJayWdlR7+YuCDklZJ+vfKfEKzhG90sl5H0u4R8ZKkAcDvgCOAxSTzxGwCfgM8EhGzJV0PXBERD0gaDtwdEftJuhA4HPg7YBDwJPBXJNM/z0/nQTerKE9VYL3RWZI+lb7eh2T6iPsi4iUASTcDH0rXTwfGJlOqADBY0qD09R0R8SbwpqT1wLCyVG+WkQPeehVJh5KE9pSIeEPSIpKz7/12skufdNu/7HAcgDdbLNqG/z1ZlXEfvPU27wX+nIb7GJJHJ+4G/K2k96dPIPpMi+1/DcxufiNpfAfH30TSZWNWcQ54620WAP0kPQp8B3gIeB64iOTpWgtJpvR9Jd3+LKBe0qOSngC+3N7BI2IjsDh92LQvslpF+SKrGclTtiLitfQMfi5wVUTMrXRdZqXwGbxZ4sJ0jv7VwB+AX1W4HrOS+QzezKygfAZvZlZQDngzs4JywJuZFZQD3sysoBzwZmYF9X+o4xc4IMTP2gAAAABJRU5ErkJggg==\n", 298 | "text/plain": [ 299 | "
" 300 | ] 301 | }, 302 | "metadata": { 303 | "needs_background": "light" 304 | }, 305 | "output_type": "display_data" 306 | } 307 | ], 308 | "source": [ 309 | "total_steps = np.sum(train_data[:,0,:],axis=1)\n", 310 | "\n", 311 | "plt.bar(range(1,n_simulations+1), total_steps)\n", 312 | "plt.axhline(np.mean(total_steps),c='red',ls='--',label='mean')\n", 313 | "plt.legend()\n", 314 | "plt.xlabel('agent')\n", 315 | "plt.ylabel('total number of steps')\n", 316 | "\n", 317 | "print('Average total number of steps: {0}'.format(np.mean(total_steps)))" 318 | ] 319 | }, 320 | { 321 | "cell_type": "markdown", 322 | "metadata": {}, 323 | "source": [ 324 | "### Test: number of steps\n", 325 | "\n", 326 | "We first plot the number of steps (average and standard deviation over 10 repetitions)" 327 | ] 328 | }, 329 | { 330 | "cell_type": "code", 331 | "execution_count": 8, 332 | "metadata": {}, 333 | "outputs": [ 334 | { 335 | "data": { 336 | "text/plain": [ 337 | "" 338 | ] 339 | }, 340 | "execution_count": 8, 341 | "metadata": {}, 342 | "output_type": "execute_result" 343 | }, 344 | { 345 | "data": { 346 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEGCAYAAABvtY4XAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOz9eZxkZX3oj78/59TW6/RMd8/OwMyAbAMCDsimokGNSHBDiNFfNMZg1ERvvDE3xghqNPfmxhijyc+L1zVqiIIxGvRGjYoLIDrsM8AAMzDMMFtP713VXcs5z/ePc55Tp6qrqk91d3VVVz/v16tfXfv5nO3zfJ7P81lEKYXBYDAY2g+r2QIYDAaDoTEYBW8wGAxtilHwBoPB0KYYBW8wGAxtilHwBoPB0KbEmi1AmIGBAXXKKac0WwyDwWBYNtx7770nlFKDld5rKQV/yimnsGvXrmaLYTAYDMsGETlQ7T3jojEYDIY2xSh4g8FgaFOMgjcYDIY2paV88AaDwVCLfD7PoUOHmJmZabYoS04qlWLz5s3E4/HI3zEK3mAwLBsOHTpET08Pp5xyCiLSbHGWDKUUw8PDHDp0iK1bt0b+nnHRGAyGZcPMzAz9/f0rSrkDiAj9/f11z1yMgjcYDMuKlabcNfPZb6PgDQaDoU0xCt5gMBgWyL6hKfYNTTVbjFkYBW8wGAxtilHwBoPBUAdPP/00Z5xxBm9729vYsWMHb3zjG7nzpz/huldeyWmnncavfvUr0uk0b33rW7nwwgs5//zz+fa3vx189wUveAEXXHABF1xwAXfddRcAd9xxB1dccQXXXnstZ5xxBm984xtZjG57DQuTFJHTga+HXtoG3KiU+mSjtmkwGFYYV1wx+7XrroN3vhMyGbjqqtnvv+Ut3t+JE3DttaXv3XFHpM0++eST3HrrrXz2s5/lwgsvJPdv3+Drt/+QPXf/mL/+67/mrLPO4iUveQlf+MIXGBsb46KLLuLKK69k7dq1/PCHPySVSvHEE0/whje8Iai/df/997Nnzx42btzIZZddxp133snll19ez9GYRcMUvFJqL3AegIjYwLPAtxq1PYPBsDhcf/PdAHz97Zc0WZLWZevWrZxzzjkAnH322Zx3yYsQEc455xyefvppDh06xHe+8x0+/vGPA1545zPPPMPGjRv5oz/6Ix544AFs2+bxxx8PfvOiiy5i8+bNAJx33nk8/fTTravgy/gNYJ9SqmrVM4PBsHisGCVdy+Lu7Kz9/sBAZIu9nGQyGTy2LIuE/9yyLAqFArZt881vfpPTTz+95Hsf+tCHWLduHQ8++CCu65JKpSr+pm3bFAqFeckWZql88L8N3FLpDRG5QUR2iciuoaGhJRLHYDAYGsfLX/5yPv3pTwd+9Pvvvx+A8fFxNmzYgGVZfOUrX8FxnIbK0XAFLyIJ4Brg1krvK6U+q5TaqZTaOThYsWa9wWAwLCs++MEPks/nOffcc9mxYwcf/OAHAXjnO9/Jl7/8ZS6++GIef/xxurq6GiqHLMZKbc0NiLwKeJdS6mVzfXbnzp3KNPxoD1aMi6BFWcjxb+Vz9+ijj3LmmWc2W4xZ6Bj47YPdJY8Xm0r7LyL3KqV2Vvr8Urho3kAV94yh8Vx/893BDWswGFYWDVXwItIJvBT4t0Zux7ByMAOWwRCdhkbRKKUyQH8jt2EwGFYWSqkVWXBsPu50k8lqMLQ4ZtZSJJVKMTw8vChZnssJXQ8+HFYZBdPww2AwLBs2b97MoUOHaLWQ6qHJLAC5E8mSx4uJ7uhUD0bBG6rSytEUhvan0vUXj8fr6mi0VHwokPW8ksfNxrhoDAaDoU0xCt5QN8YnbDAsD4yCNxgMLcNSGg8rwVAxCn6ZshIuToPBsDCMgjcsKWZgMhiWDqPgDS2JGQgMhoVjFLzBYGgqZjBvHEbBGwyGZYUZEKJjFLzBYDA0kUYOWEbBGwyGlreKW12+VsUo+AiYi8sQlSjXymJ9phVZrnK3K0bBGwwGwxKzVAOhUfALZCEnylg7BsPKoRn3u1HwbYAZKAy1MNfHysUo+GWEuVGrY45N62HOSfOZU8GLSKeIfFBE/q///DQRubrxohkMzcUoKMNyJ4oF/0UgC+iq+4eAjzZMIoPBYDAsClEU/Hal1P8G8gBKqWlg5XW8NczJcrJ4W1HWpZapFY+BYXGJouBzItIBKAAR2Y5n0RsiYmpcG1qFVrk+miVHq+z/UhGlJ+tNwH8CJ4nI14DLgLc0UiiDwWAwLJw5FbxS6ocich9wMZ5r5j1KqRMNl8xgWGTauYl4O++bYf5UVfAickHZS0f8/1tEZItS6r7GidUczE1iMBjaiVoW/N/5/1PATuBBPAv+XOAe4PLGimYwGAxLTzsZelUXWZVSL1ZKvRg4AFyglNqplHoecD7w5FIJuJxYTgs4y0lWw/LHXG/NIUoUzRlKqYf1E6XUbuC8xolkMBgMhsUgShTNoyLyOeCreKGSbwIebahUBoPBYFgwURT87wHvAN7jP/8Z8JmGSWQwGAwLpFF+9PDvLgdffZQwyRkR+Sfgv/As+L1KqXyUHxeRPuBzwA7/u29VShlHnMFgMCwBcyp4EbkC+DLwNF4UzUki8mal1M8i/P4/AP+plLpWRBJA5wJkNSwBy8EqMRgM0Yjiovk74GVKqb0AIvIc4BbgebW+JCK9wAvxs16VUjkgtxBhFwujxNqDVj+PrS5fvbT6/rSifM2WKUoUTVwrdwCl1ONAPML3tgFDwBdF5H4R+ZyIdM1TToPBYDDUSRQFv0tEPi8iV/h/nwPujfC9GHAB8Bml1PlAGvjz8g+JyA0isktEdg0NDdUlvMFgMBiqE0XBvwPYA7wbL5JmD/CHEb53CDiklLrHf34bnsIvQSn1WT+Jaufg4GA0qWvQ6gkVUeRr9X0wlGLOl6FViRJFkwU+AXxCRNYAm/3X5vreURE5KCKn+y6e3wAeWbDETabZPjWDwVDE3I+1iRJFcwdwjf/ZB4AhEfmpUuq9EX7/j4Gv+RE0+/Fi6g0Gg8GwBESJolmllJoQkbcBX1RK3SQiD0X5caXUA3iFyprCQkZ3YxksX5bi3NW7DXM9GZpBFB98TEQ2ANcBtzdYHsMyw/ifDe3Ocr7Goyj4jwDfB55USv1aRLYBTzRWLIPBYDAslCiLrLcCt4ae7wde10ihDAaDwbBwanV0+jOl1P8WkU/jN9wOo5R6d0MlMwDt4btt9X1opnytfmwMy5taFrwuCbxrKQQxGFoBo3AN7URVBa+U+g///5chqC2jlFKTSySbwWBoAtmCiyXNlsKwGESJg98JfBHo8Z7KGF7Z3yjlCgwGQ4No1GzjiWOTpOL2ov6moTlEiYP/AvBOpdTPAUTkcjyFf24jBVtszNTbYIhGtuBi+ya8uW+WN1HCJCe1cgdQSv0CMG4ag6ENcV1Fwf8zLH+iWPC/EpGb8WrAK+B64A4RuQBAKXVfA+UzGAzzYHImT3cyyu1d/r0CAI5R8PNmOucQs1tjESPKFXCe//+mstcvxVP4L1lUidoEM7U1NIs9h8d55Mgkp6/rrvu7IxmvJ49R8PPDcRWPHJlgTVei2aIA0RKdXrwUghgMhsXhmeEMADnHrfu7I+miglfKKPl62XN4nIKryBbqP/aNYE4fvIis8xt+/D//+Vki8vuNF83QSiznehztRJTzcHh8BoCCU7+CHvUVvAJm8q2hpJYTdz45DEB+HoNrI4iyyPolvFo0G/3njwP/rVECGRYPpRRjmZyxxJYhecdlKluY13ePjk8DzGuhVLtowPPj14vjqnl9r124a98JAHLLxYIHBpRS3wBcAKVUAXAaKpVhUfjZEyfYe2wqWDgzLB8OjU7z6JEJZvL132ragp+PFakteICJeSjqocksjxyZDFw9KwlXKX799AjgDa6tYMVHUfBpEenHr0cjIhcD4w2VagXSCBfIQwfHgPlZcobZLKWbajrn4Cq4/5mxur97dAEumrAFPzEPw6Dgekpt39BU3d9tFEt13qZmCszkXVZ1xAE4MTVn47uGEyWK5r3Ad4DtInInMAhc21CpDIvC7sPeOLzcFfxKjEia9i33u/0pfz0cGdMumoVZ8POZ+enom33HW0fBLxUTM3ksgf6uBOPTeYYml4GCV0rdJyIvAk4HBNirlFpRTjbXVYxPL79d3nN4AgBnHjd6LaZzDmp2gdES8o5LJjfbvbDQYzkxnacjMTuNPp0tBNmXzWR8Ok9nBfnqYSyTCwblO/cNE6tjvxxXccxXLPn5WPDpPII3XZ+Yx3nSm9x/Il33d5c7E9MFdmxaFQxyxyear+CjuGhQShWUUnuUUrtXmnIHGE7nePz4FE8to4t2LJPj0KhnyS12TPPTw2meOpGp+ZnDYzPsPTo5a7FpJOMdy6fncSxdV/HY0UkOj83Meu+J41M8M1JbpkaTyRV47Ohk4CKZL/uGvGPTmbB58OBYXedvaDKL4ypE5jdzG8vkSMY9tTAfC95doRa84yqmsgUu2d5P3PaO3/EWsOAjKfiVTrbgWaLHJxZ24y4l2nqHxVfwBVdRmGMBKZ0roIDRTOlim1b4I5n6F+Ems95vTpctPE7nHLIFd15x34uJHnjK5asX7b9e25OkUGdUyhE/gqYzbuPMY6FvJJMjFfNmIPOKovEjtlrJB78UTM7kUcBl2weI+1msxyebry9qKnjxOGmphGlVcgXvoh1eRpEBu5/1/O9WyJJbrMWmglO7VonrKjJZT8mVLzRpt8F8rEPtMiiPLNEzq3yhuWsN2nJfaPz4vqEpBBjoTpKIWYxPRz9WWoYuv0zBWKY+JT2azpGIeWphPlE02oJ/ZiSDu4LCcydmCgiw85TVWCLELGl9C155AdT/vkSyzJtGr5LnHE+hDLfAqnhUdh+eYFNfB8mY3QAL3qVQI9PxmZFMYMkNT5UOitqinJzJ1zxvJ6ays+TWvvtswS1R8tpazDtuU2P+D/vW80zemXOGU4t9x9Ok4ja2JTxvy+q6FK0OkdTrAOUzqFoopRibzhO3BduS+S2y+sffVfUPdEophiazLRFeWC8T017tn86EN7AmYlZNH3wm5yxJvkAUF80vReTChkvSwmgLfmhq+Vjwe54d5+yNvcQsWVQFny144XtAxUVUKHUPlVvwOrJjooZVemA4zb6h9KxY6vCi39PDRR++VvCeS6j0plnK0EZtPSvgoL/+MR/2n5gi5fvBL93eTybnRFZ6R8enScWtoJ77aB2zTm/QhphlEZungnddSPozgHpj+B8/NsX+E2n2HpsM3KLLgbzjks459KSKMStxWxiq4aJ5ZiQdrLU0kigK/sV4Sn6fiDwkIg+LyEONFqyV0BfbcrHgHVfx1HCaHZtWYVuyqGGS4yEFWi0aRodnQiULXrtoqlsv2q1QrtTC29t3PKzgi4+b6ffU/m+Y/yKjqxTPDGfo8BX0pacOANFdWofHZ9iwqiOIvKnHgtdx8zHfgp9fFI0KZg/1rkXoLNB01uGjtz86x6dbhyP+2ksy1CQlbltVwySVUqSzDrlC42ecUeLgX9FQCZrMXDHWkzP5wGItV1atSiZXQCnYsakX2xKc/OJdRGOhm76a62D3s+N0Jmymcw4n0uU+eO2iqa6w9O+WD0zh7YUX8fYPTXn76aqmhqYdGZ8hGbPIFtx5LzJm8577S1vg525ehSXVB9Nyjo7PsL43FVjPI+noSlrPrmKWFbhopM7IU9dVxG2LDatSdVvwdz45TDJmsaYrwVd+eYDtg10MdCfrE6AJHBz1ordSsaK9HLcthqaybF7dgZQdxMPjM8G13eh1vTkteKXUAeAk4CX+40yU77ULR0Ihb8Pp5WHBp/0Fzh0bVwWKb7EIL9qNV1jAU0qx5/AEXQmbuG2VDIpKqcBKrOVXrlaTXCu5mCVFt4xS7B9KB9PjZi5sHRmboSNhE7eF/TWm3xPT+ap1ZrTVq2P947ZFbyoe2Q9/ZGyaDX2peVnwenalffDzWWR1lMK2hG2DXUxXceFVQinFPfuH6e2Ic9LqDp6/dQ1PnUiTybV+mY2DfnhuMqTgE7ZFvkowgg6AABYcUjsXUapJ3gT8D+D9/ktx4KuNFKqV0Ao+bsuyseDTuQID3UnW9qYCH/xiTQXHQgqjklWZcxQj6RydyRgxW0rcWhPThSA9qpYFr9035QuVentdSTtQ8DnHZTrv0Osr+GZmDx4ZnyZpe/7vWhb8U8PpQCmUo63ecE/U7mSMmbw7p0WslJfktGFVCssSLKGumjBaGcVsmZcPvuC4KAW2CNsHu5nJR3dBpHMOk9kCq1IxRIRP/8752Jbw+LGplk8yfGYkg0AQfQQQ9x9XWjvZE1Lwh8fmv1YThSiW+GuAa4A0gFLqMF4D7hWBTvvuSsZaorZEFNJZhx2begGwLakYOz5fSl00sxVAxrdMAws+pGDC7ppaPni9ADvLRTNdIGYJqbjN/qE0Simmc66/vRiWNM8H77iKiZkCiZhFRw0Fn3dcsnm3ar3w6bzDut5kSfaqVhZzTefzjsJxFRtWdXjfs636FlmdUhdNvRZ8xr/GLAu2D3bjKBU5m1b7+3v9Oi5re1KctrabXMHlT299sKUroh4cnSYRs0pcMQk/Fj5XIXR3z+GJ4PwebXBuTRQFn/PDJXWxsa6GStRiaAu+K2EzMVNo+dhe11VM5x12bFwFEKTvL1ZFybkWWdM5z2/bmYjNmvWEH9eKopms4oMfn85jW0JH3CaTc8g5KrBqOxK2F5rWJAteJ3AlYp4FP5rJV7TeDgxnUHgzj0pKaybvsH2wtBOTTpw5Mce+aRk2rEoBniurnoQyz/fvKXfbEqayhboUq8590BY8RDcsxqfznL6uJ8gCBehJxdmyppMfPnKsxFXaahwcyZS4Z4BgPypdA7sPj7OqI45Aw/crioL/ht+TtU9E/gD4L+D/NlSqFuLI+DRxW4ITNp8KfccnsxWn1yPpHFOLXMpXW1Hago8FCr7+ae5IOke6zFc8Nh1W0pUUvKegbMs7ZiemsoGS0DOgZMyqaR3qmUH5sZ6YyQcWPHjKULtnYv72hhZxkdVxFUfGpyOtYegs2oRt0RGvHiZYXDuYnYSklGI677JtsNSGilvagp9DwTtawXsWfKzMgldKcXRihmwVpZt3XNZ0eq3mYpagVLG2TC10KKr2l1uWsH2ttw+VjsFM3uHZselgxuCqYpp/Oet6k1x97gYOjk7PK6pnKTg0mimJoIHqCj5XcDk2kaUrGSMRswIPQaOIssj6ceA24JvAc4AblVKfbqhULcSR8RkSMavmiFyLTK7AUyfSs3xtBcdl/1A6SI5ZLLRCPrvMgp9P6denTqSDxBnNWMZTsrZIRQs+k3XYsdEbXOK2kC24oSgkT0Gl4lbEKJrZPnhtwYNnHc7kHbav7UbEU/CL6aIZzeR4ZmSaBw7OXbK33ILX8pUTdt2Un/uC67lYyi143cD5xBxrQNk5LPijEzMcGM5UjdEvOIrVfi9Rfd3UU6hO50VYIqzvTWFJZQX/s8eHODQ6zZNDaa+Gy0wBV3kx/+WICH/zunNJxiwONVgZzgfHVZyYys2y4G1L6E7GZpXP0INgV8ImYVstYcEDPAz8HPiZ/zgSIvK0Hzf/gIjsmo+AzebI+AwJ2wqmyfUq+IMj3kU5Pl063d19eAJH1U75nw+ZnINtCZtXe1ZcoODrtH4mZ/Je04IyX/HYdN6Lk7Zn+2jzjlcPRg8uMat0UNQKKhmzI0XRuKq0M87EdJ6Yfy56kjFmcp4FrxViwl7c9HC97VoJK+WfTdgWyZj3VymTMxy/Xx5BoaNOZrtofAt+DgWfK7ik4hZ9nXH/e8JYKExSb3skk6s4OBdcN2gWbfvnrp4ILG1c2JYg4s20piscg91+Itz4dJ5/+NETwbXw/G2zFTx461+DPUkmZwotV/BP58ikYrNV6dqe5Kw1iLR/jju1Bd9sBS8ibwN+BbwWrw78L0XkrXVs48VKqfOUUjvnKWNTOerHNhct+PoUso6WyDluSbXDO5/0kjrm4/KpRTpboCthBws++kat1wevB6ZyC2Q8kw8yHcsHjWD2sKlowUNRwQ+ns74rpbZ/N+xOCruExv1FVhFh29puprIF8o4KXBpx2yKTcxYtLFQr7SiDRtZx6e9KYPnybR2oHCa4b6iYpVo+O9LW7va1pQre9iNi5kq0yzkuG1YV465jlsVktrhuFHYPfeeBZ2d9P+8oVodcNFCfgtfuQdvffkfcrjiL2fPsOB1xm4HuBJ/60RMcn8zSlbCDRhmVGPTj4b+x62BkeZYCPWtKxmaXiB7sSc4ykNLZAqf0dxKzhETM4uj4TEMXkKNY8O8DzldKvUUp9WbgeXhhk21PwfX6YiZiFjHtg6+ztrpOggC4a99w8Phu/3E9v5fOFmrGzeYdl0zeCQpNQdgHX13BV0rn14NR+YxlNJPzXDTWbBeNtk60BV8+KA5P5YIsyVr+3fACrF7UVUox4btoALYPdAXb0xav3l6UqpLpbGHOCIas/ztRkqdyBZcNfang+fbB7lnKzYvZn6I35S2wHS1z0UznHSyBDb0pyimPSKoqw6rid7VrRxsR+4amsEXoTNh8vYKiLLgqZMFL8FpU9CKrb1PQEbfJFdxZA93uw14i3Nb+Ls7a0EveUUH0TDUSMW9mctu9hxZNIWYLLs+OTS/o97L+DEWXWA6ztjdVwUXjcPYm7/5I2BY5x21oQ54oCv4QMBl6PglEHUYV8AMRuVdEbqj0ARG5QUR2iciuoaGhiD+7NISn3bZ4F1m9FvwzIxks8axZbbWX926MyoGRDAdGMjx5fLLi+08en0IpShpOFH3w9bloDvkDk6soScoZy+SDOOnySJiZvEPclsASCwZFbcFP5Yjb1pz+3clsPrD+dVimUp7i1gNW2MoNXDQ69jhCw+NnRjIcGM7UTMYpWvDRXDTreztCMnX56w/F81vwQyk74l4I6ZGyuvbTeZdU3Maq0OAjZsucYbq5gsv6EgVfapTo2cNgT5Ldz06ULKAr5fn/tQWvrfC6XDS+f1l/V1+HDx0qrmEMTWaDRUbLEv7Pm55HdzJGvz+w1GKwO8nQZLbuCpnVODYxw6HR6QW5SbIFh86EXbEpi+eiKV6LBccLjz3bX6PS12sjG3RHUfDPAveIyIf8pKdfAk+KyHtF5L1zfPcypdQFeOUO3iUiLyz/gFLqs0qpnUqpnYODg3XvQCMJL5yJCANdiXn54JMxb/p5975hlPIWlbIFl+5kDKWiFWV6+kQ6sMK/setQxc/oDLmuRNGC19ddvVE04USccPLQ+LS3yBqrYMHnHUUiFOZWdNH4UTTprK/ga/t3J2cKwZRX38x6IAwseN8tI8DJ/Z1l26t9jp4ZzgSLzvtPVE9IKvrgI1jwjsvGsAXvD0Dhc6sHk1Tcruh/nck7wQJyOeVZweUopXwXTVGGuFV6/PcPpelI2Ax0JUjESmul6OO7ussbnO35uGiyxSgaIMguDs9c9xzW16i3n1v6Ozl7Y2/JrLMafZ1xBnuSHF+kfBTtYpxPxq4mW3A5aXXnrHIE4Cl4VxWPoZ5x6hDmVlHw+/BKBusz/W3gCF6yU82EJz8pCqXUceBbwEXzlrQJhEPfAPq7Zy+azMWhUS9GtjcVZzidYzrvBL0bV/uLYVEsEu177E7G+Oa9hyrG4+85PIElBD5e8KIQ5pOVGI600I1OdDiblwgzO9Qx77glccyWCD2pGHk3ZMH7gwNUVh5KKV/Be7+jM2eDLMtAwXsKNBkvro8kAhdN7XMU9uNWq+g3k3eCbc7lg3f86Jew9azlCy+0FssQWL6CLx7jmbzXtCRVTcFbVk0LXl+XOkQSQi4aX74j4zOk4jYx2+IVO9ZzIp0L6rfr75f74KvNMCu59cp98DHboithB+5IKFYa7UzW39bQEuF1F2xmLJNfsFIcz+QDhVup5EZUZvIuJ63pqPje2l5v3UDrER1BE1jw/vWardNorIcoYZIfrvVX7Xsi0iUiPfox8DJg9+KJ3nhyBReRYiZhf3eirjrfSikvCSJuBan0E9MFxqcLnLu5L4idDS8kVvud2+49RF9HnE19KYbTuYqDwp7D43QmYrOsifLa3q7ryVWrJOvBkUwwUGgFpxWy7btoMjmnZKDJFdzgWGkG/EHRVV4v1ljIRVNJeehFUq3g9SxBu3O09b/Ft9rDFq/tL1zVsuCVUtx678HgfOyvknGq/e4icyt4rWw2hpTr1gFvhhH2w2sLXUfaHAktsOnokI4KvlzwZicj6VxVf3F5khMUo5gKTrHMgT5e1194Eo6rgjBKfV1rH7xlCQnbKhmExzK5mrMZ7YMPX369HXHuPzga/M7uZ8c5ub8zkK1err/Q6z80l7tqfDpf07X2y6eGSz47H5RSZAsOm1d3Vnx/bY93LrTLcCrrkLAt+v0FYy+/Rppuwc+XdcAvRORBvCic7yql/rOB21t0cgWXwe4kln/F9nfVZ8EXXEU655CM2STjNif3dzKa8ZKHLt3eH1hJc1nwY9N5jk9mGexJsqojzvre1CylExT5qmAZlZd+fWrYi2+vVmlQKcXB0QzdydICXkE5WcsLk4Si0s/7i0U6RVsz4A+KhbJCVuHvaq6/+W7e9Ll7gOL0tdxFo63SZMxmXW8yuFnAm60MdidrKvix6TzHJrKs602RjFlVLXgdo96ViDFcoflIGG2hhS34rmSMjrhdUrtHJzGJeMozWygusOniZNUs+JhtBZZ4JbT/e1soxDK8yKoHGv37F2/tJxnK/A1cNJ1FX3hPKlayvcPjMzxbIxY9nStgi5QYGL2pOHmn2HZw9+HxwEUxH7YOdNGZsOdUykfHZ4JIsErc5a+HwfxyRMA7Zq6Ck9ZUVvCDPd61mXdc7th7nJF0LghhBe96XdebWp4KXim1Xyn1XP/vbKXUxxq1rUaRc1w29BWtsoHuBHk3egGlYgiVbt4wwMSMV3Dr0u0DkRX88cksA91J+jrjiAiv37mZ8el8iQU+k3fJ5JwS/7um3ILXkTjVZiN5RzGTd+lKxBCKi4zFcrIyawqv/cNhFw0UB0WtdOO2VdNFU74NPbtxylw0AKf0d81anBvsSda8YYYmswx0J4PUlQYAACAASURBVOjrjHtFwarUbS+2vrNxVe0QxUrWsydLgqmsw+PHvEXxmbwTKOBy/6sOYazqoilbzyhnYjpPwrY4pb+obMIusum8lx+hZ2WWJUFs+dMn0rMsePCs7/A5ms45NYMCpnMO5YZ5T8orWTExU6DguBwcmQ7CaOdLdzJGOufUvA/14Flt7emufcOBMTRfC17f3yetruKi8RX8VLbAe/71AToTNlvKBoMNq2ZH2iwmK6bs73zIFtySkLWB7mTk9G0gSAkPd+cBgt6N2sIar+GiyRVcxjJ5rn3e5mAmcd1Of5o6Wfye9u+FI2g0sbLCUTqrttrNqgcO7d/W6f9FKzoUCeNoP7Wuulmm4Lu9hemigpfAR1tp+4EbyBJithQteKd0kbUalZJLNPpYvu4C71h2xC32n5iqqCi0BV8+i6n2uwDrysIbB7qTCPD1Xx/EdRXZghssDgcK3ikq+GTMqrp/tTKpXT86p7djtntudWeCgqOYyTlsWdMZXENQGlue94972MLsScWCczSeyQcziGozpHTOCc6txraE87esZmI6PyuMdr50Jrw2lIeqZORq1wlQ0YrPFVyeOD4VlGWYbwkEfX9v6a9swa/qiCMCRyeyuEpx2truWed3w6qO5lrwIvK/RaRXROIi8iMROSEib2qYRIvI0YmZipbXaDrHsQhV3MrDzvq7vQtCWzvj03meOD7FY0cnefMXfsXeY5MlURPlSRC61kZ3KuYtdvnmTnmbuTDa13jdzs3Bayet6aQ3FeP4VDa4kNM5z7/XUUHBV7Pgq92oYbnjManooila8KWx4olYuYJPehmxgYvG8vy7MauiBV+i4C0r5IOPqOB7q7toTkxlUcDr/QEyFbeZybsVb7Cj4zPYlgTnrpY/V4dvllvfcduL3f7W/c8GC5DFrNvZFnw16937Ld+Cr3DMHjs6ScFV9KZmx5Kv7kpQcP0aNwOlNW4SMYu+Di+2PO+4/gJ9UYawi2ZfKNqo2owzky1UDPG8dHs/6ZwTKFK9yDhfdMTNnlDnsDDD6VxQHuOZCmWZtbGzqiNeteRGJfKOy1Mn0kF4adGCr6zgdfkMgE9cd17F86st+EYlO0Xp6PQypdSfichr8GLiXw/8hEbUhN+7F664ovS1666Dd74TMhm46qrZ33nLW4DT6Zkagyuu4EZ/lX7k1nU8e/4fskZ5iqd/5Bjv+uJH4JZeYqdex1hHL1xxBRec/UruO/dy2LuXG//uXd5v3tLLfz+e5bpr/pKNQwd5lFWcfPBx+r93M5z5et701b+F2/6SHw+cxcipr6QzYTN2fISJdJ6z7v0B3PaX3Hh4gk+ffw3pM56PbQnnPPprBq5+P69dcwlnDx+AO/4nHzo8wTWv/QhjmTwXPPQLrv7hLXBLb7AP6pZehi/9U3qScbb9+Lvc+Hd/U5TPWctNl72ZD996HwCJ40c5Iz/Nh279QPAZgMTrbvJ88ONpbvzCnwLw1cuuhXXPZfOBx/HWvuHqH/wL3PJ+bjw8wb+e/iL2nX0lH/i/7+cd136Q45Mz/P5PvsaT+Tj7n3sV7//MnzGa6uEdL313oACO//MtwGm87/MfhH+DGw9PMLJ6kKc+eTMA2x++h/0bdvCB///76CpkefPVf85zDh1g5pLLPXlvuIEb79jFHZvP4W8vuo4//ueP8U8XvpbRgbPoSsTY8fBdjG48mw994o+C/XtDagu3vOYdALz35r+ge2qcH134cgonXc77P/FuOH4NrL3S+/wrXsHg+pdjp3o49bqrufHwBN+86Gq+sPn5TOddPvapd5cc+7+/4nfp795AwvaO4/EPfpQbH7wz2PaNhyf46SVXwdsvwclmOXn42ZLrj1t6ueS032DsrBey99gUZ+69DzbvYPv738ONT+7DQXjt6z7sGRFHDvCDdJKXPbWLG772/4Lf/9ZVbwEu4eSDj/Nb3/48/79X/g9e+oNbuHr/r+CWXp5z0e/w+PZzuOtHXhWQm77yIbjd2/yNhyf48nXvYWj7mRxKT5FRFtt/fDuv+uV/BPJtePEfMtKzgSeOT7F6ZpJBp1CyD//4vk/huIpLdv0X+/7rq7Ddu/9Gr/sdbtz3FH//ds/r+qK7vsuL7v4eH/2Nt7E1r7jxXz8Pt/SSeN1N5BIpLn3o53yS9UwdH2YDDgNXv5wbD0/wkf/+T8G1d8HDd5Ycf/59Hbz6Ru/xX/0VN37lW4Hcf3U0w7XXfJDdz3qffcO3PhNcuwDP/MfpsMOzQQ+NZnjzNz7JyQefCH7/Hy54FUNbzqUzYXslN37yc/jEuwKVcuPhCQ6cdBr4Xd7+6AsfYs3oEI995zSOn/O7cPQoo3/+A7I9LyJmCV2/cz03PvxUybWx+4yd8PZLWNuT5GW/+Hde+s6/ZWPo2ri6/xxuf9nvsGFVCqXgv/3jn8G/htRxFL33lrfAiROz3wsRxUWjzYKrgFuUUiMRvtN0vjVwFgU7xrDdUWLROQj7+9YzmajsN9Oc6PCmkeuTRYukP+9ZA2Mpzwq7s3cLCeWyY2Mv335hL6eOPsuDg9uCzx/rXM1JXaWj9u/v/j4XH3kM8Fw1cVRVF809PZvJuBIs1oTZeexJXrf35/zLg8e95JF4N2enj1f8nZglTCnBxa9BnfDkn0hUtjyOdfUxkEuTdAt+AS9vkJxMdiJK0ZXP0p0vdfMcx5vd9M2ULlr2d3myH+7oI+EW6Cx4v9XjZJm2Zye3ZOLejKkrP0NnoZjUko4l6S3MPetam/MszdFUabq/Avat2sBpo8UU/cGcJ2ulPISjiR56nWwQFXQ80T3rM5qsAwPTExXfW9URZ2N3nDs370CUy9aZUQBsFOsSfsMSJUzbCTZPVr9ZV+UyiFKMJ2dX677rRIFNkycqyrC6K8GoJMjbcbZPz751+zrjDMQVo6keenOl564nYRWTpFLFOjEjscr3TsZO0FGYfS2fl8ySLOSYSHZVvUbrIekW2JweLun9G+ZgsugCqtRY5cHBbVxSOBGEEE9IFDsX9qXWADDUuYr3cDozeWdWkbFyNvV18FtP/brq++v9yKuhzoW5raohc00NROR/Aa8GpvHi2PuA25VSz19sYXbu3Kl27aq/Jlm4r+r1N9+NUorx6QKPH5tEAd979wv48H/sAeBjrzmHKz/xUwCe+NgrgqgN/V39+OV//1P2Hpvitj+8hL/9/l4A/uG3z+fi//kjTunv5Cd/egWX/81PyOQKPGddD19/+yVc8j9/xNHxGXZ/+OW89Uu/5oGDY1x51rqgjnf5Nq6/+W4eOjTOFacPBp13wp/Z1NfBtx84zPlb+rjtHZdW3E/bsvjl/mEU8NFX7+A/HjwcfEYfmyPj0zwzMs3zTu4L3B6PHZ0kbgtPfOyqWb/76JEJtq/tJmFbPDs6zaGxaS48ZTUHhjPkHJfT1/XguopfHxhl8+oONvV1cOrabr7+64M87+TVJfvwJy99Dr/92V/64WGJYJFpJu/w9HCGM9b3lMh6eGyag6PT7Dx5NYdGM6SzDmdt7OXxY5Os600FWbKVjiXA21+0jbd+aRdnb+jlu+95QfC6Pncn93fy0/e9ODh+jx2dJBW32TrQVfI7Tx73+rxuHejisaOT/NZzN/DEsamK297+F99joCvBPR+4sqJMz9/Wz6d+9ATJmMXej74ieN1xFY8fm2RjXwePHZ3kzPU99HbEq+7bE8eniPky6dddpXjk8ARdyVjwevieOGfTKj73C8+6DF/L4d89b0sfN/90P6s64jx408uC13dsWsUX73yKC09ZQ29HnP965BgK+MwbL+BLdz0963eG0zlGprKc5t8P4X147od/wPh0nvf8xmn8yUufU3H/6nm8b2jKq0nku53Cn3nhcwb52+/vJRmzuOzUgcCl8vW3X8Kr/vEXPHhonA9fczbfe/gIjxyZ4Iz1Pdz6h5eiKe/THJzHrWv41I+f5OQ1nRzwB441XQnu++BL570/H3jlmVzzj3fynLXd/OC9L2I+iMi91Wp9RYmD/3PgEmCnUiqP19npVfOSZIlIZx32+goBKBnpw367ct/bxHSeR45M8Pr/c1dwAsM+eB1hUHAUz4xkeHZsuqRA0qqOOAr49dMjXmahn+VWi5glFX2aBcfluw8fob87UdXvLCJ86g3nB4u1OzZVtgLKwxKPBFE0lVv5ZQtuoIjjsWJ2aDiVvdyPfnwyO2uBFbzII/AsVb2GAV4zh2o++JhfXMsOFcuq5mMuR8cel0cmFLN8izMqEWH72u5ZFrzrKobTucBPvrYnWVKPJp0tsPfYJI8cnuDaz9yF46pZaw9hXv88b/2k3Ae7fpUXIlcewliN/gqZ1OlsgXTOqVrLZXUoKqa8SqVGL9qXp9v3pGK4qlhDRycnVVszquaDB4K8g2rXaL10JWIMTWYrrp8cHMkQs7yaO+UWvA6J1AEPlUpuVGPfUJpkzGJdb5LXXeCd07ks+LnQ+qVRyU5RFllTwO8Bt4rIN4G3A3MXyG4ix6eydMRtNvV5NanDPRB1Jh3MXiwazXgNOOK2RcK2GOhOlCSvJPwoh7zrcueTXqJEWOl0J72wwrv2DXsLJ1A1y01j21LSBk8znM6RLbhBqFU1BnuSPGddD4PdCc7aUHnxKlwawHFVUG5AAZNlDT28CITiwBREbxQUBdedNaCFsz0Tsdk3t3bRlD8uj7HWFFxFj9+XMxaKtXfcuQtSQTE0rVwR6kG+syyMdNvA7KJgQQazf/Ou7U0yFFqs33VglLFMHhdPsa/qiJdEn5Rz0ppONq/umHUuN/Z1kHW8Ylw9yViwkFoNL9Gu9JhphaUVaDnaKIlZUqLsw2wf7GZTX0cwGGv0tV1wFQeGM/QkvefVGnln/FDMSgx0JxnoTlRs6jEf9GCTrtCU++Col1yYjNkcHM2UGDFeJrZwql9KolLJjWrsG5qiI+5Vav3Ya3Yw2JOMVEOnFgNdXqRVoyJpogw//wycDXwa+EfgTOArDZFmEXBcxfBUlleeu8FPlY6VKPVwR/Ny33fBt8T+5Q8u5swNvWwf7J5lkcRtIe8o7tp3gnW9yZKyALYldKdi3LXvRFBlrjzutZyYJYxXuGGOT2Y5a0NvxbDHcrqTMbYNdle1IsNx5/pC0r9b3rMziAzwB6Zi+r+XrBRWZL2pGI6vDIcmZipa8Lo1GXg3ucYLwZt9UTuuosdXLEGkjqMoOKpmOVmNVmizFPyzE0E7ujDb13aRd0rr8odrEIE3Kwhb8PoaOmN9D//yBxdzxvqeWQNHOZv6OkpizAHW93oLbFPZAtv8piW16O9OBmUfNBPTec7a0Fvx2EMxcWmu2cHm1R30dZbKp2vJpLMFCq6iM2HXbOSdyTolYZhhEjGL7YPdQdjpQtH5Hjp7NszBkWlSMZtk3KvJHw6bzWQLdJaV0x6fzlcsvRBGKcX+E+kgSi0Vt9k20BWphk4t9Ey4mQr+dKXU7yulfuL/3YDX2aklGfFDpHRKc2fS5pEjEyjluSN2PzseKLdyCz7vqIpV4cLELS8V/u59w1y2fWDWTbkqFWfP4YnAspjbRWPNsuDT2QKZnMP1F540500fhXBpAG2d6guz/GYtD/0Kx18XXEVfBQteKcXQVGUXjWUVLfGwhdhbw0XT2+HJFg7FLLhu8HotdEOQ8no0jxwer5gEVqwZU1QU5TWI1vZ4VQy1Jbjn8DjJmDXvdHuNTozK5JwgPr4WXrG74n65rmLSz4quhh5UqhUxq4UeaKeyxb631Rp5u36xs/I4+EZhW57/vdyCV0rx7Nh00HQFinkdecdPBiwrpz2dd+bstZwtuH5DlfqP41zossGNIMoVer+IXKyfiMjzgTsbIs0iMDSZJRW32HnyasAb6TM5J4h3npgpBJZguYIvOG5QYrUaMVtIZwsMp3MVp5u9HV6FyCE/CWljX20XTcyeXdNlaCqLCLz6vE1z73AE7FoWfKaaBV+s0CgSUvAhK2+Vn+mo49wTVY6dVvzlPnjt3w1TcFXgCogFg4uXEh7FgtfbC1tEecfl8PhMxTIOQXPoUNngcgt+sCdJzl+DAPySEAu3RMNZ0tX842H6u5M4rgqulclsAaXg0lNrKXjvmHUk6h+M9ICqy0Wn4lbVRt66aFk1H3wjOGtjb5A8pckV3KCWkc5h0Nf0E8emUJSuw5SX3ADvetl7bLKk7k6xls/i54Y224J/PnCX337vaeBu4EV+K76HGiLVPFFKkc4V6OuIB5ZvV8hXpy8G7VssV24FN4IFb1tBEsWlpw7Mer8rGaMz4XWySdhWzcU3KHVDaNLZAt3JGKtq+HXroZaLprwejc7O09aliNDflQxunLCS7fUt+HCWaq3tl/vgYXY2q+P74MPf0zdolEVW8FxWkzP5IE097VugldwoJ/d3IpRZ8AWX3lQsGBjX9hYXbguOy4HhTCTX2VxsqFB9shbFRDvvmOnEoYu2VlfwWwe6Wd+bmuUeioI+3ulsgbU9Sa+TVxULXl++cywjLCo7Nq0iV3BL3HEzOkkvbocseO+1YB2mQkOc8HU4OVNgLJPn+3uOBq/p1oONsOB7UrHgml9sovzqbzZkyw1gaCqLqyjpcK5rb2dyDoJnzeqDWb64UnDUnAtd+v1T+jvZVME6t0S4aOsa7tg7VLHLSzlhN0SC4gXZ17GwxZsw4dIAOb+1nB54Kvngk6EOVuC5KJ4Z8WKkwz54bcEHWapVBrNKFrxeMC130+jFVL0Q7MnkBNuLwmBPkuOTWf7jwSNAaaPjSrIl41bJQmvWcUtmXsWFW0XBcar+Vr3oUgYKIrlo9ACpFdrEjGcI1PJr25YE9fLrJTwIbxvsQqliI+9w8AE0x4LXRcsyuUrZ4956y0B3MjBa9jw77mXrxkrXzcAvueFfXtpdcve+Yd508cmAZwCs7oxXXetYCOt6U6xb9F/1iBImeQA4CXiJ/zgNWEqpA/7zlkHXnQiHLlkinLm+Jwgn0/UgysMTcwUXR6k5/ar6BF+yfbb1rtE+0SghVMWuO94NMp1zyDsq0uAQlXBIoy6/YIsgMGu6nS3MTt5Y25tkOudd9KWLrJ4FH+58VYl44IOfbcGXK/iC6wbvec2bi23Roir4roRNR9zm679+BvDKOJy0pqOq+02XLNCUl6jQCj5XcAOf72K4aGyrmMperZ5JGL2GkXdcJmfyTGULkdYl5kt4xlRsiyiMVqhC6qhopSQWE13yINyZKlvw2h5qA+akNR2B0t9zeMIroBdaJygvuQHF8r537TsRDFzh5u7LiShhkjfh9WB9v/9SgkaUKVgEdJu58ga4Z29aRTrnkMkVgkJHtlUanqjLusbmsOC1kqi1sHWpr/wrNeKd9XtlLpriPiyupdDrhyVmC8XGzF4xr0oWfKnca3uSwQ0cnllohavdG9Wsm8o++NkuGqU8X7te3BPx2v/N+BZ8lDBJ/b21PUkePDROJlcgnS3ULFHbEbeZyTtB44fy3qbhsq/prMP63tSiWXKJmEUq5C+uhS6NnHeKLR+juq3mQ3fIbaCVW8y2mAo18tYEFvwSLbKCF+OfiFklfvhs3ru+tRxb1nSSLXi1Xh45MjGr0UisQncxPUMazXgJgaArgc49y2o1olylrwGuwbPcdZemmp2cmoVOaihXjmdv7A1cCTv8UqXlym0kUPC1D8mqjhgbVqW48szqk6qzNvSyuUJccSXKS+cebJCC14lF4bZuMcsqiaKpNnvQyUNAybqAVvDTeYfuZKyq9dbflWDz6g7Wh6otasUUvrH043BMd19HvG4LHrzBJGF7TTWyBbdmgo2OZf6TbzyA6y8ahzsjdfu13XUUxo4FlrsNs6kvxeY5Qmk1+noquC53PTmMCPQsUthhJeK2FbR83B6KGwdmxeM3w4IH75oYy+SC9ZZsobTD0kmrPQU/nXcqltOu1HwmF3LV3rXvBAXHC7VsSwseyCkv1EFB0J2pJTk4Ml3SUEITtt60BR+uVAjFcMH4HBdozLLYsqazYtVGjWUJm1Z3RFqQscsaIxfdTIu7mNOTigULpdr9ECubblebPYRr4fSVLLJ6N8t03qmZkJWM22zq65jVCAKqKfjQINKZCHpF1uOOiNsWLzt7HSf8OvVn1ahg2JWMsaW/kx8/dpynh721hrCLRkRY25tkxlcUCy13G6avM1ExWebrb78kSJXXdCdjfkST4s59w/T4jasbib6XdEmA8kbeGm3Bf/L682bJ3UjWdnt9T29/yFtvyRacktwTrex19m15JFWlRda84/VC2DbQxV37hoMF1nZV8N8QkZuBPhH5A+C/gM81Vqz58cxIpqJiPH19ccKhb/RyH7xWdHO5aBYbW7ybSFtEB0cyWFI9ImW+9KbiwUKibg5dHvJWbfYQVt7hjj+9gYvGrVgQrRaVfPD6Jusps+A19VjwAL994Zbg8VxdhNb1JHn1eRsZ8geE8kXEtT3JIGplsdLt60VEiFsW0zmHR49MRHZZLQS9DqIDCsobeWv0084Gzigq0ZX01lv+1a+5n3dUSe6JfjySznnusDKjq1Lp6rzjtZ68ZHs/9+wfZtpfd9GzmOVElEXWjwO3Ad8ETgduVEp9qtGCzYeDfoPrclJxm86ETSpmBREHVV00C0xeqRcRoS+U8q8HqcVIcAoTbt6wvte/WctC3p4ZrryGoZsHQ6kfPGxpry1rdhFFHii1nAILPrQNvagr1J+sc+l2ry1d3K5ckTOMiPDXrz0n2Mb6su5Ma3tSQXjsQuuZL4S4XUytb6T/XROzLDridjBTCDfyDqMt+M4GhBHWQsQ7tw8eHGPUPy4nlVjw3uNMzuHM9T0V1wjCJTe0OyZhC5duHyCdcziRziFU79zUysw53IrI3yil/gfwwwqvtQxKeV3j11W5kb2RvHhRel2OCiilEJFA0S21BQ+eX1vPJg6OTi+6/x1KreKwBX9iKhscg4Oj0xVnD9oHb1ul7q+wRT3YneR4hCYqmpjv363koqlkwXuWZH3nxvIrL9bqpxqmMxHj9HXdnEjnZjXH0ANEzJJZrfmWkphtoXD88MjGK9PNqzsIr6eGG3mH0T748kXMpWCgO8GR8enAxRj2wYfP1dmbVlVs0dibigX3/7B21dpWkMg4OVPwkrwaECLZaKJI/NIKr71isQVZKFmdwVbFgujrjJdkYQYXqn/zj6Rz2CJLGgUQyNYRp+B3dTk0klnUEElN2NrTVTZjftJWsMBbZfYQVm5hwguuYSs/KrYlOCFfbtFFU+qD92StfV4q+azBG4TqSfLR6wXlvm29f13JxZ9d1YMefJ+/dc2SyFFeSC3cyDuM6ycJVguVbSTeesv6INQ17KKJ2cWSBdXcdCVF8/yaQ3HbYk1XgjP9An7zKfXQClQ9GyLyDhF5GDhdRB4K/T0FtFQGK8xucD0XsbIU5dFMrinWO3gLbbrf5WS20CALvljAS/shy9vAVZs9pOJ2kDsQpjsUkTBX1ctK2JaUumhU5Sga/dlmomcxcxUVazTFPIzFqcpYiWqDJZQ28g7jKEVHonmD3/V+yWMRZrnj9DVdzbWmE/ag2JoxURYO3XYKHvgX4LeA7/j/9d/zlFIt15M1aBQdVcGXhXuNpJuo4H0LoryH62Ki3R7h0gnFYzD37CFhz56iWiGXTTiUMioxq3RxSzfwDlvwq7viwWebyTptwS9CButC0IPypTUS7RrNmq7ZZYtdV1Us5rZUXH7qAAm/wFj5IKPvp3CwRZjekAWv68/oPgiX+XV+GlGiYCmoekaUUuPAuIj8JXBUKZUVkSuAc0Xkn5VSLVUTPpt3/SbJ0RSBbZWGe41mcsSrKJFGh331dSYoOG6xjkYDLHi9cFmi4ENZtAV/9qAVajkn93dWrBQYswTHVfN20YR9uY5SiJTKqBOrmm3BX7ytny1rOqvWVNc0+loZ6E5ii3DmhualoqzuTMxKkHMU9DTB/66xLGG7X06hnA2rUqzqjFdV0qUWfNFFA/DC0wbZsqZzXrV8WoEomuSbgCMipwKfB7biWfctRbbgsrEvFXmKGC+LBhhN55voovEqK+qM0Eb44AMLPmSFx0OzmLlmD6s64iWZjZqiBT9fH3xpmGQ1P/9cReAaTdy22LAq1ZQ1mnI51vZGv84bwZqyssXQfAsevHWmSqG0HQm7ZmMOXXJDKcXxyRm/o5iOGvLOe7MNjPkS5Yy4SqmCiLwW+KRS6tMicn+jBauXbMHh9PU9kSMm7DIXzWgmtyRhZ5XQi1jpbIG+znhD3BEVXTShWu9WwTse9c4eYn6cdL0x6vq75WGS5TeS9sE3W8EbiqzunO2i0T745Yi+dh2lOD5Rua9BPSxlotdcRNmTvIi8Afhd4Hb/teZowhqE28xFIVxkyHUVmZzTNAteX2DpnFPXPtSDdnWUFmLzFH7BVUHFvWpRSNWI+e0N52NRllvwjuvOUvCrI0bRGJaONV3xipmszV6fmC+BgneU31u4fa61KAr+9/Cabn9MKfWUiGylxYqN6Tozc/U/DSMi9PrJPzqKZK4yBY1Ch2/myupoLCZnbuhh60AXa0KhoiLCGt//ny24/uyhvmOwqS81K2Y8KrbfF1a7piq5aFZ3JTh1sKukEqWhuazuSpSE14Jn/S51FutioUtgFFzF0GR2zh4Oy4k5z4hS6hHg3aHnTwH/q5FC1YuOoDkpYtEmTV9nwus36S/0NSuRIZyOf9LqToanKve8XAi6wmI5q7u8JBGl1Jz9YyuxkLBBrcy9RBLb78Qz2wrsN8q9bhrpJtBGQtiKd92lz2JdLHQAglbw/RGKBC4X2mKoKm8zF5W+zngQQQLN8/OGE0miVhZcLNZ0xb1F1rzTsNlDNexAwXtZvI6rlqynp2H+6EiisB/eUWpRauQ3A732li245By3IU09msXyPCNlZCtksEXBy2BzW8CCL1oM87GiF7TtzgR5x+tX26jZQ5iwZakH1L1HJ9k22I3jKuNrXwasDiz40gbg1Y3V4QAAEMxJREFUi9HGsBkEZa/9uvKJJbgGl2ohtlYm61f8/+9ZEkkWgO7iEqX+epi+zgSOE/LBN0m5hGuvLHVBozWdCa8hAvXPgBZKTypOR9zmA/++mwPDaVyFseCXAbqRtzaMXKVQsHwVfGexrwFUb1yzHKllwT9PRE4G3ioi/4xX0C9AKTUSZQMiYgO7gGeVUlfPW9Ia6C5E9UZy6AzSwIJvkovG8ssAFFzFpkVW8HNZCuHEnaVW8LYlnLaum/1Dad725V3Ba4bWRlvw2jAKKkk2OQ5+vuiSG1rBr5RF1v8D/CewDbiXUgWv/Nej8B7gUaBhNVZ1o+h6We374POOYlVHvKnJI7rudiPKFNRiTWd4gTfa4LKY08uOuM3HX38uf/jV+wCj4JcD2qWhDSNdQ6i8mcZyQZfc0L2F28mCr7onSqlPKaXOBL6glNqmlNoa+ouk3EVkM/BKGtggRCnFTN6ZV/anrlSYLbhNT0VO2LObESwFYQt+sWcPUfnNHRt4+wu9S6qdYpDblZhteTNOR1vw3usdy9SCh+LsvTNht5WRESVM8h0i8lzgBf5LP1NKRa0m+Ungz6jRw1VEbgBuANiyZUu1j1XFVXByfxcd81DwOjxxJu+wurO5uVtbB7rKnGBLgx7Y4rYs+ewhzPtefjo/fuz4knQpMiycmC2Biyaw4JepDx4WVnKjlZlTK4rIu4GvAWv9v6+JyB9H+N7VwHGl1L21PqeU+qxSaqdSaufg4GBEsYvYlhff3TOPMgM6PLEVLPiOhN2UkqTan5pqonIHzypc05UIaoDUKllraD4xyyq6aJa5Dx6KFvx8qqK2MlHOyNuA5yul0uB1cwLuBj49x/cuA64RkauAFNArIl9tpVLD4fjz1Z0JJmcKTZSmOeiBrREFzgztS8wu+qzdZe6Dh6IFP9ib5IRfUbIdiKLgBXBCzx0iOBOUUu8H3g/glxn+01ZS7gCrQvHnq7sSPDOSaaI0zWFNVwJh+dS7NlZ9a5CwLSam8xybmAlF0SyPa6gSusDf2p6Vp+C/CNwjIt/yn78ar2zwsqfcgm8US6mU6t1WKm5z1sbeZduxxlAfi3UtrutNcmIqy7u+dl+Q8FSPi6aaHM0awHWC3dqeFI8w0RQZGkGURdZPiMgdwOV4lvvvKaXqKheslLoDuGMe8jWUcInbNVUaXawEupdpivlypR1mIZ2JGFsHuth1YDSIfGp2PfiF0K6LrJHOiFLqPuC+Bsuy5MRtC0u8SJxGWvArkXZQYobaDHQnecFpg3zprqcBlm09eCguspb3c13uLN8hd5GI21ZLRNFUwyhKQyvzF1edyW33HiKTKyzrDFCd3LTU2dyNZsUreD01m6vXpmHlYQbXuUnELE5f30Ou4Mz94RZmdWecHRt7vXyUNqKmgvfryHxfKXXlEsmz5Oip2Rrjoll2tLoCbnX56qXa/sQsIbaM/e/g9UtYruWOa1FzTqWUcoCMiKxaInmWHF0i2GRQGgyGdiPKkDUDPCwiPwTS+kWl1Lurf6U1iGJBJWMWiZjVlPoT7WbhGQyG1iKKgv+u/9eWbOrrYF1ve62cGwwrHWM8eUSJg/+yiHQAW5RSe5dApiXFtgTbWr7hXa3EcrqpWlHWVpSpVTDHZn7MqeBF5LeAjwMJYKuInAd8RCl1TaOFa0fMhWowGJaKKC6aDwEX4WeiKqUeEJGtDZTJsASYgcbQ6phrdOFEUfAFpdR4WbcjVe3DhsZhLnjDcsJcr80nioLfLSK/A9gichrwbuCuxorVWMyFZzAYloJm65ooCv6PgQ8AWeAW4PvAXzVSKIPBUCSKkmi2IjG0JlGiaDLAB/xGH0opNdl4sZqPuWEMtaj3+jDXk6EZRImiuRD4An5fVREZB946Vys+g6EVaGfF2s771qost2MexUXzeeCdSqmfA4jI5XhNQM5tpGAGg6GxLDdl1cq06rGMouAntXIHUEr9QkRWhJvGYIhCq97cBkNVBS8iF/gPfyUiN+MtsCrgelqwO1Mr0G43ervtj8Gw0qhlwf9d2fObQo9NHPwcGOVoMETD3CuNo6qCV0q9eCkFMRgMBsPiEiWKpg/4XeCU8OeXQ7lggyGMsRQNzaQZ11+URdbvAb8EHgbcxopjMCwco8gNBo8oCj6llHpvwyUxGAyGZcRyMCSiKPiviMgfALfjlSsAQCk10jCpDAaDocEsBwW9UKIo+Bzwt3j1aHT0jAK2NUqohdAOJ60d9sFgMDSfKAr+vcCpSqkTjRbG0J60+oDVTPla/dgsFou5nyvlmC0GVoTP7AEyjRbEYDAYDItLFAveAR4QkZ9Q6oNvuzBJYxkYDIZ2IoqC/3f/b1nTKOVtBgVDO2Ou7+VNlHrwX14KQQyGVlcmrS6foTIr+bxFyWR9igq1Z5RSNaNoRCQF/AxI+tu5TSl1U63vGAwGQyNZaco+iotmZ+hxCng9sCbC97LAS5RSUyISB34hIv9PKfXLecjZMqy0C2SlYc7v8sKcr9pEcdEMl730SRH5BXDjHN9TwJT/NO7/mSqUTWQhN4O5kVoDcx4M9RDFRXNB6KmFZ9H3RPlxEbGBe4FTgX9SSt1T4TM3ADcAbNmyJcrPGgwGgyECUVw04brwBeBp4LooP66UcoDz/IqU3xKRHUqp3WWf+SzwWYCdO3caC99gMDSVdpolRXHRLLguvFJqTETuAH4T2D3Hxw110k4XZCXaff8MhkYRxUWTBF7H7HrwH5nje4NA3lfuHcCVwN8sSFpDwzHK1LASadfrPoqL5tvAOJ4vPTvHZ8NsAL7s++Et4BtKqdvrF9FQiXa9IA0Gw+IRRcFvVkr9Zr0/rJR6CDi/fpEMKwUzSBkMjSWKgr9LRM5RSj3ccGkMBkNTqHewNYPz8iCKgr8ceIuf0ZoFBC/M/dyGSmYwGAyGBRFFwb+i4VIYDAaDYdGJEiZ5YCkEMRgMBsPiEsWCN7Q5xp/a3pjzuzCW8/GL0tHJYDAYDMsQo+ANBoOhTTEK3mAwGNqUtvDBL2cfmcFgWNk0Un8ZC95gMBjaFKPgDQaDoU1pCxeNwWAwGFftbIyCNywIc1MZDK2LUfAtglGUBoNhsTE+eIPBYGhTjAVvMBgWhJl9ti7GgjcYDIY2xSh4g8FgaFOMgjcYDIY2xSh4g8FgaFOMgjcYDIY2xSh4g8FgaFOMgjcYDIY2xSh4g8FgaFOMgjcYDIY2xSh4g8FgaFOMgjcYDIY2xSh4g8FgaFNWfLExUyjJYDC0K8aCNxgMhjbFKHiDwWBoUxqm4EXkJBH5iYg8KiJ7ROQ9jdqWwWAwGGbTSB98AfjvSqn7RKQHuFdEfqiUeqSB2zQYDAaDT8MseKXUEaXUff7jSeBRYFOjtmcwGAyGUpbEBy8ipwDnA/dUeO8GEdklIruGhoaWQhyDwWBYETRcwYtIN/BN4L8ppSbK31dKfVYptVMptXNwcLDR4hgMBsOKoaEKXkTieMr9a0qpf2vktgwGg8FQSiOjaAT4PPCoUuoTjdqOwWAwGCojSqnG/LDI5cDPgYcB13/5L5RS36vxnSHgwDw3OQCcmOd3lysrcZ9hZe73StxnWJn7Xe8+n6yUqujfbpiCX2pEZJdSamez5VhKVuI+w8rc75W4z7Ay93sx99lkshoMBkObYhS8wWAwtCntpOA/22wBmsBK3GdYmfu9EvcZVuZ+L9o+t40P3mAwGAyltJMFbzAYDIYQRsEbDAZDm7LsFbyI/KaI7BWRJ0Xkz5stT6OoVn5ZRNaIyA9F5An//+pmy7rYiIgtIveLyO3+860ico+/z18XkUSzZVxsRKRPRG4Tkcf8c35Ju59rEfkT/9reLSK3iEiqHc+1iHxBRI6LyO7QaxXPrXh8ytdvD4nIBfVsa1kreBGxgX8CXgGcBbxBRM5qrlQNQ5dfPhO4GHiXv69/DvxIKXUa8CP/ebvxHrxqpJq/Af7e3+dR4PebIlVj+QfgP5VSZwDPxdv/tj3XIrIJeDewUym1A7CB36Y9z/WXgN8se63auX0FcJr/dwPwmXo2tKwVPHAR8KRSar9SKgf8K/CqJsvUEGqUX34V8GX/Y18GXt0cCRuDiGwGXgl8zn8uwEuA2/yPtOM+9wIvxCv1gVIqp5Qao83PNV5/ig4RiQGdwBHa8FwrpX4GjJS9XO3cvgr4Z+XxS6BPRDZE3dZyV/CbgIOh54dYATXny8ovr1NKHQFvEADWNk+yhvBJ4M8olrvoB8aUUgX/eTue823AEPBF3zX1ORHpoo3PtVLqWeDjwDN4in0cuJf2P9eaaud2QTpuuSt4qfBaW8d9zlV+uZ0QkauB40qpe8MvV/hou53zGHAB8Bml1PlAmjZyx1TC9zm/CtgKbAS68NwT5bTbuZ6LBV3vy13BHwJOCj3fDBxukiwNp0r55WN6yub/P94s+RrAZcA1IvI0nvvtJXgWfZ8/jYf2POeHgENKKd0g5zY8hd/O5/pK4Cml1JBSKg/8G3Ap7X+uNdXO7YJ03HJX8L8GTvNX2hN4izLfabJMDaFG+eXvAG/2H78Z+PZSy9YolFLvV0ptVkqdgnduf6yUeiPwE+Ba/2Nttc8ASqmjwEEROd1/6TeAR2jjc43nmrlYRDr9a13vc1uf6xDVzu13gN/1o2kuBsa1KycSSqll/QdcBTwO7AM+0Gx5Grifl+NNzR4CHvD/rsLzSf8IeML/v6bZsjZo/68AbvcfbwN+BTwJ3Aokmy1fA/b3PGCXf77/HVjd7uca+DDwGLAb+AqQbMdzDdyCt86Qx7PQf7/aucVz0fyTr98exosyirwtU6rAYDAY2pTl7qIxGAwGQxWMgjcYDIY2xSh4g8FgaFOMgjcYDIY2xSh4g8FgaFOMgjesaETkIyJy5SL8ztRiyGMwLCYmTNJgWAREZEop1d1sOQyGMMaCN7QdIvImEfmViDwgIjf79eSnROTvROQ+EfmRiAz6n/2SiFzrP/5fIvKIX3f74/5rJ/uff8j/v8V/fauI3C0ivxaRvyrb/vv81x8SkQ/7r3WJyHdF5EG/3vn1S3tUDCsRo+ANbYWInAlcD1ymlDoPcIA34hWvuk8pdQHwU+Cmsu+tAV4DnK2UOhf4qP/WP+KVaz0X+BrwKf/1f8ArBnYh/1979++SVRTHcfz9iYaWaHOoIRBBoaFAECJ0MZxDmsSlNgn6MbRFtEn4L6ggidDQ0BIpuPljEE0QGvoDorUIaerr8D1Xrk+JPOnS8fOCC/c5nHvPc+Dh+xzuj+8XvrXOM0bm7h4i30YdlDRC5v/+GhE3I/OdfzzzyZt1cIC32owCg8CWpN3yuZdMN/y29FkkUz+0/QB+AbOSxoH90n4bWCr7b1rH3SFfOW/aG2Nl+wTsAANkwN8D7kp6LWk4Ir6fcp5mJ3KAt9oIWIiIW2Xrj4hXf+l35OZTZM7xITJb5z2OX2HHMfvt8adb4/dFxFxEfCH/ePaAaUkvu5uWWfcc4K02q8B9ST1wWOvyOvlbb7ISTgBr7YNKnv0rEfEBeEpeXgHYIDNZQl7qaY5b72hvLAMPy/mQdE1Sj6SrwH5ELJKFLbqqrWn2Ly6e3MXs/xERnyW9AFYkXSAz9j0ii2bckLRNVgvqvMl5GXgv6RK5Cn9W2h8D85Kek1WWHpT2J8CSsvj5u9b4K+U+wGZmveUnMAn0ATOSfpfvNHW2Mzf7kx+TtHPBjzHaeeRLNGZmlfIK3sysUl7Bm5lVygHezKxSDvBmZpVygDczq5QDvJlZpQ4AVvMQeaLUmh4AAAAASUVORK5CYII=\n", 347 | "text/plain": [ 348 | "
" 349 | ] 350 | }, 351 | "metadata": { 352 | "needs_background": "light" 353 | }, 354 | "output_type": "display_data" 355 | } 356 | ], 357 | "source": [ 358 | "plt.errorbar(range(n_episodes_test),np.mean(test_data[:,0,:],axis=0),yerr=np.std(test_data[:,0,:],axis=0))\n", 359 | "plt.xlabel('episodes')\n", 360 | "plt.ylabel('number of steps per episode')\n", 361 | "plt.axhline(np.mean(test_data[:,0,:]),c='red',ls='--',label='mean')\n", 362 | "plt.legend()" 363 | ] 364 | } 365 | ], 366 | "metadata": { 367 | "kernelspec": { 368 | "display_name": "Python 3", 369 | "language": "python", 370 | "name": "python3" 371 | }, 372 | "language_info": { 373 | "codemirror_mode": { 374 | "name": "ipython", 375 | "version": 3 376 | }, 377 | "file_extension": ".py", 378 | "mimetype": "text/x-python", 379 | "name": "python", 380 | "nbconvert_exporter": "python", 381 | "pygments_lexer": "ipython3", 382 | "version": "3.7.5" 383 | } 384 | }, 385 | "nbformat": 4, 386 | "nbformat_minor": 4 387 | } 388 | -------------------------------------------------------------------------------- /Simulation2_batch32 - Training.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Simulation 2_b32 - Training the agents\n", 8 | "\n", 9 | "\n", 10 | "## Importing libraries" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 1, 16 | "metadata": {}, 17 | "outputs": [], 18 | "source": [ 19 | "import numpy as np\n", 20 | "import matplotlib.pyplot as plt\n", 21 | "from scipy.signal import savgol_filter as SGfilter\n", 22 | "from IPython.display import clear_output, display\n", 23 | "import datetime\n", 24 | "import joblib\n", 25 | "from tqdm import tqdm\n", 26 | "\n", 27 | "import const\n", 28 | "import utilities as ut\n", 29 | "\n", 30 | "\n", 31 | "import gym\n", 32 | "import ctfsql\n", 33 | "from stable_baselines3.common.vec_env import DummyVecEnv\n", 34 | "from stable_baselines3 import DQN\n", 35 | "import evaluate as ev" 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": {}, 41 | "source": [ 42 | "## Defining the parameters of the simulations" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 2, 48 | "metadata": {}, 49 | "outputs": [], 50 | "source": [ 51 | "n_simulations = 10\n", 52 | "n_episodes_training = 10**6\n", 53 | "n_episodes_test = 10**2" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": 3, 59 | "metadata": {}, 60 | "outputs": [ 61 | { 62 | "name": "stdout", 63 | "output_type": "stream", 64 | "text": [ 65 | "Using cpu device\n", 66 | "Wrapping the env in a DummyVecEnv.\n", 67 | "Using cpu device\n", 68 | "Wrapping the env in a DummyVecEnv.\n", 69 | "Using cpu device\n", 70 | "Wrapping the env in a DummyVecEnv.\n", 71 | "Using cpu device\n", 72 | "Wrapping the env in a DummyVecEnv.\n", 73 | "Using cpu device\n", 74 | "Wrapping the env in a DummyVecEnv.\n", 75 | "Using cpu device\n", 76 | "Wrapping the env in a DummyVecEnv.\n", 77 | "Using cpu device\n", 78 | "Wrapping the env in a DummyVecEnv.\n", 79 | "Using cpu device\n", 80 | "Wrapping the env in a DummyVecEnv.\n", 81 | "Using cpu device\n", 82 | "Wrapping the env in a DummyVecEnv.\n", 83 | "Using cpu device\n", 84 | "Wrapping the env in a DummyVecEnv.\n" 85 | ] 86 | } 87 | ], 88 | "source": [ 89 | "env = gym.make('ctfsql-v0')\n", 90 | "\n", 91 | "dqn_models = [DQN('MlpPolicy', env, verbose=1) for i in range(n_simulations)]" 92 | ] 93 | }, 94 | { 95 | "cell_type": "markdown", 96 | "metadata": {}, 97 | "source": [ 98 | "## Running the simulations" 99 | ] 100 | }, 101 | { 102 | "cell_type": "code", 103 | "execution_count": 134, 104 | "metadata": {}, 105 | "outputs": [ 106 | { 107 | "data": { 108 | "text/plain": [ 109 | "'train_data = np.zeros((n_simulations,3,n_episodes_training))\\ntest_data = np.zeros((n_simulations,3,n_episodes_test))\\n\\nfor i in tqdm(range(n_simulations)):\\n dqn_models[i].learn(total_timesteps=10**6)\\n timestamp = datetime.datetime.now().strftime(\"%Y%m%d%H%M%S%f\")\\n dqn_models[i].save(str(i) + \\'ignore_simul2_\\'+timestamp)\\n'" 110 | ] 111 | }, 112 | "execution_count": 134, 113 | "metadata": {}, 114 | "output_type": "execute_result" 115 | } 116 | ], 117 | "source": [ 118 | "train_data = np.zeros((n_simulations,3,n_episodes_training))\n", 119 | "test_data = np.zeros((n_simulations,3,n_episodes_test))\n", 120 | "\n", 121 | "for i in tqdm(range(n_simulations)):\n", 122 | " dqn_models[i].learn(total_timesteps=10**6)\n", 123 | " timestamp = datetime.datetime.now().strftime(\"%Y%m%d%H%M%S%f\")\n", 124 | " dqn_models[i].save(str(i) + 'ignore_simul2_'+timestamp)\n" 125 | ] 126 | }, 127 | { 128 | "cell_type": "code", 129 | "execution_count": 7, 130 | "metadata": {}, 131 | "outputs": [ 132 | { 133 | "name": "stderr", 134 | "output_type": "stream", 135 | "text": [ 136 | "\r", 137 | " 0%| | 0/10 [00:00= self.syntaxmin and action_number < self.syntaxmax): 52 | if(action_number == self.flag_cols*2 + self.setup[1] + 1 or action_number == self.flag_cols*2 + self.setup[1] + 2): 53 | if(self.verbose): print('Query with correct number of rows') 54 | return 4,self.query_reward, self.termination, "Server response is 4" 55 | 56 | if(self.verbose): print('Query has the correct escape, but contains the wrong number of rows. I return 0') 57 | return 0,self.query_reward, self.termination,'Server response is 0' 58 | else: 59 | if(self.verbose): print('Query is syntactically wrong. I return -1') 60 | return -1,self.query_reward, self.termination,'Server response is -1' 61 | 62 | 63 | def reset(self): 64 | self.termination = False 65 | if(self.verbose): print('Game reset (but not reinitialized with a new random query!)') 66 | return None,0,self.termination,'Game reset' 67 | 68 | def reveal_solution(self): 69 | #For debugging only 70 | print('Correct escapes are: \n [{0}]: {1} \n [{2}]: {3}'.format(self.setup[0],self.A[self.setup[0]],self.setup[1],self.A[self.setup[1]])) 71 | print('Correct SQL injection is: \n [{0}]: {1}'.format(self.setup[2],self.A[self.setup[2]])) 72 | -------------------------------------------------------------------------------- /utilities.py: -------------------------------------------------------------------------------- 1 | from itertools import chain, combinations 2 | 3 | def powerset(iterable): 4 | #based on: https://stackoverflow.com/questions/18035595/powersets-in-python-using-itertools 5 | "powerset([1,2,3]) --> () (1,) (2,) (3,) (1,2) (1,3) (2,3) (1,2,3)" 6 | s = list(iterable) 7 | return list(chain.from_iterable(combinations(s, r) for r in range(len(s)+1))) 8 | 9 | 10 | def getdictshape(d): 11 | return (len(d.keys()), d[()].shape) 12 | 13 | 14 | if __name__ == "__main__": 15 | x = powerset([0,1,2,3,4,5]) 16 | print(x) 17 | print(len(x)) 18 | --------------------------------------------------------------------------------