├── .gitignore
├── LICENSE
├── data
├── N-MNIST
│ └── .gitignore
├── SHD
│ └── .gitignore
└── ephys
│ └── .gitignore
├── docs
└── README.md
├── environment.yml
├── figures
├── figure2.png
├── figure4.png
└── figure5.png
├── notebooks
├── ephys.ipynb
├── getting_started.ipynb
├── library_comparison.ipynb
├── speed_benchmarks.ipynb
└── supervised_benchmarks.ipynb
├── scripts
├── ephys
│ ├── build_data.py
│ ├── run.py
│ └── train.py
├── run_benchmarks.py
└── supervised
│ ├── run_blocks_nmnist.py
│ ├── run_blocks_shd.py
│ ├── run_detach_spikes.py
│ ├── run_standard_nmnist.py
│ ├── run_standard_shd.py
│ └── train.py
├── setup.py
├── src
├── __init__.py
├── benchmark.py
├── datasets
│ ├── __init__.py
│ ├── ephys.py
│ ├── neuromorphic.py
│ ├── synthetic.py
│ └── transforms.py
├── metric.py
├── models.py
├── query.py
├── snn
│ ├── __init__.py
│ ├── block
│ │ ├── __init__.py
│ │ ├── block.py
│ │ ├── blocks.py
│ │ └── util.py
│ ├── snn.py
│ └── surrogate.py
└── train.py
└── tests
├── __init__.py
├── test_block.py
└── test_blocks.py
/.gitignore:
--------------------------------------------------------------------------------
1 | # OSX
2 | .DS_Store
3 |
4 | # Python
5 | __pycache__/
6 | *.pyc
7 | src.egg-info
8 |
9 | # PyCharm
10 | .idea
11 |
12 | # Jupyter Notebook
13 | .ipynb_checkpoints
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 | BSD 3-Clause License
2 |
3 | Copyright (c) 2023, Luke Taylor
4 | All rights reserved.
5 |
6 | Redistribution and use in source and binary forms, with or without
7 | modification, are permitted provided that the following conditions are met:
8 |
9 | 1. Redistributions of source code must retain the above copyright notice, this
10 | list of conditions and the following disclaimer.
11 |
12 | 2. Redistributions in binary form must reproduce the above copyright notice,
13 | this list of conditions and the following disclaimer in the documentation
14 | and/or other materials provided with the distribution.
15 |
16 | 3. Neither the name of the copyright holder nor the names of its
17 | contributors may be used to endorse or promote products derived from
18 | this software without specific prior written permission.
19 |
20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
--------------------------------------------------------------------------------
/data/N-MNIST/.gitignore:
--------------------------------------------------------------------------------
1 | # Ignore everything in this directory
2 | *
3 | # Except this file
4 | !.gitignore
--------------------------------------------------------------------------------
/data/SHD/.gitignore:
--------------------------------------------------------------------------------
1 | # Ignore everything in this directory
2 | *
3 | # Except this file
4 | !.gitignore
--------------------------------------------------------------------------------
/data/ephys/.gitignore:
--------------------------------------------------------------------------------
1 | # Ignore everything in this directory
2 | *
3 | # Except this file
4 | !.gitignore
--------------------------------------------------------------------------------
/docs/README.md:
--------------------------------------------------------------------------------
1 | # Addressing the speed-accuracy simulation trade-off for adaptive spiking neurons
2 |
3 | A new model for quickly training and simulating adaptive leaky integrate-and-fire spiking neural networks.
4 |
5 |
6 |
8 |
9 | ## Installing dependencies
10 |
11 | Install all required dependencies and activate the blocks environment using conda.
12 | ```
13 | conda env create -f environment.yml
14 | conda activate blocks
15 | ```
16 |
17 | ## Getting started tutorial
18 |
19 | See the [notebooks/getting_started.ipynb](../notebooks/getting_started.ipynb) notebook for getting started with our model.
20 |
21 | ## Reproducing paper results
22 |
23 | All the paper results can be reproduced using the scripts available in the `scripts` folder.
24 |
25 | ### Running benchmark experiments
26 |
27 | The `python run_benchmarks.py` script will benchmark the time of the forward and backward passes of the blocks and the standard SNN model for different numbers of neurons and simulation steps.
28 |
29 | ### Training models
30 |
31 | Ensure that the computer has a CUDA capable GPU with CUDA 11.7 installed.
32 |
33 | #### 1. Downloading and processing datasets
34 |
35 | #### Machine learning datasets:
36 | The content of the Neuromorphic-MNIST dataset can be [downloaded](https://www.garrickorchard.com/datasets/n-mnist) and unzipped into the `data/N-MNIST` directory. Thereafter, the `python convert_nmnist2h5.py` script (adapted from Perez-Nieves et al., 2021) needs to be run which processes the raw dataset. The Spiking Heidelberg Digits (SHD) dataset can be [downloaded](https://compneuro.net/posts/2019-spiking-heidelberg-digits/) and unzipped into the `data/SHD` directory.
37 |
38 | #### E-phys dataset:
39 |
40 | Running the `scripts/ephys/build_data.py` script will download and process the necessary data from the Allen Institute.
41 |
42 | #### 2. Train model
43 |
44 | You can train the blocks and standard SNN on the different datasets using the train.py scripts in the `scripts/ephys` and `scripts/supervised` folders respectively. See respective folders for different experiment run scripts.
45 |
46 | ## Building result figures
47 |
48 | Speedup plots can be built using: `notebooks/results/speed_benchmarks.ipynb`
49 |
50 | Machine learning benchmark plots can be built using: `notebooks/results/supervised_benchmarks.ipynb`
51 |
52 | Neural-fitting plots can be built using: `notebooks/results/ephys.ipynb`
53 |
54 | ### Machine learning benchmark results
55 |
56 |
58 |
59 | ### Neural-fitting results
60 |
61 |
--------------------------------------------------------------------------------
/environment.yml:
--------------------------------------------------------------------------------
1 | name: blocks
2 | dependencies:
3 | - python=3.8
4 | - pytorch::pytorch
5 | - pytorch::torchvision
6 | - nvidia::cudatoolkit=11.7
7 | - matplotlib
8 | - seaborn
9 | - pandas
10 | - conda-forge::h5py
11 | - nb_conda_kernels
12 | - ipywidgets
13 | - pytest
14 | - pip:
15 | - brainbox==0.0.6
16 | - jupyter
17 | - allensdk
18 | - --editable .
19 |
--------------------------------------------------------------------------------
/figures/figure2.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/webstorms/Blocks/540c9f28eedd58ef638dcacf75b4c27cdf52baa0/figures/figure2.png
--------------------------------------------------------------------------------
/figures/figure4.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/webstorms/Blocks/540c9f28eedd58ef638dcacf75b4c27cdf52baa0/figures/figure4.png
--------------------------------------------------------------------------------
/figures/figure5.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/webstorms/Blocks/540c9f28eedd58ef638dcacf75b4c27cdf52baa0/figures/figure5.png
--------------------------------------------------------------------------------
/notebooks/getting_started.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": 89,
6 | "id": "da7df381",
7 | "metadata": {},
8 | "outputs": [
9 | {
10 | "name": "stdout",
11 | "output_type": "stream",
12 | "text": [
13 | "The autoreload extension is already loaded. To reload it, use:\n",
14 | " %reload_ext autoreload\n"
15 | ]
16 | }
17 | ],
18 | "source": [
19 | "import time\n",
20 | "\n",
21 | "import torch\n",
22 | "import numpy as np\n",
23 | "import matplotlib.pyplot as plt\n",
24 | "import seaborn as sns\n",
25 | "import pandas as pd\n",
26 | "\n",
27 | "from src.datasets import SyntheticSpikes\n",
28 | "from src.snn.snn import SNN\n",
29 | "from src.snn.block.blocks import Blocks\n",
30 | "\n",
31 | "%load_ext autoreload\n",
32 | "%autoreload 2"
33 | ]
34 | },
35 | {
36 | "cell_type": "markdown",
37 | "id": "8702a126",
38 | "metadata": {},
39 | "source": [
40 | "## Model equivalence"
41 | ]
42 | },
43 | {
44 | "cell_type": "markdown",
45 | "id": "ca0f57e9",
46 | "metadata": {},
47 | "source": [
48 | "Let's check if the blocks model and the standard model produce the same output raster"
49 | ]
50 | },
51 | {
52 | "cell_type": "code",
53 | "execution_count": 86,
54 | "id": "286f166a",
55 | "metadata": {},
56 | "outputs": [],
57 | "source": [
58 | "n_in = 100\n",
59 | "n_out = 20\n",
60 | "rf_len = 20\n",
61 | "t_len = 1000 \n",
62 | "block_len = 40\n",
63 | "\n",
64 | "# Instantiate recurrently connected ALIF SNNs (blocks and standard version)\n",
65 | "torch.manual_seed(42)\n",
66 | "input_raster = SyntheticSpikes(t_len, n_in, min_r=0, max_r=100, n_samples=1)[0]\n",
67 | "standard_snn = SNN(n_in, n_out, rf_len, t_len, t_latency=block_len, recurrent=True, init_beta=0.99, init_p=0.99)\n",
68 | "blocks_snn = Blocks(n_in, n_out, rf_len, t_len, t_latency=block_len, recurrent=True, init_beta=0.99, init_p=0.99)\n",
69 | "\n",
70 | "# Ensure bocks model has the same weights as the standard model\n",
71 | "blocks_snn._rf_weight = standard_snn._rf_weight\n",
72 | "blocks_snn._rf_bias = standard_snn._rf_bias\n",
73 | "blocks_snn._rec_weight = standard_snn._rec_weight\n",
74 | "\n",
75 | "# Obtain spikes from blocks and standard model\n",
76 | "with torch.no_grad():\n",
77 | " blocks_spikes = blocks_snn(input_raster.unsqueeze(0), mode=\"train\")\n",
78 | " standard_spikes = standard_snn(input_raster.unsqueeze(0), mode=\"train\")"
79 | ]
80 | },
81 | {
82 | "cell_type": "code",
83 | "execution_count": 62,
84 | "id": "3c475ec8",
85 | "metadata": {},
86 | "outputs": [
87 | {
88 | "name": "stdout",
89 | "output_type": "stream",
90 | "text": [
91 | "Are model outputs the same? True\n"
92 | ]
93 | }
94 | ],
95 | "source": [
96 | "# Drum roll, the moment we have all been waiting for...\n",
97 | "print(f\"Are model outputs the same? {torch.allclose(blocks_spikes, standard_spikes)}\")"
98 | ]
99 | },
100 | {
101 | "cell_type": "code",
102 | "execution_count": 88,
103 | "id": "e476bd42",
104 | "metadata": {
105 | "scrolled": true
106 | },
107 | "outputs": [
108 | {
109 | "data": {
110 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAAEiCAYAAAAPh11JAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAA9hAAAPYQGoP6dpAADUPklEQVR4nOydf3xT1f3/Xzc3P2560962aUpvA6Q0EAEToEpt4lB0xdyqqDg3PtpusrjxQ6fO7uvq3By2HU6mTpz4Gej2UVsnm4pMtvGRisPhpti1E22ZKD/Cj1IKttEWSmlL2/P9g889S25v2gJFwJ7n43EfkHvP73PSnLzyPu83RwghYDAYDAaDwWAMiuFsN4DBYDAYDAbjfIFtnBgMBoPBYDCGCNs4MRgMBoPBYAwRtnFiMBgMBoPBGCJs48RgMBgMBoMxRNjGicFgMBgMBmOIsI0Tg8FgMBgMxhBhGycGg8FgMBiMIcI2TgwGg8FgMBhDhG2cGAzGacFxHEpLS89I2X/729/AcRxWr159RspnMM4lsrKy8O1vf/sLqevb3/42srKyvpC6vmywjdNp8Pzzz4PjONTW1p7tpgAAOjo6UFpair/97W9nuykx/O///u8Z+2BlDD/quo6+0tPTceWVV+L1118/28075/n1r3+N559//gup66OPPkJpaSn27NnzhdR3PlJfX4+vf/3rcLlcEAQBTqcTV111FZYvXx6T7uc//zlee+21s9PIEcSBAwdQWlqKDz744Aupb9WqVXjiiSeGtUy2cfoS0dHRgbKysnNy41RWVna2m8E4ScrLy/HCCy+gsrISJSUlaG5uxjXXXIO//OUvZ7tp5zRf9MaprKyMbZzi8O6772L69On48MMPMX/+fDz11FP47ne/C4PBgF/96lcxadnG6YvhwIEDKCsrO683TsZhLY1x3nP06FGIoni2mzEohBB0dnbCarWe7aZ8abn66qsxffp0+vo73/kORo0ahd///veYPXv2WWwZgzE0HnroIUiShJqaGiQnJ8c8+/TTT89Oo74gOjs7YTabYTAwfWS4YSM6zHz729+GzWZDY2Mj5syZA5vNBofDgXvvvRe9vb003Z49e8BxHB577DEsW7YMLpcLVqsVM2fOxNatW2PKvOKKK3DFFVfo1qX+Rr1nzx44HA4AQFlZGf2JZaCfyNSfZDZt2oQ77rgD6enpGD16NABg7969uOOOO3DBBRfAarXCbrfjG9/4Rr9vtsePH0dZWRkmTJgAQRBgt9sxY8YMbNiwgbbxv//7vwEg5qcflb6+PjzxxBO48MILIQgCRo0ahYULF+Lzzz+PqScrKwuzZ89GVVUVpk+fDqvViqeffjr+RDCGneTkZFitVhiNg3/f2rJlC66++mokJSXBZrMhPz8f7733Xr90ra2tKC4uRlZWFiwWC0aPHo1bb70VLS0tccvu6urC7NmzIUkS3n33XQDAkSNHcM8999By0tPTcdVVV+H9998flraWlpbGrFsV9T2kvi+ysrLw73//G5s2baJrXX3vqmnffvttLFy4EHa7HUlJSbj11lv7rfd4791oG5jnn38e3/jGNwAAV155Ja3vXFOczya7du3ChRde2G/TBADp6en0/xzH4ejRo6ioqKDjqI7zUP8WqvP7zjvv4Ac/+AEcDgdEUcSNN96I5ubmmLSEECxZsgSjR49GQkICrrzySvz73//u18bPPvsM9957L3w+H2w2G5KSknD11Vfjww8/jEmn2gL+4Q9/wAMPPACn04mEhAQcPnwYAPDaa6/B6/VCEAR4vV788Y9/PKlx/PWvf40LL7wQFosFmZmZ+N73vofW1taYNPHss6I/v/72t78hNzcXABAKhehYqwrtFVdcAa/Xi3/961+49NJLYbVaMW7cOKxcuTKmTO37TjsO6nvgiiuuwLp167B3715a13DYdTHF6QzQ29sLRVGQl5eHxx57DG+++SZ++ctfwu124/bbb49JW1lZiSNHjuB73/seOjs78atf/Qpf/epXUV9fj1GjRg25TofDgRUrVuD222/HjTfeiK997WsAgClTpgya94477oDD4cDixYtx9OhRAEBNTQ3effdd3HzzzRg9ejT27NmDFStW4IorrsBHH32EhIQEACc+UB5++GF897vfxSWXXILDhw+jtrYW77//Pq666iosXLgQBw4cwIYNG/DCCy/0q3vhwoV4/vnnEQqFcPfdd2P37t146qmnsGXLFrzzzjswmUw07SeffIJbbrkFCxcuxPz583HBBRcMeXwYJ09bWxtaWlpACMGnn36K5cuXo729Hd/85jcHzPfvf/8bl112GZKSklBSUgKTyYSnn34aV1xxBTZt2oS8vDwAQHt7Oy677DJs27YNt912Gy666CK0tLTgT3/6E/bv34+0tLR+ZR87dgw33HADamtr8eabb9I/wosWLcLq1atx5513YvLkyYhEIvjHP/6Bbdu24aKLLjrttg6VJ554AnfddRdsNht+8pOfAEC/9/Gdd96J5ORklJaW4pNPPsGKFSuwd+9e+kd/qFx++eW4++678eSTT+LHP/4xJk2aBAD0XwbgcrmwefNmbN26FV6vN266F154gf4NW7BgAQDA7XYDGPrfQpW77roLKSkpePDBB7Fnzx488cQTuPPOO/HSSy/RNIsXL8aSJUtwzTXX4JprrsH777+PYDCI7u7umLLC4TBee+01fOMb38C4ceNw6NAhPP3005g5cyY++ugjZGZmxqT/2c9+BrPZjHvvvRddXV0wm8144403cNNNN2Hy5Ml4+OGHEYlEEAqF6JfkwSgtLUVZWRlmzZqF22+/na7Zmpqafn+jB2PSpEkoLy/H4sWLsWDBAlx22WUAgEsvvZSm+fzzz3HNNddg7ty5uOWWW/Dyyy/j9ttvh9lsxm233TbkugDgJz/5Cdra2rB//34sW7YMAGCz2U6qDF0I45R57rnnCABSU1ND782bN48AIOXl5TFpc3JyyMUXX0xf7969mwAgVquV7N+/n96vrq4mAEhxcTG9N3PmTDJz5sx+9c+bN4+4XC76urm5mQAgDz744Em1f8aMGaSnpyfmWUdHR7/0mzdvJgBIZWUlvTd16lRy7bXXDljP9773PaK31P7+978TAOTFF1+Mub9+/fp+910uFwFA1q9fP6S+MU4ddV1oL4vFQp5//vl+6bVrbs6cOcRsNpNdu3bRewcOHCCJiYnk8ssvp/cWL15MAJA1a9b0K7Ovr48QQshbb71FAJBXXnmFHDlyhMycOZOkpaWRLVu2xKSXJIl873vfO+m+DrWtDz74oO4aVsdq9+7d9N6FF16o+35V01588cWku7ub3n/kkUcIALJ27Vp6L9772OVykXnz5tHXr7zyCgFA3nrrraF1eITxxhtvEJ7nCc/zJBAIkJKSElJVVRUz/iqiKMaMrcpQ/xaq8ztr1iy6fgkhpLi4mPA8T1pbWwkhhHz66afEbDaTa6+9Nibdj3/8YwIgpg2dnZ2kt7c3pu7du3cTi8US8xmjvk+ys7P7tXfatGlElmVavzouAGI+P/RQ2xoMBmPa8dRTTxEA5Nlnn6X3tGtTRfv5VVNTQwCQ5557TjctAPLLX/6S3uvq6iLTpk0j6enpdN703nfR4xD9frj22msH7efJwn6qO0MsWrQo5vVll12GcDjcL92cOXPgdDrp60suuQR5eXn43//93zPeRpX58+eD5/mYe9G2Q8ePH0ckEsH48eORnJwc8/NHcnIy/v3vf2PHjh0nXe8rr7wCSZJw1VVXoaWlhV4XX3wxbDYb3nrrrZj048aNg6IoJ10P49T47//+b2zYsAEbNmzA7373O1x55ZX47ne/izVr1sTN09vbizfeeANz5sxBdnY2vS/LMgoLC/GPf/yD/nzw6quvYurUqbjxxhv7laNVXtra2hAMBvHxxx/jb3/7G6ZNmxbzPDk5GdXV1Thw4MCQ+3cybR1OFixYEPMt/fbbb4fRaPxC3/MjhauuugqbN2/G9ddfjw8//BCPPPIIFEWB0+nEn/70pyGVMdS/hSoLFiyIWb+XXXYZent7sXfvXgDAm2++ie7ubtx1110x6e65555+ZVksFmqj1Nvbi0gkApvNhgsuuEC37nnz5sW0t6mpCR988AHmzZsHSZJixmXy5MmD9l1t6z333BNjKzV//nwkJSVh3bp1g5ZxshiNRixcuJC+NpvNWLhwIT799FP861//Gvb6TgW2cToDCIJA7Y1UUlJS+tkxAMCECRP63fN4PF/oKZlx48b1u3fs2DEsXrwYY8aMgcViQVpaGhwOB1pbW9HW1kbTlZeXo7W1FR6PBz6fDz/84Q9RV1c3pHp37NiBtrY2pKenw+FwxFzt7e39jDf12sk4c1xyySWYNWsWZs2ahaKiIqxbtw6TJ0/GnXfe2e8nBZXm5mZ0dHTo/ow6adIk9PX1oaGhAcAJ+5OBfj6J5p577kFNTQ3efPNNXHjhhf2eP/LII9i6dSvGjBmDSy65BKWlpbpfVE61rcOJ9j1vs9kgyzI7GXeGyM3NxZo1a/D555/jn//8J+6//34cOXIEX//61/HRRx8Nmn+ofwtVxo4dG/M6JSUFAOjff3UDpV0HDoeDplXp6+vDsmXLMGHChJi66+rqdOvW/o2MVxeAIZk6qPm1ac1mM7Kzs+nz4SQzM7PfASWPxwMA58x7hG2czgBa9eZ0iWf3EG1sfjronUy766678NBDD2Hu3Ll4+eWX8cYbb2DDhg2w2+3o6+uj6S6//HLs2rULzz77LLxeL37729/ioosuwm9/+9tB6+3r60N6ejpVNbRXeXn5oO1kfHEYDAZceeWVaGpqOiWF8XS44YYbQAjB0qVLY9afyty5cxEOh7F8+XJkZmbi0UcfxYUXXjhsfqfO9HtwqHzR9X2ZMJvNyM3Nxc9//nOsWLECx48fxyuvvDJovqH+LVSJ9/efEHLSbf75z3+OH/zgB7j88svxu9/9DlVVVdiwYQMuvPBC3brP5t/IL/I9crbfj8w4/Cyj9wG0ffv2GMv/lJQU3W/P2t3+yRiWDsbq1asxb948/PKXv6T3Ojs7+52kAIDU1FSEQiGEQiG0t7fj8ssvR2lpKb773e8O2C63240333wTX/nKV9im6Dyhp6cHwAnDbj0cDgcSEhLwySef9Hv28ccfw2AwYMyYMQBOzL/2BGk85syZg2AwiG9/+9tITEzEihUr+qWRZRl33HEH7rjjDnz66ae46KKL8NBDD+Hqq68+7baqSkBra2vMCS29b9yDvQ937NiBK6+8kr5ub29HU1MTrrnmGnovJSWl33utu7sbTU1NJ1UXQx/VzUb0eMYby5P5WzgUXC4XgBPrIPon4ubm5n6/SqxevRpXXnkl/ud//ifmfmtrq+7hiYHq0qK37uPl/+STT2La2t3djd27d2PWrFn0nt6aBU68R6LzDrZmDxw40M8tzvbt2wGAfi5Gvx+1dWk5E+8RpjidZV577TU0NjbS1//85z9RXV0d88fe7Xbj448/jjnS+uGHH+Kdd96JKUs93XGqb+hoeJ7v9w1p+fLl/Xb0kUgk5rXNZsP48ePR1dVF76lvAG275s6di97eXvzsZz/rV39PT8+w9IMxfBw/fhxvvPEGzGZz3JNbPM8jGAxi7dq1MbL6oUOHsGrVKsyYMQNJSUkAgJtuugkffvih7tFovW/nt956K5588kmsXLkS9913H73f29vb72eL9PR0ZGZmxqzD02mresLq7bffpunU4+taRFEccO0+88wzOH78OH29YsUK9PT09HvPR9el5tO+/+K9txgneOutt3TXkmpPFv0TVLx5G+rfwqEya9YsmEwmLF++PKZcPSeNenW/8sorMZ8ZAyHLMqZNm4aKioqY98iGDRuG9DPlrFmzYDab8eSTT8a043/+53/Q1taGa6+9lt5zu9147733Yn7G/8tf/tLv5+7B1mxPT0+Mq5nu7m48/fTTcDgcuPjii2ldQOz7sbe3F88880y/8kRR1P1Z83RgitNZZvz48ZgxYwZuv/12dHV14YknnoDdbkdJSQlNc9ttt+Hxxx+Hoij4zne+g08//RQrV67EhRdeGGO8arVaMXnyZLz00kvweDxITU2F1+sdsh1JNLNnz8YLL7wASZIwefJkbN68GW+++SbsdntMusmTJ+OKK67AxRdfjNTUVNTW1tJj4SrqYr/77ruhKAp4nsfNN9+MmTNnYuHChXj44YfxwQcfIBgMwmQyYceOHXjllVfwq1/9Cl//+tdPuu2M4eH111/Hxx9/DOCEs8BVq1Zhx44d+NGPfkQ3FHosWbIEGzZswIwZM3DHHXfAaDTi6aefRldXFx555BGa7oc//CFWr16Nb3zjG7jttttw8cUX47PPPsOf/vQnrFy5ElOnTu1X9p133onDhw/jJz/5CSRJwo9//GMcOXIEo0ePxte//nVMnToVNpsNb775JmpqamJUgtNpazAYxNixY/Gd73wHP/zhD8HzPJ599lk4HA7s27cvpsyLL74YK1aswJIlSzB+/Hikp6fjq1/9Kn3e3d2N/Px8zJ07F5988gl+/etfY8aMGbj++utpmu9+97tYtGgRbrrpJlx11VX48MMPUVVV1U9lmDZtGniexy9+8Qu0tbXBYrHgq1/9aoyPopHMXXfdhY6ODtx4442YOHEiuru78e677+Kll15CVlYWQqEQTXvxxRfjzTffxOOPP47MzEyMGzcOeXl5Q/5bOFRUv34PP/wwZs+ejWuuuQZbtmzB66+/3m9+Z8+ejfLycoRCIVx66aWor6/Hiy++GKPgDMbDDz+Ma6+9FjNmzMBtt92Gzz77DMuXL8eFF14YVzmObuv999+PsrIyFBQU4Prrr6drNjc3N8Y1yXe/+12sXr0aBQUFmDt3Lnbt2oXf/e53dJOj4na7kZycjJUrVyIxMRGiKCIvL4/aZ2VmZuIXv/gF9uzZA4/Hg5deegkffPABnnnmGXqo4sILL4Tf78f999+Pzz77DKmpqfjDH/5AFfFoLr74Yrz00kv4wQ9+gNzcXNhsNlx33XVDHj9dhvWM3ggjnjsCURT7pdUeZ1bdETz66KPkl7/8JRkzZgyxWCzksssuIx9++GG//L/73e9IdnY2MZvNZNq0aaSqqqqfOwJCCHn33XfJxRdfTMxm86CuCfTar/L555+TUChE0tLSiM1mI4qikI8//rjfkdMlS5aQSy65hCQnJxOr1UomTpxIHnrooZjjvj09PeSuu+4iDoeDcBzX71j3M888Qy6++GJitVpJYmIi8fl8pKSkhBw4cICmcblcg7o9YAwPeu4IBEEg06ZNIytWrIg5Qk2I/tH5999/nyiKQmw2G0lISCBXXnkleffdd/vVFYlEyJ133kmcTicxm81k9OjRZN68eaSlpYUQEuuOIJqSkhICgDz11FOkq6uL/PCHPyRTp04liYmJRBRFMnXqVPLrX/96SP0dalv/9a9/kby8PGI2m8nYsWPJ448/rnss+uDBg+Taa68liYmJBAA9iq2m3bRpE1mwYAFJSUkhNpuNFBUVkUgkElNXb28vue+++0haWhpJSEggiqKQnTt36h75/s1vfkOys7MJz/PMNYGG119/ndx2221k4sSJxGazEbPZTMaPH0/uuusucujQoZi0H3/8Mbn88suJ1WqNcQsw1L+F8f6e6h2R7+3tJWVlZUSWZWK1WskVV1xBtm7d2q/Mzs5O8v/+3/+j6b7yla+QzZs39zviH+99ovLqq6+SSZMmEYvFQiZPnkzWrFmj+/kRj6eeeopMnDiRmEwmMmrUKHL77beTzz//vF+6X/7yl8TpdBKLxUK+8pWvkNraWl13OmvXriWTJ08mRqMxxjXBzJkzyYUXXkhqa2tJIBAggiAQl8tFnnrqqX517dq1i8yaNYtYLBYyatQo8uMf/5hs2LCh31i3t7eTwsJCkpycPCQXDEOBI+QULNYYp82ePXswbtw4PProo7j33nvPdnMYDMYZRnX0WlNTExPKhsFgnOCKK65AS0vLkG0fzxbMxonBYDAYDAZjiLCNE4PBYDAYDMYQYRsnBoPBYDAYjCHCbJwYDAaDwWAwhghTnBgMBoPBYDCGCNs4MRgMBoPBYAwR5gATJ2KmHThwAImJiSyEwZccQgiOHDmCzMzMmGjfIxG27kcGbM3Hwtb9yOBMrnu2ccKJ2DhqTCrGyKChoQGjR48+2804q7B1P7Jga/4EbN2PLM7EumcbJwCJiYkATgzwQKEkGOc/hw8fxpgxY+icj2TYuh8ZsDUfC1v3I4Mzue7Zxgn/iZ6clJTE3kgjBCbRs3U/0mBr/gRs3Y8szsS6P6s/eL/99tu47rrrkJmZCY7j8Nprr8U8J4Rg8eLFkGUZVqsVs2bNwo4dO2LSfPbZZygqKkJSUhKSk5Pxne98Z9DAhQwGg8FgMBinwlndOB09ehRTp07Ff//3f+s+f+SRR/Dkk09i5cqVqK6uhiiKUBQFnZ2dNE1RURH+/e9/Y8OGDfjLX/6Ct99+GwsWLPiiusBgMBgMBmMEcVZ/qrv66qtx9dVX6z4jhOCJJ57AAw88gBtuuAEAUFlZiVGjRuG1117DzTffjG3btmH9+vUxQTOXL1+Oa665Bo899hgyMzO/sL4wGAwGg8H48nPOnk3dvXs3Dh48iFmzZtF7kiQhLy8PmzdvBgBs3rwZycnJMZHGZ82aBYPBgOrq6i+8zQwGg8FgML7cnLPG4QcPHgQAjBo1Kub+qFGj6LODBw8iPT095rnRaERqaipNo0dXVxe6urro68OHDw9XsxkMBoPBYHyJOWcVpzPJww8/DEmS6MV8ejAYDAaDwRgK5+zGKSMjAwBw6NChmPuHDh2izzIyMvDpp5/GPO/p6cFnn31G0+hx//33o62tjV4NDQ266crKyiCKIhYsWICioiLU1tYCAGpra1FUVITKykrdf6PTFRQUIBAIYPz48bBarSgrK9PNHwgEkJmZicrKSlq/mi66vEAggLS0NEyZMgUFBQW6bSooKIh5Fl1edP5AIEDTVVZWQpZlXHrppbr9Uvuhplf7Fq8etd3RdQ5U9vjx42GxWDBlyhSaL7rvlZWVyMrKouOjtlcdh8HGknF6aOfjVO+fTNln6tmp5h1qn06nXWeyX4yTh617tu51IecIAMgf//hH+rqvr49kZGSQxx57jN5ra2sjFouF/P73vyeEEPLRRx8RAKS2tpamqaqqIhzHkcbGxiHX3dbWRgCQtra2mPsJCQkEAOF5nhiNRlJYWEgIIaSwsJAYjUbicrl0/41Ox3EcAUCvhIQE3fzqc5fLRetX00WXF10Wx3G6beI4LuZZdHnR+aPL0LZB2z61H2p6tW/x6lHbra1zoLLVS80X3Xe1fer4RLeX47hBx3KwuR6JnMxYaOfjVO+fTNln6tmp5h1qn06nXWeiX2zNx8LWPVv3p8tZ3TgdOXKEbNmyhWzZsoUAII8//jjZsmUL2bt3LyGEkKVLl5Lk5GSydu1aUldXR2644QYybtw4cuzYMVpGQUEBycnJIdXV1eQf//gHmTBhArnllltOqh16A1xTU0PcbjfhOI4IgkACgQCpqakhFRUVJCMjgwQCAVJRUUHcbjcBQIxGI5kzZw6RZZn4/X6a1m63E6vVSgwGAzEajcTtdhOv10tkWSbz588nsiwTt9tNJEkikiQRr9dLFEUhpaWlxG6397snSRLheZ6IokjbUFhYSEpLS4nL5SKlpaVEURSiKAqpqakhNTU1xOv1ErPZTJxOJ5EkiYiiSKxWK+F5nthsNuL1eonT6SQcx5HRo0cTt9tN06vtzMjIID6fjwQCAeJ2u4nRaCRWq5XY7XZSWlpKCgsLSU1NDSGEkIqKipi2qOW53W7i9/uJz+cjdrud+Hw+WpbBYCAASEpKCvF6vbrj4XK5SEVFBa1DHVuTyUTbWlpaSrxeLzGZTESSJJp+oLkeqZzMWNTU1MTM8aneP5myz9SzU8071D6dTrvORL/Ymo+FrXu27k+Xs7pxeuutt/opIADIvHnzCCEnVKef/vSnZNSoUcRisZD8/HzyySefxJQRiUTILbfcQmw2G0lKSiKhUIgcOXLkpNqhN8BatSghIYFUVFQQs9lMABCTyUT8fr9u+wEQWZaJLMv91JToy2Qy9VOjVBVHEIR+6bX31Dp4nidpaWkEAN3gqZspVTUbqBxtG7T3zGYz4TiOKIpCAoGAbl+NRiNRFIUUFhYSRVH6qUo8z5OEhARiMBgGHBN1A6W97HY7URSFVFRUEEVRiN/vJ3a7vZ96pihKjOqUlpY26FyPVNhYjAzYPMfCxmNkcCbnmSOEEIxwDh8+DEmS0NbWRl3w19bW4oEHHkBjYyN27tyJ7u5ujBkzBnv37h20PEEQqJPOjIwMHDt2DG1tbYPm6e7uRnp6OqZOnYrGxkZs3bq1XzqLxYKurq6YOlwuF9rb2xGJRJCRkYGvfvWrePnll2E2m9HR0RG3bWazGRdccAFEUaT5Fy5ciIceegjHjx/vV7eiKPjwww9jTixyHIe8vDxkZ2cjEongr3/9K/Lz82G326EoCqqqqqAoChYvXoyGhgZwHIfe3t6Ycs1mMwwGAzo7O+F2u9Ha2opIJNKvfo7jMHbsWOzbtw/Ry1bth+q3680330RfXx8IIbDb7WhpaaFp9eZ6pMLGYmTA5jkWNh4jgzM5z+escfjZZvr06Vi/fj3q6+vx9NNPY8yYMQiFQvB6vTCbzUhPT4fZbIbD4YDZbIbT6YTdbkcgEMDTTz8Nv98Pu92OtLQ02Gw23TrS09Ph9/vh8/kgiiLy8vLw5z//GUuWLIHNZoMkSbBareB5HqIoIhAI4P7774fL5cK3vvUt2O122O12lJeX46677kJCQgKuu+46RCIR5ObmIjs7G7Iso7S0FD6fD4IgIC8vDxaLBT6fD++88w7q6uqwfPly2Gw2dHd344UXXkBCQgIkSYLb7YbdbofT6YTZbMb777+PlpYWCIKAtLQ0zJw5EwaDgRriBwIBOBwONDY2oq6uDj/4wQ+we/duTJ48GatXr8bNN9+Mn/70p3C5XJgzZw5tx29+8xtMmzYNsizjW9/6FmRZhtFohNVqjWmH3+9HeXk5gsEgfD4frFYrOI6Dw+FAZmYmlixZgiVLluC//uu/8OCDD8LlcuHxxx//IpcNg8FgML7sDLuGdR4ymKSnNXY2Go30J63on9eiDdP0DMO1V0JCAiGkv+FztOF19E9n0fVrjcDjGaxrjat5ntc1Qte2zeVy9eurtt1a43k9Q2+13YONqdqX6J/ZoDH81o6vNt1QDDKZTP8f2FiMDNg8x8LGY2RwJuf5nHWAeS5RXFyMSCSCSCSCsWPHwmAwoK+vD5IkIS0tDQcPHsTx48fR0dEBWZYxbtw4KIqCpKQkHD16FFarFUeOHAHP8yAn7MpgNBpRUlICACgvL8ePfvQjyLKM2tpaWt/hw4dx5MgRNDU1ITMzE5FIBIWFhdi9eze2b98Or9cLAKiqqqJt6unpQW5uLgghyMzMRDgcRkFBAUKhEJ5++mkkJCSgpaUFra2tmDJlCkRRBAC43W40NDRAFEW0tbWhp6cH4XAYkyZNQjgchslkos/6+vpw6aWXIhgM4k9/+hNSU1NhNBoRDAbx5z//GaIo4uDBg+js7ITVakVdXR18Ph+amprg8XiwaNEiRCIR5Ofno7i4GAAQDoexY8cOGI1GSJKEnp4e9PX1obu7G4cOHYIoirQc9SdFt9uNcDgMQRAwbdo0KIqCgoICtLW1QZIkLFmyJMarPIPBYDAYp82wb8XOQ4ayM9VTX1S1SX2tqjn4P8VGez+eOhVdvlZV0SpdhYWFMQqVVqFR7w+kSmkVpGhVJ/qZVvHS1qV1UzCQiwG99sVT6PSUJ70yopWqaOVKLUNPgWLfNv8DG4uRAZvnWNh4jAyY4nQWUI3DVaPu5uZmGI1G5Obm4t1330VCQgKKi4uxefNm7Ny5E/v27QPP87DZbBg7diw+++wzWtb48ePR0NCA3NxcvPfee/B4PFRpUfF4PDCbzRBFkaomAKgq89FHH+Gtt95CXV0denp6wHEcDh06hMTERHAcB7PZjK6uLhiNRrS2tiI3NxeLFi3CihUrsHfvXiiKgsmTJwMARFFEZWUl0tLS0N7eDgBITExEfn4+AoEAVq5cSe2GrFYr1q5dC0EQYDQaIYoiurq6MGnSJNx0001Yvnw5Ojs7IQgCcnJy0NzcjJycHLS3t6OjowPd3d3U+NtoNGLy5MlYtGgRVq5ciY0bN6KsrAybN29GW1sb8vLy0NzcjMbGRmRlZeHAgQPgeR4WiwVpaWloaWmhfW9pacG6devg9Xphs9kQDofR3t5OjQAnT57cb4wZDAaDwThthn0rdh4yFHcE6pWQkDCgvY2qhETnUW2AVPVGzzGjqrBEq1haB5cDqTjaZ9H2UvGULD3HlvHapa1LT+HR2n3p5YnXX60jTq0tVjy7r2jlKV5dg831SGWwsTgZnzQng165p1rXmWrjF8Hp+r0Zajq25mNh6/7s8mVY90xxioNqZ6SGZGlsbITNZsO9996L7du3x9jTNDc3g+d5AEBrays8Hg+8Xi+2bdsGjuOQkpKCtrY2pKSkQJIklJeXU0ULAAoLC5GYmAhBEFBUVIR9+/ahra0NHMdRG6VAIIBRo0bB4XCAEILdu3ejq6sLKSkpaG1tBcdxIIQgOTmZ2ihlZmZi9uzZcDqdUBQFlZWVWLx4MUKhEPLz89HW1gZZlgGcOOofiURQW1tL7YJqa2uRmJgIs9mMMWPGICEhAfv27QMAVFdXo7GxEcCJ04G9vb3UpUBKSgq1P1JdH2RlZSESiaCyshKPPvoo9u/fD6PRCFmW4XA4IEkSFEVBXV0dPa3Y1dWFxMREBAIBtLe3IzU1FcnJyfRkX29vL4xGI8LhMPLy8gAAR48exb59+1BVVYXKykrceuutX9yi+ZKxbNkyvPzyywCAF1988YyWe6p1nak2fhEM1Pah9ut87v+5Clv3Z5Yvxbof9q3YechgO9NoJUVlIAVooBNrWgVJtecZ6GSe3smyaNUIA6gv0QpZtG2UVomKp0zFsyGKvnie11W89Nqhd2pOT1WLHpfotHqqlrYMvXkY6lyPJNg377PLl+Gb9/kIW/dnly/DumcOMDGwo6zKykrccccd6OjoQGZmJrxeL5YsWQIAeOCBB3DgwAGEw2EcO3YMfX194HkekyZNwte//nVqK0QIoSfjVH9DAHDbbbdhx44dyMvLwzvvvAOO43DppZdSO6h7770XK1aswI4dOyAIAiKRCH70ox/h2muvjVGrHn30UezcuRPf+ta3sG/fPmzdupUqZLfccgv+8pe/wG6349ixY2hsbKRlLFu2jNpP/eAHPwDHcbjzzjupzRFwwraroaEBY8aMgdVqRVNTE1V9jh07huPHj8NiscDpdFIbJKPRCJvNhubmZjquDocDfX19mD17Nl5++WW0t7fDbDZTNQsALfvgwYPUGahqgzV27FgAJ07fdXd30xN+DocDvb29mDBhApYvXw4AuOuuu7B3714sXbq0n+LEnN/9BzYWIwM2z7Gw8RgZnNF5Hvat2HnIQDvTgRQSQvSVJz3/S2ragXwrqXmj82nLH0gtUlWp6FN8Q7H/0apWenZL0fn1AupGK0papelkTt7phXzRC0SsXqraxfw4nTxsLEYGbJ5jYeMxMjiT88w8h8ehtrYWRUVFCAaDVFFRvWi/8sorGD9+PAKBAKqrq2EymagHcbfbDb/fj+zsbJhMJgDA66+/Dp/Ph3Xr1iE5OZnaG7W3t0OSJGRkZIDneZhMJuTk5NA2VFdXQ1EU5OXlQRRFmM1mhEIheDweGI1G1NfXo7KyEnV1deA4Dk1NTeB5HjNmzEBCQgKKiooQiUSQl5dHPZQnJSWhqqoKZWVlKCoqQmVlJRITE6n/pKysLHAch/T0dCQlJdFTbE6nE36/H16vF4cOHUJvby86OjogCAK148rPz0coFEJqaiosFgs97WcymeB0Oqm/J4fDgWAwiKSkJPA8D6vVClEUIUkSZFmGKIrgeR48z8PpdCIjIwOKoqC8vBySJMFoNCI5ORkcx4HneSQkJCA1NRUejwcFBQXw+Xyw2WwQBAFlZWVnawkxGAwG48vIsG/FzkPinarT88QdrQ5FX1q7JqPRGPcEXLR6oqfIaNNqbY20ClG8tugpU2oaPYWI4zjaP20/1TIH8q80kB3UYP3Us2mKbqdWpYunhOnlH2yuRypsLEYGbJ5jYeMxMmCK01lAURQ4HA7q3ygUCsHpdOK6666DyWSCwWCA1WqlykpTUxMcDgfKysoQDochiiKMxhOHFtPT0+F0OqmSU1VVBaPRiLS0NKSkpIDjOACI8ZWkKi719fVQFAXBYBDBYBDFxcUoLy+H3W6HzWZDa2sr3G43RFGEwWCAyWRCKBQCcMJfU19fH6qrq6lHcq/XC0EQEAwG4XA4YDQakZSUBEmS4PV6kZGRAYPBQFUok8kEjuNgNBpRVlaG1tZWGAwGqgZJkgRRFGlQX9UflarQud1upKWlQZZlJCQkwGg0wuv1ory8HF6vN6YOu90Onudp+QCQm5uL/Px8erowGAwiISEBubm5MBqNVLFKTU1FKBRCXl4eJEmial8wGDwLq4fBYDAYX1qGfSt2HjKYHyee54nb7SYcxxFFUUhaWlpc1UUQhH42SYT8RykxGAz0WfT/tVd0GWlpaaSiooIUFhaSiooKoigKsdvt/eyeok/g1dTUEEEQaBpZlonf7yeyLBODwaCr2Oj1K7od0e3lOI6+NplMhOM4EggEaLmyLBOv10sEQSButzumTFmWSWFhIfH5fHH7r14WiyWmrYIgEJ7ndW2hZFkmdrudcBxH+649Wce+bf4HNhYjAzbPsbDxGBkwP05nAdWP04YNG9Db24vdu3fTZ+T/DiKq6pB6Ag044SXc6XSisbERkUgE5eXlAE7Eo1u8eDFaWlpw9OhRAEBfX19MnTzPo7e3N6YOAGhpacHixYvR2NiId955B/v27aPPBUFAeXk5Jk+ejEgkgrq6Ouzfvx/Lli3D+PHjsXXrVphMJjQ1NaGpqQkAkJCQgM7OTlgsFnR1dQEAMjIy0N3d3W8cVP9QZrM55jn5v5h7AHD8+HEAwO7du9HZ2YmEhAS4XC6899579L6KwWCAy+XCyy+/TFUlvfoEQQAAWCwWtLW10Vh8nZ2dcLlcCIVCePjhh2n7BUGg/VPn4ciRI3T8GQwGg8EYDtjGKQ7Tp0/HkiVL0NjYiB07dmD06NFIT0/Hzp070dbWBpvNBq/Xi9tvvx3l5eUIh8NwOp2w2Wx007Rw4UJUVVVh8uTJ9Fj8HXfcAYPBgMsuuwxbt26lDiKNRiPGjBmDY8eOYe/evTCZTCCEoKenB5MmTcK9996LVatWYefOndQYvbOzE7Is49FHH6X1qahuBhYvXoycnBy8/vrrkCQJra2tkGUZ6enpIITg6NGjaGlpwdKlSwEA9913H3iex6FDh2AymWA2m2mYlFWrVqGxsRHbt29Hd3c3TCYTEhISMHbsWGRmZmLs2LF44YUX4Ha7UVBQgA8++ACdnZ247LLL0NXVha1bt+LYsWPYtWsX/amzu7sbRqMRvb294DgOGRkZtF82m426I1DdHIwdOxaEEDz99NO4//778eqrr+KTTz6B0+kEAOzduxeiKOKHP/whc37JYDAYjOFn2DWs85B4kp6egTgQG0YlnqG3+lrPsBk6xuRaZ5hqunjOHfXCmmjr0wb11QYhju6bXjgUbdgT7fN499Vn2r7q/bw22DVQoGGXJsixnguIoc71SISNxciAzXMsbDxGBsw4/CxRXFyM3NxctLa2QhRFJCYmwu12QxAEXHfddXA6ndQ1gMFgQEpKCrxeLywWC4AThs25ubmoqqrCpZdeilAoBLvdTo/d19bWorKyEhs3bkR2djYyMjIQDAYxatQo+Hw++Hw+ZGRkoKOjA6IoorGxkRpT5+bmUgNy1RWCLMs0KLDa/vz8fKSkpMBgMCA1NZUabLe0tIDneeTk5NCQLAComwT1OH90W2traxEIBNDS0kKDCVdWVqKyshKyLMNoNMJutyMQCCAYDMJoNNKroKAARUVF1D0BAJhMJuqewGg0wmq1QpIk+Hw++P1++P1+ahRvMBjgdrtRXl4Ov99PDc5DoRAkSYLZbMbEiRNhMBgwevRoGj6GwWAwGIxhZdi3YuchQ3WAqVV/9EKIaBWQ6Nd6jjCj1SqtAqSmU5UfrZoV/f94TiD1Qr7oOa6M55Qz+rm2LLW/en2M5zhUW4bWKaZW/YouS8/lg15gYL1AzEOZ65EGG4uRAZvnWNh4jAyY4nQWUY/+S5IEv98Pj8cDWZZRXV1NHTlKkkTVH9U5I3DCHUAwGITZbIbVasWaNWsgCAJEUcTcuXNRXFyMUCgEQRCoWlReXo78/HxEIhF4PB44HA5kZGRAEATccMMN9Ki9wXBi6nieR1FREbxeL0wmE3WK6fP5YLFYsGHDBto2SZJo+yVJomqSJEmoq6tDQUEBdYA5depUuFwulJSUYO7cufB4PKiqqoIoivRS84dCIWRkZMDtdqOqqgrjx4+nKp0oitRlQ0dHB8LhMK3b7XYjIyMDWVlZ6O3tRVNTEwRBoA46CwoKMH78eBpYuKenhypf4XAYgiBg9erVyMnJQVpaGgRBQFpaGm2zqrwxGAwGgzFsDPtW7DxksJ2pnroBxHe6qLUlGsgpo9a5ZXR9at7oZ3oOKIfiFBMadUfPWWW0A0yt40ht+XoKkbZerR2WNjyKVlEbKG90X/UCGcezuzrZuR5JsLEYGbB5joWNx8iAKU5ngdraWhQUFCAQCCAcDsPlcuGVV15BR0cHzGYz0tPTYbFYkJOTg1GjRsHtdsNut1O7JI7jMHr0aCQmJiI1NZXaRnEch66uLpSVlcXYN6WmplLVx+Px0BNwXq+XOr6sra2ljinNZjNVkVQHmaodUDAYpCqTqnCpdlcdHR1wOBzweDyIRCLwer3UiWVSUhISExMBnFDLAoEACgoKUFtbi2AwCI7jYLVa6Qm3YDCI3NxcrF69GhzHYf/+/VQJs1qtkGUZSUlJ1KZpxowZyMvLQ0ZGBq1/0qRJEASB5nM4HJBlGUVFRUhNTaVjBoDaLnk8HqSmptJy1fao6ldBQQFtN4PBYDAYw8qwb8XOQwZzgBmtxKivVRsjvZNs2hNyevY90SrJQOFI9E7fRafTKmHRNkfRdlNAfLsiPWUnOm28ALta+6boS1WXtOm1JxW1J/7U/mrHUXu6cTA1Kl7QX/Zt8z+wsRgZsHmOhY3HyIApTmcBRVHo6bZgMIjrrrsOHMeB4ziIoghZliHLMrWnURQFRUVFNChv9Ak5v9+P4uJiGvLEYrEgJSUFra2tSEtLQ1FRERISEsBxHJKTk7F//35YLBaYTCYkJiaisrISBQUFqKuro2FLsrOzkZiYSNUjtT2hUAiJiYkQBAFFRUWYO3cuDW+iBiEOBoMIhUI0pExeXh4NCZOSkkJDpkiShKSkJITDYRrsOD09napZra2tyMnJoYoQAHpqLisrC16vlwZGFgQBoVCInvRTbbpKSkogyzKysrIgyzKCwSCysrLg8XgQDAbh9/tpP6PDxKSmpsLpdNLTeDabDVVVVfB4PHT81ZOCDAaDwWAMG8O+FTsPGSjIr54PJQD9FI3o9FpVZaCTbqoKAx3VRq0nnp2UVpEZiuIS3YbodNHKj17wX61CpG2Htr16AZL1ThVqX2ttleL5uNLWox1PdqpucNhYjAzYPMfCxmNkwBSns0BxcXHMySxFUejpOtXPUHV1NaxWKxYsWIC6ujr09fVh1apV2LRpE3Jzc2P8KlVWViIrKws33ngjXn31VepbKScnB5FIBG63GxaLhfpkig6g29TUBEIIrFYrDZqbnZ2NzMxMACdsfPLz86EoCrVbSk1NRXV1NZKTk+FwOLBgwQJkZWWhsrKS9k9VdMLhcIyvJ0mS+gX/jVaI7HY7PVWXnZ2NvLw8uN1uGoJGDYys2l2p7V+7di2mTJmCcDiM3NxcRCIRVFZWIhKJ0HFSTxkmJiaitrYWiqIgNTWVBvFVlTOn04nW1lYkJycjJycHJpMJkiQhFAohHA7D4XAwxYnBYDAYw8+wb8XOQ4ayMx1IgdI7FRbvxFm0YgL095MU78SbesU7jaZVuuKd9NOeOotXj3ppFa3osdCzsVLzxLOL0papV7bWLiz6tdZnkzqGat+YjdPQiTcWNTU1pLCwkNTU1AzpvpahphuuvKdT3+mUezL1nsqYDle/2JqP5Uys+9Odq5PNf6bW/FDKZuv+RKDWc5aenh7ywAMPkKysLCIIAsnOzibl5eWkr6+Ppunr6yM//elPSUZGBhEEgeTn55Pt27efVD3xBriiooK4XC5SWlpK/H4/kSSJCIJALBYLmT9/PvF6vUQQBDJ//nzi9/uJ0WgkAIjRaCSSJJFAIEAnv7S0lJjNZmI2m4nJZCJOp5NIkkQkSSI+n48oikJKS0uJLMvE7XYTSZKI1WolPM8Tk8lEABCLxULTBwIB4na7idlsJunp6cRisZA5c+YQWZaJ1+slfr+feL1eIooiMZvN5KKLLiI8z5M5c+YQRVGIoiikpqaG1NTU0L5JkkTcbjex2+30X5/PR/x+P21fRkYGrd/v95NAIEC8Xm9MXwKBAFEUhcyfP58YjUZiMBiIwWAgPM8Tp9MZU77aV3W8KioqiN/vJ3a7nfj9flrn6NGj6bjJskzmz59P7HY7sVqtxGAwEJPJRObPn08URaHt1XvjsQ+R/zBYqCG9n5jj/QR6KumGK+/p1Hc65Z5MvacypsPVL7bmYzkT6/505+pk85+pNT+Ustm6P8c3Tg899BCx2+3kL3/5C9m9ezd55ZVXiM1mI7/61a9omqVLlxJJkshrr71GPvzwQ3L99deTcePGkWPHjg25nngDnJaWNqAS4/f7Yz6ktQpLtLqjKAq9b7FYiMFgiCkr+l/tpadoRefXe56WlkYqKiqILMuE4zhisVj6pZEkiciyTHw+X4wSxPM8SUhIoBseAESW5ZjxEASBKjvqPbPZTDdhaj+1fdJrR/SVkJBA26OWqU3j8/kIIf3tztS2Kopy0nM9EmGK0+mVy755n58wxen0ymbr/hzfOF177bXktttui7n3ta99jRQVFRFCTqhNGRkZ5NFHH6XPW1tbicViIb///e+HXE+8Abbb7QN+yGt/mqqoqKAf3gaDgVRUVNCyojdOkiTFDXgbvSGK3qCoG46T2VxFb+TUzYxeHWrZsixTlU1ViKLTacdjoJ/41CsjI6NfPr3Nk91up5ukwTZXFouFEHLiDabXb7ZxGhpsLEYGbJ5jYeMxMhixxuGXXnop/vrXv2L79u0AgA8//BD/+Mc/cPXVVwMAdu/ejYMHD2LWrFk0jyRJyMvLw+bNm+OW29XVhcOHD8dcWmprazFhwgRqoG2xWGAwGHDRRRfBbDZDFEUaiDc1NRXhcBi7d+9GSkoK0tLScP311+P2229HWVkZAKCwsJAal0+aNAklJSU0TInZbAYAWCwWTJ48GU6nEwaDAU6nE+PHj0dqaiomTJiAQCBAXQcYDAaIogjghLPJn/70p7Db7RAEAUajET6fD+Xl5VAUBYFAAJMmTaJOOC+77DJqiC4IAq6++mrY7XZ0d3cDAFavXo2bb74ZM2bMAABwHAdJkmg+p9MJWZZxww03QBAEGmJGdZCpOtu02Wz4xS9+gccffzzGoHzChAlwOBwAToSMmTNnDmw2G+bNmweXy4X7778fiqJQFwlqEOI5c+bQIL6VlZV44IEHMGnSJKSlpcHtdkOSJNjtdhQWFqK2thZFRUXMCSaDwWAwhpdh34oNI729veS+++4jHMcRo9FIOI4jP//5z+nzd955hwAgBw4ciMn3jW98g8ydOzduuQ8++KCukhHPHYGe+wC94//RRsrRDjKjyxvIGHogB48DuQRQ64z+yS/692Ft3VrnnfEcbuq5HojOpx0bvUvPaF0b2kXrgkBbf/QY6jkX1TM0j/c7Ofu2+R/YWIwM2DzHwsZjZDBiFaeXX34ZL774IlatWoX3338fFRUVeOyxx1BRUXFa5d5///1oa2ujV0NDQ7800e4IFEWhwXUTExPh8XioCqI6yszOzobJZEJaWhrKy8tRVFQEg8FAHUKuWbMGycnJCIVCMQ4zRVGE2WxGdnY2ZFlGbm4ujEYjeJ6PUbUyMjKQk5NDj/wLgoALLrgAwAml59ChQzCbzeA4DtnZ2bT8srIybNy4Ebm5uQiFQpAkiSpIaoDfxMREqmIZjUbqcFOWZers0ufzoaSkBGlpaTCZTJg0aRLq6upQVVVFHVGq7VUdX/p8Phoixel0UrcBSUlJuO6662hw48TERKSlpSExMRGBQAA+nw9paWlISUmBxWJBdnY27YfX64XX60VLSwt1BNrT04Pe3l4cOnQIubm51C2D6uLgfOTtt9/Gddddh8zMTHAch9deey3m+be//W3qkFW9CgoKzk5jGYxhgq17xnnBsG/FhpHRo0eTp556Kubez372M3LBBRcQQgjZtWsXAUC2bNkSk+byyy8nd99995DrGUqQ33ihTlRlI1pt0uaBRoFRnw+m/kCj2EQrQNrQLNFXQkKCbrui1SFtmJfosvXUHLXdap3a9PHGTU/piu6XnnPL6Eub16UJ86JVxb4sDjD/93//l/zkJz8ha9asIQDIH//4x5jn8+bNIwUFBaSpqYlen3322UnVcb6MBeP0OJ/mma17xnBxJufZeAb2YsNGR0cHDf6qwvM8+vr6AADjxo1DRkYG/vrXv2LatGkAgMOHD6O6uhq33377addfWVmJH/zgB+js7ERSUhLGjh0Lm80GQggmTpyI4uJifPTRR6iqqkJnZydsNhva29tRWVmJ4uJiRCIRbN26FY2NjTAYDEhISEBdXR0CgQAAIDc3F0eOHEEkEsHs2bPxxhtvIBQKYf369dixYwcyMzMRiURQWFiISCSCnTt3Ys+ePQCAyZMn495778WKFSuwbds2ACdsilpaWpCSkoJ169ZBFEXYbDYaBLe6uhoGgwGEEASDQRw8eBA7duyAx+OBoihYuXIlxo0bh0WLFtFyW1tbkZGRgVdffZWGmunu7sbChQvxwgsvYPfu3cjJyaHhZqqqqui/qpqWk5MDAMjJycHrr78Oj8cDURRRWVmJ7OxsJCQkYN++fRAEAcePH0dKSgo+//xziKKIUCiEzZs3Y+vWrejs7EROTg4OHjyIlpYWHD9+HLm5ufjHP/5B10ROTs6XwgHm1VdfTW354mGxWJCRkfEFtYjBOPOwdc84Lxj2rdgwMm/ePOJ0Oqk7gjVr1pC0tDRSUlJC0yxdupQkJyeTtWvXkrq6OnLDDTcMmzuCgdSfaOVIfa7nZDKe8hRPudKzTYpWZ4DBVR5tXXrXYMpMdDlqvWo4Fj31SS/Eip6apuaPfqYd5+hwMlo7Km1Z2iDB2rxDnetzGcT55i1JEnE4HMTj8ZBFixaRlpaWkyr3fBwLxslzvs4zW/eM02HE2jgtX74cX//613HHHXdg0qRJuPfee7Fw4UL87Gc/o2lKSkpw1113YcGCBcjNzUV7ezvWr18PQRBOu/5QKASTyQSr1QpJkhAOh6mK4vF4UFlZiY0bN8LpdMJsNiM1NRUmkwmtra0oKytDIBBAVVUVvF4vfD4fPaEniiINIaLaO6k2OR0dHTAajViwYEE/O6vU1FRYLBZwHIeenh4UFBSgtrYWtbW1KCgoQCAQoKoScOKbmRoiRg2JIggCVcY8Hg/y8/Np6BOfz0dDyKh9NZlMdCzVUCrFxcWora3tF+g4FArB6XQiFAph0qRJ6Orqgs1mgyzLcDgc1Earvr6eBjYuKSlBKBSC2WymgY1TUlKQmpqK7OxsrFu3DmvWrAHP85AkCSUlJXScenp60NHRAY7jkJiYCLPZDFmWaciZcDj8pT1VV1BQgMrKSvz1r3/FL37xC2zatAlXX301ent74+YZymlSAEM6kRgvzcneP9VnZ6KNJ5Pmi+zP6fTjywZb92dv3Q/3e2K4x+YLZdi3Yuchg3mSjbb50bO10Qa6Ve9Bo+5og+Fq7Z0KCwupcsLzfL+26J2001Oj9NJEn1yL/n90H9U8PM/HLU+vzdrxilaU1LK0dlTafHp907NfUtPrtU3NFz1fX4ZTddD55q1Ftfd7880346YZymlSQk7PU/Jwewo+lfLOZN6z0Z/T6Qch5+eaJ4St+3Nt3Q/3e2K4x0bLiHWA+UURb4BLS0tJQkICmTNnDsnIyCBut5uIokhDrvj9fpKWlkbDhzidTmI0GonNZiNer5e43W5iNBqJ0WgkM2fOJBaLhbjdbuL3+4nP5yMZGRk0dIgkScTr9RKbzUYAkIsuuogQ8h8vqhUVFURRlJjwJoFAgMyfP5+YTCYiiiKtXw0LA4DYbDZSWlpKvF4vMZvNxOv10tAuakgTNayM1+ulbfT5fEQURWI0GkliYiIBQNLT0/vVHx1SxmQyEUmSaFmCINCxCwQCZObMmQQASUlJoX13uVxkzpw5xGKx0DA0oigSu91O5s+fTyRJIjzP0zFVvbU7HA66UbJYLITneWKxWIjZbCZz5swhdrud2O32GCekA831ucxQPkAIOeHpfuXKlXGfd3Z2kra2Nno1NDTojsXpeEoebk/BZ8Kb83B4gv4i+3O6ntzPxzVPCFv359q6H+73xHCPjRa2cTrDDGbjpNrVaFWkeGqU3gk5VVFRVRP1mZ7na7UMQoau7ESrXNF+pNTXWh9P0UpTvAC+0QqUVkWL7gshRHdsouuJbmu0zZS2/IHaGe9k42A2UkOZ63OZoXyANDQ0EI7jyNq1a4dc7vk4FoyT53ydZ7buGafDiD1Vd7YJhUJ45JFHEAwGUV1djYSEBOo/KCcnB83NzcjJyUFXVxccDgc+++wzHDt2DD09PUhOTkZOTg72798Pg8GAa6+9FtXV1ZBlGZWVlTAajTAYDEhNTUVHRwc6OzthNpvR29uLnp4eFBUVURuqSZMmIRwOIxAIQJIkBAIBOJ1OeDweyLKMcDgMp9MJi8WChoYG9PT0gOM48DwPAOjp6UFqaioyMzORmZlJfRtFIhE0NjbC4XDA4/GgoKAAbW1tyM/PRyAQwLJly9DR0YH09HT09fVh9uzZWLNmDTo7O2E0GqnfJUmSEAwGsXbtWmp/pZ5qmzx5Mnbv3o1169ahp6cHBoMBEyZMwI4dO8BxHCwWC66++mpqm9XV1QUAyMrKQjgcBgDk5+ejsLAQK1asoKcAFy1ahKqqKng8Hqxbtw4mk4m2c+HChdRz/Pnqx6m9vR07d+6kr3fv3o0PPvgAqampSE1NRVlZGW666SZkZGRg165dKCkpwfjx48/rk4QMBlv3jPOCYd+KnYecjI3TULxvA/0VIG15eunUS2uXpFVi9NQuPT9I0WqWnvqi2hbp2QVF2x1p1R+959o+Rdc3WH/V8rRptKqRnvoWXfZgv3kPNNfnGm+99ZauEjlv3jzS0dFBgsEgcTgcxGQyEZfLRebPn08OHjx4UnWcL2PBOD3Op3lm654xXDDF6Szh8XhgNpuRlZWFxsZGcByHYDCIo0ePQhRFvPDCC5BlGePHj6eKyLZt29DT0wNBEPC1r30Nf/zjH9Ha2oopU6ZAFEXqu0kURXR2dtITez09PRgzZgwcDgeOHj2KjRs3wu/3o7m5GUVFRaivr0d9fT26u7thNBpjVJi9e/ciGAxSNai3txd9fX245ppr8Pe//x2dnZ3gOA5VVVWorKzErbfeitraWtTV1YHneZhMJrS0tFBfVZFIBIFAAElJSThy5Ag4jkN1dTUcDkeMN+5IJIK2tjZIkoTCwkI8+uij+Pjjj2GxWDBlyhR4PB5kZWUhKyuL+mmaNm0azGYzNm3aBJPJBIvFgnXr1mHs2LHwer3YuXMnuru7MXHiRGzduhVWqxUej4f6idq9ezc2btyIsrIybN++HaFQCCtXroTD4UA4HEZBQQHGjh2LF198ESUlJXjwwQfP8io6Na644goQQuI+r6qq+gJbw2B8MbB1zzgvGPat2HlIvJ2pLMv9vvmYzWYiyzLx+Xwxyo5q/O3z+UggECAcxxFFUfqVEc+mCQBJS0sjNTU1/fIEAoF+qo3P5yMmk4mWJwhCP8XJZrMRu93eTwmaP3++bt+iL0mS+t3TU61qamqI3+/vNyYul6tfHS6Xi9TU1BCDwaBbp14edVw4jiM+n4/mFQSBGAwGIggCcbvdxGQy9ctnsViGPNcjETYWIwM2z7Gw8RgZjFg/Tmcbl8sV89pgMKC7uxtNTU3Yvn07vd/b24tNmzahq6sL9fX12L17d9wyov2NmM1m+P1+mM1mAEBLSwuWLVvWL8/u3buxf/9+mEwmAIDdbsf27dtx/PhxWl5nZydGjRoFv98PjuMAnLAXiEQiMWURQvDss8+iqalpwL53dHTQ/6vlZWRk9LMZWrZsGd57772YMREEAfv374fL5aJ2VjzPo7y8HMuWLaNevqPLNpvN/fJEtxkAtm/fTvOOHz8egiCgs7MTu3btwvHjx/v1YTh8eTEYDAaDEQ3bOA3A7bffDlEUwXEc3G43LrvsMgAnNgG33norCgsLMX/+fCQkJGDmzJmwWCxwu90QRREWiwVWqxXbtm0Dz/Ow2WxwOp0QBAEzZ85EQkICrrnmGjQ1NeGaa66BxWKBz+dDcXExli9fDr/fTx1QiqKIWbNm4be//S0URcH06dNx6623UuecVqsVZrMZixYtwvLly5GdnQ2O4+B0OuH3++F2u2E2m2Gz2QAAU6dOhd1uhyiKkCQJbrcbsixj5syZ4DgOVqsVF1xwAfx+PxRFwYMPPgiXy4WlS5di+vTpAICysjKIooiOjg7Y7XZIkoQJEyZAURQ8/fTTuOSSS1BfX4++vj5wHIfrrruOGnPb7XYIggCTyUQDEv/mN7/BrFmz0N7eTp85HA6YzWYIgoDU1FSMHj2aBg92Op0oKSmBKIp0vqxWKxwOB4ATG7K5c+d+wSuGwWAwGF96hl3DOg8ZyDgcUcbK0T+zDeQETE0TnT7a2Dmegbm2TD3DZ62BdHR90Y42BzPOjmdsPpCLguj2RTu41MujV6fWrYBav15f9C49I3U9dwzR4zHUuR6JnOxYDNV/yrnIUNs+HD6ezgYDtYmt+VjYuj+1dOfiOJytdc+Mw+NQW1uLSCQCp9OJAwcOgOM4zJgxA3//+98xbtw4KIoSE9i2uLgY06dPR3FxMerq6rB9+3Y4HA60t7fDZDLhxhtvxJo1awCcCER76NAhpKSkQJIk5OXl4Y033oAoirTMVatWwWg0wmw244ILLqA/kan/qobXwWAQdrsdPT09kGU5JiSMmra8vJwGK+Y4DoQQagiemZmJQCCA5557jhqYHzp0CDzPw+PxAADtY1VVFRYsWIA///nPSElJQV9fH771rW/h6NGjMQF+i4qKkJWVhb179wIATCYTioqKcPToUXg8HqxcuRKiKKK1tRWZmZmoq6tDcnIyjh07BuBEqBg1NIzalt7eXmRkZKCqqgqZmZnIzc2lYWE6OjrA8zxyc3Px7rvv0kDGoVDoC10zX3aWLVuGl19+GQDw4osvnuXWnBxDbftQ0p2L43AutunLwvk8tmzdnyGGfSt2HqK3M9VzHRCtYMRzIKk+0ztKHy/0iNbRpp7DSi16eaJVLK3aorZXL+xJPDcJeiFOBlPd1LKinVvGc7AZz41DtIoVrU5F163nfmEwBS/eXI9U2DfvU0t3Lo4DU5yGDlv3p5buXByHs7XuOUIGOPs5Qjh8+DAkSUJbWxuSkpIAnFCcHnjgAbS1taG5uRkNDQ3weDx47rnnMH36dFRWVmLx4sUIBoN44403EAwG8ec//xkOhwMdHR1oaGhAeno6enp6sGjRIqxfvx719fVU9ent7UV2djb+8Ic/YN26ddTR5pYtWxAKhbB+/Xps27YNRqMRjz/+OADgvvvuo2rX8uXL0dPTA5vNhubmZkiShCNHjmDy5Mn44IMPIMsyxowZAwDUXcCqVavQ2NiI3bt3o7OzE4IgYMqUKdSZpKIoWLFiBT744AN0dXXBYrHA6XTi4MGDOHbsGAwGA2w2G9ra2iAIAqZOnYqCggKqIHV0dGDhwoXYvn07RFHEiy++SJUmVZHSu//oo4/io48+ovZQhBDwPI+srCyq2kUiEcyePZuOMSEETU1NSE5OxsGDB3H8+HEEAgHU1NTQcSwvL8ett9466FyPVNhYjAzYPMfCxmNkcEbnedi3YuchJ+MAM55yorWvAfRDiURfeo4xtQ4mtc4woVFW4tUZfWltg/QcUEajpwLpXQMF4tWirVur3sWrI56qpTc2WvWNKU4Dw8ZiZMDmORY2HiMD5o7gLFFcXIy5c+ciGAzCbDYjOzsbkUgEtbW1UBQFDocDRqMRDocDRUVFkCSJvjaZTNS5pCzLyMvLg9frhSiK9JSdLMsoKyujYVUSExNp+JO6ujqYTCZ4vV4UFxejvLycnl5LSUmByWSip/0kSYIgCDCbzbjuuuuQkZEBp9MJs9mMhIQEEELw+uuv07Amqm2VGr5F62JArctqtcJgMMBgMIDneVitVlgsFnpiLxgMoqioKMZ9QE9PD2pra+nr2tpaFBQUIBwOIz8/H0VFRTCZTNi/fz/Gjx+PQCCAuro6eoJQPflnsVho22pra1FUVIQFCxZAFEWIogiv1wuTyYT6+noEg0EkJCSgqKgI+fn5kGU5xlEng8FgMBjDxrBvxc5DBtuZahWNeCFJotNBR/EhJL5SpT11Bx0FJ7rO6OdDOUmnvYYSnmSgMvTsn/TK1rZtsJNz2vGKHrNoBU87RnpBhU9lrkcSbCxGBmyeY2HjMTJgp+rOErW1tVi2bBlCoRCefvppGuR37dq1MBqN8Hq96OjoQGNjIzweDxRFoXZPL7/8MvVJpJ4GmzJlCjo6OmA0GhEMBrFv3z7s3LkTjY2NKCoqwnvvvYcdO3YgGAyivr6ehlLJyspCKBRCOBymwXoJIYhEIgiFQti8eTPy8vIgSRIURUEgEMChQ4fAcRwEQcCxY8eQnp6O9PR0ahcUHbZEPcXX2NiISCRCg+Sqp/rUUDAA0NzcjMbGRoRCIRq2xWAwoK+vj9olRSIRLFiwAJWVlUhLS6NtDofDdEzUwMhdXV3o6ekBABiNRqSlpeHAgQNYtWoVqqursXjxYlRXV6Ovrw8WiwXHjh2j9lTACWVKVfk8Hg/Wr18PSZIQDodRW1tL/U4xGAwGgzEsDPtW7DxkMBsnrcKEKKVDVT7iKUN6dkBq+ni2OtFlRatYWlVJ20b1tbaueDZM8Xwraf08xatPW1e04qMNMKznvynePe04afujd2nLi3cakX3b/A9sLEYGbJ5jYeMxMmA2TmeB2tpahMNhSJKEuro6bNq0CQDgcDggSRJsNhva29uRk5ODhIQEBINBFBQUwOfzYd26dTSMiiiK1EZKFEVqk9Ta2opNmzaB53mIoohwOExtdaLLam1tjbFrUm2eysrKIAgCNmzYAKPRCFEUUVBQgOrqavA8T+2S3G43qqurqZfx5ORkquykpKTAYDCgo6MDJpMJTqcTsiyjpKSEKlipqakQRRHJyckQBAEdHR00oK6iKPB6veB5ngYCVhQF+fn5SE1NBQCkp6fD7/cjMTERSUlJ1FN5amoqjEYjkpKSqN1SUlISVbkAYPTo0QiFQpAkCTzPQxAEml8QBHAch/T0dNjtdni9XsiyDK/XS9utKMrZWTwMBoPB+PIy7Fux85B4fpz0TsIlJCQQQgZXbaBRTeKdkINGMYmnAEWXRQjpp2LpndzTU3Ciy9G7p+2/VjHjeX5AmyXtvWhlLbpN8cYquu3RCla8E4fRYxbvBORgcz1SYWMxMmDzHAsbj5EBs3E6CxQXFyMSiWDnzp1oaGhAQkICDh8+DFmWUVtbi/LycixevJjaGLW1tcFoNCIcDlM/RABACEFDQwM99aYqS6r9T3t7OwBg7NixVHXZt28fEhIS0NPTg+7ubqSkpECWZTQ0NCAxMRFlZWU02G1KSgp4nkdGRgaOHTuGPXv2ADgRkLevrw85OTnYv38/ent7qT1QT08P9u7dC57n0dfXB6vVio6ODvT09KCyshKrVq1CW1sbVZ3Gjh2L//mf/0FfXx9SU1OpaqaeWtuwYQOam5uRnp5O7aW8Xi8OHjwIo9FIvX0fOnQIgiDAaDQiNTUVgiCgubkZoiji888/px7Dx44dC5vNhkgkgkAgAKfTiZycHDQ3N1Nv5iaTCWazGYmJicjPz6fez9X5UOeQwWAwGIxhZdi3Yuch8XamFRUVur6YzGYzCQQCpKKighQWFhJFUYjRaCSCIAzJFkeNexetqKSlpdHnevlMJpPu/YSEBKIoCjEYDDF5JUkiHMcRg8FA7ymKQux2+4DtU9uhliHLMgkEAjFpFEUhhYWFpKKigiiKQus1GAx0LAKBwKDjEd02bfkul4vwPE+VJLUsWZaJLMsx46mmNxgMxG63E1mWSUVFxUnN9UiEjcXIgM1zLGw8RgbnpOLU0tKCPXv2gOM4ZGVlwW63n2pR5yyLFy9Gb28veJ5HSUkJVq5ciYMHD6K7uxubN2/GgQMH0NjYiPz8fMydOxd1dXXYunVr3PLsdjvMZjMOHTqEMWPGxCgiVVVVAE4oVACoMqRy/Phx3TI7OzsBgNofqXR0dIAQQstT5ycSicTkV0/EqZAoR/JtbW1oa2sDIQR+vx87duygis/LL7+Md955B/v27aN51HLmzp2LjRs30rZp61DRu2cwnDC7279/P0aPHo3y8nJUVVXRsXW5XACApqYmACeUNTW9IAi0f4sXL+7nNZzBYDAYjNPlpI3D//3vf+Pyyy/HqFGjkJeXh0suuQTp6en46le/ik8++eRMtPGsUV5eDpfLhWeffRbXXnstpk6dSo2P7XY7QqEQcnNzUVtbi+rqajQ0NCAtLQ1z5syByWSCKIrw+Xzw+Xyw2+2QZRkulwt5eXmYOHEiPvroIzzwwAOorq5GW1sbRFFEeno6DAYDvvKVr8But9OfttxuN3w+H4zGE3tdi8WC0tJSXHXVVQCAYDAIg8GA5ORkmM1mjB07Fn6/H06nEwaDAV/72tdQWFhIDa3V8CmyLMNoNMJkMsFgMMDr9cJut0MURVgsFhgMBlx33XVYvnw5FEXBk08+icLCQjgcDiQmJlLHnqpDzcbGRhp812g0wmazYdy4cRBFERzHAQA16I523slxHDiOw/XXX4+2tjakp6cjFArRAMqBQAA8z8Pn82H58uXw+XwQBAGzZ8+m6UtKSuD1eulPeNGOOBkMBoPBGBZORp5qamoidrudTJw4kTzxxBNk/fr15PXXXye//OUvycSJE4nD4SCHDh0adlnsTDMUSU97DF/9fzwja62LAK1xdDwjcNUQWzXCVvPohV2Jblc8p5vRDjbjGbxHX9H1RvdHzzWDXhDjgcrWjs9ALgqix0nbD0Liu2nQa9vJzvVIgY3FyIDNcyxsPEYG54w7gmXLlsHlcmHLli34/ve/D0VRUFBQgB/84Ad4//33MWbMGCxbtuxkijxnUUOFFBQU0BArkiShqqoKHo8Hc+fOhaIoSExMhNFopIbXPp8PoVAITqcTiqJQtwapqanw+Xzw+/0wGo0wGAwwGo3Iy8ujx+uLioogyzIMBgMyMjLoM4vFAo/Hg5ycHNo+Vc1at24dkpOTY1wepKeng+M4dHR00PKCwSAikUg/hUgNzZKcnEzrVVUnp9MJo9GIxsZGiKJI+xwOh5GUlISkpCTqaNLj8cBsNsPpdMJut9O+qpfb7YbZbIbP50N5eTlyc3PR2tqKhIQEWK1WGI1GZGdn0/A0wWAwZhxLSkpoWJWCggIkJiZCkiRwHIekpCT4/X6IoohXXnkFgiDEtI3BYDAYjGHjZHZZOTk55KWXXor7/Pe//z3Jyck57d3cF81A7gj0jt1HhxvRKiZaRWqwkCN64Umiy4p2iqnnSDO6TWo9qkF7tOsAPRUs3jH+6Htq+dEKl16fte4ZtGqP1lGnnlKnzasXPiWem4RoVUrbtsHmeqTCxmJkwOY5FjYeI4NzRnEKh8O46KKL4j6fPn06wuHwyRQ5KI2NjfjmN79Jg876fL4YFYEQgsWLF0OWZVitVsyaNQs7duw47XoVRcGoUaPg9/uhKAoikQh1shgKhVBUVARFURAMBuH3++Hz+ZCRkQFFUaj6IooizZeamkptf1QD6PT0dNTV1cFms0EQBDQ2NoLnefA8j+zsbITDYRiNRgiCgJycHGojZLVaqX2QKIqw2WxoaWlBQkICDSKs2gypoWGCwSCMRiPq6+tRWVmJcDgMh8OBUChEA+N6vV5kZGTQe4mJiRAEAQaDAUVFRaitraWqlaogqe1MTEykzjNzc3OxceNGVFZWxoyn0+mEx+NBUVFRjGNLi8UCk8mElJQUOByOGKVOfa2ijq3X60VJSQkyMjLg8/kQDochyzLMZjPcbjf8fj+CwSBzScBgMBiM4eWkdlkGw4A2TAcPHiQ8z5/uZo7y2WefEZfLRb797W+T6upqEg6HSVVVFdm5cydNs3TpUiJJEnnttdfIhx9+SK6//noybtw4cuzYsSHXE09xihc4Np4SorV70jrH1HMmCY1Cgij1JV7YFq1jSGBwmyJtUFytChbPySSgr6QNFApGL3SMXh3xHIEOZKsUr+yB7JoGm+uRChuLkQGb51jYeIwMzil3BEeOHIEgCLrPDh8+HHOc/XT5xS9+gTFjxuC5556j98aNG0f/TwjBE088gQceeAA33HADAKCyshKjRo3Ca6+9hptvvvmU61aVimjFQlEUFBUVoaOjA319fdi0aRMCgQDa29uxe/duEEKwevVqHD9+HBzHITc3F3v27EFOTg4OHjyI0aNHo7Ozkwb6DYVCqK+vx/vvv4/u7m4YjUYcP36c2iS99dZbaG9vh9VqhSRJ6O7uhiAIqK6uRktLC0RRRGJiIrxeLxoaGgAAaWlpaGlpQVdXF7q7u6ktFQAamiUnJwdNTU3UIafP54PZbKY2VIqiYPLkyaivr8cnn3yCCy64gDq2zM/PR3FxMTweD5YuXYqUlBRYrVZkZmairq4OmZmZkGUZ+/btQ1ZWFgoKCgAAhYWFAE4oRs899xw8Hg/q6uroiT6z2Yy0tDS0traiuroara2tyM7ORkdHB1WpFEWBLMtobW3FoUOHYDabMXHiRNx7770Ih8PYsWMH6urqEAgEIEkSlixZwoL8MhgMBmN4OZldlupQMd6lPh8uJk2aRO655x7y9a9/nTgcDjJt2jTyzDPP0Oe7du0iAMiWLVti8l1++eXk7rvvHnI9Q92Z6gWwHehSFafoE2GIUn1U4p3MG0odA9lVxUsf3Y7o11obo2h1R091054UjC4T6B+eJXoMBzpRpx0/bVgVrS2TXp/ZqbrBGepY1NTUkMLCQlJTU3NaaU61jlN9NtDzU813Ku0fjnadzviyNR/LubLuT2cNngvr/lT6/mVZ9ye1cfrb3/42pGu4sFgsxGKxkPvvv5+8//775OmnnyaCIJDnn3+eEELIO++8QwCQAwcOxOT7xje+QebOnRu33M7OTtLW1kavhoYG3QGuqKggLpeLVFRUkJqaGuL1eokgCOSiiy4iHMcRq9VKfD4f8Xq9RBTFGO/dRqORzJ8/nwQCASKKIjGZTMTpdBJJkogkScTpdBKj0UgkSSLz588ndrudGI1GApzwqD1nzhwiSRLheZ4YDAYiiiLx+XzE7/cTr9dL7HY78fl8RFEUUlFRQfx+P7Hb7cTv91OP3m63m+a3Wq1EkiQSCARIaWkpycjIIE6nk5jNZuJ0OklaWhpxu920jJqaGlJRUUHb4HQ6id1up2mi88qyTObPn0/H4IILLiAGg4E4nU7idruJIAiktLSUEEJIaWkpSUhIIHPmzKFt4HmemEwmMn/+fNpuQRDoa7/fTwKBAPH7/URRFDJ//nzaL57nic/nixkDp9MZU2c07EPkP5zsF4aBfgYdSppTreNUnw30/FTznUr7h6NdpzO+bM3Hcq6s+9NZg+fCuj+Vvn9Z1v05HXLFZDKRQCAQc++uu+4ifr+fEHLqG6cHH3xQV43RDrBWcYlWQgbzYxQvoK9e4F1tINzoOrTKih569kjR7Y+nwujZPWnTDXSST6sK6Y2Pml9V2LT2X3oqnJ59mdb+SptX7360qqfCPkT+w7nyzXuw/Of6N+/hTH++ffM+HzlX1j1TnE6/vQNxJtc9R8jQjZIOHz48pHRJSUlDLXJAXC4XrrrqKvz2t7+l91asWIElS5ZQD9VutxtbtmzBtGnTaJqZM2di2rRp+NWvfqVbbldXF7q6uujrw4cPY8yYMWhra6Ntr62txc0334xwOAye50EIgdFoBCEE8+bNQ319PXbs2IEJEyagoKAAK1euhCiKaGlpAQBMnjwZixYtwooVK7Bt2zYaPkX1pn3gwAEQQmAwGDBu3DhYrVY0NDSgq6sLfX19mDdvHt577z189NFHtLybbrqJBrLdvn07PB4PVq5cCYfDAQA4cOAAMjMzQQhBU1MTkpOTcfDgQXR1dWHUqFHo7OzEhAkTcPvtt6OqqgodHR1Yu3YtOI7DuHHjkJCQgHA4jO7ubkycOBHPPvssHnroIbz22mvgOA7Z2dmwWq1oamqCIAg4dOgQCCGYPHky7r33Xjz66KPYuXMnvvWtb2Hfvn0AgEAggKeeegrHjx/H5MmToSgKfT127FgQQrB7924cP34cP/7xjzFu3Djcd999GDduHBYtWoRVq1bh8OHD+PTTT9HQ0IAxY8bA4XBAkiQEAgEsX74cXV1dsFgsuOuuu7B9+3aIoogXX3wRJSUlePDBB2Pm/vDhw5AkKWauRypsLEYGbJ5jYeMxMjij83xSu6wv2MbplltuITNmzIi5d88991AVqq+vj2RkZJDHHnuMPm9rayMWi4X8/ve/H3I98U7VYRCbIq2qoufFWs+eZ7BLT62Kfh1PrdE7ZTeQnyatmqTNq22D9iSbel/rxVyrjA3WD72TdtF2VQMpZ9Ht0FOshjLXIxU2FiMDNs+xsPEYGZwzp+reeuutk0l+2hQXF+PSSy/Fz3/+c8ydOxf//Oc/8cwzz+CZZ54BAHAch3vuuQdLlizBhAkTMG7cOPz0pz9FZmYm5syZc9p1V1dXY9euXeB5Hn19fXA4HOB5nvoV2rhxI8aNGwdFUagStH79euzduxcejwebN29Gfn4+VZOSk5Nx9OhROBwOevJNPVFmNBoxduxYGtMtHA5TlYrjOIiiiFAohKVLl0KWZaSnp+PTTz/F3r176bPt27dDURQ8+uij2L59Oy644IIYlWrz5s0ATpxwUxWn1157DQAwevRoyLKMxMRENDU1ITMzE5FIBMFgEM8//zwSEhKo/6ri4mIUFxejrq4OO3bsgCzLKCgoQGFhISKRCCKRCCorK1FVVUVPwrW0tKC7uxubNm1Ce3s70tLScOedd2Lz5s1oa2vDxIkTY07rGY1G1NXVweFwIBgM4uWXX8bRo0dpkN+XXnoJ69atw7Rp08BxHMxmM9rb21FWVoZIJEJP/zEYDAaDMawM+1ZsmPnzn/9MvF4vsVgsZOLEiTGn6gg5oTr99Kc/JaNGjSIWi4Xk5+eTTz755KTqiLczlWWZqhkGg4HIskw4jiOyLBNFUYjRaCSKohBFUajBdlpaGuF5nqYNBAL9TuEJgkAMBkM/FUWSJGI2m4kkSboq1OjRo+lrs9kc81ySJGo87Xa7CQDidrtj2hH9W3FNTQ0RBKFfmwRBIIFAgPh8PgKAWCwWYjAYiMvliumz3+8nsiyTQCBAlSxFUYjL5aJG22p5HMfF1KXWpxpzAyCyLJOamhoSCAQGtAkzmUzEZDINqNbp9XewuR6JsLEYGbB5joWNx8jgnLFxiqavrw87d+7Ep59+ir6+vphnl19++akUedaI91toIBDAe++9BwBISEjAlClT6GtFUWC32xGJRPDGG2/E+K9yuVyQZRnV1dUYNWoUDh06BAAghIDnefT29sbUb7fbYTab0dTURO9ZLJYYO6yhonoM7+vrg8FgoHOTkZGBlpYWzJ07Fy+++CKKioqwatUqms/r9SIcDqOjo6Nf/QkJCejq6sKsWbNon6uqqmi5U6dOpeW8+eabAIDe3t6Yvnq9XuzYsQNdXV0x7YqmsLAQGzduxMGDB2E2m3HRRRdBkiQUFhYiFArRPHpjqEXbXxVm3/Af2FiMDNg8x8LGY2RwJuf5pB1gAsB7772HwsJC7N27t5/DS47jBv1QO19Yvnw5QqEQPvnkE5hMJhQUFKC5uRm7d+/G2LFjMWPGDNx99910s2K1WjFlyhQ8+eST+Oijj9DU1ER/ImtrawPHcXS8VOPmp59+Gna7HTabDQaDAQcOHIAgCHC73bDZbOA4Dtu2bUNraytcLheamppgMpkgiiI+/fRTmEwm9PX1gRACq9VKnUY2NjZSI+0DBw5g3759EEUR4XAYCxYswCuvvAKLxYLe3l5kZGSgqakJsizjwIEDOHbsGDWG7+npAc/zSE9Pp04s7777bvA8D7PZjOPHjyMQCGDz5s1obGxEcnIyBEFAc3MzxowZAwBoaGgAx3F45plnsGrVKrS1taG5uRkNDQ1wOBxob2+H0Wik4WoeeeQRlJSU4Nprr8WyZcswefJkfOc738Fvf/tbiKKIW265BX/+85/R29uL5uZmOv7qT3aqM82pU6eyn+sYDAaDMayc0sZp0aJFmD59OtatWwdZlmkMtS8b06dPx5QpU7B161a0tbXhueeeQ3NzM/r6+vDiiy/ijTfeQFtbG02flpaGd999FwCwbNkyNDY2Yvv27bDb7fjrX/8Kp9OJxsbGGNXn4MGDaGpqAsdxGDt2LADg2LFj+Pe//41bbrkFAKht0v79+/Ff//VfePnll9HZ2QkAOH78OK3/6NGjOHz4MK1DtQXLysqi7ayurkZNTU3M5rapqQm9vb2IRCLgeR4A0N3dTZ8fOXIE7e3tqKqqwjvvvEPL6u7uxrFjx/DII4/QzVY04XAYVqsV3d3dqK+vR1VVFR0Ls9mM7u5ufP755zh27BgAUBWru7sb27dvx/bt2/Hyyy/T8niex/XXX4+jR4+ipaUFZrMZwAklb+zYsdi3bx+OHTuGY8eO4fDhw5gyZQrzHM5gMBiMYeWUNk47duzA6tWrMX78+OFuzzlFbW0tqqurAZz40O7p6UFfXx84joMsy/jqV7+K559/nv70lJiYiAULFuDFF19Ebm4uDfS7b98+5Obm4siRIzRobWVlJdauXQtCCEwmE4ATrgry8vLQ3NyMxsZGmjc5ORmtra1ITExEXV0dRFGkP3mlp6ejvb0dnZ2d6O7uRlZWFoAToU0CgQB27NgBQRCoS4XExERceeWVeP3115GUlIS2tjaq+gCA2WxGc3MzeJ6H1WpFe3s70tPTkZiYiFdffRVXX3012tvbwXEcbrzxRrzxxhsxqhpwYgO3b98+GI1GfO1rX8OqVavQ0dGB6upqLF68mLYv2mj98OHDCIfDaG9vhyAIeOmllyAIAlwuF6qqqmiw4fr6etx77720jGXLluHYsWMIBoOor69HfX09urq6kJGRQQMN33rrrV/ksmEwGAzGl5lTMYy68soryeuvvz4cNlbnBPGMyOK5JECU0bL2qL9qCK66H9AL9KvnGFO9BnM7EH1FH9lX6+V5ftDgv1qnnXrhTtQ+aB1hqq4HBiK63MLCwn6OKgfKo22rXngbPSeeWielaj6tE0xmGPof2FiMDNg8x8LGY2RwzrgjULnrrrvw//7f/8PBgwfh8/moYqIyZcqUUyn2nKK2thaRSARutxv79u2DwWBAT08PBEFAdnY2gBMOJydNmgRRFJGUlIQDBw7AYDAgEokgKSkJHR0dkGUZ48ePx9ixY1FZWQmPx4Pi4mIoioK7774bHR0dVDUCQAP4WiwWBINBVFdXw2AwoLu7GyaTibZDDQqsKAoikQi2bt2KAwcOoK+vD0lJSQgGg1i1ahWOHj0Kk8mEnp4eapwuiiICgQC2bduGhIQEcByH1tZWeL1eiKIYoxipAXaLiorwwgsvQJZlGkRXdUDZ09ODsWPHIjMzE0uWLIHH46EuHNatW4fc3Fy8/fbbEAQBdXV18Pl8NCixzWaj9lDZ2dloaGiAxWLBkSNHqN2SCs/zmDx5MjweD5KTk9HZ2YkLL7wQDQ0NdCzC4TD27t2L2bNn44033kB5efkXv3gYDAaD8eXlVHZbqpoQfZ0JB5hfFPEcYEaH+IDGyaKeE0c9xUSrMOmFAVHr0+YfiuPMaGec2rzx8gykYEW3JVpV09ajDRas19fo+obiAFQtb6Agytryo8sdSjwj9m3zP7CxGBmweY6FjcfI4JxTnHbv3n0q2c4r1NNYqh2NwWCAwWBAYmIiamtrUV5ejh/96EcwGAz49NNPcejQISQlJSEtLQ0tLS3o6emhTi3r6uqoipSTkwNZlqnjTDVkCiEEqamp9ESa6rxy5cqV6OvrQ3NzMzIzM5GamkrDmUyePJm2M1p1ys7OxuLFi/Hoo4/i448/BiEEvb29MJlMsFgs4HkebrebtrO3txd9fX0IhUIATrhaqKqqQldXF1pbWzFp0iSEw2EAQF5eHjXOlmUZ77//PjiOo+4aFEVBXV0dtftST/qpTj27urqQkpKCzz//HDzPU5cHFosFRUVFeOONNxAMBvHnP/+ZhrDp6upCT08PJk2aRMvneR4cx0EQBLS3t8NgMKC6uhqBQADt7e2IRCJYunQps29iMBgMxvAy7Fux85CBdqZDDfehXtpguXrqi97/o9Nrg9mqCkxCQsJJRczWCxystQnSC+wbL+CwNpRMtDKkDUUTnVcvCLCe7dZQInZr1TDt+Gvn4mTmeqTBxmJkwOY5FjYeI4MzOc+GU91w7dq1C3fddRdmzZqFWbNm4e6778auXbtOtbhzktraWsiyDEmSIEkSfD4f/H4/IpEIamtroSgKJEmCwWAAz/OQJAnBYBCjRo2Cz+eD1+ulzi95nofP50NJSQkyMjIQCAQQDAbpyThJkpCamopQKIT8/HyEw2EacmTGjBngeR7BYBCRSASTJk2iJ8bUdhYVFUEURRiNRtTX19P2qSqWyWSCLMvo6emBwWCgAX6NRiPS09NhsVjg8XgAnFCcRo0aBafTCUEQUFRUhGAwiGAwiI6ODvzhD39AYmIidUhpsViQmpoKRVFQXFyMYDAIt9sNs9kMWZbhcrmoLyej0UjDyvT19dETfwaDAWvXrsWll16KyspKFBUVoba2FsAJ9W/u3LnUhikhIQEmk4nWE43BYIDT6YQsy8y+icFgMBjDz6nsttavX0/MZjO55JJLSHFxMSkuLiaXXHIJsVgs5I033hjuzd0ZZ6BTdVp1R0/9QBzFSC9objTRdlJaRUerWkUHw1WVGm1QW+3ptcFOzamX9gRadHBdbf/j2R9FK1ZqGdqThnqXy+WKG5hYO17a8daepIu+f7Jzfa6xadMmMnv2bBr2549//GPMczXUUEZGBhEEgeTn55Pt27efVB3ny1gwTo/zaZ7ZumcMF+ecjdOPfvQjFBcXY+nSpf3u33fffbjqqqtOpdhzitraWoTDYUiShOrqamRmZmLp0qXweDwwm83weDy49tprEYlEqP8iNTyIGtxW9QEFAG63Gx6PB1lZWdR3UWJiItLS0iBJElpbW+HxeODxeLBy5Up6wo3jOBw5cgSSJMFoNEKSJPT29qKjowNZWVkoKChAY2MjHA4H8vLy8Je//IUGBa6vr0dqaio8Hg+8Xi8qKirAcRx6enpgMpnA8zyMRiNsNhsikQi1v3I4HMjPz4/pSyAQwN69ezF+/Hh88sknsFqtNAjx8ePHIYoiFEVBbW0tli1bBo/Hg9TUVPT09MBms+HQoUM0oLH6WhRFBINBrFmzBoIg4Pjx47DZbHR8wuEwAoFAzLx4vV7s27cPvb29aG9vhyiKsNvtOHr0KLq7u+kJwfPdf9PRo0cxdepU3Hbbbfja177W7/kjjzyCJ598EhUVFTS4taIo+OijjyAIwlloMYNx+rB1zzgvOJXdlsVi0d3lf/LJJ8RisZz2bu6LJt6pOr2TaoOdjovOH51X659JqwTFO30Xz78R/k8pimdPpVWx4tlqRStD8fwt6flHUtuo7WO0WqW1n9JTsoaiukW3Wau6qfZS8U4/DmWuz3Wg+ebd19dHMjIyyKOPPkrvtba2EovFQn7/+98PudzzcSwYJ8/5Os9s3TNOh3NOcXI4HPjggw8wYcKEmPsffPAB0tPTT6XIc47i4mKEw2Hqebu5uRmJiYkIBAJ44YUXYDQaUVBQgEAggJUrV+qeklM9fLtcLkQiEYRCITz33HP01Fj0abpo/06LFy9GKBRCIBBAfX09DAYDBEGggWvVU2ZGo5H6S+rt7UVWVhYaGxsBALIsIyEhAQcOHEA4HEYoFMLKlSshiiJVt6JP5AHA2LFjUVFRAbPZjHA4jNraWqxbtw6vvvoqnE4nurq6wHEcjQ+XlZWFlpYWdHR0IDs7OyYunKIoWLFiBXbs2IHMzExEIhEEAgE4nc4YRcloNFJbKABobGyEx+PBuHHjsHHjRjgcDupb6sCBA9RG6tixY1RZkyQJHo8HmzdvhtfrxcGDByHLMmpra7+UIVd2796NgwcPYtasWfSeJEnIy8vD5s2bcfPNN5/F1jEYZwa27hnnDKey2yorKyPJyclk6dKl5O233yZvv/02efjhh0lycjIpLy8f3q3dF0C8nWkgECAAiMVioWqI+tu7ehkMhhhFJPpZdBqDwUBVHEVRCMdxJC0tjaYxm80kLS2NVFRUkIqKCmIymfqVE133YH6ROI4jgiDQ/wcCAeJyuUggECA8zxO73U5kWSYVFRW0v4qixJSRkJBALBYLAUAEQRiw74qiEEVRiNvtJoIgkNLSUtpPQRAIz/NElmXCcRyx2+3Ebrf3a69anizLpLCwkLbTbrcTm802YF9dLhcxGAxEEASSlpZGeJ7XPaV3Pn7bhOab9zvvvEMAkAMHDsSk+8Y3vkHmzp0bt5zOzk7S1tZGr4aGhiGNRU1NDSksLCQ1NTUnnWagvMOZ53Tbeabzn2rZQ8k3GOfjmieErfvheDYcz0+1zadT9rm+7k9p49TX10cef/xx4nQ66YbC6XSSJ554gvT19Q13G8848QY4IyODfjhnZGQQRVGI3++nG514H+RGo7FfmoSEBLoI1A2FunmI3oDoGUsDIHa7ndY90GWxWIjP54vZ5MiyTPuSkZERU370T1rqxslut9NNlyRJJCEhgXi93n51iaJIN1Vqn6L7G70Rc7lcuu1X+xXdXr/fT2pqagZ1NxDdv4qKipifGl0ul+6b7nz8EBmuD5AHH3xQd/wGG4uBXGAMluZk3GecTp7TbeeZzn+qZQ8l32Ccj2ueELbuh+PZcDw/1TafTtnn+ro/6Y3T8ePHSUVFBTl48CAhhJDDhw+Tw4cPD3vDvkjiDfD8+fOJwWAgo0ePJn6/nyiKQioqKqiyYjabidPpJKIoUnsjnueJ0+kksiyT+fPnE0mSiMlkIl6vl+b1er1ElmUyc+ZMYjAYYtQUp9NJ3G439cR+0UUXEUEQiM/nI6WlpSQjI4P4fD7i9XqJ3W4nTqeTmM1m4na7ic/nIxkZGaSiooKUlpYSi8VC3G43URSFlJaWkrS0NCJJEs0vSRKx2+2ktLSUFBYWkoqKChIIBIgoijEbYrXffr+fiKJIzGYzmT9/Pm3nzJkzid1uJ6IoEkEQiMViIaWlpaSmpoZuikpLS4nf7yeSJBFRFKmKpy1f3UjV1NSQiooKYrfbiSRJxOl0EqPRSCRJInPmzImJzafW5fV6idlsJj6fj1RUVOh+YzkfP0S0HyC7du0iAMiWLVti0l1++eXk7rvvjlvOSP/mfTbzf1m/eZ9J2Lo//WfD8fxU23w6ZZ/r6/6UFCer1Ur27Nkz3G05a8Qb4IEMl6OVlXihVvScYUa/HuiYvppGq6IMtDOPNizXhoSJfg7oBxPWpoluh9pvPeeTeo4wte0byB2CXvnqa60zUO34q/0dKO9Q5vpcRvsBohrJPvbYY/ReW1sbM5Jl6HK+zjNb94zT4ZwzDr/kkkuwZcsWuFyuU8l+3hAdVqWnpwdjxoxBOBymx+C7u7shiiJ6enrgcDio4bbqTJIQgqSkJAAnwqN4vV4899xz6OvrQ2JiIq688kq8/vrrSEtLQ2trKzo7O2OMwAHQEC4AkJiYiNzcXEQiEVRWVmLVqlXYunUrent7sXbtWkyfPh379+9HMBjEjBkzsHjxYnrcf82aNTS8icFgQH19PYLBIF544QWkpKSgs7MTdXV1SExMpMbXfX194DiOhjNxOBzwer04cOAANU4XBAGjRo1Ca2srMjMzkZmZGWMkXlxcjEgkgp07d4LneRqouLe3F729vUhPT0dfXx88Hg/Wr19PDeUVRcGqVatofwsLCwGA3nc6ndQQvqOjA9XV1TSAsaIomDx5Mq3/fKS9vR07d+6kr3fv3o0PPvgAqampGDt2LO655x4sWbIEEyZMoMeyMzMzMWfOnLPXaAbjNGHrnnFecCq7rZdeeolkZ2eT5cuXk3fffZd8+OGHMdf5xkA706GoS/g/1UOrDmmP4w/kEFOr5ugpVVpnmNq2aB1Zatuv12a9I/8DKU/aPNFqVTz02qDnokBPMdKqbNH34/VroLacL98233rrLd2+zZs3jxDyH0eAo0aNIhaLheTn55NPPvnkpOo4X8aCcXqcT/PM1j1juDjnFCf12Ofdd99N73EcB0IIOI5Db2/vqRR7TqIoCtatW4eOjg6kpKTAZDIBQEwwX6vVilAohN/85jfo6Oigrzdv3owDBw6gpaUFHo8HmzZtAgAYjUZwHIdNmzYhMzMTCxcuRCQSgdfrRUtLC3JyctDY2EiVqZycHDQ1NcFsNlMnmImJicjOzsaePXvQ19cHQgh4nofNZqNuEyRJov8ePnyYzo/BYEBWVhYcDgd1A6Ae91fdFHg8Hjz00EPgeR4WiwVjx46FzWbDp59+CpPJBIfDga6uLmRkZMDpdEJRFBQUFKCtrY0G2V24cCE2b96MxsZGSJIE4ITLg46ODuzbtw9r165FRkYGRFHEoUOHaFBkreuC7du3Q1EUFBUVwePxwOl0IicnB/v370dvby8NDdPQ0EBVwYKCAixZsuS8dUdwxRVXgBAS9znHcSgvL2dhZRhfKti6Z5wXnMpua8+ePQNe5xuDKU7xHDHGs3tClOqhp5Boy4sOhKsX/Fb7Wi1DvR9tYxStisULtaINDRPdzniv1XtqGapdUbzwM3ptUcsaSC3SjoXeOMYLL6OnXJ3MXI802FiMDNg8x8LGY2RwzilOX3bbJgCorKykjijz8vKoI0dVLWlpaUFmZiaam5tRX1+Pm266iTpq5HkejY2NEAQBaWlpcDgcyMnJQUNDA7Xn2bNnD9LS0tDX14fZs2djzZo1aG1tRW5uLrxeL55//nmYzWZkZ2ejo6MDe/bsAQCqFHEch507d1KF69ixYzAYDAgGg/j73/+Ojo4O9Pb2IhgMYt++fWhra0NzczN2794NQgiysrIAAKIoIi0tDZ2dnTCZTFi9enWMc0z139raWqqKHTx4EEajEVVVVcjNzUVxcTHWrVtHbZdSUlJw7NgxFBUV0bpVJ5WyLNOgvqoNVWZmJlpbW3H8+HEUFRXh6NGjEEURFRUVWLNmDRwOB2688UY4HA5q5/Xpp59i7969EEURHo8Hq1evpsGLo22dGAwGg8EYVk5lt6U6aYx3nW/o7UyjT6jpnVwDBg4/og3+G62QaO2CTkbJiQ7FAvRXobQKU7S9k7bteuqW2j4t8U61qeXrla1VfOIpTdo+66XVC1MT3We9cpniNDBsLEYGbJ5jYeMxMjiT82w4lc3W97///ZjrjjvuwLe//W0sWLAA99xzz6kUec5RXl4Ol8uF8vJyKIpC7XiCwSA4joPVakVGRga1OcrJyYHBcGI4LRYLMjMzwfM8AICccPsA4IR908SJE2lIFuCEquP1eiEIAkKhEIqLi5Gbm4uqqipUV1cjNTUVbrcbsiwjFAqhoKAA4XAYbrcbZrMZTqcTdrsd2dnZaG1tRVJSEpxOJ0wmE1paWnDppZeirKwM7e3tMJvN4DgOPM8jOTkZRUVFsNvtEEURgiDAZDIhOzsblZWVKCoqQmVlJXw+H1566SUQQlBdXY26ujpIkkQDD2dmZiIYDEKSJFgsFlx33XV0vGpra1FUVITa2lqEQiGYzWaIogiHwwEA4HkeiYmJUBQF+fn59MSg0WikYylJEmRZRlJSEpKSkuhpPo7j4HA40N7eDrfbDbvdDrfbDVEUaSBmxqkRPW9DfR4vz2BlnU7eU8l3Jp6dqbJPtf+MU4Ote7buh8Rw7cC2b99O8vPzyfr164eryC+MwXameoqT1n4nWrkZSM1Rn6tl6NVBSKziEm2vo/VXFJ1Xe6oPGsUm+l9oVBmtDVH0v3p90Co92pN88fwvxVONtHZK0eXq2U9Ft0Nbf7RaeLJzPZIYzLbvZD0Cx8szWFmnk/dU8p2JZ2eq7FPtfzRszcfC1j1b96fLKdk46TFhwgQsXboU3/zmN/Hxxx8PV7FnlcrKStx3333geZ4Gl+3o6AAAmM1mTJkyBc3NzWhoaEBKSgp6enrQ3d2NGTNmQBAEbN26laY3mUwghGDSpEnw+/148cUXUVRUhKKiIhQXF1N/R9GKC8dxMJvNsNls8Hg8CAQC2LZtGwRBQG9vL0RRRFFREURRhNFoREpKCqxWKzweD7xeL5599ln09vbCYrGA4zjYbDbwPA+n04nm5mZccMEFMafh8vPzYbVasX//fuTk5OArX/kKFEVBeXk5du3aBZ7nkZGRgfb2dhiNRoRCIaxevRrbt2+ntlY/+tGPUFxcjN27d2PdunXo6elBcnIytTeqqqoCgJhAx1rfT2qA4G3btqG9vR0ejwd5eXnYtm0buru7cfz4caSkpKC1tZWe5kxMTKRBfXNycrB371709PR8aQP9nmm0Nm5DeR4vz2BlnU7eU8l3Jp6dqbJPtf+MU4Ote7buh8Rw7sK2bNlCEhMTh7PIL4TBPIdDo9ogSq2JTqM+V1WS6Pt6J73inWTTs5fSU35UWyntCTO9tkOjdOmpQIWFhTHKVHQ7tUqR1tt3dJv06tdTnAb7xqC1M9PabsVT5fTGY7C5HomwsRgZsHmOhY3HyOCcs3H605/+FHOtXbsWK1euxDe/+U185StfOZUih8TSpUvBcVyMHVVnZye+973vwW63w2az4aabbsKhQ4dOu67a2lrIsgy73U7thVSbGlW98Xg8VBlyu90oKSmBLMvo6enB6tWr4XA4IAgCioqKkJeXh4yMjBi7H0VRkJubi40bN6KsrAzhcBgOhwPBYJDaP3EcRxUl9d/ExEQAQG5uLvLz8+lJNY7j0NHRgYKCAhiNRmpzZTAYqF1TWloaSkpKkJ+fj3A4jLq6OiQlJVGP3bIsw2AwQJZl1NbWorKyElVVVRAEAUajEUajEdnZ2TAajVi7di02bdpEPYJzHEfzhUIhmEwmiKKIQCAAURTx6quvwuFwwGKxICcnBw6HA+FwWPf37YKCAvT09MBgMCAnJwd1dXXgeR6CIFBVrKOjA2azGenp6TCZTKivr0dtbS1KSkpgsVjg8/nYN3EGg8FgDC+nsttSVQP1MhgMZNSoUeSWW27pF7l6uPjnP/9JsrKyyJQpU8j3v/99en/RokVkzJgx5K9//Supra0lfr+fXHrppSdVtt7OVFV/FEWJUTZkWSYul4vwPB9zX5Ik4nK5SCAQoPcMBgNNpygK4Xme2O12IggCMRgMMYpV9Ak9g8Gga8+jvXieJ4qixLX70btkWab906ZLS0uLeW2324nRaBxSW6LrLiwsJIqiEI7jiN1uJ4qiEEEQYtKZTCb6fzXIr8vlIqWlpUSW5Zi02rza+qIVJlmWBwwMyb5t/gc2FiMDNs+xsPEYGZxzQX6/aI4cOUImTJhANmzYQGbOnEk3Tq2trcRkMpFXXnmFpt22bRsBQDZv3jzk8vUGWI3OHL3hkWWZ+P1+uhmqqKggdrs9ZiOQkZFBJEkiAIjb7aabLO0GLCEhgdTU1JCKigr6U57FYonZWKjlxNswcBxHFEWJyae2we/3E7fbTcxmM3E6nXQD5Pf7af8URaHtVzdKAIjZbB7yRkkURWK322lb1Y1L9IaO4zji9XpjNofR/48em+hNkNpmr9cbs9FSN5dut5soikJKS0tjxoAZyg4NNhYjAzbPsbDxGBmccz/VqXR3d+OTTz5BT0/P6RQzKN/73vdw7bXXYtasWTH3//Wvf+H48eMx9ydOnIixY8di8+bNccvr6urC4cOHYy4t06dPx4svvojCwkKMHj0ajz/+OP70pz8BANLT01FeXo5bb70V69evR2FhIS699FLqzNFoNMJms6G1tRU5OTngeR7vvfcegsEgBEEAcMLx5AMPPAAA9KcvjuNgsVhgMBhw6aWXwmq1wmazAThxbF+SJOqW4IYbboDJZMJ7770Hv98PjuPAcRwkScJ1112HpqYmfOtb38JFF12Ezs5OTJo0iRqCy7KMu+++G4FAACaTCT6fD4qi4Gtf+xoSEhJwzTXXUFcDDocDRqMRFosFPM9TdwbAiZ8As7OzsX79erz55ptQFAUulwsPPPAACgsL4ff7Ybfb4fV6YbPZMGnSJOrGINqVQ2FhIYLBIHieR25uLux2OyRJwsSJE6EoCn74wx/iggsuoI4zJUnCc889h507d6KwsBDPPfccbr31VmRkZCAQCLCf5xgMBoNx5jiV3dbRo0dJKBQiPM8TnufJrl27CCGE3HnnneThhx8e1p3d73//e+L1esmxY8cIISRGcXrxxReJ2Wzulyc3N5eUlJTELfPBBx/UVU8GO56q5wpARVVK4gXeBfoblw/kvDE6nza91iBcm3Yg1wPxjNkJIbqqj/aZ1ihbLVcdq3iBegdyJzCQmwc9A/ToOqPzDWZsTgj7thkNG4uRAZvnWNh4jAzOOcXp/vvvR11dHf72t79RBQUAZs2ahZdeeulUitSloaEB3//+9/Hiiy/G1HO63H///Whra6NXQ0ODbjo1zEh+fj48Hg/Wrl0LAHA4HNi4cSMqKytp2pKSEiQkJODyyy+HxWJBSkoKDAYDzGYzAFCDa6fTCY7j4HQ64ff7qcG1xWKhCo/ZbEZGRgYkSYIgCDTQrslkQjAYpAbhqrKVmpoK4ISzSJvNRo2qg8EgvF4vzGYz3G43IpEIQqEQVWZUlScnJwcFBQVITEyELMsoKSmh+YxGIwoKChAMBpGQkIBgMEidUwIn3Cy8/PLLGD9+PMLhMPLy8uD1erFu3TqsXbsWycnJyMrKiumzz+eDKIrgeR5JSUnUKD0tLY06upQkCU6nE6+++ipEUUReXh5VqwKBAA36GwqFaOgaNdRKQUEBCgoKmFNABoPBYAw/p7LbGjt2LLUhstlsVHHasWPHsLoj+OMf/0gVFfXC/ykRPM+TN998kwAgn3/+eb/2Pf7440OuJ97OVM/xpVo/oO9gUc8ZZXS+6GP4WpcF2iC1ek4oo4/maxUrbZ3RoUz0jv9rVR49R5vadsdTyNR02rFS50+tRy1b206tMqXOsZovnusGPRWNBfkdGic7FqrdX7Txvd69ge6fKc5UfedK/06nTrbmY/myrPszWde50L/TrfOcMw63Wq10sxS9cfrggw9IUlLSsDXu8OHDpL6+PuaaPn06+eY3v0nq6+upcfjq1atpno8//pgAp28crhp+S5JE3G43kSSJCIJATCYTcTgcRBAEMmfOHOJyucj8+fPpiTBFUUggECBut5sYjUZiNpsJz/PEYrEQs9lM85SWlhK3200AEKvVSutQ65NlmcyfPz+mDbIsk9LSUuL3+4kkScRut5P58+cTSZJo2eqpPbPZTEpLS+miq6ioiPlXNUyXZZl4vV7i8/lonRkZGcTpdBKz2RxjgK2mVfsmiiJxOp3EZDIRURTJ6NGjCc/zdJwAEIvFQmw2GwFAkpOTid/vJ16vl56Us9lsZP78+bRsv99P/H4/URSFXHTRRQQAcTgc/dqtjovf7yfz588nRqORGI1G4nQ66djoxU1kHyL/4WTHYri9Jg8nZ6q+c6V/p1MnW/OxfFnW/Zms61zo3+nWec5tnC677DLy5JNPEkJObJzC4TAh5ISNk6Iow9c6HaJtnAg54Y5g7NixZOPGjaS2tpYEAgESCAROqsyBgvzqKTiqoqEqPFrHklrlRKuIaJUbnuf7OZHUU5oI0VeC9BSqobq41ypSeqpVdL16zivVMqJtrQZyiaA3rnoqUTxHllr1T8+GiilOg/Nl+eZ9Jus7V/p3OnWyNR/Ll2XdM8VpYM65jdPf//53YrPZyKJFi4ggCOT73/8+ueqqq4goiqS2tna42xiDduN07Ngxcscdd5CUlBSSkJBAbrzxRtLU1HRSZcZTnERRpIpHWloa8fv9pKKigiiKQnw+Hz22n5ycTNLS0ojX66U+ifx+P7Hb7WT06NH0Q53n+RgFR5IkYjKZiNvtpiqOJEkxCpbb7SYGg4HMnDmTZGRkEJ/PR10NCIJA5s+fT+vyer0kEAgQr9dLZFkmc+bMIRaLhdapVZlUdcfn8xG73U7LBE64QzCZTMTr9dL+iKJIlS1ZlonT6aRtCAQCNK/VaiUOh4NwHEfS09OJKIr0p9aUlBTCcRxJSUmhGy1BEGhZpaWldA7mzJlDAJD09HTafkIIKS0tJWazmYiiSMdD7YckSdRFAlOcBoaNxciAzXMsbDxGBufcxokQQnbu3Em++93vktzcXDJp0iRSVFRE6urqhrNtXxjxBjj6dJpWqdEqUkNRYaKVkHhKT7RdUjwHmdH39Wye1P9rFSC9E2p6ypmeXZZWEdO+1qpjWnVKe+pQW5f6OtpuLN6JuXjql1aNY4rTwLCxGBmweY6FjcfI4Jw7VQcAbrcbv/nNb/DPf/4TH330EX73u9/B5/OdanHnJCUlJRAEAbIso66uDpIkoa6uDoFAAImJiRAEgYZbCYVCMSFEFEWB0+nEddddR8OhACd8Nnk8Hqxbt46eaJMkiYZxaW1tRXV1NURRRFVVFYLBIFwuF2677bYYP0WhUAiCIKCnpwdVVVVwOBwAgI6ODqSmpsLv9+O6666j/p3U02t1dXUwmUxwu93w+XxITU1FSkpKzEk+SZLg8/ng9/sRDAYRCoUwatQoOJ1OCIKA3NxcmEymmDAr6mk7k8mEnJwcjBo1Cn6/H+Xl5fQUoCAI8Hg84DgOU6dOpWFljEYjDYxsNBoRCAToGKsn7URRxLp163DppZdCURQawiYYDEKSJKxduxbr1q1DdnY2rFYrDVXDYDAYDMawcjK7LDW8ykAXz/PDvrs70wy0M9XzIQTon5DT82Gkqh7R6lW8k3B6io/eyT29dmn/rz21p5YV/X+9Mgayi4pnBxXPjktPIYoeBz1bLb1y9Xw4Rbcp+ll026IDFQ9lrkcabCxGBmyeY2HjMTI4k/P8H4c8Q+CPf/xj3GebN2/Gk08+ib6+vpMp8pxG9ePk9Xpx4MABZGZm4rPPPkNjYyNVR1QVCgCSkpJgNBrh8Xiwfv16SJJEFaiioiI8++yzKCoqAgD85je/gdFoRFFREerr67Fz505IkoSWlhYAgM1mw6FDh9DY2IgFCxagvr4ee/fuxdKlSzF58mSEw2GkpqZCEAQ0NzdDkiS0trZS5Wjjxo1YtGgR/H4/duzYAY/Hg0WLFuHRRx/F9u3bqVKVnZ2N1tZWJCcno6OjAx6PBwUFBWhsbERDQwNMJhPuvPNO5Ofno7GxEQ6HA7Nnz8aaNWvQ0dGBrq4uqpwdOnQInZ2dGDNmDBoaGuDxeAAAiqJg48aNEEURLS0tkCSJ+psaM2YMurq6sHfvXur3qqenB4IgYMqUKVAUBStXroQoijh48CC6u7tRX1+Pm266iXo1F0URnZ2d1Ku5yWTC8ePHEQwGz87CYTAYDMaXl9PdeX388cdkzpw5hOd5cuutt5I9e/YMx4buC2UwP07RypKefyYMoJLoKVDRKkm8023aU3fq/weyadKzl9KiVZmiT/DF8/KtrVNPzYoeF62fq//f3tlHN3Gd+f8raSSNPJLHtiQs2WA7liNeKkFocCxR2u4uqcZ9oUmbs+zGPk0iUrpk0+TUbJek59cS4nLatN2WnuRk+7ohdgvJUkpCE7oxKaR00xIHp0kMhQQSOUAAG1vBBtvYRuj+/mDnRhpGfgGDbfR8zpkTe+bOfX0crh499/uoY9NTNC/9v1x+enM53Lxp+6lN9ptp/PRp80NoLrIDWud0aD6yg0kZ43T8+HEsX74cwWAQiUQCb7zxBhoaGlBaWnqpVU4qGhsbsXPnTlRWVvL4pdbWVuTn58NgMMBms8Hn88HpdMLn8/HYoEAgwHP35efnw2w2Y+/evfD7/SguLoaiKIhGo7BarQgGg6irq0NLSwtisRjcbjeP2enu7kZxcTGsViuPPXK5XIhGo9wLVlBQAIfDgaqqKkQiEdTX12Pp0qWora2FKIpwOBxobGxEMBiEzWbDww8/DEVReLySxWKBxWJBMplEWVkZbzcQCKC4uBgAYDAYMH/+fN6mJEnYtGkTdu3aBUmSYDQaIQgC8vPzIUkSV0F3uVyor69HS0sLWltbuTfJaDQiPz8fgUAAsiyjt7cX8+bNA3Ah1sloNPK4rF27dqG1tRVutxuKoqC+vh6yLMNisfA4qtRYrNraWl6HLMuor6+fMPshCIIgrlHGutPq7u5mq1atYjabjYXDYfanP/1p3HdzV5vhdJy8Xq+uF0Z9plX7zuSRUsvqxT+leoGMRmOaF0dRFP5MURRWU1PDY8nwf54WRVH4tWbNmrQ+qOKT6qUKU7pcroxeM5fLxaUF8H8eL6PRmHZP7zIYDLyM1+tljDEWDod1yyqKwrxeLx/zSLpPqj6Y9p1wOMw1PlK9YEajkeQIRiDTXFyOhstYNVdGKj/e9V2JNkfz7tXSv9Frh2w+nclu95diKxNp9+M9lkvhatv9mDZO3/ve91hBQQGbM2cOe/bZZ8e9MxNFJh2n0tJSFgqFmMlkYh6PhzmdTgaAWSwWZjAYWCgUYjU1NVzbyev1MqPRyFXGi4uL+T/katk9e/ZctMh79uxhHo8nbXMjiiIzGo1MURTm8Xj4BmnPnj18Y2S1WvmGSbtxUzcWmTYk6lj0viZTn6XWM9xXlKqelaoPpY6XMZY2rtS2FEVhoVDooq/3tJtH7cYp9R0AzOPx8E3onj170vpOX9UNz2hSDY3m/ljLjKX8eNd3JdoczbuXU+fltk82n85kt/tLsZWJtPvxHsulcLXtfkzB4Q8++CBsNhsqKirQ0NCAhoYG3XJbtmwZS7WTkjvuuAMAsHLlStjtdpSVlaG6uhqPPfYY+vr6YLFYUF1djYceeggtLS3o6elBf38/cnJyIAgCHn30UbS1tWHt2rVgjPEAbpV4PI777rsPACDLMg/cPnv2LO666y4sWrQIq1evRk1NDcLhML7//e+jpKQE69atQ21tLX71q1+hoqICNTU1+MEPfgCz2YwZM2YAuJAcecaMGZg2bRreeecddHZ2wmw2w2Qy4dy5c7Db7bjvvvtw8OBB+P1+/OAHP0BfXx+sViuXX7j++uthMBhw8uRJHDt2jAexHzp0CHl5eTyIvaSkBIwxxONxHri+bt06noR3yZIlaGxshN/vx9e//nVs3LgRALB27Vr8/Oc/59ILN998M7Zv387bOXDgAP/K02AwoKWlBY2NjaiursZf//pXmEwm5OTkYMmSJdi+fTsURcH+/fsBXEh2TMHhl05dXV3af0e6P9YyYyk/3vVdiTZH8+7l1Dke7RMjM1ns/lLWcCLtfrzHcilcdbsfyy7rzjvvZHfdddeI11Qj085UL7UHUrwdqcHP0HhISktLLyo/nAilnlSANig6UyJg9T29I/7aPukl89W+rxcArheUru1L6m5fL7Bei15KFW0g+XBzqpWEyLQ+o1nrbITmIjugdU6H5iM7mDQepyeffHIsxac89fX1WLlyJQYGBpBMJnH06FEYjUaYzWYwxhCJRBAOh3Ho0CEUFxejvb0d58+fh8FggCAI+NKXvoTHHnsMiUQCLpcL3d3diMViuOeee7B37168/fbbyMvLQ3d3N3p6egBcCJCORqP47Gc/C+DCDtrv9+ORRx6B1+tFRUUFwuEwr9fhcCAQCCAej6fJBCQSCSQSCX5U32azQRAEFBQUwO/3p+3QW1tb8fbbbyOZTCI3NxclJSWIx+OoqalBLBbD4cOHeaD1tm3bcObMGbz//vsoKysDABiNF84YNDc3o7q6GmvXroWiKHjppZfgcDiwePFiKIrCZQ5OnDgBv9+P2tpa7o1KfS7LMp8Xk8mEkydPQhAERCIR7N69G4IgwGQygTGGsrIydHZ28jpWrlyJ3t5e8jgRBEEQV4Zx34pNQcYqgKmXhFfPQ6L1wKR6ZtR39WKM9L5j13qC9FKL6PVVe+mJXGo9Zpm8RNpymVK0pPYxk5ct0/xo69FKF2jr0KZq0XrJxrLW2QbNRXZA65wOzUd2MCnlCLKBlpYWNDc3AwDy8vJgMplgNptRW1uLxYsXw+FwcJmAyspKABe8L5IkIRgMIh6P87ic5uZmmEwm5ObmckkCURS51IDFYgFwQfhS9ca0tLQAuOAVCgQCsFqt3LNSWFgIn8+HgoICxGIxNDY2Ih6Po7y8nItCqkf1g8Egl00oKChAc3Mz8vLyYLfb4Xa74ff7EQqFIMsynE4notEoFi9ejFgshurqajz88MPwer1obm6GLMswm80AgHnz5kGWZZhMprT2YrEYWltbeeqXurq6NBkEk8kEk8mEvXv3orGxEbW1tfD7/SgoKIAkSZBlGaFQCIqiwOv1cvkHVXrB5/PxtaioqIDBYOAxaGpqHFUOgSAIgiDGlXHfik1BhjtlgRSPUmrckdajoo3XSY3xyZQyRCtiCZ24HRXVQ5Ua/zRa8cvh+gJcLFSp7Z9W3FKbNiWT10t7Xy9Fip7w5nCxYNpYJtXjZDKZRkzwO9xaZyM0F9kBrXM6NB/ZwaSJcco2/H4/T9CrJvpVxRgBYOfOnVwYM5lMwmAwoLi4GPF4HOFwGMXFxYhGo9i9ezdPYTI0NITNmzfjM5/5DAoLC+F2uyFJEnJzc3HkyBEe47R7927EYjEEg0GcOHECg4ODAC4k8VXbVxSFn1KrqanBxo0bsW/fPgwMDKC/vx95eXlIJBIQBAGVlZUIBAJoaGiAwWBAIpGA0WjE+fPneSyQoij485//DEVReFqXQ4cO8WTCbrcbyWQSfr8fL7/8MiKRCNrb23Ho0KG05MbqvAAXYqJsNhvOnTvH46xkWeZpVaLRKA4ePAhFUfCTn/wEBw4cwPnz59HU1IQvfvGLfI4A4MiRI9i2bRvsdjuPcVLXRxAEGAwGnDt3DqWlpYjH42hpacGCBQuuqs0QBEEQ1zjjvhWbgox0qk57Wk3rUUIGD4rWa5TqwTKZTLqpToaLCVKv4XQ49Mqr/dKeSNN6p7RaGFrvkZ5XSy+WKdMpt9S2M82PXjoY7Qk/7aWNtdImGR7NWmcjNBfZAa1zOjQf2QHFOE0Q9fX18Hq9sFgsYIzBbDZDkiQ0NTXhtddeQyKRgNPpRCgUgiRJsFgsiEQiWLp0Kerr61FZWYmmpiZMnz4dNpsNx44dg8lkgiAIKCsrw+LFixGNRiFJEo4dO4aioiJ4PB74/X6exFdN52IymQBcSOPS2tqKoqIiNDY2AriQHiY1BkmSJEiSBFEUYTKZYLfb4fV6EY1G4fF4eCqXSCQCr9cLh8OBcDjM44N27drF+2uxWHiamfnz52Px4sXIz8+H0WjkqViqqqoQCoXQ2tqKpqYmVFZWQlEUCMKHDk2DwQCTyQSfz4dIJMLT2DQ3N/Ox+P1+WCwWFBcXw+v1ora2Fm63G7FYDIqi8HlWU7wAgM1mQ0FBQdq6SZKExYsXk5YNQRAEMf6M+1ZsCjLSqTqMcEqNsfQYJBU9HScM40lJ9UJl0lPSnjLTtqPVWNLzfmnjpTJ5tlK9TWrb2r7ojUOtN7UuvdNvqe2mesS0ZVJjlrSnETMpmlOM08jQXGQHtM7p0HxkB+RxmgBaWlpQXV2NWCyG4uJiHr8UCATg9Xoxc+ZMABdO0dntdnR1dcFut6O3txeNjY1oaWmB1+uF0+lEcXEx976o3h/1NFs8HofP5+PJfN1uN6LRKKqqqviJuf7+fgDgWkbqKTw1sW0kEoHVaoXD4QBjDO3t7Zg9ezYcDgdPHKx6blTvlNfrRX19Perq6hCJRNJO3omiCIPBAJfLBbvdDsYYgAuenNbWVkiSBLPZDK/Xi9mzZ2Pnzp38NJzFYkE0GoWiKHA6nbBarTAajVi0aBHXspo7dy5isRiqqqr4fEaj0bR+tbS0cG9WJBLhyZDVJL4VFRUwmUyorKyE0+nkpxJNJhNPnkwQBEEQ4864b8WmIHo7Uz1vRyp6nqhUr4o2Xkj1jKinv7RxPpl0jwwGQ1oMj95JOvW/2vgg7am9TLFF6nj0TrfpaTWp13An7LQK4HoxVnpK49pTeMPFiKntjqRQPtJaZys0F9kBrXM6NB/ZAXmcJgBVd0jVE1J1lVpaWlBbW5sWc2MymSDLMpYtW4bS0lJEo1HE43HujXn44YdRXl4Oq9WKsrIyHhOkek+am5vTdI8URUEsFkNubi5yc3PhcDgAXPBuRSKRtMtoNKKjowN5eXn4xCc+wcupKuChUAj19fVcByoSiaCyspL3q7q6GsFgkMcmRSIRHssVCARQVlbGdaFUz5nVaoUgCCgvL4fX64UoitzzFYlEoCgKH7/BYIDZbIbD4UA0GoXT6YQkSVzBPBwOw+Vy8fgs9XSe3++H0WjEli1b4Ha70djYyPWsRFHEokWLIIoi8vPzIUkSurq6IElSWswUQRAEQYw7474Vm4Jk8jileoP0ftaW076b6o3RekX0PFajOVE3XAyVXr16cU16pwW1z9Tf9WKj1EtPx0lv/Hpj08ull1pO77Riat2ZNKn05mmktc5WaC6yA1rndGg+sgPyOE0AdXV1WLx4MeLxOPx+PyRJwubNm9Hf34/i4mL4/X5UV1dj165dSCaTkCQpLS5q8eLF3COknnLr7u6Gx+PBb37zG7z44osXKXyHQiHE43EoioJAIABBELhKt8lk4h4ntZ1EIsH7O336dN5Xp9MJWZYRDAb5CbxIJAKXy4Xe3l7Mnz8fOTk5qK2tRVVVFT8RGI1GEY1GeS44VbtJkiSUlZXx+Cqj0chPtomiCADYtWsXqqur0djYiFgsBlmW4fV6IcsyrFYrAGDz5s3YunUrRFGEIAhIJBL82bRp03isV39/P3p7e/kzq9WK7u5uLFy4EIqicNX2goICBINB+Hw+fhpPlmXetqq8ThAEQRDjxrhvxaYgmXam4XCYAWAul4t7MoxGIzMYDMxsNg97Cq2mpibtvUyXIAjM6/WyhoaGUWkgeb1eXW+L+rvFYmEAmNPpZF6vN807pPZHFEVmMpmY1+tlTqeTl1EUhSmKclGb6ljtdvuI49H2y2g0jviOOg+pKuCZytXU1KT10eVyMUVRmMlkYqIoprXp9XrZnj17RrXW2QjNRXZA65wOzUd2QB6nCaKtrQ0AwBjjsTXXXXcdGGM4d+4cAHB9JbUcAHg8HtTV1fHfAXBPjZZEIoETJ05g9erVqKurw9KlS1FXV4f6+vqLyufk5KC0tBTvv/8+CgsLEQgELmp7aGgIABCPx3HixAk4nU6IooiBgQFepqKiAtOnT8eJEycQj8dHnIdkMgkA6Ovru+iZJElpv6ttqO2q7wLgnqbUOVHnL5FI4Pz58zCZTFi2bBlycnLS6rVarQiHwxedluvq6gJwweM2MDCQ1t8TJ05g3bp1I45vKrJmzRoYDIa0a9asWRPdLYK4opDdE5MBSrkyDCtWrMB3vvMdDAwM4NSpUwCAYDCIvr4+mEwmdHR0gDGG4uJidHZ2YmhoCIIgIB6P4+c//zm8Xi96enogSRIeffRRAMD999+Pvr4+5Ofno6enB7Iso7u7GydOnMDChQsxa9YsbNu2DevXr8fdd9+N5557jn/Nl5eXh0OHDuGmm27i9d133304cOBA2td2giDAbrcjHo/jvvvuw2c/+1ncf//9OHjwIMLhMB599FHs378fDzzwAA+sFgQBNTU1aGtrw86dO2GxWODxePjGpKSkBO3t7ejs7ER+fj7fnMyYMQPAhU3m2bNnecJdt9uN3t5enmZmaGgIxcXFPH2KwWBAbm4ubDYbnnvuOcybNw/79+9HRUUFFi1ahN27d+Ott97iMg19fX1YsWIFFixYgLVr1+Kdd95BLBaDJEkIh8Po6elBb28vRFFEZ2cnZsyYgYqKimtaluAjH/kI/vCHP/DfUzelBHGtQnZPTDjj7sOagowmya96pYpUpt7XCjFmSjqbGtysV09qXcOJRWrr0ibD1dahJy6pJ7apLZcaSJ7avl46Fj35A20C5ExyBVr5hOHmI7Vf6v2R5BZGWuupxkMPPcTmzZt3WXVcK3NBDM+1tM5k98Roydqv6r773e+isrISDocD06ZNw6233oq33347rczAwADuvfdeOJ1O2O123Hbbbejo6BiX9lURRzV42mAwoKCgALIsw+FwcGHM6dOn89QseXl5/Ah+IBBAMBjkQpZq4LQqSql6m3w+H5c1MJvNPBWKIAjIy8vDtm3bUF5ezoUnHQ4HHn74YezcuROlpaXo7e1FIBCAz+eDIAjIzc1FJBJBTk4OIpEIqqur4XA44HK54HA4UFFRgU2bNkGSJESjUUQiER6YnipE2dLSgmg0ygPJA4EAzGYzOjo60oK8VVmCQCCAgoICnkBYDX4vLy+H2WzmMgZ79+6F3+9HcXEx76caqK6+r77r8/mQTCZ5ipfq6mqEw2E4HA7Isgyn04mysjIuUOr1elFZWcmT/F7LHDp0CEVFRSgvL0dtbS2OHDkybPnBwUGcPn067RoJVX5Dby4zPRvunYmo81LbHGuZ8a73csd8rUJ2f2XbHGuZSyk/5e1+3Ldi44iiKGz9+vVs37597I033mCf+cxnWElJCevt7eVlVqxYwWbMmMF27NjBWlpaWCgUYgsXLhxTO8N5nDIdfdcKUOp5S1I9M3peGWg8SKneHq23S5toOFVQU31HT0pAzxOUWq+erMJwP6e+m9q3TPOU+iz1fmr/tCKgqWVS39POgfpualD5SGKY18qnzd///vds06ZN7M0332QvvPACC4fDrKSkhJ0+fTrjOw899JDu+g03F3pyGyM9G+6diajzUtsca5nxrvdyx6xyrdg8Y2T3Y6nzUtsca5lLKT/V7X5Sb5y0nDx5kgFgu3btYowx1t3dzcxmM/vNb37Dyxw4cIABYLt37x51vZkmuKGhgXm9Xubz+ZjT6WQ+n49JksQEQWDBYJCfhFuzZg1zOp3MbrczWZaZ2+1mRqOR+Xw+1tDQwEKhEPN6vWzNmjUX/exyuVggEGCKorDly5eznJwctmbNGrZmzRpmNpuZKIrMbrczSZKYLMu8L2obdrudGY1Gduutt7JAIMAEQWCyLLPly5czp9PJZFlmwWCQ12+1WlleXh4zGAxp/ZNlOa0d9Z01a9aw0tJStmbNGqYoCnO73fzkmtvtZhaLhfl8PqYoCq9LkiR++lDtX3FxMSsuLub31PpCoRALh8MsFAqxQCDAnE4nKy4u5vX6fD4mCAKz2+0sEAiwYDDInE4nC4fDrKGhgSmKwusVRZGFw2G+Hk6nkzU0NIxqrac6p06dYrm5ueyXv/xlxjIDAwOsp6eHX0ePHh1xLvbs2cNqamouOp043LPh3pmIOi+1zbGWGe96L3fMKteqzTNGdj/Z7H605ae63U+pjdOhQ4cYALZ3717GGGM7duxgANipU6fSypWUlLAf/ehHo653OI+T1sOh9RJpy6V6gVSPTiYPTuq72vgcPcFK9Xc94Uc9b0wmj1am5LyZPEWp/82U/kU7nkx16s1L6pj0PGN66WUytaf11qWu03BrfS2wYMEC9uCDD466/LU8F8SHXOvrTHZP6HEl13nKHEdIJpP42te+ho997GP8GH57ezuPK0qlsLAQ7e3tGesaHBzE4OAg/z3Td96KomDnzp38JNiRI0cgSRJsNhsikQjKysoQjUYRi8VQUFCAZDLJT9+ZTCZYLBZs27YNJSUlqKysRCwWAwAsXryYJ62Nx+MIBALo6upCJBLB9u3b0d/fj/fff5/HO+3duxfd3d0AAIfDgaqqKnR2duK9994DYww5OTkoKyuD3W7H0aNHcfz4cSQSCQQCAbS3t8Pr9aKxsREOhwOCIODs2bM81Yt6hB8AzGYzkskkbDYbysrKcPbsWbz33nuw2WwQBAGyLOP8+fPo7u6G2WxGV1cXcnJyIAgCT5Wizpl6ElAURbS3t8NoNGLWrFl444034HK5sHPnTqxYsQJLly6FoijYuHEjP2VYUlKChoYGmM1meDwedHd38/b7+/uRTCbR39+P6upqvPPOOzAajXweBEHA1q1bYTKZYLfbUV9fPzZDm6L09vbi3XffxZe+9KWJ7gpBXDXI7okJYdy3YleIFStWsNLSUnb06FF+b8OGDcxisVxUtrKykq1atSpjXaP9zjvVO5TqwUn9PfU02HDCjZm8M6Px6uidfkv1qqR6kEbj1Rnu0osdGs2lPS2X+v2zOldagUutN0hbPnXetO2lnlpMnQftnGu5Vj5t/tu//Rv74x//yNra2tif//xndvPNNzOXy8VOnjw56jqulbkghudaWmeye2K0ZO2pOpWvfvWreP755/HSSy9h+vTp/L7H48HQ0BD3xqh0dHTA4/FkrO8b3/gGenp6+HX06FHdcoqi8PQqDocDFosF06dPx86dO1FWVgaj0chPgDHGUFFRAbPZDJvNBlmWUVxczFOldHV1oaioCGazGc3NzaiuruYny6LRKJYuXYr6+npUVlZCFEWYTKa0tCfz58+HxWJBeXn5RalVvF4v3G43FEVBJBIBcCGFSSwWQ2trKz/FV1hYyL1zBoMBkiRxb5rVaoUsywgEAvzEWkFBAQwGA6ZNmwan04lgMIhAIACn0wm32532LBAIIBaL8XQzamLkxsZGeL1euFwueDweWCwWFBQUcI/V3LlzUV1dzU9ItLS08LkOBoOor69HVVUVbDYbDAYD8vLyIIoili1bhkgkgkAgwE89qicV1aTL17K36f3338ftt9+OmTNnYunSpXA6nXjllVf4ehLEtQjZPTEpGPet2DiSTCbZvffey4qKitjBgwcveq4Gh2/evJnfe+utt8YtOFzrrdE7zQaNF0T7nva5+nMmzSGtd0YtmylhsNYTpfWEadvSaippvV3DxRfp9VN7ok7recpUb+qlFyOlvaeWzcnJuWj99NYp04kL+rT5ITQX2QGtczo0H9lB1sY43Xvvvdi4cSO2bt0Kh8PB45ZkWeZenbvvvhsrV65EQUEBcnNzcd999yEcDiMUCl12+4qioKmpCd3d3QgEAigqKkJJSQk2bNiASCSCl156Cb29vTh//jwAYN68edz7U1lZCcYYHA4Hjh49CrPZjC984Qt47rnn4Ha7YbfbwRjDrFmzoCgKqqur0dPTA4fDAa/Xi8997nPYsmULEokE5syZA0VR8LOf/QxOpxMOhwMnT57kWknl5eU4cuQItm3bhqVLl+KZZ57BuXPnuD5UZ2cnjhw5gtmzZyMUCmH9+vVgjMHr9XJFclEUkUgkIIoiCgsLeXJfWZZRU1ODn/70p2hqasLcuXMhSRIEQYDFYuEq4pFIBFu2bOHzNmfOHLS1teHgwYOYPXs2JEmC1+sFAHR2duLw4cMwm80wGo0QRRF+vx/BYBAHDx5EUVERent7sW3bNixcuBArVqxAc3Mz2trauL7UggUL0NLSgnXr1kFRFMTjcRw7dgwFBQUoKiriOk4LFiy4bDsgCIIgCM64b8XGEWSIp1m/fj0vc/bsWfav//qvLD8/n+Xk5LAvfOEL7MSJE2NqZzTK4Wq8jPaEXGq/tOrao9Wi0J7e08ZA6cVEDddu6ok8xliatyq1z5limPT6rW0z9dJTC099JzUGS3tfT9dK26/UOco0P5lOJ452rbMRmovsgNY5HZqP7CBrY5zYBbmEi6677rqLlxFFEY8//jg++OAD9PX1YcuWLcPGN42Furo6hEIhrvC9cOFCHpckSRKampq4VyYcDmPVqlXweDwIBoO6CtYtLS0Ih8PIy8vDli1bwBjD1q1b0dzcjIKCAh5D1NTUlNaWoij8BN7s2bPR29sLn88Hp9OJcDjM45oEQUA0GkV9fT1KS0sRiUTgcrmQTCZhNptRXl6eFhtVVlbG44NKS0sBADNnzuRtplJfXw+n0wmr1QqDwYD8/HwIggCbzYbc3Fw4HA6Ul5fDarWiv78fZWVlmD9/PlcFV5MXqzFMoigiEolwT6GqFG61WrFkyRKePNhut8Pv9yMWi/FTc7t27UJeXh5XL29qakJ/fz+PAXM4HDzmiyCuFSaFYjJBXGUmpd2P+1ZsCjLcznTPnj1MFEXu/RBFkZlMprR7siwzURTZmjVr2J49e1hpaSkzGo3MbDYzACwYDLI9e/Ywr9c7rNdGURRmsVjS2jIajczr9TKn08kMBgOzWq0MALNYLCwUCjFFUdLqtdlsTJZl5nK5mMvlSvPiGI1G3u/UdlIv9b4q0llaWsoaGhpG7D8A3rfUMZlMJqYoClMUhfl8PmY0Gof1eKmCmdp6hms3Ux9KS0svEkmjT5sfQnMxtRirirMKrXM6NB9Ti8lo97RxYsNPsJ5AZGlpKQsEAhdtQNSvwkwmU9rXZ1arNaPQpLop8Hq9TFGUi55rj9jLsnzR+6FQSLduVT3bYrGkBZiP9kpNrpvaf1UJXFte2zf1XUVR0jY/qRs4q9XKnE7nqPojCELafAuCwHw+30WbWHXTaTKZLvpjo/9pfgjNxdTiUtTLGaN11kLzMbWYjHY/qYPDJxI18Njv98Pj8fBg6b6+Phw5cgQdHR2wWq349Kc/jZdeegkDAwOora3F3r17kZeXB6/Xi46ODnR2diIUCiEWi8FqtWJwcBD5+fkYGhrC0NAQpk2bht7eXgwNDaGkpIS3YbPZUFFRwX8/fvw4/H4/VqxYgf/4j//A/v37AQBOpxOvvfYa7HY7ent7ed3nzp3DF7/4RfT19UGSJGzYsAGVlZV45ZVXIMsy4vE4kskkBEFAMBjE66+/DgDIyclBeXk57HY7Tp48iaNHj8LhcECSJBiNRiSTSTidTsyYMQMAcOzYMQDA9OnT8eUvfxnr1q3DmTNnYDAYYLPZ4PV6UVNTAwB45513cOTIEeTk5GDp0qXYvn07otEodu/ejePHj+P48eMQRREnT55Mk2MYHByEIAj43Oc+l3b0uLOzE6tXr0ZbWxu+853vwGKxwOVy8eTI/f399HUdcc2wYMECbNiwYaK7QRBXlUlp9+O+FZuC6O1MtcfptYHN6qVNX6KXiDdVGgD/9xWVXrC1noDjSH3Qfo2leodS39NKKGi/ItO2O9xYUtvVBqVrA7T1pAG0get68zxSELw2AD5T4LqeLAF92vwQmovsgNY5HZqP7CBrg8Mnkrq6Oi5KqQ1sFgQBoihClmWUl5dj8eLF8Pv92LZtGwwGA4xGI3JzcxGJRJCTk4NIJILc3Fxe95IlS1BWVsZ/F0URLpcLq1atgs/ng8FggM/nQzQahSzL6Orq4kHQjY2NiEajEAQBZrMZ8+fPh8Fg4AKWXq8XiqKgsrISvb29qKysxKpVq+D1elFWVsaFKA0GAwAgPz8f5eXlkGWZB7nX1dWhrq4OkUgEoVAI5eXlGBwchNF4wVzMZjNCoRBWrVoFWZZhNpvhcDjg9/tRUFCAgoICLripCmMGg0EUFRUhEomgtLSUi1OqIqOqeOXWrVuxbds2zJ49G16vlwtuBoNBWCwWAEBZWRnq6+vh9Xrh9XoRjUbh8Xi44KjJZILP50MkEkFdXd1VshiCIAgiKxj3rdgUZLQ7U+2R+EwpWYCLPUbQeHT0jtwzlu6RyRRflcmzk+q1Gc6zoxW4HCnwTvX0pL6XKUmxti/a9tT+pM5pJu+SVopBfaYVGtXzxGnbGetaZwM0F9kBrXM6NB/ZAcU4TQJaWlp4Ml+/34+6ujrs378fL730EmKxGMrKynD48GGYTCYAAGMMHR0dEEURXV1dMJvNOHfuHIxGIyRJgsfj4fFBNpsNsViMJ+K1WCzcg+N0OpFIJOByudDT0wNBELBt2zbk5ubymCiv18uT427YsAF+vx/9/f04fPgwT4ZbU1ODtrY27N+/Hzk5OTh37hwSiQQ+/vGP44033sCmTZvQ3NyclrpAlmWsXbsWq1atwiOPPAKn04ne3l4AQGtrK6qrq2Gz2cAY42llAoEAF/csKioCYwyJRALHjh1LS7rb2NiIlStXYmBgAJIkweFwIBAIoK2tDefOnUNtbS36+vpQV1eHxsZGNDU1wWKxIJFIYMmSJaiursaxY8fSZAfq6+uxcuVKGAyGazrdCkEQBDGBjPtWbAoymp1pJgFGbYLf0STG1cY8Aekikqm/Z/K6ABeLbKZ6mbSpVbReMb1kwdorU0Li1D5qx6sXY6WWSU2XovUwjSZJMDLEX43lmCp92vwQmovsgNY5HZqP7IBinCaIxsZGlJWV4eGHH0YsFkNubi4KCgq4h6Ourg5VVVXweDxYsmQJcnJysGzZMi6aKUkSF8hU428EQeCxO2rqGDXeaP78+YhEIvD5fDCbzRAEIS1hrqIovG6n04mysjIkk0ns2rUL4XAYDocDLpeLn4IDLsQwSZKETZs2IZFI8H7l5ubixRdfxMaNG3lclhobFAqFEAwG4fF40sa6dOlSRKNR5Obmwmg0wmg0wuFwAEBajJUq3KkmJp43bx5MJhNqa2sBIC1WTJIkFBcXw2w248UXX8SmTZsgiiJisRgefvhhLqRptVoRDAZRV1cHRVFQWFgIn8+HgoICtLa2IhgMwuVyoaKiAnl5eXC73WhsbLzaJkMQBEFc64z7VmwKkmlnqk0NAh0Px3DiXHopU7QxQKmeIdUjo+dh0Uu9ovX2aL1Wap3QeIT0BCXVurQxUdpx6XmoMiX21SYmzpTIVy8+TC+x8XDeL71LL86JPm1+CM1FdkDrnA7NR3ZAHqcJIhqN8lNxqodD9cK0tLQgGAxi06ZNkCQJfr8f1dXV/PRYY2MjJEnC+fPn8fTTT2Pu3LlQFAWRSASRSATRaBRutxterxeiKMJoNHKPTDQa5R4Wv9+PnTt3orKykp/si8fjWLx4MZYtWwar1Qqfz4dgMIiCggI4HA5UVVUhEAjA6/UiEonw03D5+fno7e1FUVERBEHg8VhWqxUejwcWiwWCIKC6uhqKomDx4sX8RJzdbocoipAkCaFQCKIowmAwYNq0aSgoKEAsFrvoNF9tbS1EUUR+fj5kWUZzczNPN6PGesViMX4qzu12c/0ntf9qImG3283nuLW1FW63G2VlZTAYDCguLuYePEmSIIoiLBYLotHohNnOVGW06Q1GKjeaevTKXOp7o+VS+z3W+1fqvcutl9BnPOz+cuogux/5+XjM/bgx7luxKchwSX5TPSharSEM48nRxhkh5SSaWnempLR6p/VG4wkaTjcJ0Ndx0vNSpXqQtB4drVcsNVGvtr9672uvTDFjen3KFF+l9imT9tZo1jobGU6/bKS4sZHKjaYevTKX+t5oudR+j/X+lXrvUuolm0/nStn95dRBdj/y87E+o1N1E4SqAeT3+/HYY4+hu7sbs2fPRiwWAwC43W50dnbCZrMhEongueeegyRJ6Onp4TE8sVgMjDGYzWa0trYiHA5DlmXU1NQgFovh8OHDiEajOHjw4EUeJUVR0NzcDKPRCEEQ0NjYiNbWVgiCAL/fj5aWFnzzm9/EsWPH0NbWBkEQUFhYiG3btiEvLw9z5sxBJBLBf/3XfyGZTKKiogJtbW0wGo2w2+0oKSlBUVERbDYbjh49iry8PJw5cwaSJEFRFLz88sswGAxwuVw4c+YMhoaGUFtbi8bGRvT29sJut8NkMsHr9XKVbkVR8OCDD+qe/uvs7MR7770HxhgMBgMsFgtMJhO2bduGhQsXYsWKFYjH4wCAmpoa1NfXIxaLoaioCCUlJTh58iTMZjNmzJiBiooKfopw1apVaGxsxNatW5FMJjF//nzMmjWLJ1hesGDBhNnQVEO1+ZH0r0YqN5p69Mpc6nuj5VL7Pdb7V+q9y62X0Gc87P5y6iC7H/n5eMz9uDHuW7EpyEg701TvUqqHRaucrVUQ1zt1hhFOkKXe0/Nqpf6s59HRO+Wm9yw1/ieTTlNqjJGeFyzV4zPc6T+tR0k7H9py2n6p7ev1XSV1nNr+jmWtswmai+yA1jkdmo/sgGKcJhhFUbhCtnoizuPxIBKJcAXvSCSCpUuX8pikQCCASCSCZcuWwWKxQJIkBINBHoukxgQtXboUiqKguroa4XAYsVgMixcvhiRJ+O1vfwu32w1BEGC32yEIAlf4VhXC1VN4JpMJZrMZixYt4vFLPp+P/2w2m7FkyRI+jkQigby8POTl5UGSJBgMBjDG+HuxWCztZJ7f74fb7eYeL7PZzE8JGgwGlJaWoqmpCQsXLoQkSbzd6dOnY9u2bTzXX0FBAT9laLFYUFxczMekalmFw2G4XC6eH6+yshJbt24FAJ7/7uGHH4bL5UJeXh4WLlyIaDSaVj41LowgCIIgxo1x34pNQfR2pqkZmRVFSfOMmM1mBoBZLBZ+TxRF1tDQwERR5M8URUl732KxMJfLxd+RZZk5nU4my3Ja/U6nkxmNRgaA/zf1crlczGAw8PdT60wt7/V6+c92u505nc5h443UK3VcqeMbzbvQeK6sVquuty51fOFwmN/Xa0cQhLTxCYJwUblwOMxKS0uZ0Wi86ITgSGudrdBcZAe0zunQfGQHFOM0Aaxbtw6bNm3SfXbu3DkAwNDQEL83MDCA1atXY2BggD/bvn07nE4nLzM0NISuri5YrVYMDg6ip6dHt341zsdkMkEURfT19QEALBYLzp07B8YYGGNp76t1JpNJGI1GMMZQWlqK9vZ2MMbQ19fHVb9V75KKIAhIJBJp/UzFYrHwcTmdTlx//fU4cOBAxv6z/1MSTyaTEEURg4ODsFgsWLVqFXbv3o3W1lacOHECAJBMJtHW1oZIJJJ2X8VgMKT1zWaz4ZZbbkFrayv27dvH77e1taGzsxOiKOLs2bPIyckh9XCCIAhi3KGNUwbq6uoQj8cRj8cRDofx5ptvQpIkdHV1AQBKSkoAAEeOHEEikYAoiohGo9i9ezfeeecdHD58GGazGU1NTbjvvvtw7NgxHDx4EDNnzsRtt92Gn/70p7y+RCIBQRB4ncePH0dRUREkSUJvby9OnDgBv9+PFStWoKmpCYqiYOPGjbwdURRRWFiIrq4unD9/ngdfHzhwAEVFRejq6oLL5UJ3dzcGBwdhtVoxd+5cBAIBbNiwgac3Sa23ra0NyWQSBoMB4XAYb7zxBvr7+5GXlwdZlvHoo4/iJz/5Cfbu3YvBwUEUFhais7MTQ0NDmDZtGk8cHA6HsX79eh4AHw6H0dLSwqUakskkVqxYgYceegiNjY144IEHeIC9LMvo7+9HVVUVnn/+eRgMBnzqU5/Czp07ubCoOvcrVqzAwYMHoSgKmpqaUFdXR0HhBEEQxLhDG6cMLFiwAE6nE5s2bcJbb72Frq4u/MM//AMAYNOmTQgGgwCAffv2gTGG/v5+HDx4EC+88ALKysqQSCSQSCRw9uxZrF+/Hh/72Mfw1ltvIRgM4uDBg2n1PfXUUwDA6zxw4ADsdjv3qNx+++3YsGEDAOCOO+7g/62trUUsFkNvby96e3u5BlJ/fz8fR09PD0pLS3HkyBHuZUokEjh+/DiOHz+O/v5+bN++He+99x6vV1UkBy54j15++WWcP38eAPDuu+8iFovB6XSivLwczc3NAC54rc6dO8f70NLSgqVLl+LgwYM4duwY1q9fj2PHjsFisfD+lZaW8g0lADQ1NenOS05ODpYuXYpNmzZh+/btaeMDwOdeO0cEQRAEMd7QxmkY6urq0NbWhoMHD6K0tBQ7d+7EkiVL4Ha7EYvFcM899yAej+P48ePo6uqC3+9HbW0t5s+fjyNHjgAAcnNzUV9fjzlz5gAA9+pUVlZyWYOqqioYDAbE43HU1NQAACRJwvr162EwGCBJEqqrqwGAe3Dq6+vh9/t5kLYqWaB6ltSku6IoclHMvr4+HDp0CIODgxAEAcFgEEePHsWJEye4fIGiKLpfl4miiEQigdLSUlRUVPBxVFVVcc/SunXr0N/fD0EQ+Pg6OzshCALmz5+PoaEh5OTkoKurC4IgIBKJYPv27VxQNB6Po7Kyks9DPB7H6dOn+ebQ7XajqqoKv//97/mYk8kkEokEmpubEQwGEY/H8cgjj9DmiSAIgrgyjHvU1BRkuCAyvaP3esl+9VKNQCdAWZsuJJM0gfZ4vVYCobS0NGOqEm0ak0wJfbUCnZnuaetJHUcmCQK91C9awU+thIOeeGWqhIGeMKbepRcUPpq1zjZGOxepByWGuzdaMr071vujeX4lno2lzFjfGWudoylPNp8O2T3Z/eVCGyeWeYL37NnDQqEQk2WZiaLITCYTMxqNzGazMafTyZYvX85KS0vZ8uXLmdfrZT6fj8myzCRJYpIkMVmWWSgUYmvWrGEej4cFg0EWCASY0+lkgUCAhUIhfvKuoaGBn7BTy8myzGw2W9rJuE9+8pPMaDQyn8/Hbr31VmYwGJjVamWSJDGbzcYsFgu79dZb0+oKh8NMURS2Zs0a5nQ6mdVq5ZuQ1A2HyWRiH/3oR3VPxxUXF/N69uzZw08L+nw+JooiW758OfP5fMxgMLDp06ezUCjEgsEgk2WZybLMAoEAL3vrrbey0tJStmbNGqYoSlrfJEliLpeLz+2aNWtYMBhkJpOJmc1mtnz5chYKhZjT6Uybb5vNxjdmn/zkJ/lpw4aGhlGtdTYy2rm4VKXjsdR3KfdH8/xKPBtLmbG+M9Y6R1OebD4dsnuy+8uFNk5s5JQrel4YbRJarRdEFagc7n3tH6Tec+272qTAeh6XTAl7VW+Oniin9l09L85wXqacnJy0NCx6qWr0kvam9k1vnFrPWmo/UuvPNAat94n+EfkQ+uRNn7yzEbJ7svvLxcBYyrn0LOX06dOQZRk9PT3Izc3l91taWnD//fdj//79AAC73Y7Ozk643W4MDAwgLy8PHR0dOHv2LDweD7q7u3H27FkYjUbMmjUL4XAYv/rVr+BwOPDBBx8AADweD5cFmD17Nh577DEsWLAALS0t+Od//ue0FCOMMZ6mBLhwFN9ms6GrqwsWiwUGg4HLE9hsNh4/ZDabkZubix/+8IeYM2cO7rvvPhw+fBj/8i//goMHD6K/vx/PPfcc5s2bh7/97W9wuVzo7e3lAprPP/88kskkLBYLGGMYHByE0WjEddddh2nTpiE3N5fHNJ0+fZoLZy5atAjNzc0oLi7GtGnTcPLkSX46z2azwW63czmG+fPn49FHHwUAfPOb30RPTw96e3tx9OhRCIKAL37xi3j++edRWlqKYDDI08a43W74fD709fXxODK73Y6Ojg4uFPqFL3wBW7ZsgcFgwA9/+MO0eKdMa52N0FxkB7TO6dB8ZAdXdJ3HfSs2BRlNjFOq50KNu0lNqaL11GhTsiDFk5Ias5PqddJLkDtcwl29uqHxtGjrUe8N56rVS9qbWm8mL5rJZEqLYxqur9r51HqghvNGZZrb4WKbRrPW2QbNRXZA65wOzUd2QAKYE0RLSwscDgcsFgsEQUA4HEZvb29aoty///u/x/bt21FbW4udO3dyj1FBQQGOHDkCm82GoaEhCIKAZDIJr9cLt9uNvr4+fhKvurqaJwb2er2IRqPYvHkzTCYTJEmCJEnYsmULcnNzdb1eXV1dGBoa4hpNyWQSgiCguroaJSUl/PTZrl27EA6H+Uk3SZJQW1ubpn2k6lcdP34cTU1NPCXLwMAAjEYjIpEI9u3bh9dff53Pk8PhwNDQEFwuFwYGBuD3+6EoSppHqri4GKIocg9UR0cH5s6di6KiIoTDYRQXF0NRFMyZM4frZ9lsNiSTSfT09MBkMnE5BavVikQiAYPBgKKiIuTn56O9vR1er5eS+hIEQRBXlnHfik1BhotxGsnLk8ljpD2NhxRvidabovVAaeOdMiUTTu2n+lz1+mAYz0xqvXon2VLHoo5Rb2yp9WTykGk9Xdo+pJ6y03rEtHOnF3+ljeMaKbiQPm1+CM1FdkDrnA7NR3ZASX5HweOPP46ysjKIooiqqiq8+uqrl1VfY2MjT9YbCoXg8/lgsVh40l01aW8sFkMwGITL5UJ/fz/3ykQiERiNRjgcDgiCAKvVCpPJhNzcXPj9fsRiMbjdbkQiERQWFsLn88Hj8UCSJDQ1NcFqtcJgMMDn8yESiQC4oKdUVlaG4uJiSJIEr9eLuXPnIhaLwefzwWq1wuPxoKCgAMFgEKFQCF6vFyaTiSfvDQaDPKlufn4+11NSkw3X1taisbERXq8XTqcT4XAYiqKked4cDgdkWYYoijxdy29+8xs+N1u2bMG2bdtQXl4Oj8cDRVEAfJgs2Wg08v6EQiE4HA643W4oioLGxkbu6Zo3bx5fD4PBgHnz5sFqtcLhcPB73d3d+MpXvsLfUbW0WlpaLmv9CYIgCEKXcd+KTQBPP/00s1gs7IknnmB/+9vf2PLly1leXh7r6OgY1ft6O1NtbJP2xNhwcUeZNJZSn2fSNNJ6iLR1mUymi8pl8vIMF9+kPammfZZJSym1vdR5Ue+n9j11rrT1ZOpjan16JwpH8kTpec9GWutsheYiO6B1TofmIzsgj9MI/OhHP8Ly5csRjUYxZ84c/PSnP0VOTg6eeOKJS66zvr4eLpcLXV1d3DMkyzIGBwcxe/ZsHg9UVVUFSZJgMplgtVphsVgQjUZRX18Pp9PJc6oVFxdDEATIsoxoNIqqqirk5uaio6MDjDF0dHQgLy8PtbW18Hg8cLvd3HtVX18Pi8UC4ELCXaPRCKPxwtKZTCYIgoC9e/dCURQEAgFdr4/qiVEUBUuXLkV9fT0CgQBEUUQkEkE4HEZTUxNmz54Nh8MBSZKwbds2LFy4EIqiIBKJIBQKIRgMcs9YWVlZWhJjWZYhCAJEUYTL5UJtbS2PXQLA50uWZeTm5iIWi0FRFJSXl3OF9Pr6ekiSBIPBgFmzZnGFcFmWeX1LliyBxWKByWQCAMybNw8ejwfhcBjRaJQru5PXiSAIghh3xn0rdpUZHBxkJpOJPfPMM2n377jjDvb5z39e952BgQHW09PDr6NHj+ruTLXeD6vVygAwo9HI9SMURbnIo2Q0Grm4o3rPbDZzT4uiKMzpdOp6o4xGIwsGg8zr9fJ29+zZw0RRHDbWCgBTFIW5XK6LvDRGo5EZjUYGgIXDYcbYBR0MtY3UutV+pl6q96ahoYGXVevTXur41DGqv6uCmbIsp5X3+XwXjT+1LgBs+vTpLCcnhwtsOp3OtPlT57u0tJSFw+GLvGyp0KfND7laczEWjZbL1ZiZrIyXTs2lzAHZfDpk91ePa9Xup/zG6dixYwwA+8tf/pJ2/9///d/ZTTfdpPvOQw89pPuPvnaCVTVvi8XCjEZj2j/66j/KehsndcOi98zr9WZ8J/UKhUJ801NTU8MCgcCwGxZ145S6oXC5XGzNmjVpX3l5PB7G2MWB2uqGSfs1mNPp5MaqJwug/ixJUtr4Up8pipLxa83hxjOWsuoYPR4P37jp/ZHRPyIfcrXmYiyqwJerajxZGS9l5EuZA7L5dMjurx7Xqt1npRzBN77xDaxcuZL/fvr0acyYMeOicnfccQfuuOMOtLS0YN26dTyxLXDhaycAWLt2Lf9KrrGxEbm5uThz5gxWrVqFz372s+jp6cGhQ4dQVFSEoqIirF27FgDS7qtf55WUlKCxsRF+vx+PPfYYAGDdunW8LbUPP/nJT3DgwAEAQElJCX9/7dq12L9/P1avXo36+nou/Hjddddh5cqVMBgM+N73vsf7H4/H0dPTA1mWUVNTg6amJvj9fqxfvx7RaBQHDx5EXV0dP95fX1+PBx54AG63m8sIqAmH58yZk9ZXdYxqvwAgHo/j2LFjiMfj+NznPoft27cjEongueeegyRJaG9vx9mzZwFckDhYunQptm/fjvnz53PJhyNHjqCnpydtndQkw2pfKMHv5EK1iVTbuJSyY6lnsjHWvmcqP5XnINsgu7927X7KK4cPDQ0hJycHmzdvxq233srv33nnneju7sbWrVtHrIOUZLMHWusPobnIDmid06H5yA6u5DpP+eBwi8WCG2+8ETt27OD3kskkduzYgXA4PIE9IwiCIAjiWuOa+Kpu5cqVuPPOO7FgwQLcdNNN+PGPf4y+vj5Eo9GJ7hpBEARBENcQ18TG6Z/+6Z/Q2dmJ1atXo729HTfccANeeOEFFBYWTnTXCIIgCIK4hrgmNk4A8NWvfhVf/epXJ7obBEEQBEFcw0z5GCeCIAiCIIirBW2cCIIgCIIgRsk181Xd5aAqMpw+fXqCe0JcadQ1nuIqHOMC2X12QDafDtl9dnAl7Z42TgDOnDkDALoimMS1yZkzZyDL8kR3Y0Ihu88uyOYvQHafXVwJu5/yApjjQTKZxPHjx+FwOGAwGAB8qCZ+9OhREkmbgmRaP8YYzpw5g6KiIp4oOVvR2j3Z/NRHbw3J5tMhu7/2uNp2Tx4nAEajEdOnT9d9lpubS39MUxi99aNP3RfIZPdk81Mf7RqSzX8I2f21y9Wye/r4QRAEQRAEMUpo40QQBEEQBDFKaOOUAavVioceeghWq3Wiu0JcArR+Y4fmbOpDazh2aM6mPld7DSk4nCAIgiAIYpSQx4kgCIIgCGKU0MaJIAiCIAhilNDGiSAIgiAIYpTQxikDjz/+OMrKyiCKIqqqqvDqq69OdJeyjj/96U9YsmQJioqKYDAY8Oyzz6Y9Z4xh9erV8Hq9sNlsuPnmm3Ho0KG0Mh988AFqa2uRm5uLvLw83H333ejt7U0r09raio9//OMQRREzZszA97///Ss9tEkH2fvkgez+6kF2P3mYUnbPiIt4+umnmcViYU888QT729/+xpYvX87y8vJYR0fHRHctq/j973/P/t//+39sy5YtDAB75pln0p4/8sgjTJZl9uyzz7I333yTff7zn2fXXXcdO3v2LC9TXV3N5s2bx1555RX2v//7v6yiooLdfvvt/HlPTw8rLCxktbW1bN++feypp55iNpuN/exnP7taw5xwyN4nF2T3Vwey+8nFVLJ72jjpcNNNN7F7772X/37+/HlWVFTEvvvd705gr7Ib7R9SMplkHo+H/eAHP+D3uru7mdVqZU899RRjjLH9+/czAGzPnj28zP/8z/8wg8HAjh07xhhj7D//8z9Zfn4+Gxwc5GUeeOABNnPmzCs8oskD2fvkhez+ykF2P3mZ7HZPX9VpGBoawmuvvYabb76Z3zMajbj55puxe/fuCewZkUpbWxva29vT1kmWZVRVVfF12r17N/Ly8rBgwQJe5uabb4bRaERzczMv84lPfAIWi4WXURQFb7/9Nk6dOnWVRjNxkL1PLcjuxwey+6nFZLN72jhp6Orqwvnz51FYWJh2v7CwEO3t7RPUK0KLuhbDrVN7ezumTZuW9lwQBBQUFKSV0asjtY1rGbL3qQXZ/fhAdj+1mGx2TxsngiAIgiCIUUIbJw0ulwsmkwkdHR1p9zs6OuDxeCaoV4QWdS2GWyePx4OTJ0+mPU8kEvjggw/SyujVkdrGtQzZ+9SC7H58ILufWkw2u6eNkwaLxYIbb7wRO3bs4PeSySR27NiBcDg8gT0jUrnuuuvg8XjS1un06dNobm7m6xQOh9Hd3Y3XXnuNl9m5cyeSySSqqqp4mT/96U84d+4cL/Piiy9i5syZyM/Pv0qjmTjI3qcWZPfjA9n91GLS2f2lRLxf6zz99NPMarWyJ598ku3fv5995StfYXl5eay9vX2iu5ZVnDlzhr3++uvs9ddfZwDYj370I/b666+zw4cPM8YuHE/Ny8tjW7duZa2treyWW27RPZ46f/581tzczF5++WV2/fXXpx1P7e7uZoWFhexLX/oS27dvH3v66adZTk5O1h3LJnufPJDdXx3I7icXU8nuaeOUgccee4yVlJQwi8XCbrrpJvbKK69MdJeyjpdeeokBuOi68847GWMXjqh+61vfYoWFhcxqtbLFixezt99+O62OeDzObr/9dma321lubi6LRqPszJkzaWXefPNNtmjRIma1WllxcTF75JFHrtYQJw1k75MHsvurB9n95GEq2b2BMcbG5DMjCIIgCILIUijGiSAIgiAIYpTQxokgCIIgCGKU0MaJIAiCIAhilNDGiSAIgiAIYpTQxokgCIIgCGKU0MaJIAiCIAhilNDGiSAIgiAIYpTQxokgCIIgCGKU0MZpCmEwGPDss89OdDcI4qpCdk9kI2T3kxfaOE0SOjs7cc8996CkpARWqxUejweKouDPf/4zL3PixAl8+tOfvmp9eu+992AwGPDGG29ctTaJ7ILsnshGyO6nNsJEd4C4wG233YahoSE0NDSgvLwcHR0d2LFjB+LxOC/j8XgmsIcEMf6Q3RPZCNn9FOcS8/ER48ipU6cYAPbHP/5x2HIA2DPPPMMYY6ytrY0BYP/93//NFi1axERRZAsWLGBvv/02e/XVV9mNN97IJEli1dXV7OTJkxnr/OCDD1hNTQ1zuVxMFEVWUVHBnnjiCd5e6vXJT36Sv/eLX/yCzZo1i1mtVjZz5kz2+OOP82dq35566ikWDoeZ1WplH/nIR0YcH5FdkN0T2QjZ/dSHNk6TgHPnzjG73c6+9rWvsYGBgYzl9P6QZs2axV544QW2f/9+FgqF2I033sj+7u/+jr388svsr3/9K6uoqGArVqzIWOe9997LbrjhBrZnzx7W1tbGXnzxRfa73/2OMcbYq6++ygCwP/zhD+zEiRMsHo8zxhj79a9/zbxeL/vtb3/LYrEY++1vf8sKCgrYk08+mda36dOns82bN7P9+/ezL3/5y8zhcLCurq5xmjViqkN2T2QjZPdTH9o4TRI2b97M8vPzmSiKbOHChewb3/gGe/PNN9PK6P0h/fKXv+TPn3rqKQaA7dixg9/77ne/y2bOnJmx3SVLlrBoNKr7TG3j9ddfT7vv8/nYxo0b0+59+9vfZuFwOO29Rx55hD8/d+4cmz59Ovve976XeRKIrIPsnshGyO6nNhQcPkm47bbbcPz4cfzud79DdXU1/vjHP+KjH/0onnzyyWHfmzt3Lv+5sLAQABAMBtPunTx5MuP799xzD55++mnccMMNWLVqFf7yl78M215fXx/effdd3H333bDb7fxau3Yt3n333bSy4XCY/ywIAhYsWIADBw4MWz+RXZDdE9kI2f3UhjZOkwhRFPGpT30K3/rWt/CXv/wFd911Fx566KFh3zGbzfxng8Ggey+ZTGZ8/9Of/jQOHz6Muro6HD9+HIsXL8bXv/71jOV7e3sBAL/4xS/wxhtv8Gvfvn145ZVXRjVOgkiF7J7IRsjupy60cZrEzJkzB319fVe8HbfbjTvvvBO//vWv8eMf/xg///nPAQAWiwUAcP78eV62sLAQRUVFiMViqKioSLuuu+66tHpT/7ASiQRee+01zJ49+4qPh5jakN0T2QjZ/dSB5AgmAfF4HP/4j/+IZcuWYe7cuXA4HGhpacH3v/993HLLLVe07dWrV+PGG2/ERz7yEQwODuL555/nxj5t2jTYbDa88MILmD59OkRRhCzLePjhh3H//fdDlmVUV1djcHAQLS0tOHXqFFauXMnrfvzxx3H99ddj9uzZWLduHU6dOoVly5Zd0fEQUweyeyIbIbu/BpjoICuCsYGBAfbggw+yj370o0yWZZaTk8NmzpzJvvnNb7L+/n5eDjrBgqmBfC+99BIDwE6dOsXvrV+/nsmynLHtb3/722z27NnMZrOxgoICdsstt7BYLMaf/+IXv2AzZsxgRqMx7Xjqhg0b2A033MAsFgvLz89nn/jEJ9iWLVvS+rZx40Z20003MYvFwubMmcN27tx5WfNEXFuQ3RPZCNn91MfAGGMTs2UjrlXee+89XHfddXj99ddxww03THR3COKqQHZPZCPZaPcU40QQBEEQBDFKaONEEARBEAQxSuirOoIgCIIgiFFCHieCIAiCIIhRQhsngiAIgiCIUUIbJ4IgCIIgiFFCGyeCIAiCIIhRQhsngiAIgiCIUUIbJ4IgCIIgiFFCGyeCIAiCIIhRQhsngiAIgiCIUUIbJ4IgCIIgiFHy/wGqJTc6JbNPQQAAAABJRU5ErkJggg==",
111 | "text/plain": [
112 | ""
113 | ]
114 | },
115 | "metadata": {},
116 | "output_type": "display_data"
117 | }
118 | ],
119 | "source": [
120 | "# Little util function\n",
121 | "def spike_tensor_to_points(spike_tensor):\n",
122 | " x = np.array([p[1].item() for p in torch.nonzero(spike_tensor.cpu())])\n",
123 | " y = np.array([p[0].item() for p in torch.nonzero(spike_tensor.cpu())])\n",
124 | " \n",
125 | " return x, y\n",
126 | "\n",
127 | "fig, axs = plt.subplots(1, 3, figsize=(6, 3))\n",
128 | "axs[0].scatter(*spike_tensor_to_points(input_raster), color=\"black\", s=0.5)\n",
129 | "axs[1].scatter(*spike_tensor_to_points(blocks_spikes[0]), color=\"black\", s=0.5)\n",
130 | "axs[2].scatter(*spike_tensor_to_points(standard_spikes[0]), color=\"black\", s=0.5)\n",
131 | "axs[0].set_title(\"Input raster\")\n",
132 | "axs[1].set_title(\"Blocks output\")\n",
133 | "axs[2].set_title(\"Standard output\")\n",
134 | "axs[0].set_ylabel(\"NeuronID\")\n",
135 | "axs[0].set_xlabel(\"Sim step\")\n",
136 | "axs[1].set_xlabel(\"Sim step\")\n",
137 | "axs[2].set_xlabel(\"Sim step\")\n",
138 | "fig.tight_layout()"
139 | ]
140 | },
141 | {
142 | "cell_type": "markdown",
143 | "id": "7def7ba2",
144 | "metadata": {},
145 | "source": [
146 | "## Model speed"
147 | ]
148 | },
149 | {
150 | "cell_type": "markdown",
151 | "id": "24c34073",
152 | "metadata": {},
153 | "source": [
154 | "How much faster is our blocks models model?"
155 | ]
156 | },
157 | {
158 | "cell_type": "code",
159 | "execution_count": 108,
160 | "id": "fb472914",
161 | "metadata": {},
162 | "outputs": [],
163 | "source": [
164 | "def get_training_duration(model, device=\"cpu\"):\n",
165 | " model = model.to(device)\n",
166 | " data = input_raster.unsqueeze(0).to(device).repeat(128, 1, 1) # 128 batch size\n",
167 | " torch.cuda.synchronize()\n",
168 | " start_time = time.time()\n",
169 | " output = model(data)\n",
170 | " loss = output.sum() # Arbitraty loss just so we have something to backpropogate\n",
171 | " loss.backward()\n",
172 | " torch.cuda.synchronize()\n",
173 | " training_duration = time.time() - start_time\n",
174 | " return training_duration"
175 | ]
176 | },
177 | {
178 | "cell_type": "code",
179 | "execution_count": 103,
180 | "id": "91db4382",
181 | "metadata": {},
182 | "outputs": [],
183 | "source": [
184 | "def get_training_durations_for_both_models(device):\n",
185 | " standard_training_times = [get_training_duration(standard_snn, device) for _ in range(11)][1:]\n",
186 | " block_training_times = [get_training_duration(blocks_snn, device) for _ in range(11)][1:]\n",
187 | " return pd.DataFrame({\"standard\": standard_training_times, \"block\": block_training_times})"
188 | ]
189 | },
190 | {
191 | "cell_type": "code",
192 | "execution_count": 104,
193 | "id": "b45b4bdf",
194 | "metadata": {},
195 | "outputs": [],
196 | "source": [
197 | "torch.backends.cudnn.benchmark = True # Make sure we use the best conv algorithm"
198 | ]
199 | },
200 | {
201 | "cell_type": "code",
202 | "execution_count": 107,
203 | "id": "204b6a24",
204 | "metadata": {},
205 | "outputs": [
206 | {
207 | "data": {
208 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqYAAAEpCAYAAABBdeAPAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAxEklEQVR4nO3deXiNd/7/8ddJIifIIoRYGmJfY6mQYjqZTlOK0u1bWkZSXVRLq9IaYm1oBR0E7TC2qulCazCotWlpTU0Raii6UJLJSESRCG3Cyf37o7+eaSZROdwn5448H9d1rivncz73fb/vabzmnXs7NsMwDAEAAAAe5uXpAgAAAACJxhQAAAAWQWMKAAAAS6AxBQAAgCXQmAIAAMASaEwBAABgCTSmAAAAsAQaUwAAAFgCjSkAAAAsgcYUAACYJjw8XI8++qiny0A5RWMKAAAAS6AxBQAAgCXQmAIAAMASaExR7mVkZOjxxx9X3bp1Zbfb1bBhQz399NMqKCjQsmXLZLPZ9Mknn+ipp55SjRo1FBgYqNjYWJ07d67Iemw2m1566aVi6+d6KQA3k1/LzJdeekk2m63YMj9n6YkTJ5xjhmHo5Zdf1i233KIqVarojjvu0Jdfflls2bNnz+rFF19URESE/P39FRgYqJ49e+rAgQPu3E2UUz6eLgC4Ef/5z3/UuXNnnT9/XkOGDFGLFi2UkZGhVatW6dKlS855w4cPV7Vq1fTSSy/pq6++0vz583Xy5Elt3769xBAGgJtRaTOzNCZOnKiXX35ZvXr1Uq9evbRv3z51795dBQUFReYdP35ca9eu1UMPPaSGDRsqKytLf/nLXxQdHa3Dhw+rbt26Zu4iyjkaU5RrCQkJyszM1Oeff67IyEjn+OTJk2UYhvO9r6+vUlJSVKlSJUlSgwYN9Mc//lHr169X3759y7xuAPCE0mbmtWRnZ2vGjBnq3bu31q9f7/wDf9y4cZo6dWqRuREREfr666/l5fXfk7SDBg1SixYttGTJEk2YMOEG9wo3E07lo9wqLCzU2rVr1adPnyIB+7NfHgkdMmSIsymVpKefflo+Pj7auHFjmdQKAJ7mSmZey4cffqiCggI9++yzRZZ7/vnni8212+3OptThcOj777+Xv7+/mjdvrn379rm+I7ip0Zii3MrOzlZubq7atGlzzblNmzYt8t7f31916tQpcr0UANzMXMnMazl58qSk4tlas2ZNBQcHFxkrLCzU7Nmz1bRpU9ntdoWEhKhmzZr617/+pZycnBuuBTcXGlPgGhwOh6dLAIAycbWjpjeSg1OnTlV8fLx++9vf6q233tKWLVu0bds2tW7dWoWFhde9XtycuMYU5VbNmjUVGBioQ4cOXXPuN998ozvuuMP5Pi8vT6dOnVKvXr2cY8HBwTp//nyR5QoKCnTq1CnTagYATylNZv58tPP8+fOqVq2ac/znI6Q/a9CggaSfsrVRo0bO8ezs7GJPPFm1apXuuOMOLVmypMj4+fPnFRIScl37gpsXR0xRbnl5eem+++7T+vXrtXfv3mKf//JC/oULF+ry5cvO9/Pnz9eVK1fUs2dP51jjxo31ySefFFnHwoULOWIK4KZQmsxs3LixJBXJwosXL+rNN98sMjcmJkaVKlXSvHnzimRtcnJysfV6e3sXu7Hq/fffV0ZGxo3sDm5SHDFFuTZ16lRt3bpV0dHRGjJkiFq2bKlTp07p/fff186dO53zCgoKdOedd6pfv3766quv9Oc//1m/+c1vityR/8QTT2jo0KF68MEHddddd+nAgQPasmULf9EDuGlcKzO7d++u+vXr6/HHH9eoUaPk7e2tpUuXqmbNmkpLS3Oup2bNmnrxxReVlJSke+65R7169dL+/fu1adOmYpl5zz33aPLkyRo8eLC6du2qgwcP6u233y5ypBVwMoBy7uTJk0ZsbKxRs2ZNw263G40aNTKGDRtm5OfnG2+88YYhydixY4cxZMgQIzg42PD39zcGDhxofP/990XW43A4jNGjRxshISFGlSpVjB49ehjffvut0aBBAyMuLs4zOwcAJvu1zDQMw0hNTTWioqIMX19fo379+sasWbOcWfrdd9851+NwOIzExESjTp06RuXKlY3f/e53xqFDh4pl5o8//mi88MILznndunUzdu3aZURHRxvR0dFlu/OwPJthuPDgMqCcWbZsmQYPHqw9e/aU+HgUAABgHVxjCgAAAEugMQUAAIAl0JgCAADAErjGFAAAAJbAEVMAAABYAo0pAAAALKFcN6aGYSg3N7fYN0oAAEqHHAVgJeW6Mb1w4YKCgoJ04cIFT5cCAOUSOQrASsp1YwoAAICbB40pAAAALIHGFAAAAJZAYwoAAABLoDEFAACAJdCYAgAAwBJoTAEAAGAJNKYAAACwBB9PFwDzjRgxQtnZ2ZKkmjVras6cOR6uCADKF3IU8AyPHjF1OByaMGGCGjZsqMqVK6tx48aaMmUKX413g7Kzs5WVlaWsrCxnsAIASo8cBTzDo0dMp0+frvnz5+vNN99U69attXfvXg0ePFhBQUF67rnnPFkaAAAAyphHG9PPPvtM9957r3r37i1JCg8P17vvvqvdu3d7siwAQCl0HLXc0yW4TeC5POcpxVPn8m7afU19NdbTJQBFePRUfteuXZWSkqKvv/5aknTgwAHt3LlTPXv29GRZAAAA8ACPHjEdM2aMcnNz1aJFC3l7e8vhcOiVV17RwIEDS5yfn5+v/Px85/vc3NyyKhUAUIEUVqpa4s8A3Mujjel7772nt99+W++8845at26tL774Qs8//7zq1q2ruLi4YvOTkpKUmJjogUoBABVJXnPO3AGe4NFT+aNGjdKYMWP08MMPKyIiQoMGDdLIkSOVlJRU4vyEhATl5OQ4X+np6WVcMQAAANzFo0dML126JC+vor2xt7e3CgsLS5xvt9tlt9vLojQAAACUMY82pn369NErr7yi+vXrq3Xr1tq/f79mzZqlxx57zJNlAQAAwAM82pjOmzdPEyZM0DPPPKPTp0+rbt26euqppzRx4kS3b/tmffSHxGNOAABA+eTRxjQgIEDJyclKTk72ZBkAAACwAI/e/AQAAAD8jMYUAAAAlkBjCgAAAEugMQUAAIAl0JgCAADAEmhMAQAAYAk0pgAAALAEGlMAAABYgkcfsA/3KKxUtcSfAQAArIzG9CaU17ynp0sAAABwGafyAQAAYAk0pgAAALAEGlMAAABYAo0pAAAALIHGFAAAAJZAYwoAAABLoDEFAACAJdCYAgAAwBJoTAEAAGAJNKYAAACwBBpTALCQ119/XeHh4fLz81NUVJR27979q/OTk5PVvHlzVa5cWWFhYRo5cqR+/PHHMqoWAMxFYwoAFrFy5UrFx8dr0qRJ2rdvn9q1a6cePXro9OnTJc5/5513NGbMGE2aNElHjhzRkiVLtHLlSo0dO7aMKwcAc9CYAoBFzJo1S08++aQGDx6sVq1aacGCBapSpYqWLl1a4vzPPvtM3bp104ABAxQeHq7u3bvrkUceueZRVgCwKhpTALCAgoICpaamKiYmxjnm5eWlmJgY7dq1q8RlunbtqtTUVGcjevz4cW3cuFG9evUqk5oBwGw+ni4AACCdOXNGDodDoaGhRcZDQ0N19OjREpcZMGCAzpw5o9/85jcyDENXrlzR0KFDf/VUfn5+vvLz853vc3NzzdkBADABR0wBoJzavn27pk6dqj//+c/at2+fVq9erQ8++EBTpky56jJJSUkKCgpyvsLCwsqwYgD4dRwxBQALCAkJkbe3t7KysoqMZ2VlqXbt2iUuM2HCBA0aNEhPPPGEJCkiIkIXL17UkCFDNG7cOHl5FT/2kJCQoPj4eOf73NxcmlMAlsERUwCwAF9fX3Xs2FEpKSnOscLCQqWkpKhLly4lLnPp0qVizae3t7ckyTCMEpex2+0KDAws8gIAq+CIKQBYRHx8vOLi4hQZGanOnTsrOTlZFy9e1ODBgyVJsbGxqlevnpKSkiRJffr00axZs9ShQwdFRUXp22+/1YQJE9SnTx9ngwoA5QmNKQBYRP/+/ZWdna2JEycqMzNT7du31+bNm503RKWlpRU5Qjp+/HjZbDaNHz9eGRkZqlmzpvr06aNXXnnFU7sAADfEZlztfE85kJubq6CgIOXk5Lh8OqrjqOVuqgplJfXVWE+XAJR75GjFRo7Cam7oGtNfPnIEACo6MhEAboxLjemmTZsUFxenRo0aqVKlSqpSpYoCAwMVHR2tV155Rf/5z3/cVScAWA6ZCADmKlVjumbNGjVr1kyPPfaYfHx8NHr0aK1evVpbtmzR4sWLFR0drQ8//FCNGjXS0KFDlZ2d7e66AcBjyEQAcI9S3fw0Y8YMzZ49Wz179izxuXj9+vWTJGVkZGjevHl66623NHLkSHMrBQCLIBMBwD1K1Zhe7Xua/1e9evU0bdq0GyoIAKyOTAQA9+AB+wAAALAElxvTBx98UNOnTy82PmPGDD300EOmFAUA5QWZCADmcbkx/eSTT9SrV69i4z179tQnn3xiSlEAUF6QiQBgHpcb07y8PPn6+hYbr1SpknJzc00pCgDKCzIRAMzjcmMaERGhlStXFhtfsWKFWrVqZUpRAFBekIkAYJ5S3ZX/SxMmTNADDzygY8eO6fe//70kKSUlRe+++67ef/990wsEACsjEwHAPC43pn369NHatWs1depUrVq1SpUrV1bbtm314YcfKjo62h01AoBlkYkAYB6XG1NJ6t27t3r37m1KARkZGRo9erQ2bdqkS5cuqUmTJnrjjTcUGRlpyvoBwN3MzEQAqMiu6zmm58+f1+LFizV27FidPXtWkrRv3z5lZGS4tJ5z586pW7duqlSpkjZt2qTDhw9r5syZCg4Ovp6yAMAjzMpEAKjoXD5i+q9//UsxMTEKCgrSiRMn9MQTT6h69epavXq10tLStHz58lKva/r06QoLC9Mbb7zhHGvYsKGrJQGAx5iZiQBQ0bl8xDQ+Pl6PPvqovvnmG/n5+TnHe/Xq5fIz+9atW6fIyEg99NBDqlWrljp06KBFixa5WhIAeIyZmQgAFZ3LjemePXv01FNPFRuvV6+eMjMzXVrX8ePHNX/+fDVt2lRbtmzR008/reeee05vvvlmifPz8/OVm5tb5AUAnmRmJgJARefyqXy73V5iQ/j111+rZs2aLq2rsLBQkZGRmjp1qiSpQ4cOOnTokBYsWKC4uLhi85OSkpSYmOhqyQDgNmZmIgBUdC4fMe3bt68mT56sy5cvS5JsNpvS0tI0evRoPfjggy6tq06dOsUeQN2yZUulpaWVOD8hIUE5OTnOV3p6uqvlA4CpzMxEAKjoXG5MZ86cqby8PNWqVUs//PCDoqOj1aRJEwUEBOiVV15xaV3dunXTV199VWTs66+/VoMGDUqcb7fbFRgYWOQFAJ5kZiYCQEXn8qn8oKAgbdu2Tf/4xz904MAB5eXl6dZbb1VMTIzLGx85cqS6du2qqVOnql+/ftq9e7cWLlyohQsXurwuAPAEMzMRACq663rAvvTT0c5u3bpJ+ukZftejU6dOWrNmjRISEjR58mQ1bNhQycnJGjhw4PWWBQAeYUYmAkBF5/Kp/OnTp2vlypXO9/369VONGjVUr149HThwwOUC7rnnHh08eFA//vijjhw5oieffNLldQCAp5idiQBQkbncmC5YsEBhYWGSpG3btmnbtm3atGmTevbsqVGjRpleIABYGZkIAOZx+VR+ZmamM4Q3bNigfv36qXv37goPD1dUVJTpBQKAlZGJAGAel4+YBgcHOx/TtHnzZucF/oZhyOFwmFsdAFgcmQgA5nH5iOkDDzygAQMGqGnTpvr+++/Vs2dPSdL+/fvVpEkT0wsEACsjEwHAPC43prNnz1Z4eLjS09M1Y8YM+fv7S5JOnTqlZ555xvQCAcDKyEQAMI/NMAzD00Vcr9zcXAUFBSknJ8flh+13HLXcTVWhrKS+GuvpEoByjxyt2MhRWE2prjH95z//WeoVXrp0SV9++eV1FwQAVkcmAoB7lKoxHTRokHr06KH3339fFy9eLHHO4cOHNXbsWDVu3FipqammFgkAVkImAoB7lOoa08OHD2v+/PkaP368BgwYoGbNmqlu3bry8/PTuXPndPToUeXl5en+++/X1q1bFRER4e66AcBjyEQAcA+XrzHdu3evdu7cqZMnT+qHH35QSEiIOnTooDvuuEPVq1d3V50l4tqoio1ro2AFVsrE60GOVmzkKKzG5bvyIyMjFRkZ6Y5aAKDcIRMBwDwuP2AfAAAAcAcaUwAAAFgCjSkAWMjrr7+u8PBw+fn5KSoqSrt37/7V+efPn9ewYcNUp04d2e12NWvWTBs3biyjagHAXC5fYwoAcI+VK1cqPj5eCxYsUFRUlJKTk9WjRw999dVXqlWrVrH5BQUFuuuuu1SrVi2tWrVK9erV08mTJ1WtWrWyLx4ATHBDjemPP/4oPz8/s2oBgHLtRjNx1qxZevLJJzV48GBJ0oIFC/TBBx9o6dKlGjNmTLH5S5cu1dmzZ/XZZ5+pUqVKkqTw8PDr3j4AeJrLp/ILCws1ZcoU1atXT/7+/jp+/LgkacKECVqyZInpBQKAlZmViQUFBUpNTVVMTIxzzMvLSzExMdq1a1eJy6xbt05dunTRsGHDFBoaqjZt2mjq1KlyOBxX3U5+fr5yc3OLvADAKlxuTF9++WUtW7ZMM2bMkK+vr3O8TZs2Wrx4sanFAYDVmZWJZ86ckcPhUGhoaJHx0NBQZWZmlrjM8ePHtWrVKjkcDm3cuFETJkzQzJkz9fLLL191O0lJSQoKCnK+wsLCSl0jALiby43p8uXLtXDhQg0cOFDe3t7O8Xbt2uno0aOmFgcAVufJTCwsLFStWrW0cOFCdezYUf3799e4ceO0YMGCqy6TkJCgnJwc5ys9Pd2tNQKAK1y+xjQjI0NNmjQpNl5YWKjLly+bUhQAlBdmZWJISIi8vb2VlZVVZDwrK0u1a9cucZk6deqoUqVKRRrili1bKjMzUwUFBUWO4P7MbrfLbreXui4AKEsuHzFt1aqVPv3002Ljq1atUocOHUwpCgDKC7My0dfXVx07dlRKSopzrLCwUCkpKerSpUuJy3Tr1k3ffvutCgsLnWNff/216tSpU2JTCgBW5/IR04kTJyouLk4ZGRkqLCzU6tWr9dVXX2n58uXasGGDO2oEAMsyMxPj4+MVFxenyMhIde7cWcnJybp48aLzLv3Y2FjVq1dPSUlJkqSnn35ar732mkaMGKFnn31W33zzjaZOnarnnnvO9P0EgLLgcmN67733av369Zo8ebKqVq2qiRMn6tZbb9X69et11113uaNGALAsMzOxf//+ys7O1sSJE5WZman27dtr8+bNzhui0tLS5OX13xNdYWFh2rJli0aOHKm2bduqXr16GjFihEaPHm3qPgJAWbEZhmF4uojrlZubq6CgIOXk5CgwMNClZTuOWu6mqlBWUl+N9XQJQLlHjlZs5Cis5oYesJ+Xl1fk2iZJLgcbANwsyEQAuDEu3/z03XffqXfv3qpataqCgoIUHBys4OBgVatWTcHBwe6oEQAsi0wEAPO4fMT0D3/4gwzD0NKlSxUaGiqbzeaOugCgXCATAcA8LjemBw4cUGpqqpo3b+6OegCgXCETAcA8Lp/K79SpE98UAgD/H5kIAOZx+Yjp4sWLNXToUGVkZKhNmzaqVKlSkc/btm1rWnEAYHVkIgCYx+XGNDs7W8eOHXM+8FmSbDabDMOQzWaTw+EwtUAAsDIyEQDM43Jj+thjj6lDhw569913udAfQIVHJgKAeVxuTE+ePKl169apSZMm7qgHAMoVMhEAzOPyzU+///3vdeDAAXfUAgDlDpkIAOZx+Yhpnz59NHLkSB08eFARERHFLvTv27evacUBgNWRiQBgHpcb06FDh0qSJk+eXOwzLvQHUNGQiQBgHpcb0//9HmgAqMjIRAAwj8vXmAIAAADuUKojpnPnztWQIUPk5+enuXPn/urc5557zpTCAMCqyEQAcA+bYRjGtSY1bNhQe/fuVY0aNdSwYcOrr8xm0/Hjx00t8Nfk5uYqKChIOTk5CgwMdGnZjqOWu6kqlJXUV2M9XQIqKKtm4vUgRys2chRWU6ojpt99912JPwNARUQmAoB7uHyN6eTJk3Xp0qVi4z/88EOJd6UCwM2MTAQA87jcmCYmJiovL6/Y+KVLl5SYmGhKUQBQXpCJAGAelxtTwzBK/C7oAwcOqHr16tddyLRp02Sz2fT8889f9zoAoKy5KxMBoCIq9XNMg4ODZbPZZLPZ1KxZsyJB7HA4lJeX53zQtKv27Nmjv/zlL2rbtu11LQ8AZc2dmQgAFVWpG9Pk5GQZhqHHHntMiYmJCgoKcn7m6+ur8PBwdenSxeUC8vLyNHDgQC1atEgvv/yyy8sDgCe4KxMBoCIrdWMaFxcn6afHpHTr1k0+Pi5/aVSJhg0bpt69eysmJobGFEC54a5MBICKzOUkjY6ONm3jK1as0L59+7Rnz55Szc/Pz1d+fr7zfW5urmm1AMD1MDMTAaCi89hXkqanp2vEiBF6++235efnV6plkpKSFBQU5HyFhYW5uUoAAACUFY81pqmpqTp9+rRuvfVW+fj4yMfHRzt27NDcuXPl4+Mjh8NRbJmEhATl5OQ4X+np6R6oHAAAAO7gsYui7rzzTh08eLDI2ODBg9WiRQuNHj1a3t7exZax2+2y2+1lVSIAAADKkMca04CAALVp06bIWNWqVVWjRo1i4wAAALj5udyY3n///SU+TNpms8nPz09NmjTRgAED1Lx5c1MKBAArIxMBwDwuX2MaFBSkjz76SPv27XM+XHr//v366KOPdOXKFa1cuVLt2rXTP/7xD5eL2b59u5KTk11eDgA8xZ2ZCAAVjctHTGvXrq0BAwbotddek5fXT31tYWGhRowYoYCAAK1YsUJDhw7V6NGjtXPnTtMLBgArIRMBwDwuHzFdsmSJnn/+eWcAS5KXl5eeffZZLVy4UDabTcOHD9ehQ4dMLRQArIhMBADzuNyYXrlyRUePHi02fvToUecjnvz8/Eq85goAbjZkIgCYx+VT+YMGDdLjjz+usWPHqlOnTpKkPXv2aOrUqYqNjZUk7dixQ61btza3UgCwIDIRAMzjcmM6e/ZshYaGasaMGcrKypIkhYaGauTIkRo9erQkqXv37rr77rvNrRQALIhMBADzuHwq39vbW+PGjdOpU6d0/vx5nT9/XqdOndLYsWOdD8WvX7++brnlFtOLBQCrcUcmvv766woPD5efn5+ioqK0e/fuUi23YsUK2Ww23XfffdezKwDgcTf0laSBgYEKDAw0qxYAKNfMyMSVK1cqPj5ekyZN0r59+9SuXTv16NFDp0+f/tXlTpw4oRdffFG33377DW0fADzJ5cY0KytLgwYNUt26deXj4yNvb+8iLwCoSMzOxFmzZunJJ5/U4MGD1apVKy1YsEBVqlTR0qVLr7qMw+HQwIEDlZiYqEaNGt3I7gCAR7l8jemjjz6qtLQ0TZgwQXXq1OFOUwAVmpmZWFBQoNTUVCUkJDjHvLy8FBMTo127dl11ucmTJ6tWrVp6/PHH9emnn1739gHA01xuTHfu3KlPP/1U7du3d0M5AFC+mJmJZ86ckcPhUGhoaJHx0NDQEh9J9fP2lyxZoi+++KJU28jPz1d+fr7zfW5u7nXXCwBmc/lUflhYmAzDcEctAFDueDITL1y4oEGDBmnRokUKCQkp1TJJSUkKCgpyvsLCwtxcJQCUnsuNaXJyssaMGaMTJ064oRwAKF/MzMSQkBB5e3s7Hzv1s6ysLNWuXbvY/GPHjunEiRPq06ePfHx85OPjo+XLl2vdunXy8fHRsWPHii2TkJCgnJwc5ys9Pf2G6wYAs7h8Kr9///66dOmSGjdurCpVqqhSpUpFPj979qxpxQGA1ZmZib6+vurYsaNSUlKcj3wqLCxUSkqKhg8fXmx+ixYtdPDgwSJj48eP14ULFzRnzpwSj4ba7XbZ7fZS1wQAZcnlxjQ5OdkNZQBA+WR2JsbHxysuLk6RkZHq3LmzkpOTdfHiRQ0ePFiSFBsbq3r16ikpKUl+fn5q06ZNkeWrVasmScXGAaA8cLkxjYuLc0cdAFAumZ2J/fv3V3Z2tiZOnKjMzEy1b99emzdvdt4QlZaWJi+vG3oENQBYVqka09zcXOdDo691BycP3Adws3N3Jg4fPrzEU/eStH379l9ddtmyZS5vDwCsolSNaXBwsE6dOqVatWqpWrVqJT6nzzAM2Ww2ORwO04sEACshEwHAPUrVmH700UeqXr26JOnjjz92a0EAYHVkIgC4R6ka0+jo6BJ/BoCKiEwEAPdw+eYnSTp//rx2796t06dPq7CwsMhnsbGxphQGAOUFmQgA5nC5MV2/fr0GDhyovLw8BQYGFrm2ymazEcIAKhQyEQDM4/IzR1544QU99thjysvL0/nz53Xu3Dnni4frA6hoyEQAMI/LjWlGRoaee+45ValSxR31AEC5QiYCgHlcbkx79OihvXv3uqMWACh3yEQAMI/L15j27t1bo0aN0uHDhxUREVHse6H79u1rWnEAYHVkIgCYx+XG9Mknn5QkTZ48udhnPEwaQEVDJgKAeVxuTP/3USgAUJGRiQBgHpevMQUAAADcoVRHTOfOnashQ4bIz89Pc+fO/dW5zz33nCmFAYBVkYkA4B42wzCMa01q2LCh9u7dqxo1aqhhw4ZXX5nNpuPHj5ta4K/Jzc1VUFCQcnJyFBgY6NKyHUctd1NVKCupr/LgcniGVTPxepCjFRs5Cqsp1RHT7777rsSfAaAiIhMBwD24xhQAAACW4PJd+ZL073//W+vWrVNaWpoKCgqKfDZr1ixTCgOA8oJMBABzuNyYpqSkqG/fvmrUqJGOHj2qNm3a6MSJEzIMQ7feeqs7agQAyyITAcA8Lp/KT0hI0IsvvqiDBw/Kz89Pf/vb35Senq7o6Gg99NBD7qgRACyLTAQA87jcmB45ckSxsT/dxefj46MffvhB/v7+mjx5sqZPn256gQBgZWQiAJjH5ca0atWqzmuo6tSpo2PHjjk/O3PmjHmVAUA5QCYCgHlcvsb0tttu086dO9WyZUv16tVLL7zwgg4ePKjVq1frtttuc0eNAGBZZCIAmMflxnTWrFnKy8uTJCUmJiovL08rV65U06ZNufsUQIVDJgKAeVxqTB0Oh/7973+rbdu2kn46hbVgwQK3FAYAVkcmAoC5XLrG1NvbW927d9e5c+fcVQ8AlBtkIgCYy+Wbn9q0aWP5734GgLJCJgKAeVxuTF9++WW9+OKL2rBhg06dOqXc3NwiLwCoSMhEADBPqa8xnTx5sl544QX16tVLktS3b1/ZbDbn54ZhyGazyeFwmF8lAFgMmQgA5it1Y5qYmKihQ4fq448/Nm3jSUlJWr16tY4eParKlSura9eumj59upo3b27aNgDAHdyRiQBQ0ZW6MTUMQ5IUHR1t2sZ37NihYcOGqVOnTrpy5YrGjh2r7t276/Dhw6patapp2wEAs7kjEwGgonPpcVG/PE1lhs2bNxd5v2zZMtWqVUupqan67W9/a+q2AMBsZmciAFR0LjWmzZo1u2YQnz179rqLycnJkSRVr169xM/z8/OVn5/vfM+NBQA8yd2ZCAAVjUuNaWJiooKCgtxSSGFhoZ5//nl169ZNbdq0KXFOUlKSEhMT3bJ9AHCVOzMRACoilxrThx9+WLVq1XJLIcOGDdOhQ4e0c+fOq85JSEhQfHy8831ubq7CwsLcUg8AXIs7MxEAKqJSN6buvJZq+PDh2rBhgz755BPdcsstV51nt9tlt9vdVgcAlBbXlwKA+Ur9gP2f70A1k2EYGj58uNasWaOPPvpIDRs2NH0bAOAO7sjEn73++usKDw+Xn5+foqKitHv37qvOXbRokW6//XYFBwcrODhYMTExvzofAKys1I1pYWGh6aeshg0bprfeekvvvPOOAgIClJmZqczMTP3www+mbgcAzOaOTJSklStXKj4+XpMmTdK+ffvUrl079ejRQ6dPny5x/vbt2/XII4/o448/1q5duxQWFqbu3bsrIyPD9NoAwN1shjv/7L/Wxq9yKuyNN97Qo48+es3lc3NzFRQUpJycHAUGBrq07Y6jlrs0H9aT+mqsp0sATBcVFaVOnTrptddek/RTAxwWFqZnn31WY8aMuebyDodDwcHBeu211xQbe+1/I+RoxUaOwmpcuvnJbB7siQHAcgoKCpSamqqEhATnmJeXl2JiYrRr165SrePSpUu6fPnyVR+7BwBW5tHGFADwX2fOnJHD4VBoaGiR8dDQUB09erRU6xg9erTq1q2rmJiYEj/nedAArKzU15gCAKxt2rRpWrFihdasWSM/P78S5yQlJSkoKMj54pF7AKyExhQALCIkJETe3t7KysoqMp6VlaXatWv/6rJ/+tOfNG3aNG3dulVt27a96ryEhATl5OQ4X+np6abUDgBmoDEFAIvw9fVVx44dlZKS4hwrLCxUSkqKunTpctXlZsyYoSlTpmjz5s2KjIz81W3Y7XYFBgYWeQGAVXCNKQBYSHx8vOLi4hQZGanOnTsrOTlZFy9e1ODBgyVJsbGxqlevnpKSkiRJ06dP18SJE/XOO+8oPDxcmZmZkiR/f3/5+/t7bD8A4HrQmAKAhfTv31/Z2dmaOHGiMjMz1b59e23evNl5Q1RaWpq8vP57smv+/PkqKCjQ//3f/xVZz6RJk/TSSy+VZekAcMNoTAHAYoYPH67hw4eX+Nn27duLvD9x4oT7CwKAMsI1pgAAALAEGlMAAABYAo0pAAAALIHGFAAAAJZAYwoAAABLoDEFAACAJdCYAgAAwBJoTAEAAGAJNKYAAACwBBpTAAAAWAKNKQAAACyBxhQAAACWQGMKAAAAS6AxBQAAgCXQmAIAAMASaEwBAABgCTSmAAAAsAQaUwAAAFgCjSkAAAAsgcYUAAAAlkBjCgAAAEugMQUAAIAl0JgCAADAEmhMAQAAYAk0pgAAALAEGlMAAABYAo0pAAAALIHGFAAAAJbg4+kCAHjeiBEjlJ2dLUmqWbOm5syZ4+GKANyM0iZHeLoE3KD6Ew+6df00pgCUnZ2trKwsT5cBAKjgOJUPAAAAS+CIKVBKN/MpqCvna0jy/v8//+em3Vd3n4ICANwYjpgCAADAEjhiCkDV7Y4SfwYAoCzRmALQ2A7nPV0CAACcygcAAIA1WKIxff311xUeHi4/Pz9FRUVp9+7dni4JADzG1Ux8//331aJFC/n5+SkiIkIbN24so0oBwFweb0xXrlyp+Ph4TZo0Sfv27VO7du3Uo0cPnT592tOlAUCZczUTP/vsMz3yyCN6/PHHtX//ft1333267777dOjQoTKuHABunMcb01mzZunJJ5/U4MGD1apVKy1YsEBVqlTR0qVLPV0aAJQ5VzNxzpw5uvvuuzVq1Ci1bNlSU6ZM0a233qrXXnutjCsHgBvn0ZufCgoKlJqaqoSEBOeYl5eXYmJitGvXrmLz8/PzlZ+f73yfk5MjScrNzXV52478H66jYljJ9fx3vxEXfuRu9fLuRn5nAgICZLPZTKymOFczUZJ27dql+Pj4ImM9evTQ2rVrS5xPjuKXyFG4yt056tHG9MyZM3I4HAoNDS0yHhoaqqNHjxabn5SUpMTExGLjYWFhbqsR1hU0b6inS0B5kxR03Yvm5OQoMDDQxGKKczUTJSkzM7PE+ZmZmSXOJ0fxS+QoXObmHC1Xj4tKSEgocmSgsLBQZ8+eVY0aNdx+JKO8yc3NVVhYmNLT093+f6a4OfA78+sCAgI8XYIpyNHS498EXMXvzK8rTY56tDENCQmRt7e3srKyioxnZWWpdu3axebb7XbZ7fYiY9WqVXNnieVeYGAg/zjgEn5nPMfVTJSk2rVruzSfHHUd/ybgKn5nrp9Hb37y9fVVx44dlZKS4hwrLCxUSkqKunTp4sHKAKDsXU8mdunSpch8Sdq2bRsZCqBc8vip/Pj4eMXFxSkyMlKdO3dWcnKyLl68qMGDB3u6NAAoc9fKxNjYWNWrV09JSUmSpBEjRig6OlozZ85U7969tWLFCu3du1cLFy705G4AwHXxeGPav39/ZWdna+LEicrMzFT79u21efPmYhfzwzV2u12TJk0qdsoOuBp+Z6zhWpmYlpYmL6//nuzq2rWr3nnnHY0fP15jx45V06ZNtXbtWrVp08ZTu3DT4N8EXMXvzI2zGYZheLoIAAAAwOMP2AcAAAAkGlMAAABYBI0pAAAALIHGtAL53e9+p+eff77crRvXdq3//cPDw5WcnGza9sxeH1CekKU3J3LUGmhMPejRRx/Vfffd5+kyAKBcI0uBmweNKUrNMAxduXLF02UAQLlGlgJXR2NaBlatWqWIiAhVrlxZNWrUUExMjEaNGqU333xTf//732Wz2WSz2bR9+3ZJ0ujRo9WsWTNVqVJFjRo10oQJE3T58mXn+l566SW1b99ef/3rXxUeHq6goCA9/PDDunDhgnPOxYsXFRsbK39/f9WpU0czZ84sVtdf//pXRUZGKiAgQLVr19aAAQN0+vRp5+fbt2+XzWbTpk2b1LFjR9ntdu3cubNU60bZu3LlioYPH66goCCFhIRowoQJutrT4NLS0nTvvffK399fgYGB6tevX7GvtVy/fr06deokPz8/hYSE6P7777/qthcvXqxq1aoV+wYiwExkKdyNHPU8GlM3O3XqlB555BE99thjOnLkiLZv364HHnhAkyZNUr9+/XT33Xfr1KlTOnXqlLp27SpJCggI0LJly3T48GHNmTNHixYt0uzZs4us99ixY1q7dq02bNigDRs2aMeOHZo2bZrz81GjRmnHjh36+9//rq1bt2r79u3at29fkXVcvnxZU6ZM0YEDB7R27VqdOHFCjz76aLF9GDNmjKZNm6YjR46obdu2pVo3yt6bb74pHx8f7d69W3PmzNGsWbO0ePHiYvMKCwt177336uzZs9qxY4e2bdum48ePq3///s45H3zwge6//3716tVL+/fvV0pKijp37lzidmfMmKExY8Zo69atuvPOO922f6jYyFKUBXLUAgy4VWpqqiHJOHHiRLHP4uLijHvvvfea63j11VeNjh07Ot9PmjTJqFKlipGbm+scGzVqlBEVFWUYhmFcuHDB8PX1Nd577z3n599//71RuXJlY8SIEVfdzp49ewxJxoULFwzDMIyPP/7YkGSsXbvWOed61w33io6ONlq2bGkUFhY6x0aPHm20bNnSMAzDaNCggTF79mzDMAxj69athre3t5GWluac++WXXxqSjN27dxuGYRhdunQxBg4ceNXt/by+P/7xj0adOnWMQ4cOuWGvgP8iS+Fu5Kg1cMTUzdq1a6c777xTEREReuihh7Ro0SKdO3fuV5dZuXKlunXrptq1a8vf31/jx49XWlpakTnh4eEKCAhwvq9Tp47z1NGxY8dUUFCgqKgo5+fVq1dX8+bNi6wjNTVVffr0Uf369RUQEKDo6GhJKratyMhI58+lXTfK3m233SabzeZ836VLF33zzTdyOBxF5h05ckRhYWEKCwtzjrVq1UrVqlXTkSNHJElffPHFNf9qnzlzphYtWqSdO3eqdevWJu4JUBxZirJAjnoejambeXt7a9u2bdq0aZNatWqlefPmqXnz5vruu+9KnL9r1y4NHDhQvXr10oYNG7R//36NGzdOBQUFReZVqlSpyHubzabCwsJS13Xx4kX16NFDgYGBevvtt7Vnzx6tWbNGkoptq2rVqqVeL24OlStXvuac22+/XQ6HQ++9914ZVISKjixFeUOOXh8a0zJgs9nUrVs3JSYmav/+/fL19dWaNWvk6+tb7K+wzz77TA0aNNC4ceMUGRmppk2b6uTJky5tr3HjxqpUqZI+//xz59i5c+f09ddfO98fPXpU33//vaZNm6bbb79dLVq0KHKx/o2sG57xy/8mkvTPf/5TTZs2lbe3d5Hxli1bKj09Xenp6c6xw4cP6/z582rVqpUkqW3btte8AL9z587atGmTpk6dqj/96U8m7QVwdWQp3I0c9TwfTxdws/v888+VkpKi7t27q1atWvr888+VnZ2tli1b6scff9SWLVv01VdfqUaNGgoKClLTpk2VlpamFStWqFOnTvrggw+cf32Xlr+/vx5//HGNGjVKNWrUUK1atTRu3Dh5ef3375D69evL19dX8+bN09ChQ3Xo0CFNmTLFlHXDM9LS0hQfH6+nnnpK+/bt07x580q8yzcmJkYREREaOHCgkpOTdeXKFT3zzDOKjo52nmqcNGmS7rzzTjVu3FgPP/ywrly5oo0bN2r06NFF1tW1a1dt3LhRPXv2lI+PDw8Gh9uQpSgL5KgFePoi15vd4cOHjR49ehg1a9Y07Ha70axZM2PevHmGYRjG6dOnjbvuusvw9/c3JBkff/yxYRg/XXxfo0YNw9/f3+jfv78xe/ZsIygoyLnOSZMmGe3atSuyndmzZxsNGjRwvr9w4YLxhz/8wahSpYoRGhpqzJgxw4iOji5yUf0777xjhIeHG3a73ejSpYuxbt06Q5Kxf/9+wzD+e8H+uXPnimyrNOtG2YqOjjaeeeYZY+jQoUZgYKARHBxsjB071nkR/y8v2jcMwzh58qTRt29fo2rVqkZAQIDx0EMPGZmZmUXW+be//c1o37694evra4SEhBgPPPCA87P/Xd+OHTuMqlWrGnPnznXrfqLiIkvhbuSoNdgM4yoP6AIAAADKEOcMAAAAYAk0pgAAALAEGlMAAABYAo0pAAAALIHGFAAAAJZAYwoAAABLoDEFAACAJdCYAgAAwBJoTAEAAGAJNKYAAACwBBpTAAAAWAKNKQAAACzh/wF+f/hj5kkauQAAAABJRU5ErkJggg==",
209 | "text/plain": [
210 | ""
211 | ]
212 | },
213 | "metadata": {},
214 | "output_type": "display_data"
215 | }
216 | ],
217 | "source": [
218 | "fig, axs = plt.subplots(1, 2, figsize=(8, 3), sharey=False)\n",
219 | "\n",
220 | "def plot_training_durations(ax, device):\n",
221 | " sns.barplot(get_training_durations_for_both_models(device), ax=ax)\n",
222 | " ax.set(ylabel=\"Training time (sec)\", title=device)\n",
223 | " sns.despine()\n",
224 | " \n",
225 | "plot_training_durations(axs[0], \"cpu\")\n",
226 | "plot_training_durations(axs[1], \"cuda\")"
227 | ]
228 | },
229 | {
230 | "cell_type": "markdown",
231 | "id": "f9e54af9",
232 | "metadata": {},
233 | "source": [
234 | "Voila! A condsiderable reduction in training time, on both CPUs and GPUs!"
235 | ]
236 | }
237 | ],
238 | "metadata": {
239 | "kernelspec": {
240 | "display_name": "Python [conda env:blocks] *",
241 | "language": "python",
242 | "name": "conda-env-blocks-py"
243 | },
244 | "language_info": {
245 | "codemirror_mode": {
246 | "name": "ipython",
247 | "version": 3
248 | },
249 | "file_extension": ".py",
250 | "mimetype": "text/x-python",
251 | "name": "python",
252 | "nbconvert_exporter": "python",
253 | "pygments_lexer": "ipython3",
254 | "version": "3.8.16"
255 | }
256 | },
257 | "nbformat": 4,
258 | "nbformat_minor": 5
259 | }
260 |
--------------------------------------------------------------------------------
/notebooks/library_comparison.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": 53,
6 | "id": "bef22c5e",
7 | "metadata": {},
8 | "outputs": [],
9 | "source": [
10 | "import time\n",
11 | "\n",
12 | "import torch\n",
13 | "import torch.nn as nn\n",
14 | "from spikingjelly.activation_based import neuron, layer, surrogate\n",
15 | "from norse.torch.module.lif import LIFCell\n",
16 | "\n",
17 | "from src.snn.block.blocks import Blocks\n",
18 | "from src.snn.snn import SNN"
19 | ]
20 | },
21 | {
22 | "cell_type": "markdown",
23 | "id": "f8051e64",
24 | "metadata": {},
25 | "source": [
26 | "## Setting up the different implementations"
27 | ]
28 | },
29 | {
30 | "cell_type": "markdown",
31 | "id": "4ca0b48b",
32 | "metadata": {},
33 | "source": [
34 | "Network benchmarked: 200 input units -> 100 spiking units over 1000 simulation steps using a batch size of 128."
35 | ]
36 | },
37 | {
38 | "cell_type": "code",
39 | "execution_count": 93,
40 | "id": "38193107",
41 | "metadata": {},
42 | "outputs": [],
43 | "source": [
44 | "def time_jelly():\n",
45 | " input_tensor = torch.rand(128, 200, 1000).cuda()\n",
46 | " \n",
47 | " jelly_layer = nn.Sequential(\n",
48 | " layer.Linear(200, 100, bias=False),\n",
49 | " neuron.LIFNode(tau=100.0, surrogate_function=surrogate.ATan())\n",
50 | " ).cuda()\n",
51 | " \n",
52 | " start_time = time.time()\n",
53 | " \n",
54 | " for t in range(1000):\n",
55 | " out = jelly_layer(input_tensor[:, :, t])\n",
56 | " \n",
57 | " end_time = time.time()\n",
58 | " return end_time - start_time"
59 | ]
60 | },
61 | {
62 | "cell_type": "code",
63 | "execution_count": 94,
64 | "id": "976ed321",
65 | "metadata": {},
66 | "outputs": [],
67 | "source": [
68 | "def time_norse():\n",
69 | " input_tensor = torch.rand(128, 200, 1000).cuda()\n",
70 | " \n",
71 | " norse_layer = nn.Sequential(\n",
72 | " layer.Linear(200, 100, bias=False),\n",
73 | " LIFCell()\n",
74 | " ).cuda()\n",
75 | " \n",
76 | " start_time = time.time()\n",
77 | " \n",
78 | " for t in range(1000):\n",
79 | " out = norse_layer(input_tensor[:, :, t])\n",
80 | " \n",
81 | " end_time = time.time()\n",
82 | " \n",
83 | " return end_time - start_time"
84 | ]
85 | },
86 | {
87 | "cell_type": "code",
88 | "execution_count": 95,
89 | "id": "932cb5f0",
90 | "metadata": {},
91 | "outputs": [],
92 | "source": [
93 | "def time_blocks():\n",
94 | " input_tensor = torch.rand(128, 200, 1000).cuda()\n",
95 | " \n",
96 | " blocks_snn = Blocks(200, 100, 1, 1000, t_latency=50, recurrent=False, init_beta=0.99, init_p=0.99).cuda()\n",
97 | " start_time = time.time()\n",
98 | " out = blocks_snn(input_tensor)\n",
99 | " end_time = time.time()\n",
100 | " \n",
101 | " return end_time - start_time\n",
102 | " \n",
103 | "def time_standard():\n",
104 | " input_tensor = torch.rand(128, 200, 1000).cuda()\n",
105 | " \n",
106 | " blocks_snn = SNN(200, 100, 1, 1000, t_latency=1, recurrent=False, init_beta=0.99, init_p=0.99).cuda()\n",
107 | " start_time = time.time()\n",
108 | " out = blocks_snn(input_tensor)\n",
109 | " end_time = time.time()\n",
110 | " \n",
111 | " return end_time - start_time"
112 | ]
113 | },
114 | {
115 | "cell_type": "markdown",
116 | "id": "733c9e80",
117 | "metadata": {},
118 | "source": [
119 | "## Benchmarking the differnet implementations"
120 | ]
121 | },
122 | {
123 | "cell_type": "code",
124 | "execution_count": 97,
125 | "id": "0ecf6b39",
126 | "metadata": {},
127 | "outputs": [
128 | {
129 | "name": "stdout",
130 | "output_type": "stream",
131 | "text": [
132 | "Norse=0.3080105781555176\n",
133 | "Jelly=0.1869184970855713\n",
134 | "Standard=0.3317074775695801\n",
135 | "Blocks=0.016329288482666016\n"
136 | ]
137 | }
138 | ],
139 | "source": [
140 | "print(f\"Norse={time_norse()}\")\n",
141 | "print(f\"Jelly={time_jelly()}\")\n",
142 | "print(f\"Standard={time_standard()}\")\n",
143 | "print(f\"Blocks={time_blocks()}\")"
144 | ]
145 | }
146 | ],
147 | "metadata": {
148 | "kernelspec": {
149 | "display_name": "Python [conda env:blocks] *",
150 | "language": "python",
151 | "name": "conda-env-blocks-py"
152 | },
153 | "language_info": {
154 | "codemirror_mode": {
155 | "name": "ipython",
156 | "version": 3
157 | },
158 | "file_extension": ".py",
159 | "mimetype": "text/x-python",
160 | "name": "python",
161 | "nbconvert_exporter": "python",
162 | "pygments_lexer": "ipython3",
163 | "version": "3.8.16"
164 | }
165 | },
166 | "nbformat": 4,
167 | "nbformat_minor": 5
168 | }
169 |
--------------------------------------------------------------------------------
/scripts/ephys/build_data.py:
--------------------------------------------------------------------------------
1 | import argparse
2 |
3 | from src import datasets
4 |
5 |
6 | def main():
7 | parser = argparse.ArgumentParser()
8 | parser.add_argument("--root", type=str, default=".")
9 | args = parser.parse_args()
10 |
11 | # Builds current and spike tensors with DT=0.1ms
12 | # Note: You might want to set manifest_file if you have already downloaded the data from the Allen Institute
13 | builder = datasets.NoiseBuilder()
14 | builder.build(f"{args.root}/Blocks/data/ephys/train", noise_type="noise1")
15 | builder.build(f"{args.root}/Blocks/data/ephys/test", noise_type="noise2")
16 |
17 | # Builds current tensors with DT=0.05ms
18 | builder = datasets.NoiseBuilder()
19 | builder.build(f"{args.root}/Blocks/data/ephys/train", noise_type="noise1", target_sampling_rate=20000)
20 | builder.build(f"{args.root}/Blocks/data/ephys/test", noise_type="noise2", target_sampling_rate=20000)
21 |
22 |
23 | if __name__ == "__main__":
24 | main()
25 |
--------------------------------------------------------------------------------
/scripts/ephys/run.py:
--------------------------------------------------------------------------------
1 | import os
2 | root = "" # TODO: Change this to the project folder
3 |
4 | from src import datasets
5 |
6 |
7 | def launch(method, abs_refac_ms, downsample):
8 | neuron_idx_query = datasets.ValidNeuronQuery(f"{root}/data/ephys")
9 |
10 | dir_name = f"{method}_{abs_refac_ms}_{downsample}"
11 | os.makedirs(f"{root}/results/ephys/{dir_name}")
12 |
13 | for neuron_idx in neuron_idx_query.idx:
14 | os.system(f"python {root}/scripts/ephys/train.py --root={root} --method={method} --abs_refac_ms={abs_refac_ms} --downsample={downsample} --neuron_idx={neuron_idx} --dir_name={dir_name} --id={neuron_idx}")
15 |
16 |
17 | # Blocks: Different dt
18 | for downsample in [0.5, 1, 5, 10, 20, 40]:
19 | launch("blocks", abs_refac_ms=2, downsample=downsample)
20 |
21 | # Blocks: Different ARP
22 | for abs_refac_ms in [1, 4, 6, 8, 16]:
23 | launch("blocks", abs_refac_ms=abs_refac_ms, downsample=1)
24 |
25 | # Standard
26 | launch("standard", abs_refac_ms=2, downsample=1)
27 |
--------------------------------------------------------------------------------
/scripts/ephys/train.py:
--------------------------------------------------------------------------------
1 | import sys
2 | import ast
3 | import logging
4 | import argparse
5 |
6 | from src import datasets, models, train
7 |
8 | logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
9 |
10 |
11 | def eval(v):
12 | return ast.literal_eval(v)
13 |
14 |
15 | def get_dataset(args):
16 | if args.downsample == 0.5:
17 | # Used dt=0.05ms for neural fits
18 | return datasets.EphysDataset(f"{args.root}/data/ephys", "train", args.neuron_idx, target_sampling_rate=20000)
19 | else:
20 | # Used dt>=0.1ms for neural fits (used by most experiments)
21 | return datasets.EphysDataset(f"{args.root}/data/ephys", "train", args.neuron_idx)
22 |
23 |
24 | def get_model(args):
25 | if args.downsample == 0.5:
26 | # Used dt=0.05ms for neural fits
27 | return models.Neuron(method=args.method, abs_refac_ms=args.abs_refac_ms, downsample=1, dt01ref=True)
28 | else:
29 | # Used dt>=0.1ms for neural fits (used by most experiments)
30 | return models.Neuron(method=args.method, abs_refac_ms=args.abs_refac_ms, downsample=int(args.downsample))
31 |
32 |
33 | def get_trainer(args, model, train_dataset):
34 | n_epochs = 200
35 | batch_size = 5
36 | lr = 0.0001
37 | dt = 0.1 * args.downsample
38 | return train.EphysTrainer(f"{args.root}/results/ephys/{args.dir_name}", model, train_dataset, n_epochs, batch_size, lr, gamma=0.1, dt=dt, epoch_scan=5, max_decay=0, val_dataset=None, device="cuda", id=args.id)
39 |
40 |
41 | def main():
42 | parser = argparse.ArgumentParser()
43 | parser.add_argument("--root", type=str, default=".")
44 |
45 | # Model
46 | parser.add_argument("--method", type=str, default="standard")
47 | parser.add_argument("--abs_refac_ms", type=int, default=10)
48 | parser.add_argument("--downsample", type=float, default=1)
49 |
50 | # Dataset
51 | parser.add_argument("--neuron_idx", type=int, default="")
52 |
53 | # Trainer
54 | parser.add_argument("--dir_name", type=str, default="")
55 | parser.add_argument("--id", type=str, default="")
56 |
57 | args = parser.parse_args()
58 |
59 | train_dataset = get_dataset(args)
60 | model = get_model(args)
61 | model_trainer = get_trainer(args, model, train_dataset)
62 | model_trainer.train(save=True)
63 |
64 |
65 | if __name__ == "__main__":
66 | main()
67 |
--------------------------------------------------------------------------------
/scripts/run_benchmarks.py:
--------------------------------------------------------------------------------
1 | import torch
2 | torch.backends.cudnn.benchmark = True # Important in order to use best conv algorithm
3 |
4 | from src.benchmark import Benchmarker
5 |
6 |
7 | def run_different_sim_lengths(root):
8 | n_in = 1000
9 | n_hidden = 128
10 |
11 | for abs_refac in [10, 20, 30, 40, 50, 60, 70, 80, 90, 100]:
12 | for t_len in [2**9, 2**10, 2**11]:
13 | for batch_size in [32, 64, 128]:
14 | for method in ["standard", "blocks"]:
15 | bencher = Benchmarker(method, t_len, abs_refac, n_in, n_hidden, n_layers=1, batch_size=batch_size)
16 | bencher.benchmark()
17 | bencher.save(root)
18 |
19 |
20 | def run_different_layers(root):
21 | n_in = 1000
22 |
23 | for abs_refac in [40]:
24 | for t_len in [2**10]:
25 | for n_hidden in [128, 256, 512]:
26 | for batch_size in [64]:
27 | for n_layers in [2, 3, 4, 5]:
28 | for method in ["standard", "blocks"]:
29 | bencher = Benchmarker(method, t_len, abs_refac, n_in, n_hidden, n_layers=n_layers, batch_size=batch_size)
30 | bencher.benchmark()
31 | bencher.save(root)
32 |
33 |
34 | if __name__ == "__main__":
35 | root = "" # TODO: Change this to the project folder
36 | run_different_sim_lengths(f"{root}/benchmarks/sim_lengths")
37 | run_different_layers(f"{root}/benchmarks/layers")
38 |
39 |
--------------------------------------------------------------------------------
/scripts/supervised/run_blocks_nmnist.py:
--------------------------------------------------------------------------------
1 | import os
2 | root = "" # TODO: Change this to the project folder
3 |
4 |
5 | def launch(abs_refac, surr_grad, dt, method):
6 | for i in range(3):
7 | id = f"nmnist_{method}_{surr_grad}_{abs_refac}_{dt}_{i}"
8 | os.system(f"python {root}/scripts/supervised/train.py --root={root} --method={method} --abs_refac={abs_refac} --surr_grad={surr_grad} --name=nmnist --dt={dt} --id={id}")
9 |
10 |
11 | for abs_refac in [10, 20, 30, 40, 50]:
12 | launch(abs_refac, "mg", 1, "blocks")
13 |
--------------------------------------------------------------------------------
/scripts/supervised/run_blocks_shd.py:
--------------------------------------------------------------------------------
1 | import os
2 | root = "" # TODO: Change this to the project folder
3 |
4 |
5 | def launch(abs_refac, surr_grad, dt, method):
6 | for i in range(3):
7 | id = f"shd_{method}_{surr_grad}_{abs_refac}_{dt}_{i}"
8 | os.system(f"python {root}/scripts/supervised/train.py --root={root} --method={method} --abs_refac={abs_refac} --surr_grad={surr_grad} --name=shd --dt={dt} --id={id}")
9 |
10 |
11 | for abs_refac in [10, 20, 30, 40, 50]:
12 | launch(abs_refac, "mg", 1, "blocks")
13 |
--------------------------------------------------------------------------------
/scripts/supervised/run_detach_spikes.py:
--------------------------------------------------------------------------------
1 | import os
2 | root = "" # TODO: Change this to the project folder
3 |
4 |
5 | def launch(abs_refac, surr_grad, dt, detach_spike_grad):
6 | for i in range(3):
7 | id = f"shd_blocks_{surr_grad}_{abs_refac}_{dt}_{detach_spike_grad}_{i}"
8 | os.system(f"python {root}/scripts/supervised/train.py --root={root} --method=blocks --abs_refac={abs_refac} --surr_grad={surr_grad} --name=shd --dt={dt} --id={id} --detach_spike_grad={detach_spike_grad}")
9 |
10 |
11 | launch(30, "mg", dt=2, detach_spike_grad=False)
12 | launch(30, "fast_sigmoid", dt=2, detach_spike_grad=False)
13 | launch(30, "box_car", dt=2, detach_spike_grad=False)
14 |
--------------------------------------------------------------------------------
/scripts/supervised/run_standard_nmnist.py:
--------------------------------------------------------------------------------
1 | import os
2 | root = "" # TODO: Change this to the project folder
3 |
4 |
5 | def launch(abs_refac, surr_grad, dt, method):
6 | for i in range(3):
7 | id = f"nmnist_{method}_{surr_grad}_{abs_refac}_{dt}_{i}"
8 | os.system(f"python {root}/scripts/supervised/train.py --root={root} --method={method} --abs_refac={abs_refac} --surr_grad={surr_grad} --name=nmnist --dt={dt} --id={id}")
9 |
10 |
11 | launch(0, "mg", 1, "standard")
12 |
--------------------------------------------------------------------------------
/scripts/supervised/run_standard_shd.py:
--------------------------------------------------------------------------------
1 | import os
2 | root = "" # TODO: Change this to the project folder
3 |
4 |
5 | def launch(abs_refac, surr_grad, dt):
6 | for i in range(3):
7 | id = f"shd_standard_{surr_grad}_{abs_refac}_{dt}_{i}"
8 | os.system(f"python {root}/scripts/supervised/train.py --root={root} --method=standard --abs_refac={abs_refac} --surr_grad={surr_grad} --name=shd --dt={dt} --id={id}")
9 |
10 |
11 | launch(0, "mg", dt=2)
12 |
--------------------------------------------------------------------------------
/scripts/supervised/train.py:
--------------------------------------------------------------------------------
1 | import sys
2 | import ast
3 | import logging
4 | import argparse
5 |
6 | from src import datasets, models, train
7 |
8 | logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
9 |
10 |
11 | def eval(v):
12 | return ast.literal_eval(v)
13 |
14 |
15 | def get_dataset(args, train=True):
16 | if args.name == "shd":
17 | return datasets.SHDDataset(f"{args.root}/data/SHD", train=train, dt=args.dt)
18 | elif args.name == "nmnist":
19 | return datasets.NMNISTDataset(f"{args.root}/data/N-MNIST", train=train, dt=args.dt)
20 |
21 |
22 | def get_model(args, dataset):
23 | abs_refac = int(args.abs_refac / args.dt)
24 |
25 | if args.name == "shd":
26 | n_in = 700
27 | n_out = 20
28 | elif args.name == "nmnist":
29 | n_in = 1156
30 | n_out = 10
31 |
32 | return models.AuditoryModel(args.method, n_in, args.n_hidden, n_out, dataset.t_len, abs_refac, eval(args.recurrent), args.dt, args.surr_grad, detach_spike_grad=eval(args.detach_spike_grad))
33 |
34 |
35 | def get_trainer(args, model, train_dataset, val_dataset):
36 | gamma = 0.1
37 | if args.name == "shd":
38 | milestones = [15, 15]
39 | epochs = 40
40 | elif args.name == "nmnist":
41 | milestones = [30]
42 | epochs = 20
43 | return train.Trainer(f"{args.root}/results", model, train_dataset, epochs, args.batch_size, args.lr, milestones, gamma, val_dataset, id=args.id)
44 |
45 |
46 | def main():
47 | parser = argparse.ArgumentParser()
48 | parser.add_argument("--root", type=str, default=".")
49 |
50 | # Model
51 | parser.add_argument("--method", type=str, default="standard")
52 | parser.add_argument("--n_hidden", type=int, default=256)
53 | parser.add_argument("--abs_refac", type=float, default=10)
54 | parser.add_argument("--recurrent", type=str, default="True")
55 | parser.add_argument("--detach_spike_grad", type=str, default="True")
56 | parser.add_argument("--surr_grad", type=str, default="fast_sigmoid")
57 |
58 | # Dataset
59 | parser.add_argument("--name", type=str, default="shd")
60 | parser.add_argument("--dt", type=float, default=1)
61 |
62 | # Trainer
63 | parser.add_argument("--batch_size", type=int, default=64)
64 | parser.add_argument("--lr", type=float, default=0.001)
65 | parser.add_argument('--id', type=str, default="")
66 |
67 | args = parser.parse_args()
68 |
69 | train_dataset = get_dataset(args, train=True)
70 | val_dataset = get_dataset(args, train=False)
71 | if args.name == "nmnist":
72 | val_dataset = None
73 | model = get_model(args, train_dataset)
74 | model_trainer = get_trainer(args, model, train_dataset, val_dataset)
75 | model_trainer.train(save=True)
76 |
77 |
78 | if __name__ == "__main__":
79 | main()
80 |
--------------------------------------------------------------------------------
/setup.py:
--------------------------------------------------------------------------------
1 | from setuptools import find_packages, setup
2 |
3 | setup(
4 | name="src",
5 | packages=find_packages(),
6 | )
7 |
--------------------------------------------------------------------------------
/src/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/webstorms/Blocks/540c9f28eedd58ef638dcacf75b4c27cdf52baa0/src/__init__.py
--------------------------------------------------------------------------------
/src/benchmark.py:
--------------------------------------------------------------------------------
1 | import os
2 | import time
3 |
4 | import torch
5 | import pandas as pd
6 |
7 | from src import datasets
8 | from src import models
9 |
10 |
11 | class Benchmarker:
12 |
13 | def __init__(self, method, t_len, abs_refac, n_in, n_hidden, n_layers, batch_size=16, min_r=0, max_r=200, n_samples=11):
14 | self._method = method
15 | self._t_len = t_len
16 | self._abs_refac = abs_refac
17 | self._n_in = n_in
18 | self._n_hidden = n_hidden
19 | self._n_layers = n_layers
20 | self._batch_size = batch_size
21 | self._model = models.ModelBuilder(method, t_len, abs_refac, n_in, n_hidden, n_layers)
22 |
23 | self._data_loader = self._get_data_loader(t_len, n_in, min_r, max_r, batch_size, n_samples*batch_size)
24 | self._benchmark_results = None
25 |
26 | def benchmark(self, device="cuda"):
27 | timing_list = []
28 |
29 | self._model = self._model.to(device)
30 |
31 | for i, data in enumerate(self._data_loader):
32 | # Benchmark forward pass
33 | data = data.to(device)
34 |
35 | start_time = time.time()
36 | output = self._model(data)
37 | torch.cuda.synchronize()
38 | forward_pass_time = time.time() - start_time
39 |
40 | # Benchmark backward pass
41 | start_time = time.time()
42 | loss = output.sum()
43 | loss.backward()
44 | torch.cuda.synchronize()
45 | backward_pass_time = time.time() - start_time
46 |
47 | # Ignore first run (as this usually loads things which slows things down)
48 | # e.g. cudnn finds best conv algorithm
49 | if i > 0:
50 | timing_row = {"forward_time": forward_pass_time, "backward_time": backward_pass_time}
51 | timing_list.append(timing_row)
52 |
53 | self._benchmark_results = timing_list
54 |
55 | def save(self, path):
56 | results_df = self._to_df()
57 | results_df.to_csv(os.path.join(path, f"{self._get_df_name()}.csv"), index=False)
58 |
59 | def _get_description(self):
60 | return {"method": self._method, "t_len": self._t_len, "abs_refac": self._abs_refac, "units": self._n_hidden, "layers": self._n_layers, "batch": self._batch_size}
61 |
62 | def _get_df_name(self):
63 | return f"{self._method}_{self._t_len}_{self._abs_refac}_{self._n_hidden}_{self._n_layers}_{self._batch_size}"
64 |
65 | def _get_data_loader(self, t_len, n_units, min_r, max_r, batch_size, n_samples):
66 | spikes_dataset = datasets.SyntheticSpikes(t_len, n_units, min_r, max_r, n_samples)
67 | return torch.utils.data.DataLoader(spikes_dataset, batch_size, shuffle=False)
68 |
69 | def _to_df(self):
70 | assert self._benchmark_results is not None
71 | results = []
72 |
73 | for results_row in self._benchmark_results:
74 | results.append({**results_row, **self._get_description()})
75 |
76 | return pd.DataFrame(results)
77 |
--------------------------------------------------------------------------------
/src/datasets/__init__.py:
--------------------------------------------------------------------------------
1 | from .synthetic import SyntheticSpikes
2 | from .neuromorphic import NMNISTDataset, SHDDataset
3 | from .ephys import ValidNeuronQuery, Builder, NoiseBuilder, EphysDataset
4 |
--------------------------------------------------------------------------------
/src/datasets/ephys.py:
--------------------------------------------------------------------------------
1 | import os
2 | import glob
3 | import pickle
4 | from collections import defaultdict
5 |
6 | import numpy as np
7 | import pandas as pd
8 | import torch
9 | from allensdk.core.cell_types_cache import CellTypesCache
10 | from allensdk.ephys.ephys_extractor import EphysSweepFeatureExtractor
11 |
12 |
13 | class ValidNeuronQuery:
14 |
15 | """
16 | This class builds an idx set of neurons which meet the requirement of having 4 repeats.
17 | """
18 |
19 | def __init__(self, root):
20 | self.root = root
21 | self.query_df = self.build_query_df()
22 |
23 | @property
24 | def idx(self):
25 | query = self.query_df["n_trials"] == 4 # Want 4 repeats
26 | return self.query_df[query]["idx"].values
27 |
28 | def build_query_df(self):
29 | train_idxs = [v.split("/")[-1] for v in glob.glob(f"{self.root}/train/*")]
30 | test_idxs = [v.split("/")[-1] for v in glob.glob(f"{self.root}/test/*")]
31 | joint_idxs = list(set(train_idxs) & set(test_idxs))
32 |
33 | data_list = []
34 |
35 | for idx in joint_idxs:
36 | try:
37 | v = torch.load(f"{self.root}/test/{idx}/v.pt")
38 | n_trials = v.shape[0]
39 | data_list.append({"idx": idx, "n_trials": n_trials})
40 | except:
41 | pass
42 |
43 | return pd.DataFrame(data_list)
44 |
45 |
46 | class EphysDataset:
47 |
48 | """
49 | This is the PyTorch friendly e-phys dataset, with all the current and spike tensors.
50 | """
51 |
52 | LENGTH = 10000 # = 1 second (with DT=0.1)
53 |
54 | def __init__(self, root, dataset, neuron_idx, target_sampling_rate=None):
55 | # dataset: str which is train or test
56 | info_df = pd.read_csv(f"{root}/info_df.csv").set_index("idx")
57 |
58 | if target_sampling_rate is None:
59 | self.i = torch.load(f"{root}/{dataset}/{neuron_idx}/i.pt")[:, :, :1*EphysDataset.LENGTH]
60 | elif target_sampling_rate == 20000:
61 | self.i = torch.load(f"{root}/{dataset}/{neuron_idx}/i_20k.pt")[:, :, :2*EphysDataset.LENGTH]
62 | self.v = torch.load(f"{root}/{dataset}/{neuron_idx}/v.pt")[:, :, :EphysDataset.LENGTH]
63 | self.s = torch.load(f"{root}/{dataset}/{neuron_idx}/s.pt")[:, :, :EphysDataset.LENGTH]
64 |
65 | vrest = info_df.loc[neuron_idx]["vrest"]
66 | vthresh = info_df.loc[neuron_idx]["vthresh"]
67 | self.i = self.i / (100 * self.i.max())
68 | self.v = (self.v - vrest) / (vthresh - vrest)
69 |
70 | @property
71 | def hyperparams(self):
72 | return {}
73 |
74 | def __getitem__(self, item):
75 | x = self.i[0, item].unsqueeze(0) # Input current is the same across all trials
76 | trace = self.v[0, item].unsqueeze(0)
77 | trace = torch.clamp(trace, -1, 1)
78 | spikes = self.s[0, item].unsqueeze(0) # Take first spike trial (all are the same)
79 |
80 | return x, (trace, spikes)
81 |
82 | def __len__(self):
83 | return 3
84 |
85 |
86 | class Builder:
87 |
88 | def __init__(self, manifest_file="data/plot_data/allen-brain-observatory/cell_types/manifest.json"):
89 | self.ctc = CellTypesCache(manifest_file=manifest_file)
90 | ephys_df = self.generate_ephys_df()
91 | self.info_df = self.generate_info_df(ephys_df).set_index("idx")
92 |
93 | def save_info_df(self, path):
94 | self.info_df.to_csv(f"{path}/info_df.csv")
95 |
96 | def build(self, path, **kwargs):
97 | for neuron_idx in self.info_df.index:
98 | print(f"Building {neuron_idx}...")
99 | try:
100 | meta, i, v = self.generate_all_sweep_tensor(neuron_idx, **kwargs)
101 | os.mkdir(f"{path}/{neuron_idx}")
102 | torch.save(i, f"{path}/{neuron_idx}/i.pt")
103 | torch.save(v, f"{path}/{neuron_idx}/v.pt")
104 | with open(f"{path}/{neuron_idx}/meta.pkl", "wb") as f:
105 | pickle.dump(meta, f)
106 | except Exception as e:
107 | print(f"Failed {neuron_idx}: {e}")
108 |
109 | def generate_all_sweep_tensor(self, neuron_idx, target_sampling_rate=10000, start_s=1.02, end_s=1.3):
110 | sweep_idxs = self.info_df.loc[neuron_idx]["long_square"]
111 | sweep_dict = {}
112 |
113 | for sweep_idx in sweep_idxs:
114 | i, v, spike_times = self.generate_sweep_tensor(neuron_idx, sweep_idx, target_sampling_rate, start_s, end_s)
115 | sweep_dict[i] = (i, v, spike_times)
116 |
117 | # Sort from lowest to highest current
118 | sweep_dict = {k: v for k, v in sorted(sweep_dict.items(), key=lambda item: item[0])}
119 |
120 | meta = pd.DataFrame([{"i": v[0], "spikes": v[2]} for k, v in sweep_dict.items()])
121 | v = torch.stack([sweep_dict[key][1] for key in sweep_dict.keys()])
122 |
123 | return meta, v
124 |
125 | def generate_sweep_tensor(self, neuron_idx, sweep_number, target_sampling_rate=10000, start_s=1.02, end_s=1.3):
126 | data_set = self.ctc.get_ephys_data(neuron_idx)
127 | sweep_data = data_set.get_sweep(sweep_number)
128 |
129 | index_range = sweep_data["index_range"]
130 | i = sweep_data["stimulus"][0:index_range[1]+1] # in A
131 | v = sweep_data["response"][0:index_range[1]+1] # in V
132 | i *= 1e12 # to pA
133 | v *= 1e3 # to mV
134 |
135 | sampling_rate = int(sweep_data["sampling_rate"]) # in Hz
136 | t = np.arange(0, len(v)) * (1.0 / sampling_rate)
137 | downsample_factor = sampling_rate / target_sampling_rate
138 |
139 | sweep_ext = EphysSweepFeatureExtractor(t=t, v=v, i=i, start=start_s, end=end_s)
140 | sweep_ext.process_spikes()
141 | spike_times = sweep_ext.spike_feature("threshold_t")
142 |
143 | start_idx = int(start_s*sampling_rate)
144 | end_idx = int(end_s*sampling_rate)
145 |
146 | downsampled_v, downsampled_i = [], []
147 | assert v.shape == i.shape
148 |
149 | idx = start_idx
150 | while idx < end_idx:
151 | downsampled_v.append(v[int(idx)])
152 | downsampled_i.append(i[int(idx)])
153 | idx += downsample_factor
154 |
155 | return torch.Tensor(downsampled_i), torch.Tensor(downsampled_v), spike_times
156 |
157 | def generate_ephys_df(self):
158 | cells = {cell["id"]: cell for cell in self.ctc.get_cells()}
159 | ephys_features = self.ctc.get_ephys_features()
160 | ephys_df = pd.DataFrame(ephys_features)
161 | ephys_df['id'] = pd.Series([idx for idx in ephys_df['specimen_id']], index=ephys_df.index)
162 | ephys_df['species'] = pd.Series([cells[idx]['species'] for idx in ephys_df['specimen_id']], index=ephys_df.index)
163 | ephys_df['dendrite_type'] = pd.Series([cells[idx]['dendrite_type'] for idx in ephys_df['specimen_id']], index=ephys_df.index)
164 | ephys_df['structure_layer_name'] = pd.Series([cells[idx]['structure_layer_name'] for idx in ephys_df['specimen_id']], index=ephys_df.index)
165 | ephys_df['disease_state'] = pd.Series([cells[idx]['disease_state'] for idx in ephys_df['specimen_id']], index=ephys_df.index)
166 | query = ephys_df["structure_layer_name"] == "4"
167 | query &= ephys_df["species"] == "Mus musculus"
168 | query &= ephys_df["disease_state"] == ""
169 |
170 | return ephys_df[query]
171 |
172 | def generate_info_df(self, ephys_df):
173 | info_list = []
174 |
175 | neuron_idxs = ephys_df["id"].values
176 |
177 | for neuron_idx in neuron_idxs:
178 | # Sweep info
179 | sweeps = self.ctc.get_ephys_sweeps(neuron_idx)
180 | sweep_numbers = defaultdict(list)
181 | for sweep in sweeps:
182 | sweep_numbers[sweep['stimulus_name']].append(sweep['sweep_number'])
183 |
184 | neuron_type = ephys_df[ephys_df["id"] == neuron_idx]["dendrite_type"].values[0]
185 | vrest = ephys_df[ephys_df["id"] == neuron_idx]["vrest"].values[0]
186 | vthresh = ephys_df[ephys_df["id"] == neuron_idx]["threshold_v_long_square"].values[0]
187 |
188 | info_list.append({"idx": neuron_idx, "type": neuron_type, "long_square": sweep_numbers.get("Long Square"), "noise1": sweep_numbers.get("Noise 1"), "noise2": sweep_numbers.get("Noise 2"), "test": sweep_numbers.get("Test"), "vrest": vrest, "vthresh": vthresh})
189 |
190 | return pd.DataFrame(info_list)
191 |
192 |
193 | class NoiseBuilder(Builder):
194 |
195 | def build(self, path, **kwargs):
196 | for neuron_idx in self.info_df.index:
197 | print(f"Building {neuron_idx}...")
198 | try:
199 | if not os.path.exists(f"{path}/{neuron_idx}"):
200 | os.makedirs(f"{path}/{neuron_idx}")
201 |
202 | i, v, s = self.generate_all_sweep_tensor(neuron_idx, **kwargs)
203 |
204 | # Default used for all experiments
205 | if kwargs.get("target_sampling_rate") is None:
206 | torch.save(i, f"{path}/{neuron_idx}/i.pt")
207 | torch.save(v, f"{path}/{neuron_idx}/v.pt")
208 | torch.save(s, f"{path}/{neuron_idx}/s.pt")
209 | # Re-ran some experiments with DT=0.05ms as requested by one reviewer
210 | elif kwargs.get("target_sampling_rate") == 20000:
211 | torch.save(i, f"{path}/{neuron_idx}/i_20k.pt")
212 | torch.save(v, f"{path}/{neuron_idx}/v_20k.pt")
213 | torch.save(s, f"{path}/{neuron_idx}/s_20k.pt")
214 |
215 | except Exception as e:
216 | print(f"Failed {neuron_idx}: {e}")
217 |
218 | def generate_all_sweep_tensor(self, neuron_idx, target_sampling_rate=10000, noise_type="noise1"):
219 | sweep_idxs = self.info_df.loc[neuron_idx][noise_type]
220 | assert len(sweep_idxs) is not None
221 |
222 | i_list = []
223 | v_list = []
224 | s_list = []
225 |
226 | for sweep_idx in sweep_idxs:
227 | i, v, s = self.generate_noise_sweep_tensor(neuron_idx, sweep_idx, target_sampling_rate)
228 | i_list.append(i)
229 | v_list.append(v)
230 | s_list.append(s)
231 |
232 | return torch.stack(i_list), torch.stack(v_list), torch.stack(s_list)
233 |
234 | def generate_noise_sweep_tensor(self, neuron_idx, sweep_number, target_sampling_rate=10000):
235 | i1, v1, t1 = self.generate_sweep_tensor(neuron_idx, sweep_number, target_sampling_rate, start_s=2, end_s=5)
236 | i2, v2, t2 = self.generate_sweep_tensor(neuron_idx, sweep_number, target_sampling_rate, start_s=10, end_s=13)
237 | i3, v3, t3 = self.generate_sweep_tensor(neuron_idx, sweep_number, target_sampling_rate, start_s=18, end_s=21)
238 | t1 -= 2
239 | t2 -= 10
240 | t3 -= 18
241 | s1 = NoiseBuilder.to_spike_target(t1, target_sampling_rate)
242 | s2 = NoiseBuilder.to_spike_target(t2, target_sampling_rate)
243 | s3 = NoiseBuilder.to_spike_target(t3, target_sampling_rate)
244 |
245 | return torch.stack([i1, i2, i3]), torch.stack([v1, v2, v3]), torch.stack([s1, s2, s3])
246 |
247 | @staticmethod
248 | def to_spike_target(spike_times, target_sampling_rate):
249 | dt = 0.0001
250 | spike_target = torch.zeros(target_sampling_rate)
251 | spike_idx = [int(spike_time // dt) for spike_time in spike_times if int(spike_time // dt) < target_sampling_rate]
252 | spike_target[spike_idx] = 1
253 |
254 | return spike_target
255 |
--------------------------------------------------------------------------------
/src/datasets/neuromorphic.py:
--------------------------------------------------------------------------------
1 | import os
2 | import tables
3 |
4 | import torch
5 | import numpy as np
6 | from brainbox.datasets import BBDataset
7 |
8 | from src.datasets.transforms import SpikeTensorBuilder
9 |
10 |
11 | class H5Dataset(BBDataset):
12 |
13 | def __init__(self, root, train, n_in, n_out, t_len, train_name, test_name, dt):
14 | self._file = None
15 | self._train_name = train_name
16 | self._test_name = test_name
17 | super().__init__(root, train, lambda dataset: H5Dataset.preprocess(dataset), SpikeTensorBuilder(n_in, t_len, dt))
18 | self._file.close()
19 |
20 | self._n_in = n_in
21 | self._n_out = n_out
22 | self._t_len = t_len
23 | self._dt = dt
24 |
25 | @property
26 | def hyperparams(self):
27 | return {**super().hyperparams, "t_len": self._t_len, "dt": self._dt}
28 |
29 | @property
30 | def n_in(self):
31 | return self._n_in
32 |
33 | @property
34 | def n_out(self):
35 | return self._n_out
36 |
37 | @property
38 | def t_len(self):
39 | return self._t_len
40 |
41 | @property
42 | def dt(self):
43 | return self._dt
44 |
45 | @staticmethod
46 | def preprocess(dataset):
47 | processed_dataset = []
48 | units, times = dataset
49 |
50 | for i in range(len(units)):
51 | item_units = torch.Tensor(np.array(units[i], dtype=np.int))
52 | item_times = torch.Tensor(np.array(times[i], dtype=np.float))
53 | processed_dataset.append((item_units, item_times))
54 |
55 | return processed_dataset
56 |
57 | @staticmethod
58 | def _open_file(hdf5_file_path):
59 | fileh = tables.open_file(hdf5_file_path, mode="r")
60 | units = fileh.root.spikes.units
61 | times = fileh.root.spikes.times
62 | labels = fileh.root.labels
63 |
64 | return fileh, units, times, labels
65 |
66 | def _load_dataset(self, train):
67 | name = self._train_name if train else self._test_name
68 | fileh, units, times, labels = H5Dataset._open_file(os.path.join(self._root, name))
69 | targets = torch.Tensor(labels)
70 | self._file = fileh
71 |
72 | return (units, times), targets
73 |
74 |
75 | class NMNISTDataset(H5Dataset):
76 |
77 | T_LEN = 400
78 |
79 | def __init__(self, root, train=True, dt=1):
80 | t_len = int(NMNISTDataset.T_LEN / dt)
81 | super().__init__(root, train, n_in=1156, n_out=10, t_len=t_len, train_name="train.h5", test_name="test.h5", dt=dt)
82 |
83 | @property
84 | def name(self):
85 | return "nmnist"
86 |
87 |
88 | class SHDDataset(H5Dataset):
89 |
90 | T_LEN = 1200
91 |
92 | def __init__(self, root, train=True, dt=2):
93 | t_len = int(SHDDataset.T_LEN / dt)
94 | super().__init__(root, train, n_in=700, n_out=20, t_len=t_len, train_name="shd_train.h5", test_name="shd_test.h5", dt=dt)
95 |
96 | @property
97 | def name(self):
98 | return "shd"
99 |
--------------------------------------------------------------------------------
/src/datasets/synthetic.py:
--------------------------------------------------------------------------------
1 | import torch
2 | from torch.distributions.poisson import Poisson
3 | from brainbox.datasets import BBDataset
4 |
5 |
6 | class SyntheticSpikes(BBDataset):
7 |
8 | """
9 | This is the synthetic spike dataset with which the model was benchmarked.
10 | """
11 |
12 | def __init__(self, t_len, n_units, min_r, max_r, n_samples):
13 | super().__init__(None)
14 | self.t_len = t_len
15 | self.n_units = n_units
16 | self.min_r = min_r
17 | self.max_r = max_r
18 | self.n_samples = n_samples
19 |
20 | def __getitem__(self, i):
21 | rate = torch.FloatTensor(1).uniform_(self.min_r, self.max_r).item()
22 | x = self._create_spikes(rate, self.n_units, self.t_len)
23 |
24 | return x
25 |
26 | def __len__(self):
27 | return self.n_samples
28 |
29 | def _load_dataset(self, train):
30 | return None, None
31 |
32 | def _create_spikes(self, rate, n_units, t_len):
33 | pois_dis = Poisson(rate/t_len)
34 | if type(n_units) == tuple:
35 | samples = pois_dis.sample(sample_shape=(*n_units, t_len))
36 | else:
37 | samples = pois_dis.sample(sample_shape=(n_units, t_len))
38 | samples[samples > 1] = 1
39 |
40 | return samples
--------------------------------------------------------------------------------
/src/datasets/transforms.py:
--------------------------------------------------------------------------------
1 | import torch
2 | import numpy as np
3 | import brainbox
4 | from brainbox.datasets.transforms import BBTransform
5 |
6 |
7 | class SpikeTensorBuilder(BBTransform):
8 |
9 | def __init__(self, n_units, t_len, dt):
10 | self._n_units = n_units
11 | self._t_len = t_len
12 | self._dt = dt
13 |
14 | def __call__(self, args):
15 | units, times = args[0], args[1]
16 | units = units % self._n_units
17 | times = torch.round(times * 1000. / self._dt).int()
18 |
19 | # Constrain spike length
20 | idxs = (times < self._t_len)
21 | units = units[idxs]
22 | times = times[idxs]
23 |
24 | # Build COO tensor
25 | indices = torch.stack([torch.Tensor(units.tolist()), torch.Tensor(times.tolist())], dim=0).long()
26 | shape = torch.Size([self._n_units, self._t_len, ])
27 | spikes = torch.FloatTensor(np.ones(len(indices[0])))
28 |
29 | return torch.sparse.FloatTensor(indices, spikes, shape).to_dense()
30 |
31 |
32 | class List:
33 |
34 | @staticmethod
35 | def get_nmnist_transform(t_len, use_augmentation=False):
36 | if use_augmentation:
37 | raise NotImplementedError
38 | else:
39 | transform_list = [SpikeTensorBuilder(n_units=1156, t_len=t_len, dt=1)]
40 |
41 | return brainbox.datasets.transforms.Compose(transform_list)
42 |
43 | @staticmethod
44 | def get_shd_transform(t_len):
45 | transform_list = [SpikeTensorBuilder(n_units=700, t_len=t_len, dt=2)]
46 |
47 | return brainbox.datasets.transforms.Compose(transform_list)
48 |
--------------------------------------------------------------------------------
/src/metric.py:
--------------------------------------------------------------------------------
1 | import torch
2 | import pandas as pd
3 | from brainbox.physiology.spiking import SpikeToPSTH
4 |
5 | from src import datasets, train
6 |
7 |
8 | class SpikeTrainEV:
9 |
10 | def __init__(self, st_len, sig):
11 | self._spike_smoother = SpikeToPSTH(st_len, sig)
12 |
13 | def __call__(self, input, target):
14 | # input: b x 3 x t
15 | smooth_input_trains = self.smooth_spike_trains(input)
16 | smooth_target_trains = self.smooth_spike_trains(target)
17 | flatten_input_trains = self.flatten_spike_trains(smooth_input_trains)
18 | flatten_target_trains = self.flatten_spike_trains(smooth_target_trains)
19 |
20 | return SpikeTrainEV.ev(flatten_input_trains, flatten_target_trains.mean(0).unsqueeze(0)).mean()
21 |
22 | def smooth_spike_trains(self, spike_trains):
23 | # spike_trains: b x 3 x t
24 | b = spike_trains.shape[0]
25 |
26 | return self._spike_smoother(spike_trains.view(b*3, -1).unsqueeze(1))[:, 0].view(b, 3, -1)
27 |
28 | def flatten_spike_trains(self, spike_trains):
29 | # spike_trains: b x 3 x t
30 | return spike_trains.flatten(1, 2)
31 |
32 | @staticmethod
33 | def ev(input, target):
34 | # input: b x t
35 | return (input.var(1) + target.var(1) - (input - target).var(1)) / (input.var(1) + target.var(1))
36 |
37 |
38 | class EphysAnalysis:
39 |
40 | def __init__(self, root, method, abs_refac_ms, downsample, taus=[100]):
41 | self.root = root
42 | self.method = method
43 | self.abs_refac_ms = abs_refac_ms
44 | self.downsample = downsample
45 | self.taus = taus
46 |
47 | self.metrics = {tau: SpikeTrainEV(10000, tau) for tau in taus}
48 | self.neuron_idxs = datasets.ValidNeuronQuery(f"{root}/data/ephys").idx
49 |
50 | # Init test dataset
51 | self.test_dataset = {}
52 | self.norm_factors = {tau: {} for tau in taus}
53 |
54 | for neuron_idx in self.neuron_idxs:
55 | if downsample == 0.5:
56 | self.test_dataset[neuron_idx] = datasets.EphysDataset(f"{root}/data/ephys", "test", int(neuron_idx), target_sampling_rate=20000)
57 | else:
58 | self.test_dataset[neuron_idx] = datasets.EphysDataset(f"{root}/data/ephys", "test", int(neuron_idx))
59 | target_spikes = self.test_dataset[neuron_idx].s.cpu()
60 | for tau in taus:
61 | self.norm_factors[tau][neuron_idx] = self.metrics[tau](target_spikes, target_spikes).item()
62 |
63 | self._norm_df = {tau: pd.Series(self.norm_factors[tau]).to_frame().rename(columns={0: "score"}) for tau in taus}
64 | self._ev_df = None
65 |
66 | def ev_df(self, tau, normalise):
67 | if self._ev_df is None:
68 | self._ev_df = self._build_ev_df().set_index("neuron_idx")
69 |
70 | query = self._ev_df["tau"] == tau
71 | ev_df = self._ev_df[query]["score"].to_frame()
72 |
73 | if normalise:
74 | df = ev_df / self._norm_df[tau]
75 | else:
76 | df = ev_df
77 |
78 | df.index = df.index.map(int)
79 |
80 | return df
81 |
82 | def get_times_df(self):
83 | dir_name = f"{self.method}_{self.abs_refac_ms}_{self.downsample}"
84 |
85 | times_list = []
86 |
87 | for neuron_idx in self.neuron_idxs:
88 | times_csv = pd.read_csv(f"{self.root}/results/ephys/{dir_name}/{neuron_idx}/times.csv")
89 | forward_pass, backward_pass = times_csv.sum()
90 | times_list.append({"neuron_idx": neuron_idx, "forward_pass": forward_pass, "backward_pass": backward_pass})
91 |
92 | return pd.DataFrame(times_list)
93 |
94 | def _build_ev_df(self):
95 | dir_name = f"{self.method}_{self.abs_refac_ms}_{self.downsample}"
96 |
97 | metric_list = []
98 |
99 | for i, neuron_idx in enumerate(self.neuron_idxs):
100 | print(f"Building {i}/{len(self.neuron_idxs)} {neuron_idx}...")
101 | dt01ref = self.downsample == 0.5
102 | neuron = train.EphysTrainer.load_model(f"{self.root}/results/ephys/{dir_name}", neuron_idx, dt01ref=dt01ref)
103 |
104 | with torch.no_grad():
105 | test_dataset = self.test_dataset[neuron_idx]
106 | pred_spikes = neuron(test_dataset.i[0].unsqueeze(1).cuda()).permute(1, 0, 2).cpu()
107 | target_spikes = test_dataset.s.cpu()
108 |
109 | for tau in self.taus:
110 | score = self.metrics[tau](target_spikes, pred_spikes).item() # note: model spikes are reference
111 | metric_list.append({"neuron_idx": neuron_idx, "score": score, "tau": tau})
112 |
113 | return pd.DataFrame(metric_list)
114 |
115 | def load_prediction(self, neuron_idx): # Load prediction for a certain fit and neuron
116 | test_dataset = self.test_dataset[str(neuron_idx)]
117 |
118 | with torch.no_grad():
119 | dir_name = f"{self.method}_{self.abs_refac_ms}_{self.downsample}"
120 | dt01ref = self.downsample == 0.5
121 | neuron = train.EphysTrainer.load_model(f"{self.root}/results/ephys/{dir_name}", neuron_idx, dt01ref=dt01ref)
122 |
123 | output = neuron(test_dataset.i[0].unsqueeze(1).cuda(), mode="val")
124 | spikes = output[0].permute(1, 0, 2).cpu()
125 | mem = output[1].permute(1, 0, 2).cpu()
126 |
127 | return spikes, mem, test_dataset.s, test_dataset.v, test_dataset.i
128 |
129 | def load_params_df(self):
130 | dir_name = f"{self.method}_{self.abs_refac_ms}_{self.downsample}"
131 |
132 | params_list = []
133 |
134 | for i, neuron_idx in enumerate(self.neuron_idxs):
135 | print(f"Building {i}/{len(self.neuron_idxs)} {neuron_idx}...")
136 | neuron = train.EphysTrainer.load_model(f"{self.root}/results/ephys/{dir_name}", neuron_idx).neuron
137 |
138 | params_list.append({"neuron_idx": neuron_idx, "beta": neuron.beta.item(), "p": neuron.p.item(), "b": neuron.b.item()})
139 |
140 | return pd.DataFrame(params_list)
141 |
--------------------------------------------------------------------------------
/src/models.py:
--------------------------------------------------------------------------------
1 | import numpy as np
2 | import torch
3 | import torch.nn as nn
4 | import torch.nn.functional as F
5 | from brainbox import models
6 |
7 | import src.snn.block.blocks as blocks
8 | import src.snn.snn as snn
9 |
10 |
11 | class AuditoryModel(models.BBModel):
12 |
13 | # AuditoryModel name comes from the SHD dataset being auditory
14 |
15 | HIDDEN_MEM_TIME = 20
16 | HIDDEN_ADAPT_TIME = 150
17 | READOUT_MEM_TIME = 20
18 |
19 | def __init__(self, method, n_in, n_hidden, n_out, t_len, abs_refac, recurrent=True, dt=1, surr_grad="fast_sigmoid", detach_spike_grad=True):
20 | super().__init__()
21 | self._method = method
22 | self._n_in = n_in
23 | self._n_hidden = n_hidden
24 | self._n_out = n_out
25 | self._t_len = t_len
26 | self._abs_refac = abs_refac
27 | self._recurrent = recurrent
28 | self._dt = dt
29 | self._surr_grad = surr_grad
30 |
31 | init_hidden_beta = np.exp(-dt / AuditoryModel.HIDDEN_MEM_TIME)
32 | init_hidden_p = np.exp(-dt / AuditoryModel.HIDDEN_ADAPT_TIME)
33 | init_readout_beta = np.exp(-dt / AuditoryModel.READOUT_MEM_TIME)
34 |
35 | if method == "standard":
36 | self._thalamic_layer = snn.SNN(n_in, n_hidden, 1, t_len, abs_refac, recurrent, beta_grad=True, adapt=True, init_beta=init_hidden_beta, init_p=init_hidden_p, surr_grad=self._surr_grad)
37 | self._cortical_layer = snn.SNN(n_hidden, n_hidden, 1, t_len, abs_refac, recurrent, beta_grad=True, adapt=True, init_beta=init_hidden_beta, init_p=init_hidden_p, surr_grad=self._surr_grad)
38 | self._output = snn.SNNIntegrator(n_hidden, n_out, t_len, init_beta=init_readout_beta)
39 | else:
40 | self._thalamic_layer = blocks.Blocks(n_in, n_hidden, 1, t_len, abs_refac, recurrent, beta_grad=True, adapt=True, init_beta=init_hidden_beta, init_p=init_hidden_p, surr_grad=self._surr_grad, detach_spike_grad=detach_spike_grad)
41 | self._cortical_layer = blocks.Blocks(n_hidden, n_hidden, 1, t_len, abs_refac, recurrent, beta_grad=True, adapt=True, init_beta=init_hidden_beta, init_p=init_hidden_p, surr_grad=self._surr_grad, detach_spike_grad=detach_spike_grad)
42 | self._output = blocks.BlocksIntegrator(n_hidden, n_out, t_len, init_beta=init_readout_beta)
43 |
44 | @property
45 | def hyperparams(self):
46 | return {**super().hyperparams, "method": self._method, "n_in": self._n_in, "n_hidden": self._n_hidden, "n_out": self._n_out, "t_len": self._t_len, "abs_refac": self._abs_refac, "recurrent": self._recurrent, "dt": self._dt, "surr_grad": self._surr_grad}
47 |
48 | def forward(self, x, mode="train"):
49 | # x: b x n x t
50 | thalamic_output = self._thalamic_layer(x, mode)
51 | cortical_output = self._cortical_layer(thalamic_output if mode == "train" else thalamic_output[0], mode)
52 |
53 | if mode == "train":
54 | return self._output(cortical_output, mode).sum(2)
55 | else:
56 | return self._output(cortical_output[0], mode).sum(2), cortical_output, thalamic_output
57 |
58 |
59 | class ModelBuilder(models.BBModel):
60 |
61 | def __init__(self, method, t_len, abs_refac, n_in, n_hidden, n_layers):
62 | super().__init__()
63 | self._layers = nn.ModuleList()
64 |
65 | for i in range(n_layers):
66 | n_in = n_in if i == 0 else n_hidden
67 | if method == "standard":
68 | self._layers.append(snn.SNN(n_in, n_hidden, 1, t_len, abs_refac, recurrent=True, beta_grad=True, adapt=True, init_beta=0.9, init_p=0.9, surr_grad="mg"))
69 | else:
70 | self._layers.append(blocks.Blocks(n_in, n_hidden, 1, t_len, abs_refac, recurrent=True, beta_grad=True, adapt=True, init_beta=0.9, init_p=0.9, surr_grad="mg"))
71 |
72 | def forward(self, x):
73 | for layer in self._layers:
74 | x = layer(x)
75 |
76 | return x
77 |
78 |
79 | class Neuron(models.BBModel):
80 |
81 | def __init__(self, method, abs_refac_ms, downsample=1, dt01ref=False):
82 | super().__init__()
83 | self.method = method
84 | self.abs_refac_ms = abs_refac_ms
85 | self.downsample = downsample
86 | self.dt01ref = dt01ref
87 | self.dt_ms = downsample * 0.1 # Larger dt_ms == downsample temporal resolution
88 |
89 | if not dt01ref:
90 | self.neuron = Neuron.get_neuron(method, abs_refac_ms, self.dt_ms, downsample)
91 | else:
92 | self.neuron = Neuron.get_neuron(method, abs_refac_ms, 0.5*self.dt_ms, 0.5*downsample)
93 | upsample_kernel = torch.zeros(downsample)
94 | upsample_kernel[0] = 1
95 | self.upsample_kernel = nn.Parameter(upsample_kernel.view(1, 1, -1), requires_grad=False)
96 |
97 | @property
98 | def hyperparams(self):
99 | return {**super().hyperparams, "method": self.method, "abs_refac_ms": self.abs_refac_ms, "downsample": self.downsample, "dt01ref": self.dt01ref}
100 |
101 | def forward(self, x, mode="train"):
102 | x = F.avg_pool1d(x, self.downsample, self.downsample) # Down sample the signal (if need be)
103 | spikes = self.neuron(x, mode)
104 |
105 | # Return data for plotting
106 | if mode == "val":
107 | spikes, mem = spikes[0], spikes[1]
108 |
109 | return spikes, mem
110 |
111 | # Return predicted spike train for training
112 | if self.dt01ref: # When running in DT=0.05ms (was added on to run additional experiments for a reviewer)
113 | spikes = F.max_pool1d(spikes, 2, 2)
114 | return spikes
115 |
116 | if self.downsample == 1:
117 | return spikes
118 | else:
119 | return F.conv_transpose1d(spikes, self.upsample_kernel, stride=self.downsample)
120 |
121 | @staticmethod
122 | def get_neuron(method, abs_refac_ms, dt_ms, downsample):
123 | init_beta = np.exp(-dt_ms / 20)
124 | init_p = np.exp(-dt_ms / 100)
125 | t_len = int(1000 / dt_ms)
126 | abs_refac_ms = int(abs_refac_ms / dt_ms)
127 |
128 | if method == "blocks":
129 | neuron = blocks.Blocks(1, 1, rf_len=1, t_len=t_len, t_latency=abs_refac_ms, recurrent=False, beta_grad=True, adapt=True, init_beta=init_beta, init_p=init_p, detach_spike_grad=True, surr_grad="mg")
130 | else:
131 | neuron = snn.SNN(1, 1, rf_len=1, t_len=t_len, t_latency=abs_refac_ms, recurrent=False, beta_grad=True, adapt=True, init_beta=init_beta, init_p=init_p, detach_spike_grad=True, surr_grad="mg")
132 |
133 | neuron.init_weight(neuron._rf_weight, "constant", c=downsample)
134 | neuron._b = nn.Parameter(data=torch.Tensor([0.1 / downsample]), requires_grad=True)
135 |
136 | return neuron
137 |
--------------------------------------------------------------------------------
/src/query.py:
--------------------------------------------------------------------------------
1 | import glob
2 |
3 | import pandas as pd
4 | from brainbox import trainer
5 |
6 | from src import models, train
7 |
8 |
9 | models.snn.BaseSNN.MIN_BETA = 0.01
10 | models.snn.BaseSNN.MAX_BETA = 0.99
11 |
12 |
13 | class BenchmarkQuery:
14 |
15 | def __init__(self, root, batches=[32, 64, 128]):
16 | self._root = root
17 |
18 | self._results_df = self._build_df()
19 | self._results_df = pd.concat([self._query_results(batch=b) for b in batches])
20 |
21 | def _build_df(self):
22 | results_df_list = []
23 |
24 | for path in self._get_paths(self._root):
25 | results_df_list.append(pd.read_csv(path))
26 |
27 | results_df = pd.concat(results_df_list)
28 | results_df["total_time"] = results_df["forward_time"] + results_df["backward_time"]
29 |
30 | return results_df
31 |
32 | def _get_paths(self, root):
33 | return [path for path in glob.glob(f"{root}/*")]
34 |
35 | def _query_results(self, **kwargs):
36 | query = True
37 | for key, value in kwargs.items():
38 | query &= self._results_df[key] == value
39 |
40 | if len(kwargs) > 0:
41 | return self._results_df[query]
42 |
43 | return self._results_df
44 |
45 | def get_speedup(self):
46 | results_df = self._build_df()
47 | standard_times = results_df[results_df["method"] == "standard"].set_index(["t_len", "units", "batch", "abs_refac", "layers"])[["forward_time", "backward_time", "total_time"]]
48 | blocks_times = results_df[results_df["method"] == "blocks"].set_index(["t_len", "units", "batch", "abs_refac", "layers"])[["forward_time", "backward_time", "total_time"]]
49 |
50 | speedup_df = standard_times / blocks_times
51 | speedup_df.rename(columns={"forward_time": "forward_speedup", "backward_time": "backward_speedup", "total_time": "total_speedup"}, inplace=True)
52 |
53 | return speedup_df
54 |
55 |
56 | class SupervisedQuery:
57 |
58 | def __init__(self, root):
59 | self.root = root
60 |
61 | def get_average_duration_per_batch(self, models_root, model_id):
62 | durations_list = []
63 |
64 | duration = trainer.load_log(models_root, model_id)["duration"][1:].mean()
65 | durations_list.append({"model_id": model_id, "duration": duration})
66 |
67 | return pd.DataFrame(durations_list).set_index("model_id").values[0][0]
68 |
69 | def build_results(self, dataset, methods, sgs, abs_refacs, repeats, detach=True, batch_size=500):
70 | results_list = []
71 |
72 | for method in methods:
73 | for sg in sgs:
74 | for abs_refac in abs_refacs:
75 | for i in range(repeats):
76 | if detach:
77 | name = f"{dataset.name}_{method}_{sg}_{abs_refac}_{dataset.dt}_{i}"
78 | else:
79 | name = f"{dataset.name}_{method}_{sg}_{abs_refac}_{dataset.dt}_{detach}_{i}"
80 | print(f"Loading {name}...")
81 | model = train.Trainer.load_model(f"{self.root}/results/supervised", name)
82 | val_acc = train.Trainer.get_acc(model, dataset, batch_size)
83 | avg_time = self.get_average_duration_per_batch(f"{self.root}/results/supervised", name)
84 | results_list.append({"dataset": dataset.name, "method": method, "sg": sg, "abs_refac": abs_refac, "dt": dataset.dt, "i": i, "val_acc": val_acc, "avg_time": avg_time})
85 |
86 | return pd.DataFrame(results_list)
--------------------------------------------------------------------------------
/src/snn/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/webstorms/Blocks/540c9f28eedd58ef638dcacf75b4c27cdf52baa0/src/snn/__init__.py
--------------------------------------------------------------------------------
/src/snn/block/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/webstorms/Blocks/540c9f28eedd58ef638dcacf75b4c27cdf52baa0/src/snn/block/__init__.py
--------------------------------------------------------------------------------
/src/snn/block/block.py:
--------------------------------------------------------------------------------
1 | import torch
2 | import torch.nn as nn
3 | import torch.nn.functional as F
4 |
5 | from src.snn.block.util import bconv1d
6 | from src.snn import surrogate
7 |
8 |
9 | class Block(nn.Module):
10 |
11 | def __init__(self, n_in, t_len, surr_grad):
12 | super().__init__()
13 | self._n_in = n_in
14 | self._t_len = t_len
15 | self._surr_grad = surr_grad
16 |
17 | self._beta_ident_base = nn.Parameter(torch.ones(n_in, t_len), requires_grad=False)
18 | self._beta_exp = nn.Parameter(torch.arange(t_len).flip(0).unsqueeze(0).expand(n_in, t_len).float(), requires_grad=False)
19 | self._phi_kernel = nn.Parameter((torch.arange(t_len) + 1).flip(0).float().view(1, 1, 1, t_len), requires_grad=False)
20 |
21 | @staticmethod
22 | def g(faulty_spikes):
23 | negate_faulty_spikes = faulty_spikes.clone().detach()
24 | negate_faulty_spikes[faulty_spikes == 1.0] = 0
25 | faulty_spikes -= negate_faulty_spikes
26 |
27 | return faulty_spikes
28 |
29 | def forward(self, current, beta, v_init=None, v_th=1, mode="train"):
30 |
31 | if v_init is not None:
32 | current[:, :, 0] += beta * v_init
33 |
34 | pad_current = F.pad(current, pad=(self._t_len - 1, 0)).unsqueeze(1)
35 |
36 | # compute membrane potential without reset
37 | beta_kernel = self.build_beta_kernel(beta)
38 | membrane = bconv1d(pad_current, beta_kernel)
39 |
40 | # map no-reset membrane potentials to output spikes
41 | v_th = v_th.unsqueeze(1)
42 | faulty_spikes = surrogate.spike(membrane - v_th, self._surr_grad)
43 |
44 | pad_spikes = F.pad(faulty_spikes, pad=(self._t_len - 1, 0))
45 | z = F.conv2d(pad_spikes, self._phi_kernel)
46 | z_copy = z.clone().squeeze(1)
47 |
48 | if mode == "train":
49 | return Block.g(z).squeeze(1), z_copy, membrane.squeeze(1)
50 | elif mode == "val":
51 | return Block.g(z).squeeze(1), z_copy, faulty_spikes, membrane.squeeze(1)
52 |
53 | def build_beta_kernel(self, beta):
54 | beta_base = beta.unsqueeze(1).multiply(self._beta_ident_base)
55 | return torch.pow(beta_base, self._beta_exp).unsqueeze(1).unsqueeze(1)
56 |
--------------------------------------------------------------------------------
/src/snn/block/blocks.py:
--------------------------------------------------------------------------------
1 | import math
2 | import torch
3 | import torch.nn as nn
4 | import torch.nn.functional as F
5 |
6 | from src.snn.snn import BaseSNN
7 | from src.snn.block.block import Block
8 | from src.snn.block.util import time_cat, bconv1d
9 |
10 |
11 | class Blocks(BaseSNN):
12 |
13 | def __init__(self, n_in, n_out, rf_len, t_len, t_latency, recurrent=True, beta_grad=True, adapt=True, init_beta=1, init_p=1, detach_spike_grad=True, surr_grad="fast_sigmoid"):
14 | super().__init__(n_in, n_out, rf_len, t_len, t_latency, recurrent, beta_grad, adapt, init_beta, init_p, detach_spike_grad, surr_grad)
15 |
16 | self._t_len_block = t_latency + 1
17 | self._block = Block(n_out, self._t_len_block, surr_grad)
18 | self._n_blocks = math.ceil(t_len / self._t_len_block)
19 | self._t_pad = self._n_blocks * self._t_len_block - self._t_len
20 |
21 | self._p_ident_base = nn.Parameter(torch.ones(n_out, self._t_len_block), requires_grad=False)
22 | self._p_exp = nn.Parameter(torch.arange(1, self._t_len_block + 1).float(), requires_grad=False)
23 |
24 | def process(self, x, mode="train"):
25 | x_init = x
26 | if self._t_pad != 0:
27 | x = F.pad(x, pad=(0, self._t_pad))
28 |
29 | mem_list = []
30 | spikes_list = []
31 | z_list = []
32 |
33 | z = torch.zeros_like(x[:, :, self._t_len_block:])
34 | v_init = torch.zeros_like(x[:, :, 0]).to(x.device)
35 | int_mem = torch.zeros_like(x[:, :, 0]).to(x.device)
36 |
37 | a_kernel = torch.zeros_like(x).to(x.device)[:, :, :self._t_len_block]
38 | v_th = torch.ones_like(x).to(x.device)[:, :, :self._t_len_block]
39 | v_th_list = []
40 |
41 | for i in range(self._n_blocks):
42 | x_slice = x[:, :, i * self._t_len_block: (i+1) * self._t_len_block]
43 |
44 | # Recurrent current and refractory mask only included after first block
45 | if i > 0:
46 | # Add recurrent current to input
47 | if self._recurrent:
48 | rec_current = self.get_rec_input(spikes)
49 | x_slice = x_slice + rec_current
50 |
51 | # Apply refractory mask to input
52 | if self._detach_spike_grad:
53 | spike_mask = spikes.detach().amax(dim=2).bool()
54 | else:
55 | spike_mask = spikes.amax(dim=2).bool()
56 | refac_mask = (z < spike_mask.unsqueeze(2)) * x_slice
57 | x_slice -= refac_mask
58 |
59 | # Set initial membrane potentials
60 | v_init = int_mem[:, :, -1] * ~spike_mask # if spiked -> zero initial membrane potential
61 |
62 | # Set initial adaptive params
63 | if self._adapt:
64 | # Get a at time of spike + spike (which is equal to 1/p to account for raising v_th by 1 next step
65 | # do the math or see paper if this is not clear)
66 | if self._detach_spike_grad:
67 | a_at_spike = (a_kernel * spikes.detach()).sum(dim=2) + (1 / self.p)
68 | else:
69 | a_at_spike = (a_kernel * spikes).sum(dim=2) + (1 / self.p)
70 | decay_steps = (z > 1).sum(dim=2) # Compute number of decay steps
71 | new_a = a_at_spike * torch.pow(self.p.unsqueeze(0), decay_steps)
72 | a = (a_kernel[:, :, -1] * ~spike_mask) + (new_a * spike_mask)
73 |
74 | # Update a for neurons that spiked
75 | a_kernel = self.compute_a_kernel(a, self.p)
76 | v_th = 1 + self.b.view(1, -1, 1) * a_kernel
77 |
78 | if mode == "train":
79 | spikes, z, int_mem = self._block(x_slice, self.beta, v_init=v_init, v_th=v_th, mode="train")
80 | spikes_list.append(spikes)
81 | elif mode == "val":
82 | spikes, z, _, int_mem = self._block(x_slice, self.beta, v_init=v_init, v_th=v_th, mode="val")
83 | spikes_list.append(spikes)
84 | mem_list.append(int_mem)
85 | z_list.append(z)
86 | v_th_list.append(v_th)
87 |
88 | if mode == "train":
89 | return time_cat(spikes_list, self._t_pad)
90 | elif mode == "val":
91 | return time_cat(spikes_list, self._t_pad), time_cat(mem_list, self._t_pad), x_init, time_cat(z_list, self._t_pad), time_cat(v_th_list, self._t_pad)
92 |
93 | def compute_a_kernel(self, a, p):
94 | # a: b x n
95 | # p: n
96 | # output: b x n x t
97 |
98 | return torch.pow(p.unsqueeze(-1) * self._p_ident_base, self._p_exp).unsqueeze(0) * a.unsqueeze(-1)
99 |
100 |
101 | class BlocksIntegrator(BaseSNN):
102 |
103 | def __init__(self, n_in, n_out, t_len, init_beta=1):
104 | super().__init__(n_in, n_out, 1, t_len, t_latency=0, recurrent=False, beta_grad=True, adapt=False, init_beta=init_beta, init_p=1, detach_spike_grad=True, surr_grad="fast_sigmoid")
105 | self._block = Block(n_out, t_len, "fast_sigmoid")
106 |
107 | def process(self, x, mode="train"):
108 | pad_current = F.pad(x, pad=(self._t_len - 1, 0)).unsqueeze(1)
109 |
110 | # compute membrane potential without reset
111 | beta_kernel = self._block.build_beta_kernel(self.beta)
112 | membrane = bconv1d(pad_current, beta_kernel)
113 |
114 | return membrane.squeeze(1)
115 |
--------------------------------------------------------------------------------
/src/snn/block/util.py:
--------------------------------------------------------------------------------
1 | import torch
2 | import torch.nn.functional as F
3 |
4 |
5 | def bconv1d(x, weight, stride=1, dilation=1, padding=0):
6 | # Would be useful if PyTorch provided batched 1D convs in their library
7 | b, c, n, h = x.shape
8 | n, out_channels, in_channels, kernel_width_size = weight.shape
9 |
10 | out = x.view(b, c * n, h)
11 | weight = weight.view(n * out_channels, in_channels, kernel_width_size)
12 |
13 | out = F.conv1d(out, weight=weight, bias=None, stride=stride, dilation=dilation, groups=n, padding=padding)
14 |
15 | return out.view(b, c, n, -1)
16 |
17 |
18 | def time_cat(tensor_list, t_pad):
19 | tensor = torch.cat(tensor_list, dim=2)
20 |
21 | if t_pad > 0:
22 | tensor = tensor[:, :, :-t_pad]
23 |
24 | return tensor
25 |
--------------------------------------------------------------------------------
/src/snn/snn.py:
--------------------------------------------------------------------------------
1 | import numpy as np
2 | import torch
3 | import torch.nn as nn
4 | import torch.nn.functional as F
5 | from brainbox.models import BBModel
6 |
7 | from src.snn import surrogate
8 |
9 | # SNN control models.
10 |
11 |
12 | class BaseSNN(BBModel):
13 |
14 | MIN_BETA = 0.001
15 | MAX_BETA = 0.999
16 |
17 | def __init__(self, n_in, n_out, rf_len, t_len, t_latency, recurrent=True, beta_grad=True, adapt=True, init_beta=1, init_p=1, detach_spike_grad=True, surr_grad="fast_sigmoid"):
18 | super().__init__()
19 | self._n_in = n_in
20 | self._n_out = n_out
21 | self._rf_len = rf_len
22 | self._t_len = t_len
23 | self._t_latency = t_latency
24 | self._recurrent = recurrent
25 | self._beta_grad = beta_grad
26 | self._adapt = adapt
27 | self._detach_spike_grad = detach_spike_grad
28 | self._surr_grad = surr_grad
29 |
30 | self._beta = nn.Parameter(data=torch.Tensor(n_out * [init_beta]), requires_grad=beta_grad)
31 | self._rf_weight = nn.Parameter(torch.rand(n_out, 1, n_in, self._rf_len), requires_grad=True)
32 | self._rf_bias = nn.Parameter(torch.zeros(n_out), requires_grad=True)
33 |
34 | self._rec_weight = nn.Parameter(torch.rand(n_out, n_out), requires_grad=recurrent)
35 |
36 | self._p = nn.Parameter(data=torch.Tensor(n_out * [init_p]), requires_grad=adapt)
37 | self._b = nn.Parameter(data=torch.Tensor(n_out * [1.8]), requires_grad=adapt)
38 |
39 | self.init_weight(self._rf_weight, "uniform", a=-1 / np.sqrt(n_in * rf_len), b=1 / np.sqrt(n_in * rf_len))
40 | self.init_weight(self._rec_weight, "identity")
41 |
42 | @property
43 | def hyperparams(self):
44 | return {**super().hyperparams, "n_in": self._n_in, "n_out": self._n_out, "rf_len": self._rf_len, "t_len": self._t_len, "t_latency": self._t_latency, "recurrent": self._recurrent, "beta_grad": self._beta_grad, "adapt": self._adapt, "detach_spike_grad": self._detach_spike_grad, "surr_grad": self._surr_grad}
45 |
46 | @property
47 | def p(self):
48 | return torch.clamp(self._p.abs(), min=0, max=0.999)
49 |
50 | @property
51 | def b(self):
52 | return torch.clamp(self._b.abs(), min=0.001, max=1)
53 |
54 | @property
55 | def beta(self):
56 | return torch.clamp(self._beta, min=BaseSNN.MIN_BETA, max=BaseSNN.MAX_BETA)
57 |
58 | @property
59 | def rec_weight(self):
60 | return self._rec_weight
61 |
62 | def get_rec_input(self, spikes):
63 | return torch.einsum("ij, bj...->bi...", self.rec_weight, spikes.detach() if self._detach_spike_grad else spikes)
64 |
65 | def forward(self, x, mode="train"):
66 | # x: b x n x t
67 |
68 | x = F.pad(x, (self._rf_len - 1, 0))
69 | x = x.unsqueeze(1) # Add channel dim
70 | x = F.conv2d(x, self._rf_weight, self._rf_bias)[:, :, 0] # Slice out height dim
71 |
72 | return self.process(x, mode)
73 |
74 | def process(self, x, mode):
75 | raise NotImplementedError
76 |
77 |
78 | class SNN(BaseSNN):
79 |
80 | def __init__(self, n_in, n_out, rf_len, t_len, t_latency, recurrent=False, beta_grad=True, adapt=True, init_beta=1, init_p=1, detach_spike_grad=True, surr_grad="fast_sigmoid"):
81 | super().__init__(n_in, n_out, rf_len, t_len, t_latency, recurrent, beta_grad, adapt, init_beta, init_p, detach_spike_grad, surr_grad)
82 |
83 | def process(self, x, mode="train"):
84 | # x: b x n x t
85 |
86 | mem_list = []
87 | spikes_list = []
88 | spikes = torch.zeros_like(x).to(x.device)[:, :, 0]
89 | rec_current = torch.zeros_like(x)
90 | mem = torch.zeros_like(x).to(x.device)[:, :, 0]
91 | refac_times = torch.zeros_like(x).to(x.device)[:, :, 0] + self._t_latency
92 |
93 | v_th = torch.ones_like(x).to(x.device)[:, :, 0]
94 | a = torch.zeros_like(x).to(x.device)[:, :, 0]
95 | v_th_list = []
96 |
97 | for t in range(x.shape[2]):
98 | stimulus_current = x[:, :, t]
99 | rec_current[:, :, t] = self.get_rec_input(spikes)
100 |
101 | # Recurrent latency
102 | if t >= self._t_latency and self._recurrent:
103 | input_current = stimulus_current + rec_current[:, :, t-self._t_latency]
104 | else:
105 | input_current = stimulus_current
106 |
107 | # Apply absolute refractory period
108 | refac_times[spikes > 0] = 0
109 | refac_mask = refac_times < self._t_latency
110 | input_current[refac_mask] = 0
111 | refac_times += 1
112 |
113 | new_mem = torch.einsum("bn...,n->bn...", mem, self.beta) + input_current
114 | spikes = surrogate.spike(new_mem - v_th, self._surr_grad)
115 |
116 | mem_list.append(new_mem)
117 | if self._detach_spike_grad:
118 | mem = new_mem * (1 - spikes.detach())
119 | else:
120 | mem = new_mem * (1 - spikes)
121 | # new_mem -= new_mem * spikes (should be same as above?)
122 | spikes_list.append(spikes)
123 |
124 | if self._adapt:
125 | a = self.p * a + spikes
126 | v_th = 1 + self.b * a
127 | v_th_list.append(v_th)
128 |
129 | if mode == "train":
130 | return torch.stack(spikes_list, dim=2)
131 | elif mode == "val":
132 | v_th = torch.stack(v_th_list, dim=2)
133 | v_th = torch.roll(v_th, 1, dims=2)
134 | v_th[:, :, :1] = 1
135 |
136 | return torch.stack(spikes_list, dim=2), torch.stack(mem_list, dim=2), x, v_th
137 |
138 |
139 | class SNNIntegrator(SNN):
140 |
141 | def __init__(self, n_in, n_out, t_len, init_beta=1):
142 | super().__init__(n_in, n_out, 1, t_len, t_latency=0, recurrent=False, beta_grad=True, adapt=False, init_beta=init_beta, init_p=1, detach_spike_grad=True, surr_grad="fast_sigmoid")
143 |
144 | def process(self, x, mode="train"):
145 | mem_list = []
146 | mem = torch.zeros_like(x).to(x.device)[:, :, 0]
147 |
148 | for t in range(x.shape[2]):
149 | input_current = x[:, :, t]
150 |
151 | new_mem = torch.einsum("bn...,n->bn...", mem, self.beta) + input_current
152 | mem_list.append(new_mem)
153 | mem = new_mem
154 |
155 | return torch.stack(mem_list, dim=2)
156 |
--------------------------------------------------------------------------------
/src/snn/surrogate.py:
--------------------------------------------------------------------------------
1 | import math
2 |
3 | import torch
4 |
5 |
6 | class FastSigmoid(torch.autograd.Function):
7 |
8 | @staticmethod
9 | def forward(ctx, input, scale=10):
10 | ctx.scale = scale
11 | ctx.save_for_backward(input)
12 |
13 | return input.gt(0).float()
14 |
15 | @staticmethod
16 | def backward(ctx, grad_output):
17 | input, = ctx.saved_tensors
18 | grad_input = grad_output.clone()
19 | grad = grad_input / (ctx.scale * torch.abs(input) + 1.0) ** 2
20 |
21 | return grad, None
22 |
23 |
24 | class BoxCar(torch.autograd.Function):
25 |
26 | @staticmethod
27 | def forward(ctx, input):
28 | ctx.save_for_backward(input)
29 |
30 | return input.gt(0).float()
31 |
32 | @staticmethod
33 | def backward(ctx, grad_output):
34 | input, = ctx.saved_tensors
35 | grad = grad_output.clone()
36 | grad[input <= -0.5] = 0
37 | grad[input > 0.5] = 0
38 |
39 | return grad
40 |
41 |
42 | class MG(torch.autograd.Function):
43 |
44 | @staticmethod
45 | def forward(ctx, input):
46 | ctx.save_for_backward(input)
47 | return input.gt(0).float()
48 |
49 | @staticmethod
50 | def backward(ctx, grad_output):
51 | input, = ctx.saved_tensors
52 | grad = grad_output.clone()
53 | lens = 0.5
54 | hight = 0.15
55 | scale = 6
56 | gamma = 0.5
57 |
58 | temp = MG.gaussian(input, mu=0., sigma=lens) * (1. + hight) - MG.gaussian(input, mu=lens, sigma=scale * lens) * hight - MG.gaussian(input, mu=-lens, sigma=scale * lens) * hight
59 |
60 | return gamma * grad * temp.float()
61 |
62 | @staticmethod
63 | def gaussian(x, mu=0., sigma=.5):
64 | return torch.exp(-((x - mu) ** 2) / (2 * sigma ** 2)) / torch.sqrt(2 * torch.tensor(math.pi)) / sigma
65 |
66 |
67 | def spike(x, type):
68 | if type == "fast_sigmoid":
69 | return FastSigmoid.apply(x)
70 | elif type == "box_car":
71 | return BoxCar.apply(x)
72 | elif type == "mg":
73 | return MG.apply(x)
74 |
--------------------------------------------------------------------------------
/src/train.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import time
4 | import logging
5 |
6 | import torch
7 | import torch.nn.functional as F
8 | import numpy as np
9 | import pandas as pd
10 | from brainbox import trainer
11 | from brainbox.physiology.spiking import VanRossum
12 |
13 | from src import datasets, models
14 |
15 |
16 | torch.backends.cudnn.benchmark = True
17 | logger = logging.getLogger("trainer")
18 | logging.basicConfig(stream=sys.stdout, level=logging.INFO)
19 |
20 |
21 | class Trainer(trainer.Trainer):
22 |
23 | def __init__(self, root, model, dataset, n_epochs, batch_size, lr, milestones=[-1], gamma=0.1, val_dataset=None, device="cuda", id=None):
24 | super().__init__(root, model, dataset, n_epochs, batch_size, lr, torch.optim.Adam, device=device, optimizer_kwargs={"eps": 1e-5}, loader_kwargs={"shuffle": True, "pin_memory": True, "num_workers": 16}, id=id)
25 | self._milestones = milestones
26 | self._gamma = gamma
27 | self._val_dataset = val_dataset
28 |
29 | self._times = {"forward_pass": [], "backward_pass": []}
30 | self._train_acc = []
31 | self._val_acc = []
32 | self._min_loss = np.inf
33 | self._milestone_idx = 0
34 |
35 | @staticmethod
36 | def accuracy_metric(output, target):
37 | _, predictions = torch.max(output, 1)
38 | return (predictions == target).sum().cpu().item()
39 |
40 | @staticmethod
41 | def spike_count(output, target):
42 | _, cortical_output, thalamic_output = output
43 |
44 | count = cortical_output[0].sum().cpu().item()
45 | count += thalamic_output[0].sum().cpu().item()
46 |
47 | return count
48 |
49 | @property
50 | def times_path(self):
51 | return os.path.join(self.root, self.id, "times.csv")
52 |
53 | @property
54 | def train_acc_path(self):
55 | return os.path.join(self.root, self.id, "train_acc.csv")
56 |
57 | @property
58 | def val_acc_path(self):
59 | return os.path.join(self.root, self.id, "val_acc.csv")
60 |
61 | def save_model_log(self):
62 | super().save_model_log()
63 |
64 | # Save times
65 | times_df = pd.DataFrame(self._times)
66 | times_df.to_csv(self.times_path, index=False)
67 |
68 | # Save acc
69 | train_acc_df = pd.DataFrame(self._train_acc)
70 | train_acc_df.to_csv(self.train_acc_path, index=False)
71 | val_acc_df = pd.DataFrame(self._val_acc)
72 | val_acc_df.to_csv(self.val_acc_path, index=False)
73 |
74 | def loss(self, output, target, model):
75 | target = target.long()
76 | loss = F.cross_entropy(output, target, reduction="mean")
77 |
78 | return loss
79 |
80 | def train_for_single_epoch(self):
81 | epoch_loss = 0
82 | n_samples = 0
83 | n_correct = 0
84 |
85 | for batch_id, (data, target) in enumerate(self.train_data_loader):
86 | data = data.to(self.device).type(self.dtype)
87 | target = target.to(self.device).type(self.dtype)
88 | torch.cuda.synchronize()
89 |
90 | # Forward pass
91 | start_time = time.time()
92 | output = self.model(data)
93 | torch.cuda.synchronize()
94 | forward_pass_time = time.time() - start_time
95 | self._times["forward_pass"].append(forward_pass_time)
96 |
97 | # Compute accuracy
98 | _, predictions = torch.max(output, 1)
99 | n_correct += (predictions == target).sum().cpu().item()
100 |
101 | # Compute loss
102 | loss = self.loss(output, target, self.model)
103 |
104 | # Backward pass
105 | start_time = time.time()
106 | loss.backward()
107 | torch.cuda.synchronize()
108 | backward_pass_time = time.time() - start_time
109 | self._times["backward_pass"].append(backward_pass_time)
110 |
111 | self.optimizer.step()
112 | self.optimizer.zero_grad()
113 |
114 | with torch.no_grad():
115 | epoch_loss += (loss.item() * data.shape[0])
116 | n_samples += data.shape[0]
117 |
118 | train_acc = n_correct/n_samples
119 | logging.info(f"Train acc: {train_acc}")
120 | self._train_acc.append(train_acc)
121 |
122 | if self._val_dataset is not None and len(self.log["train_loss"]) % 5 == 0:
123 | val_acc = Trainer.get_acc(self.model, self._val_dataset, self.batch_size)
124 | logging.info(f"Val acc: {val_acc}")
125 | self._val_acc.append(val_acc)
126 |
127 | return epoch_loss / n_samples
128 |
129 | @staticmethod
130 | def get_acc(model, dataset, batch_size):
131 | scores = trainer.compute_metric(model, dataset, Trainer.accuracy_metric, batch_size=batch_size)
132 | return np.sum(scores) / len(dataset)
133 |
134 | @staticmethod
135 | def get_spike_count(model, dataset, batch_size):
136 | scores = trainer.compute_metric(model, dataset, Trainer.spike_count, batch_size=batch_size)
137 | return np.sum(scores) / len(dataset)
138 |
139 | def on_epoch_complete(self, save):
140 | if save:
141 | self.save_model_log()
142 |
143 | epoch_loss = self.log["train_loss"][-1]
144 | if epoch_loss < self._min_loss:
145 | logging.info(f"Saving model...")
146 | self._min_loss = epoch_loss
147 | self.save_model()
148 |
149 | n_epoch = len(self.log["train_loss"])
150 |
151 | if n_epoch == self._milestones[self._milestone_idx]:
152 | logging.info(f"Decaying lr...")
153 | self.lr *= self._gamma
154 | # Load best model
155 | self.model = Trainer.load_model(self.root, self.id, self.device, self.dtype)
156 | self.optimizer = self.optimizer_func(
157 | self.model.parameters(), self.lr, **self.optimizer_kwargs
158 | )
159 |
160 | if self._milestone_idx != len(self._milestones) - 1:
161 | logging.info(f"New milestone target...")
162 | self._milestone_idx += 1
163 |
164 | def on_training_complete(self, save):
165 | pass
166 |
167 | @staticmethod
168 | def hyperparams_loader(hyperparams):
169 | model_params = hyperparams["model"]
170 | del model_params["name"]
171 | del model_params["weight_initializers"]
172 |
173 | return models.AuditoryModel(**model_params)
174 |
175 | @staticmethod
176 | def load_model(root, id, device="cuda", dtype=torch.float):
177 | return trainer.load_model(root, id, Trainer.hyperparams_loader, device, dtype)
178 |
179 |
180 | class EphysTrainer(Trainer):
181 |
182 | def __init__(self, root, model, dataset, n_epochs, batch_size, lr, gamma=0.1, dt=0.1, epoch_scan=5, max_decay=1, val_dataset=None, device="cuda", id=None):
183 | super().__init__(root, model, dataset, n_epochs, batch_size, lr, [-1], gamma, val_dataset, device, id)
184 | self.epoch_scan = epoch_scan
185 | self.van_rossum = VanRossum(datasets.EphysDataset.LENGTH, tau=100, dt=dt).to(device)
186 | self.max_decay = max_decay
187 |
188 | self._decay_count = 0
189 |
190 | def loss(self, spikes_pred, spikes):
191 | spike_loss = self.van_rossum(spikes_pred, spikes)
192 |
193 | return spike_loss
194 |
195 | def train_for_single_epoch(self):
196 | epoch_loss = 0
197 | n_samples = 0
198 |
199 | for batch_id, (data, target) in enumerate(self.train_data_loader):
200 | data = data.to(self.device).type(self.dtype)
201 | trace = target[0].to(self.device).type(self.dtype)
202 | spikes = target[1].to(self.device).type(self.dtype)
203 | torch.cuda.synchronize()
204 |
205 | # Forward pass
206 | start_time = time.time()
207 | spikes_pred = self.model(data)
208 | torch.cuda.synchronize()
209 | forward_pass_time = time.time() - start_time
210 | self._times["forward_pass"].append(forward_pass_time)
211 |
212 | # Compute loss
213 | loss = self.loss(spikes_pred, spikes)
214 |
215 | # Backward pass
216 | start_time = time.time()
217 | loss.backward()
218 | torch.cuda.synchronize()
219 | backward_pass_time = time.time() - start_time
220 | self._times["backward_pass"].append(backward_pass_time)
221 |
222 | self.optimizer.step()
223 | self.optimizer.zero_grad()
224 |
225 | with torch.no_grad():
226 | epoch_loss += (loss.item() * data.shape[0])
227 | n_samples += data.shape[0]
228 |
229 | return epoch_loss / n_samples
230 |
231 | def on_epoch_complete(self, save):
232 | if save:
233 | self.save_model_log()
234 |
235 | epoch_loss = self.log["train_loss"][-1]
236 | if epoch_loss < self._min_loss:
237 | logging.info(f"Saving model...")
238 | self._min_loss = epoch_loss
239 | self.save_model()
240 |
241 | min_lost_over_last_epochs = np.array(self.log["train_loss"][-self.epoch_scan:]).min()
242 |
243 | if min_lost_over_last_epochs > self._min_loss:
244 | if self._decay_count < self.max_decay:
245 | self._min_loss = np.inf
246 | self._last_train_scores = []
247 | logging.info(f"Decaying lr...")
248 | self._decay_count += 1
249 | self.lr *= self._gamma
250 | # Load best model
251 | self.model = EphysTrainer.load_model(self.root, self.id, self.device, self.dtype)
252 | self.optimizer = self.optimizer_func(
253 | self.model.parameters(), self.lr, **self.optimizer_kwargs
254 | )
255 | else:
256 | self.exit = True
257 |
258 | @staticmethod
259 | def load_model(root, id, device="cuda", dtype=torch.float, dt01ref=False):
260 |
261 | def model_loader(hyperparams):
262 | model_params = hyperparams["model"]
263 | del model_params["name"]
264 | del model_params["weight_initializers"]
265 |
266 | model_params = {**model_params, "dt01ref": dt01ref}
267 |
268 | return models.Neuron(**model_params)
269 |
270 | return trainer.load_model(root, id, model_loader, device, dtype)
271 |
--------------------------------------------------------------------------------
/tests/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/webstorms/Blocks/540c9f28eedd58ef638dcacf75b4c27cdf52baa0/tests/__init__.py
--------------------------------------------------------------------------------
/tests/test_block.py:
--------------------------------------------------------------------------------
1 | import torch
2 | import pytest
3 |
4 | from src.snn.block.block import Block
5 |
6 |
7 | @pytest.fixture
8 | def block():
9 | return Block(2, 4, "fast_sigmoid")
10 |
11 |
12 | def test_beta_kernel(block):
13 | # Use a single beta
14 | assert torch.allclose(block.build_beta_kernel(torch.Tensor([0.1]))[0, 0, 0], torch.Tensor([0.0010, 0.0100, 0.1000, 1.0000]))
15 | assert torch.allclose(block.build_beta_kernel(torch.Tensor([0.1]))[1, 0, 0], torch.Tensor([0.0010, 0.0100, 0.1000, 1.0000]))
16 |
17 | # Use multiple beta
18 | assert torch.allclose(block.build_beta_kernel(torch.Tensor([0.1]))[0, 0, 0], torch.Tensor([0.0010, 0.0100, 0.1000, 1.0000]))
19 | assert torch.allclose(block.build_beta_kernel(torch.Tensor([0.5]))[1, 0, 0], torch.Tensor([0.1250, 0.2500, 0.5000, 1.0000]))
20 |
21 |
22 | def test_phi_kernel(block):
23 | assert torch.allclose(block._phi_kernel, torch.Tensor([[[[4., 3., 2., 1.]]]]))
24 |
25 |
26 | def test_g(block):
27 | phi_spikes = torch.zeros(2, 4)
28 | phi_spikes[0, 0] = 1
29 | phi_spikes[0, 2] = 2
30 | phi_spikes[1, 1] = 1
31 | phi_spikes[1, 3] = 3
32 | assert block.g(phi_spikes).sum() == 2
33 |
34 |
35 | def test_differentiable_vars(block):
36 | assert not block._beta_ident_base.requires_grad
37 | assert not block._beta_exp.requires_grad
38 | assert not block._phi_kernel.requires_grad
--------------------------------------------------------------------------------
/tests/test_blocks.py:
--------------------------------------------------------------------------------
1 | import torch
2 | import pytest
3 |
4 | from src.snn.snn import SNN
5 | from src.snn.block.blocks import Blocks
6 |
7 |
8 | @pytest.fixture
9 | def in_spikes(b=4, n=100, t=200):
10 | return torch.rand(b, n, t)
11 |
12 |
13 | def get_models(n_in=100, n_out=100, rf_len=10, t_len=200, t_latency=0, recurrent=True, adapt=False):
14 | blocks = Blocks(n_in, n_out, rf_len, t_len, t_latency, recurrent=recurrent, adapt=adapt)
15 | snn = SNN(n_in, n_out, rf_len, t_len, t_latency, recurrent=recurrent, adapt=adapt)
16 |
17 | blocks._rf_weight = snn._rf_weight
18 | blocks._rf_bias = snn._rf_bias
19 | blocks._rec_weight = snn._rec_weight
20 |
21 | return blocks, snn
22 |
23 |
24 | def test_networks_none(in_spikes):
25 | for t_latency in [0, 1, 2, 4, 8]:
26 | blocks, snn = get_models(t_latency=t_latency, recurrent=False)
27 | spikes1 = blocks(in_spikes, mode="train")
28 | spikes2 = snn(in_spikes, mode="train")
29 |
30 | assert torch.allclose(spikes1, spikes2)
31 |
32 |
33 | def test_networks_recurrent(in_spikes):
34 | for t_latency in [0, 1, 2, 4, 8]:
35 | blocks, snn = get_models(t_latency=t_latency, recurrent=True)
36 | spikes1 = blocks(in_spikes, mode="train")
37 | spikes2 = snn(in_spikes, mode="train")
38 |
39 | assert torch.allclose(spikes1, spikes2)
40 |
41 |
42 | def test_adaption(in_spikes):
43 | for t_latency in [0, 1, 2, 4, 8]:
44 | blocks, snn = get_models(t_latency=t_latency, recurrent=True, adapt=True)
45 | spikes1, mem1, x1, z1, v_th1 = blocks(in_spikes, mode="val")
46 | spikes2, mem2, x2, v_th2 = snn(in_spikes, mode="val")
47 |
48 | # v_th should equal at the start of every block
49 | for i in range(blocks._t_len // blocks._t_len_block):
50 | assert torch.allclose(v_th1[:, :, i*blocks._t_len_block], v_th2[:, :, i*blocks._t_len_block])
51 |
--------------------------------------------------------------------------------
7 | 
57 | 
62 |