├── .gitignore ├── README.md ├── curves ├── imdb-curves │ ├── acclip-imdb-lstm │ ├── adam-imdb-lstm │ └── sgd-imdb-lstm ├── mrpc-curves │ ├── acclip-mrpc-bert │ ├── adam-mrpc-bert │ └── sgd-mrpc-bert ├── squad-curves │ ├── acclip-squad-bert │ └── adam-squad-bert └── sst-curves │ ├── acclip-sst-lstm │ ├── adam-sst-lstm │ └── sgd-sst-lstm ├── download_mrpc_data.py ├── models └── lstm_attn.py ├── noise_plot.ipynb ├── optimizers └── ACClip.py ├── plot.ipynb ├── run_mrpc.py ├── run_mrpc.sh ├── run_text_classifier.py └── utils.py /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | *.egg-info/ 24 | .installed.cfg 25 | *.egg 26 | MANIFEST 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | *.cover 47 | .hypothesis/ 48 | .pytest_cache/ 49 | 50 | # Translations 51 | *.mo 52 | *.pot 53 | 54 | # Django stuff: 55 | *.log 56 | local_settings.py 57 | db.sqlite3 58 | 59 | # Flask stuff: 60 | instance/ 61 | .webassets-cache 62 | 63 | # Scrapy stuff: 64 | .scrapy 65 | 66 | # Sphinx documentation 67 | docs/_build/ 68 | 69 | # PyBuilder 70 | target/ 71 | 72 | # Jupyter Notebook 73 | .ipynb_checkpoints 74 | 75 | # pyenv 76 | .python-version 77 | 78 | # celery beat schedule file 79 | celerybeat-schedule 80 | 81 | # SageMath parsed files 82 | *.sage.py 83 | 84 | # Environments 85 | .env 86 | .venv 87 | env/ 88 | venv/ 89 | ENV/ 90 | env.bak/ 91 | venv.bak/ 92 | 93 | # Spyder project settings 94 | .spyderproject 95 | .spyproject 96 | 97 | # Rope project settings 98 | .ropeproject 99 | 100 | # mkdocs documentation 101 | /site 102 | 103 | # mypy 104 | .mypy_cache/ 105 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # ACClip-Pytorch 2 | The PyTorch implementation of ICLR2020 submission: Why ADAM Beats SGD for Attention Models (https://openreview.net/pdf?id=SJx37TEtDH) 3 | 4 | Github Repo: https://github.com/rivercold/ACClip-Pytorch 5 | 6 | ## Requirments 7 | Strongly recommend that you have a Anaconda enviroment. (https://www.anaconda.com/distribution/) 8 | * Python >= 3.6 9 | * PyTorch >= 1.0 (https://pytorch.org/get-started/locally/) 10 | * torchtext >= 0.4.0 (conda install -c pytorch torchtext) 11 | * numpy >= 1.16.4 12 | * matplotlib >= 3.1.0 13 | * transformers >= 2.1.1 (pip install transformers) 14 | 15 | ## Core Implementaions 16 | `ACClip` optimizer: https://github.com/rivercold/ACClip-Pytorch/blob/master/optimizers/ACClip.py 17 | `print_noise` function for LSTM: https://github.com/rivercold/ACClip-Pytorch/blob/master/run_text_classifier.py#L22 18 | `print_noise` function for BERT: https://github.com/rivercold/ACClip-Pytorch/blob/master/run_mrpc.py#L224 19 | 20 | ## Setup 21 | 22 | ### Text Classification 23 | 24 | #### IMDB 25 | ```shell script 26 | $ python run_text_classifier.py --optimizer=acclip --lr=0.001 --epoch=30 27 | ``` 28 | optmizers chosen from `acclip`, `adam` and `sgd`. 29 | 30 | ##### Plot Noise 31 | ```shell script 32 | $ python run_text_classifier.py --optimizer=sgd --lr=0.1 --epoch=30 --mode=plot 33 | ``` 34 | 35 | #### SST 36 | ```shell script 37 | $ python run_text_classifier.py --optimizer=acclip --lr=0.001 --epoch=20 --dataset=sst 38 | ``` 39 | 40 | ### GLUE task 41 | * Donwload the datasets 42 | ```shell script 43 | $ python download_mrpc_data.py 44 | ``` 45 | 46 | * Run Bert model, where you can specify your optimizers and learning rates in ```run_mrpc.sh``` 47 | ```shell script 48 | $ sh ./run_mrpc.sh 49 | ``` 50 | 51 | * To plot noise 52 | switch `do_train` with `do_plot` in `./run_mrpc.sh` 53 | 54 | ## Plot for visualization 55 | For training, the evaluation results will be written to the `curves` folder. 56 | Install Jupyter notebook to run ```plot.ipynb```. 57 | 58 | For plotting noise norm, the evaluation results will be written to `noises` folder. 59 | Run ```noise_plot.ipynb``` 60 | -------------------------------------------------------------------------------- /curves/imdb-curves/acclip-imdb-lstm: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rivercold/ACClip-Pytorch/8141bc87b1493e31c0a29999db2e315a4787721f/curves/imdb-curves/acclip-imdb-lstm -------------------------------------------------------------------------------- /curves/imdb-curves/adam-imdb-lstm: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rivercold/ACClip-Pytorch/8141bc87b1493e31c0a29999db2e315a4787721f/curves/imdb-curves/adam-imdb-lstm -------------------------------------------------------------------------------- /curves/imdb-curves/sgd-imdb-lstm: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rivercold/ACClip-Pytorch/8141bc87b1493e31c0a29999db2e315a4787721f/curves/imdb-curves/sgd-imdb-lstm -------------------------------------------------------------------------------- /curves/mrpc-curves/acclip-mrpc-bert: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rivercold/ACClip-Pytorch/8141bc87b1493e31c0a29999db2e315a4787721f/curves/mrpc-curves/acclip-mrpc-bert -------------------------------------------------------------------------------- /curves/mrpc-curves/adam-mrpc-bert: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rivercold/ACClip-Pytorch/8141bc87b1493e31c0a29999db2e315a4787721f/curves/mrpc-curves/adam-mrpc-bert -------------------------------------------------------------------------------- /curves/mrpc-curves/sgd-mrpc-bert: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rivercold/ACClip-Pytorch/8141bc87b1493e31c0a29999db2e315a4787721f/curves/mrpc-curves/sgd-mrpc-bert -------------------------------------------------------------------------------- /curves/squad-curves/acclip-squad-bert: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rivercold/ACClip-Pytorch/8141bc87b1493e31c0a29999db2e315a4787721f/curves/squad-curves/acclip-squad-bert -------------------------------------------------------------------------------- /curves/squad-curves/adam-squad-bert: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rivercold/ACClip-Pytorch/8141bc87b1493e31c0a29999db2e315a4787721f/curves/squad-curves/adam-squad-bert -------------------------------------------------------------------------------- /curves/sst-curves/acclip-sst-lstm: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rivercold/ACClip-Pytorch/8141bc87b1493e31c0a29999db2e315a4787721f/curves/sst-curves/acclip-sst-lstm -------------------------------------------------------------------------------- /curves/sst-curves/adam-sst-lstm: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rivercold/ACClip-Pytorch/8141bc87b1493e31c0a29999db2e315a4787721f/curves/sst-curves/adam-sst-lstm -------------------------------------------------------------------------------- /curves/sst-curves/sgd-sst-lstm: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rivercold/ACClip-Pytorch/8141bc87b1493e31c0a29999db2e315a4787721f/curves/sst-curves/sgd-sst-lstm -------------------------------------------------------------------------------- /download_mrpc_data.py: -------------------------------------------------------------------------------- 1 | ''' Script for downloading all GLUE data. 2 | Note: for legal reasons, we are unable to host MRPC. 3 | You can either use the version hosted by the SentEval team, which is already tokenized, 4 | or you can download the original data from (https://download.microsoft.com/download/D/4/6/D46FF87A-F6B9-4252-AA8B-3604ED519838/MSRParaphraseCorpus.msi) and extract the data from it manually. 5 | For Windows users, you can run the .msi file. For Mac and Linux users, consider an external library such as 'cabextract' (see below for an example). 6 | You should then rename and place specific files in a folder (see below for an example). 7 | mkdir MRPC 8 | cabextract MSRParaphraseCorpus.msi -d MRPC 9 | cat MRPC/_2DEC3DBE877E4DB192D17C0256E90F1D | tr -d $'\r' > MRPC/msr_paraphrase_train.txt 10 | cat MRPC/_D7B391F9EAFF4B1B8BCE8F21B20B1B61 | tr -d $'\r' > MRPC/msr_paraphrase_test.txt 11 | rm MRPC/_* 12 | rm MSRParaphraseCorpus.msi 13 | 1/30/19: It looks like SentEval is no longer hosting their extracted and tokenized MRPC data, so you'll need to download the data from the original source for now. 14 | 2/11/19: It looks like SentEval actually *is* hosting the extracted data. Hooray! 15 | ''' 16 | 17 | import os 18 | import sys 19 | import shutil 20 | import argparse 21 | import tempfile 22 | import urllib.request 23 | import zipfile 24 | # TASKS = ["CoLA", "SST", "MRPC", "QQP", "STS", "MNLI", "SNLI", "QNLI", "RTE", "WNLI", "diagnostic"] 25 | TASKS = ["MRPC"] 26 | TASK2PATH = {"CoLA":'https://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2FCoLA.zip?alt=media&token=46d5e637-3411-4188-bc44-5809b5bfb5f4', 27 | "SST":'https://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2FSST-2.zip?alt=media&token=aabc5f6b-e466-44a2-b9b4-cf6337f84ac8', 28 | "MRPC":'https://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2Fmrpc_dev_ids.tsv?alt=media&token=ec5c0836-31d5-48f4-b431-7480817f1adc', 29 | "QQP":'https://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2FQQP.zip?alt=media&token=700c6acf-160d-4d89-81d1-de4191d02cb5', 30 | "STS":'https://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2FSTS-B.zip?alt=media&token=bddb94a7-8706-4e0d-a694-1109e12273b5', 31 | "MNLI":'https://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2FMNLI.zip?alt=media&token=50329ea1-e339-40e2-809c-10c40afff3ce', 32 | "SNLI":'https://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2FSNLI.zip?alt=media&token=4afcfbb2-ff0c-4b2d-a09a-dbf07926f4df', 33 | "QNLI": 'https://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2FQNLIv2.zip?alt=media&token=6fdcf570-0fc5-4631-8456-9505272d1601', 34 | "RTE":'https://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2FRTE.zip?alt=media&token=5efa7e85-a0bb-4f19-8ea2-9e1840f077fb', 35 | "WNLI":'https://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2FWNLI.zip?alt=media&token=068ad0a0-ded7-4bd7-99a5-5e00222e0faf', 36 | "diagnostic":'https://storage.googleapis.com/mtl-sentence-representations.appspot.com/tsvsWithoutLabels%2FAX.tsv?GoogleAccessId=firebase-adminsdk-0khhl@mtl-sentence-representations.iam.gserviceaccount.com&Expires=2498860800&Signature=DuQ2CSPt2Yfre0C%2BiISrVYrIFaZH1Lc7hBVZDD4ZyR7fZYOMNOUGpi8QxBmTNOrNPjR3z1cggo7WXFfrgECP6FBJSsURv8Ybrue8Ypt%2FTPxbuJ0Xc2FhDi%2BarnecCBFO77RSbfuz%2Bs95hRrYhTnByqu3U%2FYZPaj3tZt5QdfpH2IUROY8LiBXoXS46LE%2FgOQc%2FKN%2BA9SoscRDYsnxHfG0IjXGwHN%2Bf88q6hOmAxeNPx6moDulUF6XMUAaXCSFU%2BnRO2RDL9CapWxj%2BDl7syNyHhB7987hZ80B%2FwFkQ3MEs8auvt5XW1%2Bd4aCU7ytgM69r8JDCwibfhZxpaa4gd50QXQ%3D%3D'} 37 | 38 | MRPC_TRAIN = 'https://dl.fbaipublicfiles.com/senteval/senteval_data/msr_paraphrase_train.txt' 39 | MRPC_TEST = 'https://dl.fbaipublicfiles.com/senteval/senteval_data/msr_paraphrase_test.txt' 40 | 41 | def download_and_extract(task, data_dir): 42 | print("Downloading and extracting %s..." % task) 43 | data_file = "%s.zip" % task 44 | urllib.request.urlretrieve(TASK2PATH[task], data_file) 45 | with zipfile.ZipFile(data_file) as zip_ref: 46 | zip_ref.extractall(data_dir) 47 | os.remove(data_file) 48 | print("\tCompleted!") 49 | 50 | def format_mrpc(data_dir, path_to_data): 51 | print("Processing MRPC...") 52 | mrpc_dir = os.path.join(data_dir, "MRPC") 53 | if not os.path.isdir(mrpc_dir): 54 | os.mkdir(mrpc_dir) 55 | if path_to_data: 56 | mrpc_train_file = os.path.join(path_to_data, "msr_paraphrase_train.txt") 57 | mrpc_test_file = os.path.join(path_to_data, "msr_paraphrase_test.txt") 58 | else: 59 | print("Local MRPC data not specified, downloading data from %s" % MRPC_TRAIN) 60 | mrpc_train_file = os.path.join(mrpc_dir, "msr_paraphrase_train.txt") 61 | mrpc_test_file = os.path.join(mrpc_dir, "msr_paraphrase_test.txt") 62 | urllib.request.urlretrieve(MRPC_TRAIN, mrpc_train_file) 63 | urllib.request.urlretrieve(MRPC_TEST, mrpc_test_file) 64 | assert os.path.isfile(mrpc_train_file), "Train data not found at %s" % mrpc_train_file 65 | assert os.path.isfile(mrpc_test_file), "Test data not found at %s" % mrpc_test_file 66 | urllib.request.urlretrieve(TASK2PATH["MRPC"], os.path.join(mrpc_dir, "dev_ids.tsv")) 67 | 68 | dev_ids = [] 69 | with open(os.path.join(mrpc_dir, "dev_ids.tsv"), encoding="utf8") as ids_fh: 70 | for row in ids_fh: 71 | dev_ids.append(row.strip().split('\t')) 72 | 73 | with open(mrpc_train_file, encoding="utf8") as data_fh, \ 74 | open(os.path.join(mrpc_dir, "train.tsv"), 'w', encoding="utf8") as train_fh, \ 75 | open(os.path.join(mrpc_dir, "dev.tsv"), 'w', encoding="utf8") as dev_fh: 76 | header = data_fh.readline() 77 | train_fh.write(header) 78 | dev_fh.write(header) 79 | for row in data_fh: 80 | label, id1, id2, s1, s2 = row.strip().split('\t') 81 | if [id1, id2] in dev_ids: 82 | dev_fh.write("%s\t%s\t%s\t%s\t%s\n" % (label, id1, id2, s1, s2)) 83 | else: 84 | train_fh.write("%s\t%s\t%s\t%s\t%s\n" % (label, id1, id2, s1, s2)) 85 | 86 | with open(mrpc_test_file, encoding="utf8") as data_fh, \ 87 | open(os.path.join(mrpc_dir, "test.tsv"), 'w', encoding="utf8") as test_fh: 88 | header = data_fh.readline() 89 | test_fh.write("index\t#1 ID\t#2 ID\t#1 String\t#2 String\n") 90 | for idx, row in enumerate(data_fh): 91 | label, id1, id2, s1, s2 = row.strip().split('\t') 92 | test_fh.write("%d\t%s\t%s\t%s\t%s\n" % (idx, id1, id2, s1, s2)) 93 | print("\tCompleted!") 94 | 95 | def download_diagnostic(data_dir): 96 | print("Downloading and extracting diagnostic...") 97 | if not os.path.isdir(os.path.join(data_dir, "diagnostic")): 98 | os.mkdir(os.path.join(data_dir, "diagnostic")) 99 | data_file = os.path.join(data_dir, "diagnostic", "diagnostic.tsv") 100 | urllib.request.urlretrieve(TASK2PATH["diagnostic"], data_file) 101 | print("\tCompleted!") 102 | return 103 | 104 | def get_tasks(task_names): 105 | task_names = task_names.split(',') 106 | if "all" in task_names: 107 | tasks = TASKS 108 | else: 109 | tasks = [] 110 | for task_name in task_names: 111 | assert task_name in TASKS, "Task %s not found!" % task_name 112 | tasks.append(task_name) 113 | return tasks 114 | 115 | def main(arguments): 116 | parser = argparse.ArgumentParser() 117 | parser.add_argument('--data_dir', help='directory to save data to', type=str, default='glue_data') 118 | parser.add_argument('--tasks', help='tasks to download data for as a comma separated string', 119 | type=str, default='all') 120 | parser.add_argument('--path_to_mrpc', help='path to directory containing extracted MRPC data, msr_paraphrase_train.txt and msr_paraphrase_text.txt', 121 | type=str, default='') 122 | args = parser.parse_args(arguments) 123 | 124 | if not os.path.isdir(args.data_dir): 125 | os.mkdir(args.data_dir) 126 | tasks = get_tasks(args.tasks) 127 | 128 | for task in tasks: 129 | if task == 'MRPC': 130 | format_mrpc(args.data_dir, args.path_to_mrpc) 131 | elif task == 'diagnostic': 132 | download_diagnostic(args.data_dir) 133 | else: 134 | download_and_extract(task, args.data_dir) 135 | 136 | 137 | if __name__ == '__main__': 138 | sys.exit(main(sys.argv[1:])) -------------------------------------------------------------------------------- /models/lstm_attn.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | from torch.autograd import Variable 4 | from torch.nn import functional as F 5 | 6 | class LSTM_Attn(torch.nn.Module): 7 | def __init__(self, batch_size, output_size, hidden_size, vocab_size, embedding_length, weights): 8 | super(LSTM_Attn, self).__init__() 9 | 10 | """ 11 | Arguments 12 | --------- 13 | batch_size : Size of the batch which is same as the batch_size of the data returned by the TorchText BucketIterator 14 | output_size : 2 = (pos, neg) 15 | hidden_sie : Size of the hidden_state of the LSTM 16 | vocab_size : Size of the vocabulary containing unique words 17 | embedding_length : Embeddding dimension of GloVe word embeddings 18 | weights : Pre-trained GloVe word_embeddings which we will use to create our word_embedding look-up table 19 | 20 | -------- 21 | """ 22 | self.batch_size = batch_size 23 | self.output_size = output_size 24 | self.hidden_size = hidden_size 25 | self.vocab_size = vocab_size 26 | self.embedding_length = embedding_length 27 | 28 | self.word_embeddings = nn.Embedding(vocab_size, embedding_length) 29 | self.word_embeddings.weights = nn.Parameter(weights, requires_grad=False) 30 | self.lstm = nn.LSTM(embedding_length, hidden_size) 31 | self.classify = nn.Linear(hidden_size, output_size) 32 | 33 | # self.attn_fc_layer = nn.Linear() 34 | 35 | def attention_net(self, lstm_output, final_state): 36 | 37 | """ 38 | Arguments 39 | --------- 40 | 41 | lstm_output : Final output of the LSTM which contains hidden layer outputs for each sequence. 42 | final_state : Final time-step hidden state (h_n) of the LSTM 43 | --------- 44 | 45 | Returns : It performs attention mechanism by first computing weights for each of the sequence present in lstm_output and and then finally computing the 46 | new hidden state. 47 | 48 | Tensor Size : 49 | hidden.size() = (batch_size, hidden_size) 50 | attn_weights.size() = (batch_size, num_seq) 51 | soft_attn_weights.size() = (batch_size, num_seq) 52 | new_hidden_state.size() = (batch_size, hidden_size) 53 | """ 54 | 55 | hidden = final_state.squeeze(0) 56 | attn_weights = torch.bmm(lstm_output, hidden.unsqueeze(2)).squeeze(2) 57 | soft_attn_weights = F.softmax(attn_weights, 1) 58 | new_hidden_state = torch.bmm(lstm_output.transpose(1, 2), soft_attn_weights.unsqueeze(2)).squeeze(2) 59 | 60 | return new_hidden_state 61 | 62 | def forward(self, input_sentences): 63 | 64 | """ 65 | Parameters 66 | ---------- 67 | input_sentence: input_sentence of shape = (batch_size, num_sequences) 68 | batch_size : default = None. Used only for prediction on a single sentence after training (batch_size = 1) 69 | 70 | Returns 71 | ------- 72 | Output of the linear layer containing logits for pos & neg class which receives its input as the new_hidden_state which is basically the output of the Attention network. 73 | final_output.shape = (batch_size, output_size) 74 | 75 | """ 76 | input = self.word_embeddings(input_sentences) 77 | num_ins = input.size(0) 78 | input = input.permute(1, 0, 2) 79 | 80 | if torch.cuda.is_available(): 81 | h_0 = Variable(torch.zeros(1, num_ins, self.hidden_size)).cuda() 82 | c_0 = Variable(torch.zeros(1, num_ins, self.hidden_size)).cuda() 83 | else: 84 | h_0 = Variable(torch.zeros(1, num_ins, self.hidden_size)) 85 | c_0 = Variable(torch.zeros(1, num_ins, self.hidden_size)) 86 | 87 | output, (final_hidden_state, final_cell_state) = self.lstm(input, ( 88 | h_0, c_0)) # final_hidden_state.size() = (1, batch_size, hidden_size) 89 | output = output.permute(1, 0, 2) # output.size() = (batch_size, num_seq, hidden_size) 90 | 91 | attn_output = self.attention_net(output, final_hidden_state) 92 | logits = self.classify(attn_output) 93 | 94 | return logits -------------------------------------------------------------------------------- /noise_plot.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "** The Plot script for all optimizers and comparisons **" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 151, 13 | "metadata": {}, 14 | "outputs": [], 15 | "source": [ 16 | "import numpy as np\n", 17 | "import matplotlib.mlab as mlab\n", 18 | "import matplotlib.pyplot as plt" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 152, 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "import pickle\n", 28 | "def read_pickle(path):\n", 29 | " with open(path, \"rb\") as f:\n", 30 | " d = pickle.load(f)\n", 31 | " return d" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": 370, 37 | "metadata": { 38 | "scrolled": false 39 | }, 40 | "outputs": [ 41 | { 42 | "name": "stdout", 43 | "output_type": "stream", 44 | "text": [ 45 | "20\n", 46 | "1\n" 47 | ] 48 | }, 49 | { 50 | "data": { 51 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZxcVZ338c8XAiIBWZJAFAkBBgiKbPYAEZAAiriz+qgPq2BUYCIIjyDqsMyogAsKyKYCsozKgOCjMIAssorYAVT2NewBQth3yG/+OKexKGq71VVdN13f9+tVr9t17rnn/qo76V+fe889RxGBmZlZ2SzQ6wDMzMxqcYIyM7NScoIyM7NScoIyM7NScoIyM7NScoIyM7NScoIyM7NSKkWCknS4pEskPSDpRUlzJd0g6SBJ4wq0M0tS1HnN7uZnMDOzzlIZHtSV9ApwPXAL8BgwFtgAGAAeBjaIiAdaaGcWsCTw4xq7n4uIH3QqZjMz666yJKhFIuKlGuXfAQ4EjouIPVpoZxZAREzudIxmZjaySnGJr1Zyys7M21VGKhYzMyuHMb0OoIlP5u3fCxzzNkk7AJOA5/OxV0TE650OzszMuqcUl/iGSNoPWAxYgnT/aSNSgvlQRDzewvGzgBVq7LoX2DUiLu9ctGZm1k1lS1CzgWUrii4AdomIR1s8/iDgSuBm4FlgJWAvYDrwEjA1Iv5W59jpuR5jx459/5QpU9r9GGZmVmXmzJlzImJCkWNKlaCGSFoW+ABwGLA48ImIuH4Y7f0A2Bc4NyK2blZ/YGAgBgcH2z2dmZlVkTQzIgaKHFOKQRLVIuLRiDgH2AIYB5w6zCaPz9sPDrMdMzMbIaVMUEMi4j7Ss1HvlTR+GE09lrdjhx+VmZmNhFInqOxdeTucUXhT8/aeYcZiZmYjpOcJStIUSRNrlC+QH9RdBrgmIp7M5QvlY1auqv9eSUvXaGcF4Jj89vTOfwIzM+uGMjwHtSXwfUlXAHcDT5BG8m1CGoU3G/hiRf3lgFuB+4DJFeXbAwdIuow0rPxZYGXg48AiwPmApzoyM5tPlCFBXQycCGwIrEWaS+954A7gNOCoiJjbQjuXAasB65Au6Y0FngKuyu2cFmUcsmhmZjX1PEFFxE3AngXqzwJUo/xywA/impmNEj2/B2VmZlaLE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZVSzxOUpHGSdpd0jqS7JL0o6WlJV0naTVKhGCV9XNJFkh7Mbd0j6b8lTe3WZzAzs84b0+sAgO2B44BHgMuA+4FlgW2AnwMflbR9RESzhiQdDnwdeAI4F5gD/AvwaWBbSTtFxOld+RRmZtZRZUhQdwCfAs6LiHlDhZIOBK4DtiUlq7MbNSJpIrAf8CiwZkQ8VrFvU+BS4FDACcrMbD7Q80t8EXFpRPy+Mjnl8tnA8fnttBaaWoH0ef5SmZxyW5cBzwIThh+xmZmNhJ4nqCZezdvXWqh7J/AKsJ6k8ZU7JH0QWBy4uLPhmZlZt5ThEl9NksYAO+W3FzSrHxFzJe0P/Ai4RdK5pHtRK5MuIf4R+FKXwjUzsw4rbYICDgPWAM6PiAtbOSAifixpFnAS8MWKXXcBp1Rf+qskaTowHWDSpEntxmxmZh1Sykt8kmYA+wK3ATsWOO7rwFnAKaSe01jg/cA9wBmSjqh3bEScGBEDETEwYYJvVZmZ9VrpEpSkPYGfALcAm0bE3BaPmwYcDvz/iPhaRNwTES9ExPXA1sBDwL6SVupS6GZm1kGlSlCS9gaOAW4iJafZBQ7/RN5eVr0jIl4gDVlfAFhnuHGamVn3lSZB5QEORwI3kpJT3ftFdbwtb+tdnxsqf6WN8MzMbISVIkFJ+jZpUMRMYPOImNOg7kKSpkhauWrXlXk7XdJyVcd8FNgQeAm4pnORm5lZt/R8FJ+knUkzPLxOSjIzJFVXmxURp+SvlwNuBe4DJlfUOYv0nNOHgFslnQPMBlYnXf4TcEBEPNGVD2JmZh1VKEFJWpzUE5kEjAdeBB4DboyIm9uMYcW8XRDYu06dy0kj8+qKiHmSPgbsCXyWNDBiUWAucD5wVERc1GaMZmY2wtRsDlZJbwc+B+wGrMc/LwsOdXOGGphDmi/vuIj4R+dDHTkDAwMxODjY6zDMzEYNSTMjYqDIMXV7UHkmhxnAN4GlSPdvrgX+Srp0Nhd4OzAOmAJsAHwZ+JKki4F9I+KmNj6HmZlZw0t8t5Euv10A/BL4XUS83KgxSasCu5CmKLpB0m4RcWqHYjUzsz7SKEHdAmwbEX9rtbGIuAM4UNLBwB6kHpaZmVlhdRNURHyq3UYj4hXgx+0eb2ZmVornoMzMzKoN6zkoSSsC78lvb4mIe4cfkpmZWZsJKj8P9XNgu6EiICSdCUyPiGc7FJ+ZmfWpdntQxwBbAAeRpidahLQo4M7AC6RnpszMzNrWMEFJWjTPBF5ta+ArEXFGRdk5khbN+5ygzMxsWJoNkviHpE1rlI8Bal3Ge5YSzO9nZmbzv2bJ5FrgYkk/A/5fxb2lS4BjJI0FbiAtdfFJ0gO6f+hWsGZm1j8a9qAi4v8CnwY+Dtwsacu8aw/S3HtnADcD15NmJP8bsFfXojUzs77R9HJcRPxB0hWkxQTPl3QqsHdErCvpw6R5+ATcHBGXdDdcMzPrFy3dL4qIZ4DdJP0aOBG4RdIeEXEu8MduBmhmZv2p0EwSEfFHYA3gHOBsSb+WNK4rkZmZWV9rKUFJGi/p/ZLGR8TzEbEnsCmwDmn12s92NUozM+s7DROUpMUk/QZ4FLgOeFTSmZIWi4grgLWA04DTJZ0raWL3QzYzs37QrAf1PWB70npQe5KWXd8OOAwgIl6KiH1Jy8CvQro3tUu3gjUzs/7RLEF9GjgrIr4QEcdHxG6kZd0/XVkpIv4CrA0cB5zQlUjNzKyvNEtQY4EHq8oeABatrhgRr0bEN4H1OhSbmZn1sWYJ6lpgR0kbSlpY0lRgh1xeU5EVeM3MzOpplqC+CrwIXJG3VwEvAXt3OS4zM+tzDR/UjYi7JE0hzbM3Cbgf+ENEPD8SwZmZWf9qZaqjF4DfjEAsZmZmbyg0k4SZmdlIqZugJO0raZF2G5a0rqSPtnu8mZn1t0Y9qO8Cd0vaX9JyrTSm5COSzgH+SpppwszMrLBG96DeB/yINJvEf0q6hjSKbxB4BHgSWAQYR1pyYwNgc2Ai8ARpXaimD+3myWa3Jq059T5gOeAV4B/AycDJETGvlQ8jaTtgE9JDw2sBiwNnRMQOrRxvZmblUTdBRcQdwCckfYA0zdG2wMZA1KiuvL0dOJyUVGotCV/L9qQZKB4BLiONFFwW2Ab4OfBRSdtHRK3zVvsWKTE9R3rAeEqLMZiZWcm0MorvGuAaSV8GPghsRBpyPo70bNRjwN+BP0XEzW3EcAfwKeC8yp6SpANJE9RuS0pWZ7fQ1j6kxHQXqSd1WRvxmJlZCbS0YCFA7hGdl18dExGX1imfLel44DvANFpIUBHxRkKS1KiqmZmVXNmHmb+at6/1NIr5kPOzmc3vSpugJI0BdspvL+hlLGZmNvJKm6BIa06tAZwfERd2+2SSpksalDT4+OOPd/t0ZmbWRCkTlKQZwL7AbcCOI3HOiDgxIgYiYmDChAkjcUozM2ugdAlK0p7AT4BbgE0jYm6PQzIzsx4oVYKStDdwDHATKTnN7nFIZmbWI6VJUJL2B44EbiQlp8d6HJKZmfVQywlK0taSFuxGEJK+TRoUMRPYPCLmNKi7kKQpklbuRixmZlYOLT+oS3pQ9mFJJwE/j4j7OxGApJ2BQ4HXgSuBGTUesp0VEafkr5cDbgXuAyZXtbUVsFV+OzFvp0oaOnZOROzXibjNzKy7iiSoY4HPk+a7+4akC0iTwZ7X4jx59ayYtwtSfyn5y4FTWmhrbWDnqrKV8gtSUnOCMjObD6hIbsnrQ30W+BKwPmni2IdIk7r+IiIe6kaQI21gYCAGBwd7HUZDElT/6CrLau1vt66Z2XBJmhkRA0WOKTRIIiJeiohTImIqsCZpFvLFgIOBeyWdI2nLIm2amZnV0vYovoi4KSL2At4F7Ao8Sp6VXNK9kvaTNLZDcZqZWZ8Z1jDznIB2AmaQBi8I+BtpKY4jgNskrT3cIM3MrP+0laAkrZOXwngYOB5YlXQfat2IWJfUqzoAGA8c1aFYzcysj7Q8ik/SosDnSAMk3k/qLd1KSlC/jIhnhupGxHPAEZKWB3braMRmZtYXigwzfxhYnPS80tnAsRHxpybHPAQs0l5oNj/yiEAz65QiCepZ4IfAzwrMkXcs8KvCUZmZWd8rkqBWiIh5RRrPl/2eaVrRzMysSpFBEhdL2qlRBUk7SLp0mDGZmZkVSlDTqJr7roYVgE3aDcbMzGxIp5fbeDvwWofbNDOzPlTkHhSkuffeQmn68UnAx4AHhhuUmZlZwx6UpHmSXpf0ei46eOh95YvUa7qHNJv4r7scs5mZ9YFmPagr+Gev6YPA/cCsGvVeB54ALiHNKGFmZjYsDRNUREwb+lrSPODkiDi020GZmZkVuQe1IvBUtwKx4pqtCdVKeaO2wbNCmFnvtJygIuK+bgZiZmZWqW6CkvTvpPtPP42Iufl9KyIi/qMj0ZmZWd9q1IM6mJSgfgPMze9bEYATlJmZDUujBLVp3t5f9d7MzKzr6iaoiLi80XubP3j5CzObX3V6qiMzM7OOaDlBSZos6WOSxlaUjZF0iKS/SbpG0tbdCdPMzPpNkeegDgI+BSxbUfYt4NsV78+UtHFEXNuJ4MzMrH8VucQ3FbgkIl4DkLQAsAdwG2mi2PWA54F9Oh2kmZn1nyIJalmg8mHdtYHxpOekHoyIQeB3wL92MD7rgqFZIoa2RY4xMxspRRLUQrx5uY0N8/vKFXQfBN7ZgbjMzKzPFUlQDwJrVrz/GDAnIm6tKFsGeKZoEJK2k3S0pCslPSMpJJ1etJ0a7e6Y2wpJuw+3PTMzGzlFBkn8AdhH0g+Al4APAydX1ZnCmy8DtupbwFrAc6REOKWNNt5E0vLA0bnNxYbbnpmZjawiPagjgHuBrwEHAo+QRvYBIGkF4AOkNaSK2gdYFXgH8JU2jn+TvMLvyaQ1qo4fbntmZjbyisxm/pik9wGb56LLI+LZiiqLkZLXhUWDiIjLhr5WZ+7GzwA2A6blbV9pZxBEp8/v2SvMbLiKXOIjIl4kXeqrte9m4OZOBDUcklYHDgN+EhFXSOq7BGVmNhqMqqmOJI0BTiNNcHtgj8MxM7NhKNSDkrQ08AXSQ7lLAQvWqBYRsXmN8pHw78A6wEa5t9cySdOB6QCTJk3qQmhmZlZEywlK0hTgT8AEoNHdjZ7cfZC0HqnX9MOI+HPR4yPiROBEgIGBAd9BMTPrsSKX+H5Aes7pcGAlYKGIWKDGq1avqqsqLu3dwZvnBhxVpNoDH+qVt9N+L441M6ulyCW+jYHzIqKM93YWIw1TB3ipzkjAn0n6GWnwxN4jFpmZmbWlSIIScEu3Ahmml4Ff1Nm3Lum+1FXA7UDhy39mZjbyiiSomcBq3QqkVZIWAlYGXo2Iu+GN4e81pzKSdDApQf0yIn4+UnGamdnwFElQhwIXSpoWEX/qZBCStgK2ym8n5u1USafkr+dExH756+WAW0lTKk3uZBxmZlYeRRLU8qTlNC6S9CtSj+qpWhUj4tSCcawN7FxVtlJ+QUpG+2HD0o0ZHjxrhJl1i6LF3y6S5pGGkFeOQKg+WKTnoEZ8JF8nDQwMxODgYK/DeIuhsR8Rtb+up1H9ofeV2+pzDb2v9U+l+tjq48zMACTNjIiBIscU6UHtWjAeMzOzthWZLPaX3QzEzMys0qiai8/MzEaPQnPxAUiaAGwLrA6MjYjdK8pXBP5RdB48a1/RGRzq1e/GTBC+H2Vmw1F0stjdgKOARcgDIvjn80fLkh6CnU79h2bNzMxa0vIlPkkfJk2megewNXBc5f6IuIm0HtRWbz3azMysmCI9qP1Jy7xvEhHPSFqnRp2/A1M7EpmZmfW1IoMkBoA/RMQzDeo8yD9ngjAzM2tbkQS1MPB8kzpLAq+3H47NL7q1vIeX7TCzIUUS1Czg/U3qrE+aMdzMzGxYiiSo3wEbS9q+1k5JuwJrAmd3IjAzM+tvRQZJHAF8FviVpO2AJQAk7UVazHAb4E7g6E4HaWZm/afIVEdPStoEOBWo7EUdlbdXAp+PiGb3qczMzJoq9KBuRNwPTJO0Jmk4+TjgaeDaiJjZhfjMzKxPFZ7qCCAi/k565sl6oFPTG7V7fLP3tY73dEdmVlQ7c/GtAEwgTXP0eO5VmZmZdVRLo/gkjZf0I0mPAPcAfwGuA+6V9LCk70taupuBmplZf2maoCStAgwCXyVNCPs68BjweP56IvA1YFDSSvXaMTMzK6JhgpK0AHAGMAm4HPgQsFhEvDMiJgKLA1sAVwCTgdO7Gq2ZmfWNZj2oLUhz8J0JbB4Rl0bEK0M7I+LliLgY2Aw4C1g/z3puwzQ08EAa2el/ig6AGE7bZmaNNEtQ2wIvA/8WUX8cVt63F/AqsF3nwjMzs37VLEGtC1wdEY83aygiHgOuyseYmZkNS7MEtTxpEcJW3Qys0H44ZmZmSbME9Q7gqQLtPUUaOGFmZjYszRLUwhRb32lePsY6rJcDJTrRRuX7el93KxYzmz+18qCuJ6kxM7MR18pURwdLOrjbgZiZmVVqpQelgq+2SHq3pJPy1EkvS5ol6ceSlmrx+GmSooXX8u3GaGZmI6dhDyoiiqy42zZJKwPXAMuQVu69DViPNL3SlpI2jIgnmjQzCzikzr73kRZUvDkiHuhI0GZm1lVtLbfRBceSktOMiHhjRV5JPwL2Ab4DfLlRAxExCzi41j5Jv8pfntiBWEeVIoMQerlsRmWcXrrDrD+MSA+pkTzB7BakHtBPq3YfBDwP7ChpbJvtjwO2Bl4ETms/UjMzG0k9T1CkefwALoqIeZU7IuJZ4GpgUWCDNtvfBXgb8N8R8WS7QZqZ2cgqQ4JaLW/vqLP/zrxdtc32d8/bE9o83szMeqAMCWqJvH26zv6h8iWLNixpE2AKaXDENU3qTpc0KGnw8cebTj1oZmZdVoYE1czQ7fF2bo1Pz9umvaeIODEiBiJiYMKECW2cqj80mgWilQEXlcuIdCqOdmNo9/hOHmdm9ZUhQQ31kJaos/8dVfVakpeg3xYPjjAzmy+VIUHdnrf17jGtkrf17lHVszNpcMSZEVFkwlszMyuBMiSoy/J2i7zE/BskLQ5sSOoFXVuw3S/mrZ99MjObD/U8QUXE3cBFwGRgz6rdhwBjgVMj4vmhQklTJE2p16akjYHVgZuaDY4wM7NyKstMEnuQpjo6StLmwK3A+sCmpEt736yqf2ve1rs1PTQ4Yr7uPfXbjffhDFAoOrvEcGfFGDq+l7NrmI12Pe9BwRu9qAHgFFJi2hdYGTgKmNrCPHxvyJPLbocHR5iZzdfK0oMiT+K6a4t16/6tnWeLeHun4jIzs94oRQ/KzMysmhOUmZmVkhOUlY7U2owVjWaTqNdGq7NfNKpXlsErZYnDrFucoMzMrJScoMzMrJScoMzMrJScoMzMrJScoErIN7+Tkfo+NDtPq3E0W8aj1n7/rM3qc4IyM7NScoIyM7NScoIyM7NScoIyM7NScoIqkX69YV5vAEEv2mhU1k77rc5UUSSOds9pNr9xgjIzs1JygjIzs1JygjIzs1JygjIzs1JygiqJZktIWG2tzALRyve01cEQrc460Up7rS790aj9ov9eOvHvq+j306xdTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlA95JFOvdWp7/9wR9JVjjSstW1lxF69Op0Y5dcshuGcw/8HrBEnKDMzKyUnKDMzKyUnKDMzK6XSJChJ75Z0kqSHJb0saZakH0taqmA7S+fjZuV2Hs7tvrtbsZuZWeeN6XUAAJJWBq4BlgF+B9wGrAd8FdhS0oYR8UQL7YzL7awKXAr8GpgC7Ap8XNLUiLinO5+iPb5JPHJ6MS3QcNoczhRJjepIEPHmsoi31q2uU6uNWu/rnauyfivtVcZW6/ha8VaX1zq2UVvV7dWLu9HxjdqsFWc3tBNjGZWlB3UsKTnNiIitIuKAiNgMOBJYDfhOi+18l5ScjoyIzXM7W5ES3TL5PGZmNh9Q9DjNSloJuBuYBawcEfMq9i0OPAIIWCYinm/QzljgcWAe8M6IeLZi3wL5HJPzORr2ogYGBmJwcLDdj9Qy957mb9U9jOryevt7FU+jslq9n2bxVx9T632tXkz196ZeL6c6Dvegip2vbD0oSTMjYqDIMWXoQW2WtxdVJieAnGSuBhYFNmjSzlTg7cDVlckptzMPuCi/3XTYEZuZWdeVIUGtlrd31Nl/Z96uOkLtmJlZCZRhkMQSeft0nf1D5Ut2sx1J04Hp+e3Lkm5qcj4rv/HAnG413mwQw0hfwm11UEWtsnZmiGjWRqO1q2p93SD+8cCcovE0arudtbeKHt+s3W4r4S2E1ZpXebMyJKhmhr7Nw72i2rCdiDgROBFA0mDRa6VWPv45jg7+OY4Okgrf2C/DJb6hns0Sdfa/o6pet9sxM7MSKEOCuj1v690bWiVv691b6nQ7ZmZWAmVIUJfl7RZ5OPgb8jDzDYEXgWubtHNtrrdhPq6ynQWALarO18iJLdSx8vPPcXTwz3F0KPxz7HmCioi7SUPAJwN7Vu0+BBgLnFr5DJSkKZKmVLXzHHBarn9wVTt75fYvbGUmiXw/yuZz/jmODv45jg7t/Bx7/qAu1Jzq6FZgfdIzS3cAH6ic6khSAESEqtqpnuroOmB14NPAY7mdu7v9eczMbPhKkaAAJC0PHApsCYwjzSBxLnBIRMytqlszQeV9SwMHAVsB7wSeAP4H+PeIeLCbn8HMzDqnNAnKzMysUs/vQZVFp5b7sN6RtJ2koyVdKekZSSHp9F7HZa2TNE7S7pLOkXSXpBclPS3pKkm7VQ+ksvKSdLikSyQ9kH+OcyXdIOmgfDumeRvuQTVc7mNT0vD1lpb7sN6SdCOwFvAc8CBpqZUzImKHngZmLZP0ZeA40iX+y4D7gWWBbUjPOJ4NbB/+xVV6kl4BrgduIY0BGEuaU3UAeBjYICIeaNiGf84g6ULSMPQZEXF0RfmPgH2AEyLiy72Kz1ojaVNSYroL2IT0C84Jaj4iaTPSL7LzqlY2mEga9LQ8sF1EnN2jEK1FkhaJiJdqlH8HOBA4LiL2aNRG33eX83IfW5CW+/hp1e6DgOeBHfNyHlZiEXFZRNzpv67nXxFxaUT8vsbKBrOB4/PbaSMemBVWKzllZ+btKnX2v6HvExSdW+7DzLrr1bx9radR2HB9Mm//3qzi/DBZbLe1skzHFqRnqy4ZkYjM7E0kjQF2ym8v6GUsVoyk/YDFSPcQB4CNSMnpsGbHOkF1brkPM+uew4A1gPMj4sJeB2OF7Eca6DLkAmCXiHi82YG+xNdcp5b7MLM2SJoB7EsaXbtjj8OxgiJiYp5UYSJpNOZKwA2S1m12rBOUl+kwKy1JewI/IQ1V3rR6Vhmbf0TEoxFxDumWyTjg1GbHOEF5mQ6zUpK0N3AMcBMpOc3ucUjWARFxH+kPjvdKGt+orhNU55b7MLMOkbQ/cCRwIyk5PdbjkKyz3pW3rzeq1PcJqp3lPsyseyR9mzQoYiaweUTM6XFIVlBeEmlijfIF8oO6ywDXRMSTDdvxM43Fl/uwcpK0FWkWe0g3ZD8C3ANcmcvmRMR+vYjNWiNpZ+AU0l/WR1P73u+siDhlBMOygvLl2e8DVwB3k1aVWJY0w8tKwGzSHx+3NGzHCSopstyHlZOkg0mzf9RzX0RMHplorB0t/AwBLo+Iad2PxtolaQ3gK6RbJO8mPabzPOkP/vOAo1r5veoEZWZmpdT396DMzKycnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMOkjSLEmzeh2H2WjgBGWjnqTIr/skLVKnzqxcx4t4mpWEE5T1k0nA3l0+x+b5ZWbD5KmObNSTFMCTpFWRxwArV8+QnS/LrQAsFBGvjXiQZvYW7kFZv3gB+A/SCsnNJiN9E0mfkXSFpKclvSjpH5K+IeltNeq+5R6UpIUlzZB0vaQnJb2Q6/1O0odqtDFF0imSHpD0sqRHJf2XpNUKxDwtX7I8WNLaks6T9FQ+9+WSPlDnuCUkfU/S7ZJeyvFeWCfOynOsl88xN5dNrvx+SFpM0pH5M70o6cY8+zySxkg6UNKd+Zx3S9qr1c9qo5d7UDbq5R7UQ8CKpKVUJgFrRMQdFXVmUaMHJem7wDeAOcBZwHPAR4H3ApcDH46IV6vaoXLWdEn/BXyOtDLspaQFMN8FbAT8tnIJEElbAr8FFgJ+D9xFmg16G+Bl0uJ917fwmaeRFuM8D9gM+DNwQ/7s2wKvAGtHxO0VxywJXA28B/hrjnU88BlgMeArEXFCjXNcBEwDrsrnGA8cGBEP5+/HQsD9wNLAH4GF8/djUdLy33uQlrf5n/wZtyctffPZiPhNs89qo1hE+OXXqH6RLu09mL/eLr//bVWdWbl8TEXZ1Fx2PzCxonwMKXkE6RdxdTuzKt4vAcwDBoEFa8Q2ruLrpUiXIucA76mq915Scry+xc88LccXwC5V+76Uy4+tKj8hl59A/uM1l69CWpfpZWBynXN8qU4cQ9/X3wNvqyjfOJfPJSXDJSv2rURKoDf0+t+OX719+RKf9ZWIOIvUm9ha0kZNqn8hb/8zImZXtPEasC8p8eze7JSASL/c59WIp3IhzJ1I6+YcFFULuUXEzcDPgHUkvafJOStdHW9d3O8k4DVgvaECSQsBO5CS4Dci4o1LKxFxJ3AUqeezU41z3BgVPas69o6IlyvavBK4l5SU94+Ipyr23ZjvB7sAAALsSURBVEPqyb1P0oJNP6GNWh5Sa/1oX9IKyj+UtEHlL+Mq6+btpdU7IuIOSQ8CK0pasvIXbFW9ZyT9HvgkcKOks0kr/P4lIl6oqj41b9fKC/dVWzVvVwcarkRaYbBGTK9KepSUHIZMIV1yuzpqLyR3KfAtYJ0a+65rEsNTEXF3jfKHSZddZ9bY9xCwIGll5IeatG+jlBOU9Z2I+LOks0iX+z4D1LvPsUTePlJn/yOkezpLADUTVPZ/gP2BzwOH5LKXcgz7RcSjuWxc3n6xyUdYrMn+SvXieo2UAIa08lkh9fCqza5RVqnWsu1DMRARtfYP3QdcqEnbNor5Ep/1qwOAV4HvSVq4Tp2hX5wT6+x/Z1W9miLixYg4OCJWJSW0HUgDCnYgDbyoPt9aEaEGr182+WztGM5n9Ugr6wonKOtL+ZLTsaRLTP9Wp9oNeTuteoekfyGNrru33uW9Oud9ICLOAD4C3AlsJGmo53Rt3m7cansddDtpKP7akpaqsX/TvG06gtCsU5ygrJ8dSroE9k1qXzY7KW+/JWnCUGG+cf8D0v+fXzQ6gaQJktavsWsssDjpUtYruezkHM9BktarPkDSAnlod8dFxCvAGaTvw6FV510ZmEHqcZ7WjfOb1eJ7UNa3ImJufs7piDr7r5F0BPB14KZ8z+h50nNQa5Au032/yWmWA66VdCup9/EA6WHhT5Aupx0VEc/m8z0haTvgnHzMJcDNpNF/k0iDKMYBNecT7IADSL23vST9K+kZp6HnoBYH9oqIe7t0brO3cIKyfncU6UHRybV2RsT+km4A9iINsV4IuJs0ou2HuefRyCzSzBXTSJfJxpOe/bmdlBB+XXW+SyStCexHugy4MamH9TBpJN3ZBT9fy3LCnkp6MHkb4Gukh4qvA74fERd169xmtXgmCTMzKyXfgzIzs1JygjIzs1JygjIzs1JygjIzs1JygjIzs1JygjIzs1JygjIzs1JygjIzs1JygjIzs1L6X5wDqTHsbWrXAAAAAElFTkSuQmCC\n", 52 | "text/plain": [ 53 | "
" 54 | ] 55 | }, 56 | "metadata": { 57 | "needs_background": "light" 58 | }, 59 | "output_type": "display_data" 60 | }, 61 | { 62 | "name": "stdout", 63 | "output_type": "stream", 64 | "text": [ 65 | "10\n" 66 | ] 67 | }, 68 | { 69 | "data": { 70 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZxcVZ338c8XAiIBWZJAFAmBDBAU2ewBIiBBFHFn9VEfVsGowEQQHlHUYZlRAXdANhWQZVQGBB+FAWSRVcQOoBL2JewBQljDTn7zxzkNRVHVVbe6qut21/f9etXrdt3l3F91Q/1ylnuOIgIzM7OyWaTbAZiZmdXiBGVmZqXkBGVmZqXkBGVmZqXkBGVmZqXkBGVmZqXkBGVmZqVUigQl6QhJl0i6X9LzkuZLukHSwZLGFShnjqSo85rbyc9gZmbtpTI8qCvpJeB64GbgUWAssDHQBzwEbBwR9zdRzhxgWeAnNQ4/GxE/aFfMZmbWWWVJUEtExAs19n8HOAg4LiL2aqKcOQARMbndMZqZ2fAqRRNfreSUnZm3qw9XLGZmVg5juh1AA5/I238UuOYtknYCJgEL8rVXRMSr7Q7OzMw6pxRNfAMkHQAsBSxD6n/alJRgPhgRjzVx/RxglRqH7gF2j4jL2xetmZl1UtkS1FxgxYpdFwC7RcQjTV5/MHAlMBt4BlgN2AeYAbwATIuIv9e5dkY+j7Fjx7536tSprX4MMzOrMmvWrHkRMaHINaVKUAMkrQi8DzgcWBr4eERcP4TyfgDsD5wbEds2Or+vry/6+/tbvZ2ZmVWRNCsi+opcU4pBEtUi4pGIOAfYChgHnDrEIo/P2/cPsRwzMxsmpUxQAyLiXtKzUe+WNH4IRT2at2OHHpWZmQ2HUieo7B15O5RReNPy9u4hxmJmZsOk6wlK0lRJE2vsXyQ/qLsCcE1EPJH3L5avmVJ1/rslLV+jnFWAY/Lb09v/CczMrBPK8BzU1sD3JV0B3AU8ThrJtzlpFN5c4AsV568E3ALcC0yu2L8j8HVJl5GGlT8DTAE+BiwBnA94qiMzsxGiDAnqYuBEYBNgXdJceguA24HTgKMiYn4T5VwGrAmsT2rSGws8CVyVyzktyjhk0czMaup6goqIm4C9C5w/B1CN/ZcDfhDXzGyU6HoflJmZWS1OUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpdT1CSxknaU9I5ku6U9LykpyRdJWkPSYVilPQxSRdJeiCXdbek/5Y0rVOfYaik119mZpaM6XYAwI7AccDDwGXAfcCKwHbAL4CPSNoxIqJRQZKOAL4GPA6cC8wD/gX4FLC9pF0i4vSOfAozM2urMiSo24FPAudFxMKBnZIOAq4Dticlq7MHK0TSROAA4BFgnYh4tOLYFsClwGGAE5SZ2QjQ9Sa+iLg0Iv5QmZzy/rnA8fnt9CaKWoX0ef5amZxyWZcBzwAThh6xmZkNh64nqAZezttXmjj3DuAlYENJ4ysPSHo/sDRwcXvDMzOzTilDE19NksYAu+S3FzQ6PyLmSzoQ+BFws6RzSX1RU0hNiH8CvtihcM3MrM1Km6CAw4G1gfMj4sJmLoiIn0iaA5wEfKHi0J3AKdVNf5UkzQBmAEyaNKnVmM3MrE1K2cQnaSawP3ArsHOB674GnAWcQqo5jQXeC9wNnCHpyHrXRsSJEdEXEX0TJrirysys20qXoCTtDfwUuBnYIiLmN3nddOAI4P9HxFcj4u6IeC4irge2BR4E9pe0WodCNzOzNipVgpK0L3AMcBMpOc0tcPnH8/ay6gMR8RxpyPoiwPpDjdPMzDqvNAkqD3D4MXAjKTnV7S+q4y15W699bmD/Sy2E13aePcLMbHClSFCSvk0aFDEL2DIi5g1y7mKSpkqaUnXoyrydIWmlqms+AmwCvABc077IzcysU7o+ik/SrqQZHl4lJZmZenO1Yk5EnJJ/Xgm4BbgXmFxxzlmk55w+CNwi6RxgLrAWqflPwNcj4vGOfBAzM2urQglK0tKkmsgkYDzwPPAocGNEzG4xhlXzdlFg3zrnXE4amVdXRCyU9FFgb+AzpIERSwLzgfOBoyLiohZjNDOzYaZGc7BKeivwWWAPYENebxYcqOYMFDCPNF/ecRHxz/aHOnz6+vqiv7+/o/eo1/fUeEpcM7ORR9KsiOgrck3dGlSeyWEm8E1gOVL/zbXA30hNZ/OBtwLjgKnAxsCXgC9KuhjYPyJuauFzmJmZDdrEdyup+e0C4FfA7yPixcEKk7QGsBtpiqIbJO0REae2KVYzM+shgyWom4HtI+LvzRYWEbcDB0k6BNiLVMMyMzMrrG6CiohPtlpoRLwE/KTV683MzErxHJSZmVm1IT0HJWlV4F357c0Rcc/QQzIzM2sxQeXnoX4B7DCwCwhJZwIzIuKZNsVnZmY9qtUa1DHAVsDBpOmJliAtCrgr8BzpmSkzM7OWDZqgJC2ZZwKvti3w5Yg4o2LfOZKWzMecoMzMbEgaDZL4p6QtauwfA9RqxnuGEszvZ2ZmI1+jZHItcLGknwP/r6Jv6RLgGEljgRtIS118gvSA7h87FayZmfWOQWtQEfF/gU8BHwNmS9o6H9qLNPfeGcBs4HrSjOR/B/bpWLRmZtYzGjbHRcQfJV1BWkzwfEmnAvtGxAaSPkSah0/A7Ii4pLPhmplZr2iqvygingb2kPQb4ETgZkl7RcS5wJ86GaCZmfWmQjNJRMSfgLWBc4CzJf1G0riORGZmZj2tqQQlabyk90oaHxELImJvYAtgfdLqtZ/paJRmZtZzBk1QkpaS9FvgEeA64BFJZ0paKiKuANYFTgNOl3SupImdD3nkkl5/mZnZ4BrVoL4H7EhaD2pv0rLrOwCHA0TECxGxP2kZ+NVJfVO7dSpYMzPrHY0S1KeAsyLi8xFxfETsQVrW/VOVJ0XEX4H1gOOAEzoSqZmZ9ZRGCWos8EDVvvuBJatPjIiXI+KbwIZtis3MzHpYowR1LbCzpE0kLS5pGrBT3l9TkRV4zczM6mmUoL4CPA9ckbdXAS8A+3Y4LjMz63GDPqgbEXdKmkqaZ28ScB/wx4hYMBzBmZlZ72pmqqPngN8OQyxmZmavKTSThJmZ2XCpm6Ak7S9piVYLlrSBpI+0en2v8sO8ZmbJYDWo7wJ3STpQ0krNFKbkw5LOAf5GmmnCzMyssMH6oN4D/Ig0m8R/SrqGNIqvH3gYeAJYAhhHWnJjY2BLYCLwOGldqIYP7ebJZrclrTn1HmAl4CXgn8DJwMkRsbCZDyNpB2Bz0kPD6wJLA2dExE7NXG9mZuVRN0FFxO3AxyW9jzTN0fbAZkDUOH2gQeo24AhSUqm1JHwtO5JmoHgYuIw0UnBFYDvgF8BHJO0YEbXuW+1bpMT0LOkB46lNxmBmZiXTzCi+a4BrJH0JeD+wKWnI+TjSs1GPAv8A/hwRs1uI4Xbgk8B5lTUlSQeRJqjdnpSszm6irP1IielOUk3qshbiMTOzEmhqwUKAXCM6L7/aJiIurbN/rqTjge8A02kiQUXEawlJXR5lUHn7pup+g5TR6vVmZiNZ2YeZv5y3r3Q1CjMzG3alTVCSxgC75LcXdDMWMzMbfqVNUKQ1p9YGzo+ICzt9M0kzJPVL6n/sscc6fTszM2uglAlK0kxgf+BWYOfhuGdEnBgRfRHRN2HChOG4pZmZDaJ0CUrS3sBPgZuBLSJifpdDMjOzLihVgpK0L3AMcBMpOc3tckhmZtYlpUlQkg4EfgzcSEpOj3Y5JDMz66KmE5SkbSUt2okgJH2bNChiFrBlRMwb5NzFJE2VNKUTsZiZWTk0/aAu6UHZhySdBPwiIu5rRwCSdgUOA14FrgRm1njIdk5EnJJ/Xgm4BbgXmFxV1jbANvntxLydJmng2nkRcUA74jYzs84qkqCOBT5Hmu/uG5IuIE0Ge16T8+TVs2reLkr9peQvB05poqz1gF2r9q2WX5CSmhOUmdkIoCK5Ja8P9Rngi8BGpIljHyRN6vrLiHiwE0EOt76+vujv7x9SGbWmOmp19iVPdWRmI52kWRHRV+SaQoMkIuKFiDglIqYB65BmIV8KOAS4R9I5krYuUqaZmVktLY/ii4ibImIf4B3A7sAj5FnJJd0j6QBJY9sUp5mZ9ZghDTPPCWgXYCZp8IKAv5OW4jgSuFXSekMN0szMek9LCUrS+nkpjIeA44E1SP1QG0TEBqRa1deB8cBRbYrVzMx6SNOj+CQtCXyWNEDivaTa0i2kBPWriHh64NyIeBY4UtLKwB5tjdjMzHpCkWHmDwFLk55XOhs4NiL+3OCaB4ElWgvNzMx6WZEE9QzwQ+DnBebIOxb4deGozMys5xVJUKtExMIihedmv6cbnmhmZlalyCCJiyXtMtgJknaSdOkQYzIzMyuUoKZTNfddDasAm7cazGgltT6LhJlZr2r3chtvBV5pc5lmZtaDivRBQZp7702Uph+fBHwUuH+oQZmZmQ1ag5K0UNKrkl7Nuw4ZeF/5ItWa7ibNJv6bDsdsZmY9oFEN6gperzW9H7gPmFPjvFeBx4FLSDNKmJmZDcmgCSoipg/8LGkhcHJEHNbpoMzMzIr0Qa0KPNmpQMzMzCo1naAi4t5OBmL11Vr80MxstKuboCT9O6n/6WcRMT+/b0ZExH+0JTozM+tZg9WgDiElqN8C8/P7ZgTgBNUhrk2ZWa8YLEFtkbf3Vb23OjxbhJlZ+9RNUBFx+WDvzczMOqndUx2ZmZm1RdMJStJkSR+VNLZi3xhJh0r6u6RrJG3bmTDNzKzXFHkO6mDgk8CKFfu+BXy74v2ZkjaLiGvbEdxI4H4nM7POKNLENw24JCJeAZC0CLAXcCtpotgNgQXAfu0O0szMek+RBLUiUPmw7nrAeNJzUg9ERD/we+Bf2xifmZn1qCJNfIvxxuU2NsnvK1fQfQB4exvisib4mSgzG82K1KAeANapeP9RYF5E3FKxbwXg6aJBSNpB0tGSrpT0tKSQdHrRcmqUu3MuKyTtOdTyzMxs+BSpQf0R2E/SD4AXgA8BJ1edM5U3NgM261vAusCzpEQ4tYUy3kDSysDRucylhlqemZkNryI1qCOBe4CvAgcBD5NG9gEgaRXgfaQ1pIraD1gDeBvw5Rauf4O8wu/JpDWqjh9qeWZmNvyKzGb+qKT3AFvmXZdHxDMVpyxFSl4XFg0iIi4b+FntGbc9E/gAMD1vzcxshCnSxEdEPE9q6qt1bDYwux1BDYWktYDDgZ9GxBWSnKDMzEagUTXVkaQxwGmkCW4P6nI4ZmY2BIVqUJKWBz5Peih3OWDRGqdFRGxZY/9w+HdgfWDTXNtrmqQZwAyASZMmdSA0MzMroukEJWkq8GdgAjBYR1FXnsiRtCGp1vTDiPhL0esj4kTgRIC+vj4/VWRm1mVFmvh+QHrO6QhgNWCxiFikxqtWraqjKpr2bueNcwOamdkIVaSJbzPgvIgoY9/OUqRh6gAv1BkJ+HNJPycNnth32CLrIs80YWYjWZEEJeDmTgUyRC8Cv6xzbANSv9RVwG1A4eY/MzMbfkUS1CxgzU4F0ixJiwFTgJcj4i54bfh7zamMJB1CSlC/iohfDFecZmY2NEUS1GHAhZKmR8Sf2xmEpG2AbfLbiXk7TdIp+ed5EXFA/nkl4BbSlEqT2xmHmZmVR5EEtTJpOY2LJP2aVKN6staJEXFqwTjWA3at2rdafkFKRgdgZmY9Q9Fk77mkhaQh5JUjEKovFuk5qGEfyddOfX190d/f39S5ZVlRt9af0YMkzKwsJM2KiL4i1xSpQe1eMB4bRgPJyInIzEaLIpPF/qqTgZiZmVUaVXPxmZnZ6FFoLj4ASROA7YG1gLERsWfF/lWBfxadB8/MzKxa0cli9wCOApYgD4jg9eePViQ9BDuD+g/NmpmZNaXpJj5JHyJNpno7sC1wXOXxiLiJtB7UNm++2szMrJgiNagDScu8bx4RT0tav8Y5/wCmtSUyMzPraUUGSfQBf4yIpwc55wFenwnCzMysZUUS1OLAggbnLAu82no4ZmZmSZEENQd4b4NzNiLNGG5mZjYkRRLU74HNJO1Y66Ck3YF1gLPbEZiZmfW2IoMkjgQ+A/xa0g7AMgCS9iEtZrgdcAdwdLuDNDOz3lNkqqMnJG0OnApU1qKOytsrgc9FRKN+Kuugskxea2Y2VIUe1I2I+4DpktYhDScfBzwFXBsRszoQn5mZ9ajCUx0BRMQ/SM88mZmZdUQrc/GtAkwgTXP0WK5VmZmZtVVTo/gkjZf0I0kPA3cDfwWuA+6R9JCk70tavpOBmplZb2mYoCStDvQDXyFNCPsq8CjwWP55IvBVoF/SavXKMTMzK2LQBCVpEeAMYBJwOfBBYKmIeHtETASWBrYCrgAmA6d3NFozM+sZjWpQW5Hm4DsT2DIiLo2IlwYORsSLEXEx8AHgLGCjPOu5mZnZkDRKUNsDLwL/FhFR76R8bB/gZWCH9oVnZma9qlGC2gC4OiIea1RQRDwKXJWvMTMzG5JGCWpl0iKEzZoNrNJ6OGZmZkmjBPU24MkC5T1JGjhhZmY2JI0e1F2cYus7LczXWInVmq+vfg+jmVl3NPOgrr+6zMxs2DUz1dEhkg7pdCBmZmaVmqlBqeCrJZLeKemkPHXSi5LmSPqJpOWavH66pGjitXKrMZqZ2fAZtAYVEUVW3G2ZpCnANcAKpJV7bwU2JE2vtLWkTSLi8QbFzAEOrXPsPaQFFWdHxP1tCdrMzDqqpeU2OuBYUnKaGRGvrcgr6UfAfsB3gC8NVkBEzAEOqXVM0q/zjye2IVYzMxsGw1JDGkyeYHYrUg3oZ1WHDwYWADtLGtti+eOAbYHngdNaj3R0k15/mZmVQdcTFGkeP4CLImJh5YGIeAa4GlgS2LjF8ncD3gL8d0Q80WqQlUbil/lIjNnMelsZEtSaeXt7neN35O0aLZa/Z96e0OL1ZmbWBWVIUMvk7VN1jg/sX7ZowZI2B6aSBkdc0+DcGZL6JfU/9ljDqQfNzKzDypCgGhlolGrlgeEZeduw9hQRJ0ZEX0T0TZgwoYVbmZlZO5UhQQ3UkJapc/xtVec1JS9Bvz0eHGFmNiKVIUHdlrf1+phWz9t6fVT17EoaHHFmRBSZ8NYqeHCFmXVLGRLUZXm7VV5i/jWSlgY2IdWCri1Y7hfy1s8+mZmNQF1PUBFxF3ARMBnYu+rwocBY4NSIWDCwU9JUSVPrlSlpM2At4KZGgyPMzKycyjKTxF6kqY6OkrQlcAuwEbAFqWnvm1Xn35K39RqeBgZHuPZkZjZCdb0GBa/VovqAU0iJaX9gCnAUMK2JefhekyeX3QEPjmiZ+53MrAzKUoMiT+K6e5Pn1v3qzLNFvLVdcZmZWXeUogZlZmZWzQnKzMxKqTRNfDYyVfZTRStzfZiZ1eEalJmZlZITlJmZlZITlJmZlZL7oKxp7m8ys+HkBGUd4WRmZkPlBGUt8SwTZtZp7oMyM7NScoIyM7NScoIyM7NScoIyM7NScoKyYeVlPMysWR7FZ23jxGNm7eQalJmZlZITlJmZlZKb+ApwE5aZ2fBxgrIRxVMomfUON/GZmVkpOUGZmVkpOUGZmVkpuQ/KuqJeX5L7mMxsgBOUlZ5HT5r1JicoK61Gicm1LbPRzQnKus41JDOrxYMkzMyslEqToCS9U9JJkh6S9KKkOZJ+Imm5guUsn6+bk8t5KJf7zk7FboMbmMHcNSUzK6IUTXySpgDXACsAvwduBTYEvgJsLWmTiHi8iXLG5XLWAC4FfgNMBXYHPiZpWkTc3ZlPYd3UKPkV6aNqpm+r1f6vIte5j816XSkSFHAsKTnNjIijB3ZK+hGwH/Ad4EtNlPNdUnL6cUR8taKcmcBP8322bmPcNkI0U3urlQQ6WeurVXankl0nOem+Wa98zk5TdPm3J2k14C5gDjAlIhZWHFsaeBgQsEJELBiknLHAY8BC4O0R8UzFsUXyPSbnewxai+rr64v+/v4a92j6Y1mPa6bmNdTy6pU1HP9Lt3rvWl/cw/VM3EB5w/37aWfsQymvW0nz9ftqVkT0Fbm2DH1QH8jbiyqTE0BOMlcDSwIbNyhnGvBW4OrK5JTLWQhclN9uMeSIzcys48qQoNbM29vrHL8jb9cYpnLMhqxyYEg7BokUKavRuc3E1ui6VmNv5z2G8vmGWl47Yq51XdHfcad+h0XOLfL3LaoMfVDL5O1TdY4P7F+2k+VImgHMyG9flHRTg/tZ+Y0H5nU7iDJo9guoleuGcn6TxkuD/x2L3LfV30Uzx1v9HRa5bqjnFi1jqOVWWHPQozWUIUE1MvCRh9pqOmg5EXEicCKApP6ibaVWPv47jg7+O44Okt7csd9AGZr4Bmo2y9Q5/raq8zpdjpmZlUAZEtRteVuvb2j1vK3Xt9TucszMrATKkKAuy9ut8nDw1+Rh5psAzwPXNijn2nzeJvm6ynIWAbaqut9gTmziHCs//x1HB/8dR4fCf8euJ6iIuIs0BHwysHfV4UOBscCplc9ASZoqaWpVOc8Cp+XzD6kqZ59c/oXNzCSR+6NshPPfcXTw33F0aOXv2PUHdQFqTHV0C7AR6Zml24H3VU51JCkAIkJV5VRPdXQdsBbwKeDRXM5dnf48ZmY2dKVIUACSVgYOI01FNI40g8S5wKERMb/q3JoJKh9bHjgY2AZ4O/A48D/Av0fEA538DGZm1j6lSVBmZmaVut4HVRbtWu7DukfSDpKOlnSlpKclhaTTux2XNU/SOEl7SjpH0p2Snpf0lKSrJO1RPZDKykvSEZIukXR//jvOl3SDpINzd0zjMlyDqtkHNrDcxxak4etNLfdh3SXpRmBd4FngAdJSK2dExE5dDcyaJulLwHGkJv7LgPuAFYHtSM84ng3sGP7iKj1JLwHXAzeTxgCMJc2p2gc8BGwcEfcPWob/ziDpQtIw9HrLfZwQEc0s92FdJGkLUmK6E9ic9AXnBDWCSPoA6YvsvKqVDSaSBj2tDOwQEWd3KURrkqQlIuKFGvu/AxwEHBcRew1WRs9Xl/NyH1uRlvv4WdXhg4EFwM55OQ8rsYi4LCLu8L+uR66IuDQi/lBjZYO5wPH57fRhD8wKq5WcsjPzdvU6x1/T8wmK9i33YWad9XLevtLVKGyoPpG3/2h04kiYLLbTmlmmYyvSs1WXDEtEZvYGksYAu+S3F3QzFitG0gHAUqQ+xD5gU1JyOrzRtU5Q7Vvuw8w653BgbeD8iLiw28FYIQeQBroMuADYLSIea3Shm/gaa9dyH2bWAkkzgf1Jo2t37nI4VlBETMyTKkwkjcZcDbhB0gaNrnWC8jIdZqUlaW/gp6ShyltUzypjI0dEPBIR55C6TMYBpza6xgnKy3SYlZKkfYFjgJtIyWlul0OyNoiIe0n/4Hi3pPGDnesE1b7lPsysTSQdCPwYuJGUnB7tckjWXu/I21cHO6nnE1Qry32YWedI+jZpUMQsYMuImNflkKygvCTSxBr7F8kP6q4AXBMRTwxajp9pLL7ch5WTpG1Is9hD6pD9MHA3cGXeNy8iDuhGbNYcSbsCp5D+ZX00tft+50TEKcMYlhWUm2e/D1wB3EVaVWJF0gwvqwFzSf/4uHnQcpygkiLLfVg5STqENPtHPfdGxOThicZa0cTfEODyiJje+WisVZLWBr5M6iJ5J+kxnQWkf/CfBxzVzPeqE5SZmZVSz/dBmZlZOTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmbWRpDmS5nQ7DrPRwAnKRj1JkV/3Slqizjlz8jlexNOsJJygrJdMAvbt8D22zC8zGyJPdWSjnqQAniCtijwGmFI9Q3ZullsFWCwiXhn2IM3sTVyDsl7xHPAfpBWSG01G+gaSPi3pCklPSXpe0j8lfUPSW2qc+6Y+KEmLS5op6XpJT0h6Lp/3e0kfrFHGVEmnSLpf0ouSHpH0X5LWLBDz9NxkeYik9SSdJ+nJfO/LJb2vznXLSPqepNskvZDjvbBOnJX32DDfY37eN7ny9yFpKUk/zp/peUk35tnnkTRG0kGS7sj3vEvSPs1+Vhu9XIOyUS/XoB4EViUtpTIJWDsibq84Zw41alCSvgt8A5gHnAU8C3wEeDdwOfChiHi5qhwqZ02X9F/AZ0krw15KWgDzHcCmwO8qlwCRtDXwO2Ax4A/AnaTZoLcDXiQt3nd9E595OmkxzvOADwB/AW7In3174CVgvYi4reKaZYGrgXcBf8uxjgc+DSwFfDkiTqhxj4uA6cBV+R7jgYMi4qH8+1gMuA9YHvgTsHj+fSxJWv57L9LyNv+TP+OOpKVvPhMRv230WW0Uiwi//BrVL1LT3gP55x3y+99VnTMn7x9TsW9a3ncfMLFi/xhS8gjSF3F1OXMq3i8DLAT6gUVrxDau4uflSE2R84B3VZ33blJyvL7Jzzw9xxfAblXHvpj3H1u1/4S8/wTyP17z/tVJ6zK9CEyuc48v1olj4Pf6B+AtFfs3y/vnk5LhshXHViMl0Bu6/d+OX919uYnPekpEnEWqTWwradMGp38+b/8zIuZWlPEKsD8p8ezZ6JaASF/uC2vEU7kQ5i6kdXMOjqqF3CJiNvBzYH1J72pwz0pXx5sX9zsJeAXYcGCHpMWAnUhJ8BsR8VrTSkTcARxFqvnsUuMeN0ZFzaqOfSPixYoyrwTuISXlAyPiyYpjd5Nqcu+RtGjDT2ijlofUWi/an7SC8g8lbVz5ZVxlg7y9tPpARNwu6QFgVUnLVn7BVp33tKQ/AJ8AbpR0NmmF379GxHNVp0/L23Xzwn3V1sjbtYBBVyKt0F8jppclPUJKDgOmkprcro7aC8ldCnwLWL/GsesaxPBkRNxVY/9DpGbXWTWOPQgsSloZ+cEG5dso5QRlPSci/iLpLFJz36eBev0cy+Ttw3WOP0zq059aZhEAAAJASURBVFkGqJmgsv8DHAh8Djg073shx3BARDyS943L2y80+AhLNTheqV5cr5ASwIBmPiukGl61uTX2Vaq1bPtADEREreMD/YCLNSjbRjE38Vmv+jrwMvA9SYvXOWfgi3NineNvrzqvpoh4PiIOiYg1SAltJ9KAgp1IAy+q77duRGiQ168afLZWDOWzeqSVdYQTlPWk3OR0LKmJ6d/qnHZD3k6vPiDpX0ij6+6p17xX5773R8QZwIeBO4BNJQ3UnK7N282aLa+NbiMNxV9P0nI1jm+Rtw1HEJq1ixOU9bLDSE1g36R2s9lJefstSRMGduaO+x+Q/v/55WA3kDRB0kY1Do0FliY1Zb2U952c4zlY0obVF0haJA/tbruIeAk4g/R7OKzqvlOAmaQa52mduL9ZLe6Dsp4VEfPzc05H1jl+jaQjga8BN+U+owWk56DWJjXTfb/BbVYCrpV0C6n2cT/pYeGPk5rTjoqIZ/L9Hpe0A3BOvuYSYDZp9N8k0iCKcUDN+QTb4Ouk2ts+kv6V9IzTwHNQSwP7RMQ9Hbq32Zs4QVmvO4r0oOjkWgcj4kBJNwD7kIZYLwbcRRrR9sNc8xjMHNLMFdNJzWTjSc/+3EZKCL+put8lktYBDiA1A25GqmE9RBpJd3bBz9e0nLCnkR5M3g74Kumh4uuA70fERZ26t1ktnknCzMxKyX1QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSv8L0fJRmLQS8/AAAAAASUVORK5CYII=\n", 71 | "text/plain": [ 72 | "
" 73 | ] 74 | }, 75 | "metadata": { 76 | "needs_background": "light" 77 | }, 78 | "output_type": "display_data" 79 | }, 80 | { 81 | "name": "stdout", 82 | "output_type": "stream", 83 | "text": [ 84 | "19\n" 85 | ] 86 | }, 87 | { 88 | "data": { 89 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZxcVZ338c+XTSQgSxKIIiGQAYIimz1ABCQBRdxZfdSHVTAqMBGERxR1CMyogAsKyKYCsozKgOCjMIAssorYAVTCvoQ9QAj7DvnNH+d0KIqqrrrVVV03Xd/361Wv23WXc3+Vgv71OfcsigjMzMzKZpFuB2BmZlaLE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZVSKRKUpCMkXSrpAUkvSpon6UZJh0gaXaCc2ZKizmtOJz+DmZm1l8owUFfSK8ANwC3AY8AoYBOgD3gY2CQiHmiinNnAcsBPahx+LiJ+2K6Yzcyss8qSoJaMiJdq7P8ucDBwfETs3UQ5swEiYkK7YzQzs+FViia+WskpOytv1xiuWMzMrBwW63YADXwyb/9R4Jq3SdoZGA88n6+9MiJeb3dwZmbWOaVo4hsg6UBgaWBZ0vOnzUgJ5kMR8XgT188GVq1x6F5gj4i4on3RmplZJ5UtQc0BVqrYdSGwe0Q82uT1hwBXAbOAZ4HVgX2BacBLwOSI+Huda6fl8xg1atT7J02a1OrHMDOzKjNnzpwbEWOLXFOqBDVA0krAB4DDgWWAT0TEDUMo74fAAcB5EbFdo/P7+vqiv7+/1duZmVkVSTMjoq/INaXoJFEtIh6NiHOBrYHRwGlDLPKEvP3gEMsxM7NhUsoENSAi7iONjXqvpDFDKOqxvB019KjMzGw4lDpBZe/K26H0wpuct/cMMRYzMxsmXU9QkiZJGldj/yJ5oO6KwLUR8WTev3i+ZmLV+e+VtEKNclYFjs1vz2j/JzAzs04owziobYAfSLoSuBt4gtSTbwtSL7w5wBcrzl8ZuBW4D5hQsX8n4BuSLid1K38WmAh8HFgSuADwVEdmZguJMiSoS4CTgE2B9Uhz6T0P3AGcDhwdEfOaKOdyYC1gA1KT3ijgKeDqXM7pUcYui2ZmVlPXE1RE3AzsU+D82YBq7L8C8EBcM7MRousJaiRRVdp0fc3MrHVd7yRhZmZWixOUmZmVkhOUmZmVkhOUmZmVkhOUmZmVknvxDVF1zz0zM2sP16DMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUvNxGQV5ew8xseLgGZWZmpeQEZWZmpeQEZWZmpeQEZWZmpeQEZWZmpdT1BCVptKS9JJ0r6S5JL0p6WtLVkvaUVChGSR+XdLGkB3NZ90j6b0mTO/UZzMys/crQzXwn4HjgEeBy4H5gJWB74BfARyXtFBHRqCBJRwBfB54AzgPmAv8CfBrYQdKuEXFGRz6FmZm1VRkS1B3Ap4DzI2L+wE5JBwPXAzuQktU5gxUiaRxwIPAosG5EPFZxbCpwGXAY4ARlZrYQ6HoTX0RcFhF/qExOef8c4IT8dkoTRa1K+jx/rUxOuazLgWeBsUOP2MzMhkPXE1QDr+bta02ceyfwCrCRpDGVByR9EFgGuKS94ZmZWaeUoYmvJkmLAbvmtxc2Oj8i5kk6CPgxcIuk80jPoiaSmhD/BHypQ+GamVmblTZBAYcD6wAXRMRFzVwQET+RNBs4GfhixaG7gFOrm/4qSZoGTAMYP358qzGbmVmblLKJT9J04ADgNmCXAtd9HTgbOJVUcxoFvB+4BzhT0pH1ro2IkyKiLyL6xo71oyozs24rXYKStA/wU+AWYGpEzGvyuinAEcD/j4ivRcQ9EfFCRNwAbAc8BBwgafUOhW5mZm1UqgQlaT/gWOBmUnKaU+DyT+Tt5dUHIuIFUpf1RYANhhqnmZl1XmkSVO7gcBRwEyk51X1eVMfb8rZe+9zA/ldaCM/MzIZZKRKUpO+QOkXMBLaKiLmDnLu4pEmSJlYduipvp0laueqajwKbAi8B1xaP742XmZkNj6734pO0G2mGh9dJSWa63poJZkfEqfnnlYFbgfuACRXnnE0a5/Qh4FZJ5wJzgLVJzX8CvhERT3Tkg5iZWVsVSlCSliHVRMYDY4AXgceAmyJiVosxrJa3iwL71TnnClLPvLoiYr6kjwH7AJ8ldYxYCpgHXAAcHREXtxijmZkNMzWag1XS24HPAXsCG/FGs+BANWeggLmk+fKOj4h/tj/U4dPX1xf9/f0L3rfatNd4elszs94gaWZE9BW5pm4NKs/kMB34FrA86fnNdcDfSE1n84C3A6OBScAmwJeBL0m6BDggIm5u4XOYmZkN2sR3G6n57ULgV8DvI+LlwQqTtCawO2mKohsl7RkRp7UpVjMz6yGDJahbgB0i4u/NFhYRdwAHS5oB7E2qYRlvbSZ085+Z2eDqJqiI+FSrhUbEK8BPWr3ezMys693MRzKPmzIza92QEpSk1YD35Le3RMS9Qw/JzMysxQSVx0P9AthxYBcQks4CpkXEs22Kz8zMelSrNahjga2BQ0jTEy1JWhRwN+AF0pgpMzOzlg2aoCQtlWcCr7Yd8JWIOLNi37mSlsrHnKDMzGxIGk0W+09JU2vsXwyo1Yz3LO54YWZmbdAomVwHXCLp58D/q3i2dClwrKRRwI2kpS4+SRqg+8dOBWtmZr1j0BpURPxf4NPAx4FZkrbJh/Ymzb13JjALuIE0I/nfgX07Fq2ZmfWMhs1xEfFHSVeSFhO8QNJpwH4RsaGkD5Pm4RMwKyIu7Wy4ZmbWK5p6XhQRzwB7SvoNcBJwi6S9I+I84E+dDNDMzHpToRV1I+JPwDrAucA5kn4jaXRHIjMzs57WVIKSNEbS+yWNiYjnI2IfYCqwAWn12s92NEozM+s5gyYoSUtL+i3wKHA98KiksyQtHRFXAusBpwNnSDpP0rjOh2xmZr2gUQ3q+8BOpPWg9iEtu74jcDhARLwUEQeQloFfg/RsavdOBWtmZr2jUYL6NHB2RHwhIk6IiD1Jy7p/uvKkiPgrsD5wPHBiRyI1M7Oe0ihBjQIerNr3ALBU9YkR8WpEfAvYqE2xmZlZD2uUoK4DdpG0qaQlJE0Gds77ayqyAq+ZmVk9jRLUV4EXgSvz9mrgJWC/Dsc14klvvMzM7K0GHagbEXdJmkSaZ288cD/wx4h4fjiCMzOz3tXMVEcvAL8dhljMzMwWKDSThJmZ2XCpm6AkHSBpyVYLlrShpI+2er2ZmfW2wWpQ3wPulnSQpJWbKUzJRySdC/yNNNOEmZlZYYM9g3of8GPSbBL/KelaUi++fuAR4ElgSWA0acmNTYCtgHHAE6R1oRoO2s2TzW5HWnPqfcDKwCvAP4FTgFMiYn4zH0bSjsAWpEHD6wHLAGdGxM7NXG9mZuVRN0FFxB3AJyR9gDTN0Q7A5kDUOH2gs/TtwBGkpFJrSfhadiLNQPEIcDmpp+BKwPbAL4CPStopImrdt9q3SYnpOdIA40lNxmBmZiXTTC++a4FrJX0Z+CCwGanL+WjS2KjHgH8Af46IWS3EcAfwKeD8ypqSpINJE9TuQEpW5zRR1v6kxHQXqSZ1eQvxmJlZCTS1YCFArhGdn19tExGX1dk/R9IJwHeBKTSRoCJiQUKSR8CamS3Uyt7N/NW8fa2rUZiZ2bArbYKStBiwa357YTdjMTOz4VfaBEVac2od4IKIuKjTN5M0TVK/pP7HH3+807czM7MGSpmgJE0HDgBuA3YZjntGxEkR0RcRfWPHjh2OW5qZ2SBKl6Ak7QP8FLgFmBoR87ockpmZdUGpEpSk/YBjgZtJyWlOl0MyM7MuKU2CknQQcBRwEyk5PdblkMzMrIuaTlCStpO0aCeCkPQdUqeImcBWETF3kHMXlzRJ0sROxGJmZuXQ9EBd0kDZhyWdDPwiIu5vRwCSdgMOA14HrgKm1xhkOzsiTs0/rwzcCtwHTKgqa1tg2/x2XN5OljRw7dyIOLAdcZuZWWcVSVDHAZ8nzXf3TUkXkiaDPb/JefLqWS1vF6X+UvJXAKc2Udb6wG5V+1bPL0hJzQnKzGwhoCK5Ja8P9VngS8DGpIljHyJN6vrLiHioE0EOt76+vujv71/wvtOzJg0pvZuZLQQkzYyIviLXFOokEREvRcSpETEZWJc0C/nSwAzgXknnStqmSJlmZma1tNyLLyJujoh9gXcBewCPkmcll3SvpAMljWpTnGZm1mOG1M08J6BdgemkzgsC/k5aiuNI4DZJ6w81SDMz6z0tJShJG+SlMB4GTgDWJD2H2jAiNiTVqr4BjAGOblOsZmbWQ5ruxSdpKeBzpA4S7yfVlm4lJahfRcQzA+dGxHPAkZJWAfZsa8RmZtYTinQzfxhYhjRe6RzguIj4c4NrHgKWbC00MzPrZUUS1LPAj4CfF5gj7zjg14WjMjOznlckQa0aEfOLFJ6b/Z5peKKZmVmVIp0kLpG062AnSNpZ0mVDjMnMzKxQgppC1dx3NawKbNFqML1KevPLzMzav9zG24HX2lymmZn1oCLPoCDNvfcWStOPjwc+Bjww1KDMzMwGrUFJmi/pdUmv510zBt5Xvki1pntIs4n/psMxm5lZD2hUg7qSN2pNHwTuB2bXOO914AngUtKMEmZmZkMyaIKKiCkDP0uaD5wSEYd1OigzM7Miz6BWA57qVCBmZmaVmk5QEXFfJwMxMzOrVDdBSfp30vOnn0XEvPy+GRER/9GW6MzMrGcNVoOaQUpQvwXm5ffNCMAJyszMhmSwBDU1b++vem9mZtZxdRNURFwx2HszM7NOavdUR2ZmZm3RdIKSNEHSxySNqti3mKRDJf1d0rWStutMmGZm1muKjIM6BPgUsFLFvm8D36l4f5akzSPiunYEZ2ZmvatIE99k4NKIeA1A0iLA3sBtpIliNwKeB/Zvd5BmZtZ7iiSolYDKwbrrA2NI46QejIh+4PfAv7YxPjMz61FFEtTivHm5jU3z+8oVdB8E3tmGuMzMrMcVSVAPAutWvP8YMDcibq3YtyLwTNEgJO0o6RhJV0l6RlJIOqNoOTXK3SWXFZL2Gmp5ZmY2fIp0kvgjsL+kHwIvAR8GTqk6ZxJvbgZs1reB9YDnSIlwUgtlvImkVYBjcplLD7U8MzMbXkVqUEcC9wJfAw4GHiH17ANA0qrAB0hrSBW1P7Am8A7gKy1c/yZ5hd9TSGtUnTDU8szMbPgVmc38MUnvA7bKu66IiGcrTlmalLwuKhpERFw+8HPKLUM2HdgSmJK3Zma2kCnSxEdEvEhq6qt1bBYwqx1BDYWktYHDgZ9GxJWSnKDMzBZChRJU2UlaDDidNMHtwV0Op2XVlciI2ueZmY1khRKUpBWAL5AG5S4PLFrjtIiIrWrsHw7/DmwAbJZre02TNA2YBjB+/PgOhGZmZkU0naAkTQL+DIwFBntQ1JW/9yVtRKo1/Sgi/lL0+og4CTgJoK+vz3UWM7MuK9KL74ekcU5HAKsDi0fEIjVetWpVHVXRtHcHb54b0MzMFlJFmvg2B86PiDI+21ma1E0d4KU6PQF/LunnpM4T+w1bZGZm1pIiCUrALZ0KZIheBn5Z59iGpOdSVwO3A4Wb/8zMbPgVSVAzgbU6FUizJC0OTARejYi7YUH395pTGUmaQUpQv4qIXwxXnGZmNjRFEtRhwEWSpkTEn9sZhKRtgW3z23F5O1nSqfnnuRFxYP55ZeBW0pRKE9oZh5mZlUeRBLUKaTmNiyX9mlSjeqrWiRFxWsE41gd2q9q3en5BSkYHYmZmPUPR5ChQSfNJXcgreyBUXyzSOKhh78nXTn19fdHf37/gfXtmX2qdB+qa2cJO0syI6CtyTZEa1B4F4zEzM2tZkclif9XJQMzMzCqNqLn4RqrKJkY395lZryicoCSNBXYA1gZGRcReFftXA/5ZdB48MzOzakUni90TOBpYktwhgjfGH61EGgQ7jfqDZs3MzJrS9Fx8kj5Mmkz1DmA74PjK4xFxM2k9qG3ferWZmVkxRWpQB5GWed8iIp6RtEGNc/4BTG5LZGZm1tOKzGbeB/wxIp4Z5JwHeWMmCDMzs5YVSVBLAM83OGc54PXWwzEzM0uKJKjZwPsbnLMxacZwMzOzISmSoH4PbC5pp1oHJe0BrAuc047AzMystxXpJHEk8Fng15J2BJYFkLQvaTHD7YE7gWPaHaSZmfWeIlMdPSlpC+A0oLIWdXTeXgV8PiIaPacyMzNrqNBA3Yi4H5giaV1Sd/LRwNPAdRExswPxWZXqmdU99ZGZjVQtzcUXEf8gjXkyMzPriFbm4lsVGEua5ujxXKsyMzNrq6Z68UkaI+nHkh4B7gH+ClwP3CvpYUk/kLRCJwM1M7Pe0jBBSVoD6Ae+SpoQ9nXgMeDx/PM44GtAv6TV65VjZmZWxKAJStIiwJnAeOAK4EPA0hHxzogYBywDbA1cCUwAzuhotGZm1jMa1aC2Js3BdxawVURcFhGvDByMiJcj4hJgS+BsYOM867mZmdmQNEpQOwAvA/8WUb9Dcz62L/AqsGP7wjMzs17VKEFtCFwTEY83KigiHgOuzteYmZkNSaMEtQppEcJmzQJWbT0cMzOzpFGCegfwVIHyniJ1nDAzMxuSRglqCYqt7zQ/X2NmZjYkzcwk4dneFhKep8/MRpJmEtQMSTM6HYiZmVmlZqY6UsFXSyS9W9LJeeqklyXNlvQTScs3ef0USdHEa5VWYzQzs+EzaA0qIoqsuNsySROBa4EVSSv33gZsRJpeaRtJm0bEEw2KmQ0cWufY+0gLKs6KiAfaErSZmXVUS8ttdMBxpOQ0PSIWrMgr6cfA/sB3gS8PVkBEzAZm1Dom6df5x5PaEKuZmQ2DYakhDSZPMLs1qQb0s6rDhwDPA7tIGtVi+aOB7YAXgdNbj7ScpDdeZmYjSdcTFGkeP4CLI2J+5YGIeBa4BlgK2KTF8ncH3gb8d0Q82WqQZmY2vMqQoNbK2zvqHL8zb9dssfy98vbEFq83M7MuKEOCWjZvn65zfGD/ckULlrQFMInUOeLaBudOk9Qvqf/xxxtOPWhmZh1WhgTVyMDTlVaGnU7L24a1p4g4KSL6IqJv7NixLdzKzMzaqQwJaqCGtGyd4++oOq8peQn6HRihnSPMzEa6MiSo2/O23jOmNfK23jOqenYjdY44KyKKTHhrZmYlUIYEdXnebp2XmF9A0jLApqRa0HUFy/1i3nrsk5nZQqjrCSoi7gYuBiYA+1QdPhQYBZwWEc8P7JQ0SdKkemVK2hxYG7i5UecIMzMrp7LMJLE3aaqjoyVtBdwKbAxMJTXtfavq/Fvztt7w1IHOEa49mZktpLpeg4IFtag+4FRSYjoAmAgcDUxuYh6+BfLksjvizhFmZgu1stSgyJO47tHkuXUn9smzRby9XXGZmVl3lKIGZWZmVs0JyszMSskJyszMSskJyszMSqk0nSSs/SrXiIpWZjI0M+si16DMzKyUnKDMzKyU3MTXo9z8Z2Zl5xqUmZmVkhOUmZmVkpv4eoTqTg5lZlZOrkGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpuZOEvYXHSJlZGbgGZWZmpeQEZWZmpeQmPvMYKTMrJScoG1R18vIzKTMbLm7iMzOzUnKCMjOzUnKCMjOzUvIzKCtksGdSg3W28LMrMyvKNSgzMysl16BsWLTaG7BRF3jXzMxGLicoGxKPoTKzTilNE5+kd0s6WdLDkl6WNFvSTyQtX7CcFfJ1s3M5D+dy392p2M3MrP1KUYOSNBG4FlgR+D1wG7AR8FVgG0mbRsQTTZQzOpezJnAZ8BtgErAH8HFJkyPins58CmvVcAwG7sQ9PIjZrLPKUoM6jpScpkfEthHxjYjYEjgKWAv4bpPlfI+UnI6KiK1yOduSEt2K+T5WAtIbr06UWf0qm+GOrez/Hma1KLr8Z5+k1YG7gdnAxIiYX3FsGeARQMCKEfH8IOWMAh4H5gPvjIhnK44tku8xId9j0FpUX19f9Pf3V5Rd+GPZEDTbdX047jHY/x5FYqsup9NLmoyUziWupY4ckmZGRF+Ra8pQg9oyby+uTE4AOclcAywFbNKgnMnA24FrKpNTLmc+cHF+O3XIEVtHDcdf+s3eo2w1jzLXvMpUgx3K/cv0fXdbt7/TMiSotfL2jjrH78zbNYepHLO2G+x/7m78EmjHPTsV63CX2a7P0ew9Wi2zXbF1qsxG57aiDJ0kls3bp+scH9i/XCfLkTQNmJbfvizp5gb3s/IbA8xtZ4HD/Vf1YPfrVCyduOcQY13wPbbjMxdNwu2+x1CTQru1q8wmylmr4RlVypCgGhn42ENtfR60nIg4CTgJQFJ/0bZSKx9/jyODv8eRQVJ/47PerAxNfAM1m2XrHH9H1XmdLsfMzEqgDAnq9ryt92xojbyt92yp3eWYmVkJlCFBXZ63W+fu4AvkbuabAi8C1zUo57p83qb5uspyFgG2rrrfYE5q4hwrP3+PI4O/x5Gh8PfY9QQVEXeTuoBPAPapOnwoMAo4rXIMlKRJkiZVlfMccHo+f0ZVOfvm8i9qZiaJ/DzKFnL+HkcGf48jQyvfY9cH6gLUmOroVmBj0pilO4APVE51JCkAIkJV5VRPdXQ9sDbwaeCxXM7dnf48ZmY2dKVIUACSVgEOA7YBRpNmkDgPODQi5lWdWzNB5WMrAIcA2wLvBJ4A/gf494h4sJOfwczM2qc0CcrMzKxS159BlUW7lvuw7pG0o6RjJF0l6RlJIemMbsdlzZM0WtJeks6VdJekFyU9LelqSXtWd6Sy8pJ0hKRLJT2Qv8d5km6UdEh+HNO4DNegaj4DG1juYyqp+3pTy31Yd0m6CVgPeA54kLTUypkRsXNXA7OmSfoycDypif9y4H5gJWB70hjHc4Cdwr+4Sk/SK8ANwC2kPgCjSHOq9gEPA5tExAODluHvGSRdROqGPj0ijqnY/2Ngf+DEiPhyt+Kz5kiaSkpMdwFbkH7BOUEtRCRtSfpFdn7VygbjSJ2eVgF2jIhzuhSiNUnSkhHxUo393wUOBo6PiL0HK6Pnq8t5uY+tSct9/Kzq8CHA88AueTkPK7GIuDwi7vRf1wuviLgsIv5QY2WDOcAJ+e2UYQ/MCquVnLKz8naNOscX6PkERfuW+zCzzno1b1/rahQ2VJ/M2380OnFhmCy205pZpmNr0tiqS4clIjN7E0mLAbvmtxd2MxYrRtKBwNKkZ4h9wGak5HR4o2udoNq33IeZdc7hwDrABRFxUbeDsUIOJHV0GXAhsHtEPN7oQjfxNdau5T7MrAWSpgMHkHrX7tLlcKygiBiXJ1UYR+qNuTpwo6QNG13rBOVlOsxKS9I+wE9JXZWnVs8qYwuPiHg0Is4lPTIZDZzW6BonKC/TYVZKkvYDjgVuJiWnOV0OydogIu4j/cHxXkljBjvXCap9y32YWZtIOgg4CriJlJwe63JI1l7vytvXBzup5xNUK8t9mFnnSPoOqVPETGCriJjb5ZCsoLwk0rga+xfJA3VXBK6NiCcHLcdjGosv92HlJGlb0iz2kB7IfgS4B7gq75sbEQd2IzZrjqTdgFNJf1kfQ+1nv7Mj4tRhDMsKys2zPwCuBO4mrSqxEmmGl9WBOaQ/Pm4ZtBwnqKTIch9WTpJmkGb/qOe+iJgwPNFYK5r4DgGuiIgpnY/GWiVpHeArpEck7yYN03me9Af/+cDRzfxedYIyM7NS6vlnUGZmVk5OUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZtJGm2pNndjsNsJHCCshFPUuTXfZKWrHPO7HyOF/E0KwknKOsl44H9OnyPrfLLzIbIUx3ZiCcpgCdJqyIvBkysniE7N8utCiweEa8Ne5Bm9hauQVmveAH4D9IKyY0mI30TSZ+RdKWkpyW9KOmfkr4p6W01zn3LMyhJS0iaLukGSU9KeiGf93tJH6pRxiRJp0p6QNLLkh6V9F+S1ioQ85TcZDlD0vqSzpf0VL73FZI+UOe6ZSV9X9Ltkl7K8V5UJ87Ke2yU7zEv75tQ+e8haWlJR+XP9KKkm/Ls80haTNLBku7M97xb0r7NflYbuVyDshEv16AeAlYjLaUyHlgnIu6oOGc2NWpQkr4HfBOYC5wNPAd8FHgvcAXw4Yh4taocKmdNl/RfwOdIK8NeRloA813AZsDvKpcAkbQN8DtgceAPwF2k2aC3B14mLd53QxOfeQppMc7zgS2BvwA35s++A/AKsH5E3F5xzXLANcB7gL/lWMcAnwGWBr4SESfWuMfFwBTg6nyPMcDBEfFw/vdYHLgfWAH4E7BE/vdYirT8996k5W3+J3/GnUhL33w2In7b6LPaCBYRfvk1ol+kpr0H88875ve/qzpndt6/WMW+yXnf/cC4iv2LkZJHkH4RV5czu+L9ssB8oB9YtEZsoyt+Xp7UFDkXeE/Vee8lJccbmvzMU3J8AexedexLef9xVftPzPtPJP/xmvevQVqX6WVgQp17fKlOHAP/rn8A3laxf/O8fx4pGS5XcWx1UgK9sdv/7fjV3Zeb+KynRMTZpNrEdpI2a3D6F/L2PyNiTkUZrwEHkBLPXo1uCYj0y31+jXgqF8LclbRuziFRtZBbRMwCfg5sIOk9De5Z6Zp46+J+JwOvARsN7JC0OLAzKQl+MyIWNK1ExJ3A0aSaz6417nFTVNSs6tgvIl6uKPMq4F5SUj4oIp6qOHYPqSb3PkmLNvyENmK5S631ogNIKyj/SNImlb+Mq2yYt5dVH4iIOyQ9CKwmabnKX7BV5z0j6Q/AJ4GbJJ1DWuH3rxHxQtXpk/N2vbxwX7U183ZtYNCVSCv014jpVUmPkpLDgEmkJrdrovZCcpcB3wY2qHHs+gYxPBURd9fY/zCp2XVmjWMPAYuSVkZ+qEH5NkI5QVnPiYi/SDqb1Nz3GaDec45l8/aROscfIT3TWRaomaCy/wMcBHweODTveynHcGBEPJr3jc7bLzb4CEs3OF6pXlyvkRLAgGY+K6QaXrU5NaoP7j4AAAIMSURBVPZVqrVs+0AMRESt4wPPARdvULaNYG7is171DeBV4PuSlqhzzsAvznF1jr+z6ryaIuLFiJgREWuSEtrOpA4FO5M6XlTfb72I0CCvXzX4bK0Yymd1TyvrCCco60m5yek4UhPTv9U57ca8nVJ9QNK/kHrX3Vuvea/OfR+IiDOBjwB3AptJGqg5XZe3mzdbXhvdTuqKv76k5Wscn5q3DXsQmrWLE5T1ssNITWDfonaz2cl5+21JYwd25gf3PyT9//PLwW4gaaykjWscGgUsQ2rKeiXvOyXHc4ikjaovkLRI7trddhHxCnAm6d/hsKr7TgSmk2qcp3fi/ma1+BmU9ayImJfHOR1Z5/i1ko4Evg7cnJ8ZPU8aB7UOqZnuBw1uszJwnaRbSbWPB0iDhT9Bak47OiKezfd7QtKOwLn5mkuBWaTef+NJnShGAzXnE2yDb5Bqb/tK+lfSGKeBcVDLAPtGxL0durfZWzhBWa87mjRQdEKtgxFxkKQbgX1JXawXB+4m9Wj7Ua55DGY2aeaKKaRmsjGksT+3kxLCb6rud6mkdYEDSc2Am5NqWA+TetKdU/DzNS0n7MmkgcnbA18jDSq+HvhBRFzcqXub1eKZJMzMrJT8DMrMzErJCcrMzErJCcrMzErJCcrMzErJCcrMzErJCcrMzErJCcrMzErJCcrMzErJCcrMzErpfwH7Bz8jmN3iKAAAAABJRU5ErkJggg==\n", 90 | "text/plain": [ 91 | "
" 92 | ] 93 | }, 94 | "metadata": { 95 | "needs_background": "light" 96 | }, 97 | "output_type": "display_data" 98 | } 99 | ], 100 | "source": [ 101 | "# lstm plot\n", 102 | "d = read_pickle(\"./sgd-imdb-lstm_noise\")\n", 103 | "print(len(d))\n", 104 | "for i in [1,10,19]:#[3,9,15,19]:\n", 105 | " noises=d[i]\n", 106 | " num_bins = len(noises)\n", 107 | " plt.yticks(np.linspace(0,3.5,6))\n", 108 | " plt.xticks(np.linspace(0,3,4))\n", 109 | " plt.xlim(0,3)\n", 110 | " plt.ylim(0,3.5)\n", 111 | " n, bins, patches = plt.hist(noises, 500, density=1, facecolor='blue')\n", 112 | " plt.ylabel('Density (%)')\n", 113 | " plt.xlabel('Noise norm')\n", 114 | " plt.xticks(fontsize=20)\n", 115 | " plt.yticks(fontsize=20)\n", 116 | " plt.rc('axes', labelsize=20)\n", 117 | " #plt.savefig('text.png')\n", 118 | " plt.tight_layout()\n", 119 | " print(i)\n", 120 | " \n", 121 | " plt.savefig('lstm'+str(i)+'.png',dpi=300)\n", 122 | " plt.show()" 123 | ] 124 | }, 125 | { 126 | "cell_type": "code", 127 | "execution_count": 337, 128 | "metadata": { 129 | "scrolled": false 130 | }, 131 | "outputs": [ 132 | { 133 | "data": { 134 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3debxcRZn/8c8XAYEQQsgijkwIMEBUVJYriwEMIExEURBxG0BABhhgoog/ZYSRhBF3FsMYBRUQQVFBYBCXyCIoixgWxRAIBMK+RyCEJGzP74+qJk2n+3afe7tvn5v7fb9e/ar0qdN1nr6E+6Tq1KlSRGBmZlY2K3U7ADMzs3qcoMzMrJScoMzMrJScoMzMrJScoMzMrJRW7nYAZTR69OgYP358t8MwMxsSbrrppicjYkztcSeoOsaPH8+sWbO6HYaZ2ZAg6b56xz3EZ2ZmpeQEZWZmpeQEZWZmpeQEZWZmpeQEZWZmpeQEZWZmpeQEZWZmpeQEZWZmpeQEZWZmpeSVJGzASMv+7H0yzawZ96DMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUSpOgJK0n6UxJD0taKmm+pFMljSzQxh8kRS+v1Tr5HczMrH1KsZq5pI2A64CxwCXAHcDWwKeByZImRsRTBZqc1uD4S/0K1MzMBkwpEhQwg5ScpkTEaZWDkk4GjgJOBA5rtbGImNruAM3MbGB1fYhP0obAbsB84Ds11ccDi4D9JA0b4NDMzKyLytCD2jmXMyPileqKiFgo6VpSAtsWuKKVBiV9FNgAeAGYA1wZEUvbF7KZmXVaGRLUprmc26D+LlKC2oQWExRwfs37xyUdEREX9CE+MzPrgq4P8QEjcvlMg/rK8bVbaOsSYA9gPWB1YALw1fzZn0l6b6MPSjpE0ixJs5544omWAjczs84pQ4JqRrmMZidGxCkR8auIeCgilkTEnRHxReBo0nf9Si+fPSMieiKiZ8yYMe2J3MzM+qwMCarSQxrRoH6tmvP64gekKeabSxrej3bMzGyAlCFB3ZnLTRrUb5zLRveomoqIJcDC/NazAc3MBoEyJKircrmbpNfEk3s7E4HFwA19vYCkTYGRpCT1ZF/bGeqkZa8ytWVmK6auJ6iImAfMBMYDR9RUTyP1eM6JiEWVg5ImSJpQfaKkDSW9qbZ9SaOBs/Lb8yPCq0mYmQ0CZZhmDnA4aamj6ZJ2IT27tA2wE2lo79ia8+fksvrf3zsCP5B0NTAPWACMA3Yn3d+aBXy+U1/AzMzaqxQJKiLmSeoBTgAmk5LKI8B0YFpELGihmZuAc4GtgM1JkysWArcBPwdOj4gXOhC+Vakdsoumcy/NzOorRYICiIgHgANbPHe5OxcRcRtwQJvDMjOzLun6PSgzM7N6StODss6rHn7z0JuZlZ17UGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpeScI6yvs9mVlfuQdlZmal5ARlZmal5ARlZmalVOgelKThwETSTrWjgcXA48CtETG7/eGZmdlQ1TRBSVod+DjwKWBrlvW6Kre/I5/3JHAh8N28eaCZmVmfNUxQklYGpgDHAiOBJcANwF+AR4EFwOrAKGACsC1wGHCopMuBoyPi7x2N3szMVli99aDuADYAfgv8CLgkIpb21pikTUjbru8P3CLpUxFxTptitQHijQ3NrAx6S1C3A3tHxF9bbSwi5gJflDQVOJzUwzIzMyusYYKKiA/0tdGIeAE4ta+fNzMz80oS1m9eLcLMOqFfCUrSBsBb8tvbI+Le/odkZmbWxwd1JQ2X9DPgbuD/gEuBuyX9ND8r1Zc215N0pqSHJS2VNF/SqZJG9qW93OaOkl6WFJK+3Nd2zMxs4PW1B/W/wG7A8cBNwGrAB4BPAs+TnplqmaSNgOuAscAlpBmEWwOfBiZLmhgRTxVsczhp9uHzwJpFPmtmZt3Xa4KStEZEPF+nai/gPyLivKpjF0laI9cVSlDADFJymhIRp1Vd/2TgKOBE0jNWRXwbGAF8NX/ezMwGkWZDfLdJ2qnO8ZWBhXWOL6T48kkbknpj84Hv1FQfDywC9pM0rECbHwQOJD1o/HCReMzMrByaJagbgMslfa/m3tIVwP9K+rikCZLeIek40gO6lxeMYedczoyIV6orImIhcC2wBmmliqYkjQW+D1wcEecWjMXMzEqi1wQVEf8GfBB4HzBb0uRcdTjwJHAeMBu4GTgB+CtwZMEYNs3l3Ab1d+VykxbbO4P0vYoOCVqXSMteZmYVTYfjIuJXkq4BTgF+Lekc4DMRsaWkXUnr8AmYHRFX9CGGEbl8pkF95fjazRqSdBApoX40Ih4rEoSkQ4BDAMaNG1fko2Zm1gEt3S+KiGeBT0k6n9RDuV3S4RFxMfD7TgZIzarpDU+SxpNWr/hFRPy86EUi4gzSd6Onp8cr0JmZdVmh56Ai4vfAZsBFwIWSzpc0qp8xVHpIIxrUr1VzXiNnkvanOryf8ZiZWQm0lKAkjZa0laTREbEoIo4AdgK2AOZI+lg/Yrgzl43uMW2cy0b3qCq2JE1VfyI/mBuSAjgr1x+bj13cj1jNzGyANHsOak3gh8CHq45dCBwUEddIegfpGaNzc5I6LCIeLRjDVbncTdJK1TP5qnbwXUyaUdibc0iz/WptDOwI3Ep6qPiWgvGZmVkXNLsH9VVgH+Bs4EbgnaTnix4HjoyIJcDRkn5OGmK7XdJnI+LsVgOIiHmSZpKehToCOK2qehowDDg9IhZVDkqakD97R1U7U+q1L+kAUoK6LCKOazUuMzPrrmYJ6oPABRFxUH7/PUlr5eOvTiePiD9L2hyYCpxOSmhFHE5a6mi6pF2AOcA2pGHEuaRdfavNyaUnJpuZraCa3YMaBjxYc+wB6gylRcSLEXEsaQ29QiJiHtBDSmzbAEcDGwHTge2KrsNnZmaDX7Me1A2kZYYuBP4CbAXsSy/3g4rswFvzuQdIw4etnNtyzykPN57dl5hWBP19+NUPz5pZtzTrQX2aNEHhmlz+CVgCfKbDcZmZ2RDXaw8qIu7OExL2AMYB9wO/qp6wYGZm1gmtLHX0PPCzAYjFzMzsVf3a8t0Gr+p7S1GihZ16i6usMZtZZzS8ByXpaEmr9bVhSVtKem9fP29mZkNbb5MkvgLMk/QFSW9qpTEl/yrpItKsv3e0I0gzMxt6ehviextwMmk1iS9Luo40i28W8AjwD2A1YBRpy41tgV2AdYGnSA/ynt6xyK1tPJXczMqoYYKKiLnA+yW9i7QE0d7ADtTf9qLyK+5O4OvAWXk3XDMzsz5pZRbfdcB1kg4jrWm3PWnK+SjSs1GPA38D/hARszsYq5mZDSEtz+LLPaLL8svMzKyjCm1YaGZmNlCcoMzMrJScoMzMrJS8koQNel5hwmzF5B6UmZmVkhOUmZmVkof4rLR6W+HCq1+Yrfha7kFJ2kvS6zoZjJmZWUWRIb4LgfsknSBpXKcCMjMzg2IJagawBnAcaZXzSyW9X/Jgi5mZtV/LCSoijgT+CTiItKL5+4BLSL2qL7W6JYeZmVkrCs3ii4glEXF2RGwHvB34LrAmMBW4V9JFkia3P0wzMxtq+jzNPCL+XtWrOhB4DPgAcJmkeyV9TtKwVtuTtJ6kMyU9LGmppPmSTpU0skAb/0/Sr/Nnn5P0rKTbJJ0sab3CX9Iakpa9zMw6oV/PQeUEtD8wBXgTaV+ov5K24vgGcIekzVtoZyPgJlKiuxE4BbgH+DRwvaRRLYZ0KClhXk26Z/ZD0uaJRwGzJW3R8pczM7Ou6tNzUPkX/aHAx4HhwCLgB8CMiLhV0prA4cA0YDppH6nezADGAlMi4rSq65xMSi4nAoe1ENpmEbGkTrz/DpyR29m9hXbMzKzLFC0uXiZpDVJCOhTYitRbmgN8D/hRRDxb5zOnAZ+KiDV6aXdDYB4wH9goIl6pqhtO2l5ewNiIWNTa11ruGiOAp4G7I2LjZuf39PTErFmz+nKp0hlqQ3Bei89s8JF0U0T01B4v0oN6mNRbepn0TNSMiPhDk888BKzW5JydczmzOjlB2iRR0rXAbsC2wBUF4q22Ry7/1sfPm5nZACuSoBYCJwHfj4hHW/zMDOCnTc7ZNJdzG9TfRUpQm9BigpJ0MLAeaYbh24D3APcBx7TyeTMz674iCWr92h5OM3nYb7mhvxojcvlMg/rK8bULXPpgYJuq938BPhERdzf6gKRDgEMAxo3zQhlmZt1WZBbf5ZL27+0ESftKurKfMS3XbC5bvrsQEdtGhIDRpN4XwE29PaMVEWdERE9E9IwZM6bv0ZqZWVsUSVCTgPFNzlkfeHfBGCo9pBEN6teqOa9lEfFURPyelKQWA+dIWr1oO2ZmNvDavR/U6sBLBT9zZy43aVBfmXXX6B5VUxHxNHA9MAZ4a1/bMTOzgVM0QdUdZlOyPukZowcKtnlVLneT9Jp48jTziaTezw0F261VWSuwaAI1M7Mu6DVBSXpF0suSXs6HplbeV79Iv/TvATYHzi8SQETMA2aShg+PqKmeBgwDzql+BkrSBEkTamJdPz9TVe97HAq8k5Q8bysSn5mZdUezWXzXsKzXtCNwP+mB2lovk5YUuoK0okRRhwPXAdMl7UJ6AHgbYCfS0N6xNefPyWX1Y6hbAL+UdF3+zGOkJZe2JU01fw7YLyJexszMSq/XBBURkyp/lvQKcFZEnNDuICJinqQe4ARgMmmo8BHSMknTImJBC83cTFrDbwfSViDrAEtIPbuTgG9HRNHhRzMz65Iiz0FtQFouqCNy8jiwxXOXW8AnIu4Hjm53XLZiql4CyssjmZVTywkqIu7rZCBmZmbVGiYoSV8i3X/6TkQsyO9bERHxP22JzszMhqzeelBTSQnqZ8CC/L4VAThBWakMtVXdzVYEvSWonXJ5f817MzOzjmuYoCLi6t7em5mZdVK7lzoyMzNri5YTlKTxknaXNKzq2MqSpkn6q6TrJO3VmTDNzGyoKfIc1PHAB4A3VB07Dvjvqvc/l7RDRPR33TwzMxviigzxbQdcEREvAeSFXQ8H7gDGAVsDi4Cj2h2kFSMte5mZDVZFEtQbSNumV2xO2hDwOxHxYETMAi4hLcpqZmbWL0US1Cq8druNifl99Q66DwJvbENcZmY2xBVJUA8Cb696vzvwZETMqTo2Fni2HYGZmdnQVmSSxK+AoyR9i7RK+K7AWTXnTOC1w4BmA8qLwJqtOIokqG8AewKfze8fIs3sA9KGgcC7SFtemJmZ9UuR1cwfl/Q2YJd86OqIWFh1ypqk5PW7NsZnZmZDVJEeFBGxmDTUV69uNjC7HUGZmZl5qSMzMyulQj0oSesAB5Eeyh0JvK7OaRERu9Q5bmZm1rKWE5SkCcAfgDFAb2sUeO6UmZn1W5Ehvm+RnnP6OrAhsEpErFTnVa9XZWZmVkiRIb4dgMsi4oudCsbMzKyiSA9KwO2dCsTMzKxakR7UTcCmnQrE6vPKCH3X39Xc/bM3664iPagTgN0lTepEIJLWk3SmpIclLZU0X9Kpkka2+Plhkv5N0k8k3SFpkaSFkmZJOlrSqp2I28zMOqNID+qfSdtpzJT0U1KP6ul6J0bEOUWCkLQRcB1pEsYlpD2mtgY+DUyWNDEinmrSzA7AucAC4CrgYmAdYA/SBI8PSdolIpYUic3MzLqjSII6mzSFXMB++VU78KF8rFCCAmaQktOUiDjt1cakk0kbIJ4IHNakjUeBfYFfRMQLVW0MJ02PfxdwBHBSwdgGnDcaXDF4iNCsfxQt/p8j6ZOtNhoRP2o5AGlDYB4wH9goIl6pqhsOPEJKfGMjYlGr7dZc4xPAecCvImKPZuf39PTErFmz+nKptmiUoFr9JecEV0yjn2t/E4wTlFlrJN0UET21x4ssFtty0ilo51zOrE5O+ZoLJV0L7AZsC1zRx2u8mMuX+vh5MzMbYGVYi68yM3Bug/q7crlJP65xUC5/2482zMxsABVaiw9A0hhgb+DNwLCIOLjq+AbAbXnV81aNyOUzDeorx9cuGmuO60hgMnArcGYv5x0CHAIwbty4vlzKzMzaqFAPStKnSPeKvgP8J3BgVfUbgOuBT7QruMplc1l4FF/Sh4BTSRMo9o6IFxudGxFnRERPRPSMGTOmb5GamVnbtJygJO0KnEEaitsL+G51fUT8nbQf1J4FY6j0kEY0qF+r5ryWSNoTOB94HJgUEfcUjKt0pGUvG1j+2ZsNvCJDfF8gzah7d0Q8K2mLOuf8DdiuYAx35rLRPaaNc9noHtVyJO0D/ITUc9o5Iu5q8hEzMyuZIkN8PaRp2s/2cs6DwLoFY7gql7tJek08eZr5RGAxcEMrjeUp5T8FHiYlUycnM7NBqEiCWhVo9hzS2sDLRQKIiHnATGA86UHaatOAYcA51c9ASZqQ96d6jfys1o+B+4EdV4RhPTOzoarIEN98YKsm52zDsiG7Ig4nLXU0XdIuwJzc1k6kob1ja86fk8tX7whI2ok0S28lUq/sQC1/w+DpiDi1D/GZ9Ysf2jUrrkiCugT4vKR9IuIXtZWSDgTezvLJpKmImCeph7Qg7WRgd9L9runAtIhY0EIz67OsR3hQg3PuI83qMzOzkiuy1NFI4GbSorEXkmbd7Upa0HUH4EOkJYu26uuSRGVR1qWOqvX2n80zzYppZamjVs5v9+fNhop2LHX0D0nvJi0Eu09V1fRc/hH4xGBPTja0ObmblUehlSQi4n5gkqS3k6aTjyI9n3RDRNzUgfjMzGyIKrzUEUBE/I30zJOZmVlH9GUtvvWBMaSlh57IvSozM7O2auk5KEmjJZ0s6RHgHuDPwI3AvXmL9m9KWqeTgZqZ2dDSNEFJ2hiYRZqt9wbSg7iPA0/kP68LfBaYlTcfNDMz67deE1Reeug8YBxwNfAeYM2IeGNErAsMJ20meA1pJYhzOxqtmZkNGc16ULuR1uD7ObBLRFwZES9UKiNiaURcTtoV9wJgm7zqudmQUb3SeSurnntldLPWNEtQewNLgf+MXp7ozXVHkrZW/3D7wjMzs6GqWYLaErg2Ip5o1lBEPA78KX/GzMysX5olqH8mbULYqtmkNfGsw1odTjIzG6yaJai1gKcLtPc0aeKEmZlZvzRLUKtSbH+nV/JnzMzM+qWVlSS89rJZFQ+pmg2MVhLUVElTOx2ImZlZtVYSVNF/L7rHZWZm/dZrgoqIltbqMzMza7c+bbdhtiLxPSWzcnIPyczMSskJyszMSslDfF3koSWr/TvQeMVLs6HHPSgzMyslJygzMyul0iQoSetJOjNvIb9U0nxJp0oaWaCNXSWdJOkKSQskhaQ/dTJus3byAsBmy5TiHpSkjYDrgLHAJcAdwNakbeYnS5oYEU+10NQRwAeBJcDdQMvJzczMyqUsPagZpOQ0JSL2jIhjImJn4BRgU+DEFtv5OrAZsCawR0ciNTOzAdH1BCVpQ9LW8vOB79RUHw8sAvaTNKxZWxFxfUTMjogiK7CvEDw0NHT4v7UNFV1PUMDOuZwZEa9UV0TEQuBaYA1g24EOzMzMuqcMCWrTXM5tUH9XLjfpZBCSDpE0S9KsJ55ousO9mZl1WBkS1IhcPtOgvnJ87U4GERFnRERPRPSMGTOmk5cyM7MWlGIWXxOVkXY/Y29DSvU9Jq8wYUNRGXpQlR7SiAb1a9WcZ2ZmQ0AZEtSduWx0j2njXDa6R2VmZiugMgzxXZXL3SStVD2TT9JwYCKwGLihG8GZlUGjKeUeBrQVWdd7UBExD5gJjCetBFFtGjAMOCciFlUOSpogacKABWlmZgOuDD0ogMNJSx1Nl7QLMAfYBtiJNLR3bM35c3L5mn9XStoeODi/XTOXG0s6u3JORBzQzsDNzKwzSpGgImKepB7gBGAysDvwCDAdmBYRC1ps6l+AT9YcG1tz7ID+RWs2+LQyFOjhQiubUiQogIh4ADiwxXPrjshHxNnA2e2LyszMuqXr96DMzMzqcYIyM7NScoIyM7NScoIyM7NScoIyM7NScoIyM7NScoIyM7NScoIyM7NScoIyM7NScoIyM7NScoIyM7NSKs1afENBoz19zAaSF4W1wcI9KDMzKyUnKDMzKyUnKDMzKyXfgzJbQfTlHqfvi1qZuQdlZmal5ARlZmal5CG+DvMQiq0IWpma3pfp657ybr1xD8rMzErJCcrMzErJQ3wd4GE9W5H1d1iulf8/Gl2j9rMeblyxlaYHJWk9SWdKeljSUknzJZ0qaWTBdtbJn5uf23k4t7tep2I3M7P2K0UPStJGwHXAWOAS4A5ga+DTwGRJEyPiqRbaGZXb2QS4EjgfmAAcCLxP0nYRcU9nvoWZmbVTWXpQM0jJaUpE7BkRx0TEzsApwKbAiS228xVScjolInbJ7exJSnRj83XMzGwQUHR5sFXShsA8YD6wUUS8UlU3HHgEEDA2Ihb10s4w4AngFeCNEbGwqm6lfI3x+Rq99qJ6enpi1qxZff1Kvgdlg17tr4X+/J1uta3e7jW10nYr7fam6D2obtyz6uZ9st7+m/Q3Fkk3RURP7fEy9KB2zuXM6uQEkJPMtcAawLZN2tkOWB24tjo55XZeAWbmtzv1O2IzM+u4MiSoTXM5t0H9XbncZIDaMTOzEijDJIkRuXymQX3l+NqdbEfSIcAh+e1zku5scr1uGg082e0gmnCM7TPgcfZhSK9hjK221anFbmvOaelnWTSWNg/rdyTGDng1zjbEsn69g2VIUM1Uvnp/R1x7bScizgDO6Oc1BoSkWfXGa8vEMbbPYIhzMMQIgyPOwRAjDEycZRjiq/RsRjSoX6vmvE63Y2ZmJVCGBFUZSmt0b2jjXDa6t9TudszMrATKkKCuyuVueTr4q/I084nAYuCGJu3ckM+bmD9X3c5KwG411xvMBsNQpGNsn8EQ52CIEQZHnIMhRhiAOLv+HBSApN+REsiUiDit6vjJwFHA6RFxWNXxCQARcUdNO6eTJjqcHBFHVx2fAnwb+F1ETO7kdzEzs/YoS4KqXepoDrAN6ZmlucC7qpc6khQAEaGadmqXOroReDPwQeDx3M68Tn8fMzPrv1IkKABJ/wycAEwGRpFWkLgYmBYRC2rOrZugct06wPHAnsAbgaeA3wBfiogHO/kdzMysfUqToMzMzKqVYZKE9ULShyWdJumPkp6VFJLO7XZc1SSNknSwpIsk3S1psaRnJP1J0qdqJ790k6SvS7pC0gM5zgWSbpF0fB4iLiVJ++X/9iHp4G7HA5C3tIkGr0e7HV81STtIulDSI3kbnkckzZS0ewliO6CXn2Pl9XK34wSQ9L78c3sw//9zj6RfSNquE9cbDA/qDnXHAe8AngMeJG0fUjb7AN8lDcteBdwPvAH4EPAD4L2S9olydNePAm4Gfk+6LzmMtM7jVOAQSdtGxAPdC295efj7NNLfgTW7HE6tZ4BT6xx/bqADaUTSccD/kFY9+BXp7+loYAtgEvDrrgWX3ApMa1C3A2m90t8MXDj1Sfo68HnSbZOLST/PfyHd499b0v4R0d5/PEeEXyV+kSaKbExaCWMSaSWMc7sdV02MOwN7ACvVHF+XlKwC2LvbceaYVmtw/MQc54xux1gTl4DLSavxfzPHeHC348qxzQfmdzuOJjHuk39mvweG16lfpdsxNon/+hz/B7ocx7rAy8CjpJ0lqut2yjHe0+7rlmboxeqLiKsi4q7IfxPKKCKujIhLY/nV6B8FvpffThrwwOqIiCUNqn6ey40b1HfLFNI/AA4EGm43Y8vLQ8tfB54HPhE1uxwARMSLAx5YiyRtRurdPwRc1uVw1ifdEvpzRDxeXRERVwELgTHtvqiH+KzTKr8AXupqFM3tkcu/dTWKKpLeDHwN+HZEXCNp52af6YLXS9oXGEdKoH8DromIMtwzeRewAXAB8A9J7wM2A5YAN0bE9d0MrgWH5vKHJfh53gW8AGwtaXREvLqYraQdgeGkYb+2coKyjpG0MrB/fvvbbsZSS9LnSPdzRgA9wPakX65f62ZcFfln92PSEOkXuxxOb9YlxVntXkkHRsTV3Qioyjtz+RjpvuPbqislXQN8OCKeGOjAmpG0OrAvaQPWH3Q5HCJigaQvACcDt0u6mHQvaiPgA6Qh1EN7aaJPnKCsk75G+hfrryPid90OpsbnSBM5Kn4LHFCiX1ZfIt3E3z4iFnc7mAbOAv4IzCYN8WwIHElazeU3kraLiL92Mb6xuTwMuBd4D/Bn0nDVScC/Ar+gJMPPNT5C2hrosijJpJ2IOFXSfOBM4N+rqu4Gzq4d+msH34OyjsjLSx0N3AHs1+VwlhMR60Z60Htd0mzDDYFbJG3Z3chA0takXtNJZR6Giohp+f7jYxHxfET8PdKSZCeTdree2t0IeV0uReopXRERz0XEbGAv0qzYd3dqinQ/VfamO72rUVSR9HnScOnZpJ7TMGAr4B7gPEnfaPc1naCs7SQdQVr78HZgp6hZCaRM8i/Xi0hrQY4CzulmPFVDe3OB/+5mLP1QmRizY1ejgH/k8p7anlzulVZ69VsPaFRNSHoL6f7Zg3R/CjwAkiaRJpz8X0R8NiLuyf8ouZmU7B8Cjpa0YTuv6wRlbSXpM8D/An8nJadSPbDZSETcR0qob5U0uouhrElaS/LNwJLqhzVJS3gBfD8fq/f8URlUhnqGdTWKZVvwPN2gvpLAVh+AWIoo0+SIivfncrndICLiedK6pyuRhqXbxvegrG3yTdSvkR483LV6ps8g8U+57OYvhaXADxvUbUn6BfAn0i/fsg7/VYbM7ulqFHANafboxpJWjYgXauo3y+X8AY2qF5JWIw2Jv0Ljvwfd8PpcNppKXjle+zPun24+/OVX4YflJlHCB3VzbP+dY5sFrNPteBrEOAFYt87xlVj2oO613Y6zl/inUpIHdYG31vvvTJqAcFeO84sliPPcHMuXa47vSkoCTwNrdzvOqrj2y/Fe2u1YauL6SI7rUeBNNXXvzT/LxcCodl7XPaiSk7QnaWV2SDf0AbaTdHb+85MR8bkBD2eElEwAAAboSURBVKyKpE+SVqJ/mTSra4q03ELz8yPi7AEOrdZk4Jt5evE80jTZNwDvJk2SeJTXzk6yxvYBjpF0FWmG3ELSjfP3AauR7p18q3vhveqzpK17js3P69xISqJ7kf6+/ntENBoC7IbK5IiybVp4AWlFk/cAcyRdRPr/5c2k4T8Bx0TVtkjt4ARVfpsDn6w5tmF+AdxHmjLdTRvk8nXAZxqcczVp9k83XU76H38iaX3DtUkPl84lTUyYHiWe0FEyVwGbkoYctyPdb3qaNPz4Y+DHkf953U0R8bikbUhrWu5FWplhIWllhq9GRLOdugdMfjB7e0o0OaIiIl7JC+seAXyM9LNcA1hAinV6RMxs93W93YaZmZWSZ/GZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZdYik+Xn/HDPrAycoG1KqVge/Ly/MWe+c+fkcr7Ri1kVOUDZUjaPxskztskt+mVkfeKkjG1Lyvkr/IK3MvDKwUdRsC5KH5dYHVomIlwY8SDMD3IOyoel54H+AtVi2CWBLJH1E0jWSnpG0WNJtkv5L0uvrnLvcPShJq0qaIulmSf+Q9Hw+7xJJ76nTxgRJZ0t6QNJSSY9J+omkTQvEPCkPWU6VtLmkyyQ9na99taR3NfjcCElflXSnpCU53t81iLP6GlvnayzIx8ZX/zwkrSnplPydFku6Na/aj6SVJX1R0l35mvMkHdnqd7UVi3tQNqTkHtRDpBXY55CG+jaLiLlV58ynTg9K0leA/wKeJG0/8BxpL5y3klZr3zUiXqxph4gYX3XsJ8DHSTsOX0naQ+efSKtY/7J66xRJk4FfAqsAlwJ3A+sBHyJtbLhTpC23m33nSaTVxy8DdiZtdHhL/u57kzaZ2zwi7qz6zNrAtcBbgL/kWEeT9gVaE/iPiDi9zjVmkvYt+1O+xmjSvlAP55/HKsD9wDrA74FV889jDWA34HDS9hi/yd9xH2As8LGI+Fmz72ormG5vhOWXXwP5Ig3tPZj//OH8/pc158zPx1euOrZdPnY/VZsekoYJL6XOBn25nflV70eQNnabBbyuTmyjqv48kjQU+STwlprz3kpKjje3+J0n5fgCOKCm7tB8fEbN8dPz8dPJ/5DNxzcGniElj/ENrnFogzgqP9dLgddXHd8hH19ASoZrV9VtSEqgt3T7745fA//yEJ8NWRFxAak3sZek7ZucflAuvxwRj1a18RJwNCnxHNzskqSN3Zbm82vjqd7sbX/SflXHR8TtNefNBr4PbCHpLU2uWe3aWH7TyDNJ26JvXTkgaRVgX1IS/K+IeHWYJSLuAqaTej7717nGrVHVs2rgMxGxtKrNP5I2PRwJfCGqNhCMiHtIPbm3SXpd029oKxRPo7Wh7mjgOuAkSdtW/zKusWUur6ytiIi5kh4ENpC0djTYoTUinpV0KbAHcKukC0k7EP85Ip6vOX27XL5D0tQ6zW2SyzcDt9epr2dWnZhelPQYKTlUTCANuV0b9TdwvJK0AeAWdepubBLD0xExr87xh0nDrjfVqXuItBnmuvnPNkQ4QdmQFhHXS7qANNz3EaDRfY4RuXykQf0jpHs6I0g7yzbyUeALwCeAafnYkhzD5yLisXxsVC6bbUG/ZpP6ao3ieomUACpa+a6Qeni1Hq1zrNozvcRARNSrr9wHXKVJ27aC8RCfGRwDvAh8VdKqDc6p/OJct0H9G2vOqysiFkfE1IjYhJTQ9iVNKNiXNPGi9nrviAj18vpRk+/WF/35rp51ZW3jBGVDXh5ymkEaYvrPBqfdkstJtRWS/oU0u+7eRsN7Da77QEScB/wrcBewvaRKz+mGXO7QanttdCdpKv7mkkbWqd8pl01nEJr1hxOUWXICaQjsWOoPm52Zy+MkjakczDfuv0X6f+mHvV1A0hhJ29SpGgYMJw1lvZCPnZXjOV7S1rUfkLRSntrddhHxAnAe6edwQs11NwKmkHqcP+7E9c0qfA/KDIiIBfk5p280qL9O0jeAzwN/z/eMFpGeg9qMNEz3zSaXeRNwg6Q5pN7HA6SHhd9PGk6bHhEL8/WekvRh4KL8mSuA2aTZf+NIkyhGAXXXE2yDY0i9tyMlvZP0jFPlOajhwJERcW+Hrm0GOEGZVZtOelB0fL3KiPiCpFuAI0lTrFcB5pFmtJ2Uex69mU9auWISaZhsNOnZnztJCeH8mutdIentwOdIw4A7kHpYD5Nm0l1Y8Pu1LCfs7UgPJn8I+CzpoeIbgW9GxMxOXduswitJmJlZKfkelJmZlZITlJmZlZITlJmZlZITlJmZlZITlJmZlZITlJmZlZITlJmZlZITlJmZlZITlJmZldL/B4mVk68mZMkHAAAAAElFTkSuQmCC\n", 135 | "text/plain": [ 136 | "
" 137 | ] 138 | }, 139 | "metadata": { 140 | "needs_background": "light" 141 | }, 142 | "output_type": "display_data" 143 | }, 144 | { 145 | "data": { 146 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3debxcRZn/8c8XAYEEwpIgjhgCGIiKChjDEsGEjDGiLAou4wCCYGSACSiOMMIMBHeUxTCgIEJEUEAQ+CkqUVYFowZBEUKCgUDYIWEJkIQlz++Pqoam6b693O7b5977fb9e/TrpU6frPCe56edWnTpVigjMzMyKZpVuB2BmZlaNE5SZmRWSE5SZmRWSE5SZmRWSE5SZmRXSqt0OoIiGDx8eo0aN6nYYZmYD2s033/x4RIyoVe4EVcWoUaOYM2dOt8MwMxvQJN3bU7m7+MzMrJCcoMzMrJCcoMzMrJCcoMzMrJCcoMzMrJCcoMzMrJCcoMzMrJCcoMzMrJCcoMzMrJAKk6AkbSzpHEkPSlohaaGkUyWt10Qd10mKHl5rdPIaik565WVmVnSFmOpI0ubATcCGwBXAncA44HBgiqTxEbG4iSqn19j/Yq8CNTOzPlOIBAWcQUpO0yLitNJOSScDnwe+BhzcaGURcXy7AzQzs77V9S4+SZsBk4GFwOkVxccBzwL7ShrSx6GZmVkXFaEFtUvezoqIleUFEbFU0o2kBLY9cHUjFUr6BLAp8DwwF7gmIla0L2QzM+u0IiSoLfN2fo3yu0gJagsaTFDAhRXvH5V0aERcUusDkqYCUwFGjhzZ4GnMzKxTut7FBwzL26dqlJf2r9tAXVcAuwEbA2sCY4Bv5M9eJOmDtT4YEWdFxNiIGDtiRM31s8zMrI8UoQVVT2lQdNQ7MCJOqdg1D/iypAeB04CvA79ub3hmZtYJRWhBlVpIw2qUr1NxXCvOJg0x31rS2r2ox8zM+kgREtS8vN2iRvnovK11j6quiFgOLM1vPRrQzKwfKEKCujZvJ0t6VTy5tTMeWAbMbvUEkrYE1iMlqcdbrcfMzPpO1xNURCwAZgGjgEMriqeTWjznRcSzpZ2SxkgaU36gpM0kvamyfknDgXPz2wsjwrNJmJn1A0UZJHEIaaqjGZImkZ5d2g6YSOraO6bi+Ll5Wz6r3M7A2ZKuBxYAS4CRwK6k+1tzgC916gLMzKy9CpGgImKBpLHACcAUUlJ5CJgBTI+IJQ1UczNwPvBuYGvS4IqlwG3AxcCZEfF8B8I3M7MOKESCAoiIRcABDR77mvm4I+I2YP82h2VmZl3S9XtQZmZm1RSmBWXdU74+VNR9HNrMrG+4BWVmZoXkBGVmZoXkBGVmZoXkBGVmZoXkBGVmZoXkBGVmZoXkBGVmZoXkBGVmZoXkBGVmZoXkBGVmZoXkBGVmZoXkBGVmZoXkBGVmZoXkBGVmZoXkBGVmZoXkBGVmZoXkBGVmZoXkBGVmZoXkBGVmZoXkBGVmZoXkBGVmZoXkBGVmZoXkBGVmZoVUmAQlaWNJ50h6UNIKSQslnSppvV7UubOklySFpK+2M97+TnrlZWZWRKt2OwAASZsDNwEbAlcAdwLjgMOBKZLGR8TiJutcG/gR8BwwtL0Rm5lZpxWlBXUGKTlNi4g9I+LoiNgFOAXYEvhaC3V+FxgGfKN9YZqZWV9pqgWVWyXjgZHAcGAZ8Chwa0Tc3koAkjYDJgMLgdMrio8DpgL7SjoyIp5tsM49gAOAfSlIK9HMzJpT98tb0prAvwEHkrrdSq2u0t2LyMc9DlwKfC8ibmsihl3ydlZErCwviIilkm4kJbDtgasbiHdD4AfA5RFxvqT9m4jFzMwKomaCkrQqMA04BlgPWA7MBv4CPAwsAdYENgDGkBLIwcDnJP0OODIi/tFADFvm7fwa5XeREtQWNJCggLNISfTgBo41M7OC6qkFdSewKfAb0mCDKyJiRU+VSdoC2B/YD7hF0oERcV6dGIbl7VM1ykv7161TD5I+A+wBfCIiHql3fMVnp5K6Exk5cmQzHzUzsw7oaZDEHcC2EfGhiLi4XnICiIj5EfFlYDPgv0gtrN56VVdizYOkUcCpwM8i4uJmTxIRZ0XE2IgYO2LEiKaDNDOz9qrZgoqI3VutNCKeJyWLRpRaSMNqlK9TcVwt55AGbRzS4HnNzKzAijDMfF7eblGjfHTe1rpHVbItaaj6Y/nB3JAUwLm5/Ji87/LehWtmZn2hV0OwJW0KvC2/vSMi7mmhmmvzdrKkVcpH8pUNa19GGqDRk/OAtarsHw3sDNwK3Azc0kKMZmbWx1pKUDlxnA3sXdoFhKSLgakRsbTRuiJigaRZpJF6hwKnlRVPB4YAZ5Y/AyVpTP7snWX1TKsR6/6kBHVlRBzbaFxmZtZdrbag/o+UUI4jtUrWAHYHPk2aWujAJus7hDTV0QxJk4C5wHbARFLX3jEVx8/NW88kVwDl8/lFj0NZzMwa12OCkrRWRDxXpegjwH9ExAVl+y6TtFYuaypB5VbUWOAEYAqwK/AQMAOYHhFLmqnPzMz6v3otqNskHRQR11bsXxWo1o23tIE6q4qIRaTpiRo5tuGWU0TMBGa2EpOZmXVPvWQyG/idpB8A/1V2b+lq4P8kDSENOng9sBvpAd1fdipYMzMbPHocZh4R/06ameFDwO2SpuSiQ4DHgQuA24G/krrn/gYc1rFozcxs0KjbHRcRv5R0A2npi19JOg84IiK2lfR+0jx8Am6PiEbmyjMzM6uroftFEfE0cKCkC0mTsd4h6ZCIuBz4bScDNDOzwampmSQi4rfAVsBlwKWSLpS0QUcis5aUL+XeynLuXgrezIqioQQlabikd0saHhHPRsShpGeUtgHmSvpkR6M0M7NBp8cEJWmopIuAR4A/A49IuljS0Ii4AXgX8GPgfEmXS9qo8yGbmdlgUK8F9Q3gY6T1oA4lPU+0N/BNgIhYHhFHkubLG026N7V/p4I1M7PBo16C2gO4JCI+ExHfj4gDScu671F+UET8Cdga+B5wZkciNTOzQaVeghoC3F+xbxFVZg2PiBci4hhgXJtiMzOzQaxegpoN7CtpvKTVJe0A7EMPS19ExN/aGaCZmQ1O9RLU4aS1mG7I2z8Ay4EjOhyXtYmHjZtZf9Xjg7oR8c+89tJuwEjgPuCX5WszmZmZdUIjUx09B1zUB7GYmZm9rFdLvtvg4oUJzawv1bwHJelISWu0WrGkbSV9sNXPm5nZ4NbTIImvAwskHSXpTY1UpuQDki4D/kKaacLMzKxpPXXxvQM4mTSbxFcl3UQaxTeHtBz7E8AawAakJTe2ByYBGwGLSetC+aFdMzNrSc0EFRHzgQ9L2pE0zdFewE5AtbsPpbsT84BvAeeWrb5rZmbWtEZG8d0E3CTpYGBn4L2kIecbkJ6NehT4O3BdRNzewVjNzGwQaXgUX24RXZlfZmZmHdXUgoVmZmZ9xQnKzMwKyQ/q9iNFelDWc/uZWae5BWVmZoVUmAQlaWNJ50h6UNIKSQslnSppvSbq+C9Jv8qffUbS05Juk3SypI07Gb+ZmbVXIbr4JG0O3ARsCFwB3Ela+PBwYIqk8RGxuIGqPgc8A1wPPAKsBmwDfB44UNKEiLilA5dgZmZtVogEBZxBSk7TIuK00k5JJ5OSy9eAgxuoZ6uIWF65U9JngbNyPbu2JWIzM+uohrv4JH1E0uvaHYCkzYDJwELg9Iri44BnSav6DqlXV7XklF2ct6NbDNPMzPpYM/egLgXulXSCpJFtjGGXvJ0VESvLC/LDwTcCa5Hm+mvVbnn7917UYWZmfaiZBHUGKVEcS5rl/BeSPiz1esDxlnk7v0b5XXm7RaMVSjpI0vGSviPpKuBHwL3A0a2HaWZmfamZqY4Ok/RF4JOkwQgfIt3PeUDS2cAPI+KBFmIYlrdP1Sgv7V+3iToPArYre/8X4FMR8c9aH5A0FZgKMHJkOxuIZmbWiqaGmUfE8oiYGRE7AO8EvgcMBY4H7pF0maQpbY6x1EJr+NHUiNg+IgQMJ93fAri5p9gi4qyIGBsRY0eMGNF6tGZm1hYtPwcVEf+IiMOAfwEOIA3r3h24UtI9kr7YyMAGXmkhDatRvk7Fcc3EuDgifktKUsuA8ySt2Ww9ZmbW93r1oG5OQPsB04A3kVo7fyMtxXEicKekretUMy9va91jKo28q3WPqq6IeBL4IzACeHur9ZiZWd9pKUFJ2kbS94EHge+TksvZwLYRsS2pVXU0qYttRp3qrs3byZJeFY+ktYHxpNbP7FZiLVNatv7FXtZjDZJeeZmZNauZ56DWknSgpD+Tln2fCtxPmu3hTRExNSJuBYiIZyLiRFLSGttTvRGxAJgFjCKt3FtuOjAEOC8ini2LZYykMRXxbZKfqaoW++eA9wCLgNsavGQzM+uiZmaSeBBYG3iJ9EzUGRFxXZ3PPACs0UDdh5CmOpohaRIwlzQKbyKpa++YiuPn5m357+bbAD+XdFP+zCOkrsbtgXeQpkDaNyJeaiAeMzPrsmYS1FLgJOAHEfFwg585A/hpvYMiYoGkscAJwBTS8PWHSN2D0yNiSQPn+itwCrATaQj8+sBy4O4c93cjYlGDcZuZWZc1k6A2qZzpoZ6IeBp4usFjF5FGAzZy7GvuakTEfcCRzcRnZmbF1cwgid9J2q+nAyTtI+maXsZkDfAABDMb6JpJUBNIAxl6sgnwvlaDMTMzK2n3chtr4mHcfc6tKDMbiJpNUFWnG8oTxo4kDW7wQAQzM+u1Hrv4JK2U9JKk0tDs40vvy1+kVtPdwNbAhR2O2czMBoF6LagbeKXVtDNwH2lhwUovAYuBq0kP55qZmfVKjwkqIiaU/ixpJXBuRJzQ6aDMzMyauQe1KfBkpwIxMzMr18yChfd2MhAzM7NyNROUpP8l3X86PSKW5PeNiIj4SluiMzOzQaunFtTxpAR1EbAkv29EAE5QZmbWKz0lqIl5e1/FezMzs46rmaAi4vqe3puZmXVSr5Z8NzMz65RmVtQdJWlXSUPK9q0qabqkv0m6SdJHOhOmdUP5jOme78/M+lozz0EdB+wOvKFs37HA/5S9v1jSThExux3BmZnZ4NVMF98OwNUR8SKApFVIS7XfSZoodhzwLPD5dgdpZmaDTzMJ6g1A+cO6WwPDSc9J3R8Rc4ArgPe0MT4zMxukmklQq/Hq5TbG5/flK+jeD7yxDXGZmdkg18w9qPuBd5a93xV4PCLmlu3bEHi6HYFZ/9TOwRTldUXVlcjMbCBrJkH9Evi8pO8Ay4H3A+dWHDOGV3cDmpmZtaSZBHUisCfwhfz+AdLIPgAkbQLsCJzStuhswOhta8itKbPBp5nZzB+V9A5gUt51fUQsLTtkKCl5XdXG+MzMbJBqpgVFRCwjdfVVK7sduL0dQZmZmXmqIzMzK6SmWlCS1gc+Q3oodz3gdVUOi4iYVGV/vbo3Bk4ApgAbAA8BlwPTI+KJBj4/hHSP7EPAtsCbgZXAPOCnwGkR8XyzcZmZWXc0nKAkjQGuA0YAPQ0mbvoWtqTNgZtIw9SvIM1OMQ44HJgiaXxELK5TzU7A+aS1q64lJbf1gd2A7wAflTQpIpY3G183eQ48MxusmmlBfYeUQL4JnAUsioiX2hTHGbnuaRFxWmmnpJNJUyd9DTi4Th0PA/sAPytvKUlam5RYdwQOBU5qU8xmZtZBzdyD2gm4MiK+HBEL25WcJG0GTAYWAqdXFB9Hmt9v3/JZ1KuJiFsj4oLKbrw80rCUlCa0I2YzM+u8ZhKUgDs6EMMueTsrIlaWF+TkciOwFrB9L87xQt6+2Is6zMysDzWToG4GtuxADKU659covytvt+jFOT6Tt7/pRR1mZtaHmklQJwC7SprQ5hiG5e1TNcpL+9dtpXJJh5FGBt4KnNPDcVMlzZE057HHHmvlVGZm1kbNDJJ4M2mE3SxJPyW1qJ6sdmBEnNeG2EpK49haGR34UeBU0gCKvSLihVrHRsRZpMEfjB071pPpmJl1WTMJaiYpSQjYN78qv8iV9zWToEotpGE1ytepOK4hkvYELgQeBSZGxN3NfN7MzLqrmQR1QIdimJe3te4xjc7bWveoXkPSx4CfkFpOu0TEXXU+YmZmBdPMZLE/6lAM1+btZEmrlI/ky88wjQeWAbMbqUzSp0gtuAdwy8nMrN/q+lx8EbEAmAWMIj1IW246MAQ4LyKeLe2UNCbPbPEqkj4N/Bi4D9i5aMlJeuXVTJmZ2WDU1Fx8AJJGAHsBbwWGRMRBZfs3BW7Ls5434xDSVEczJE0C5gLbARNJXXvHVBxfWsX35a9zSRNJo/RWIbXKDtBrv+2fjIhTm4zNzMy6oNnJYg8EZgBr8MqAiINy8RuAPwJTgR82U29ELJA0llcmi92VNFnsDNJksUsaqGYTXmkRfqbGMfeSRvWZmVnBNdzFJ+n9pGHY84GPAN8rL4+If5DWg9qzlUAiYlFEHBARb4yI1SNik4g4vFpyighFhCr2zSzt7+E1qpXYzMys7zXTgjqK1Kp5X0Q8LWmbKsf8HdihLZHZoOHl3M2smmYGSYwFfhkRT/dwzP3ARr0LyczMrLkEtTppZvGerAu0awkOMzMbxJpJUAuBd9c5ZjteefDWzMysZc0kqCuAnfIsDa8h6QDgncCl7QjMzMwGt2YGSZwIfBL4qaS9yXPn5dnCdwI+Sloa47SaNZiZmTWomamOnpD0PtI0QuWtqBl5+3vgU+UzPpiZmbWqqQd1I+I+YIKkd5KGk29AmmV8dkTc3IH4zMxskGp6qiOAiPg76Zkns6b1NN+g5yI0s5JW5uLbBBhBmubosdyqMjMza6uGRvFJGi7pZEkPAXcDfwL+DNwj6UFJ35a0ficDNTOzwaVugpI0GpgDHE6aEPYl0iq1j+U/bwR8AZgjabPOhWpmZoNJjwlK0irABcBI4HrgX4GheULXjYC1gcnADaT1nM7vaLRmdXhdLbOBo14LajJpDr6LgUkRcU1EPF8qjIgVEfE7YBfgEmC7POu5mZlZr9RLUHsBK4D/jKg9z3QuOwx4Adi7feGZmdlgVW8U37bAjRHxWL2KIuJRSX/In7E63AVlZtazei2oN5MWIWzU7aSVbc3MzHqlXoJaB3iyifqeJA2cMDMz65V6CWp1mlvfaWX+jJmZWa80MpOEF+HuY74/ZWbWWII6XtLxnQ7EzMysXCMJqtnf593iMjOzXusxQUVEMyvumpmZtY0TkJmZFVJL60GZdVP5IJLa85uYWX/nFpSZmRVSYRKUpI0lnZPXl1ohaaGkUyWt10Qd75d0kqSrJS2RFHn6JTMz62cK0cUnaXPgJmBD4ArgTmAcaQ2qKZLGR8TiBqo6FNgDWA78E2g4uZmZWbEUpQV1Bik5TYuIPSPi6IjYBTgF2BL4WoP1fAvYChgK7NaRSM3MrE90PUHlVXgnAwuB0yuKjwOeBfaVNKReXRHxx4i4PSKamZ6po7yAnplZa7qeoEiLHQLMioiV5QURsRS4EVgL2L6vAzMzs+4pQoLaMm/n1yi/K2+36GQQkqZKmiNpzmOP1V3+yszMOqwICWpY3j5Vo7y0f91OBhERZ0XE2IgYO2LEiE6eyszMGlCEBFVP6e6NH8k0MxtEijDMvNRCGlajfJ2K48xe1ujgk1qzT3hWCrPiKkILal7e1rrHNDpva92jMjOzAagICeravJ0s6VXxSFobGA8sA2b3dWBmZtY9XU9QEbEAmAWMIs0EUW46MAQ4LyKeLe2UNEbSmD4L0szM+lwR7kEBHEKa6miGpEnAXGA7YCKpa++YiuPn5u2r7kBIei9wUH47NG9HS5pZOiYi9m9n4FZcfjjarH8rRIKKiAWSxgInAFOAXYGHgBnA9IhY0mBVbwE+XbFvw4p9+/cuWjMz6wuFSFAAEbEIOKDBY6v+bhwRM4GZ7YvKzMy6pev3oMzMzKpxgjIzs0IqTBef2UDiB4DNes8tKDMzKyQnKDMzKyR38ZllPXXLucvOrO+5BWVmZoXkFpQNOr2dYcKtKbO+4RaUmZkVkhOUmZkVkhOUmZkVkhOUmZkVkhOUmZkVkhOUmZkVkoeZm1XRjcUOPXzd7NXcgjIzs0JygjIzs0JyF59Zm3SqW9BdfzZYuQVlZmaF5BZUB3TjBrt1Ryv/1m4RmTXGLSgzMyskt6DMOqynVlZvWtuVn3VrzAYat6DMzKyQnKDMzKyQ3MVnVkDtHHzRSl1F7C50l+bgU5gWlKSNJZ0j6UFJKyQtlHSqpPWarGf9/LmFuZ4Hc70bdyp2MzNrv0K0oCRtDtwEbAhcAdwJjAMOB6ZIGh8RixuoZ4NczxbANcCFwBjgAOBDknaIiLs7cxVmA1Mjw+L729D5TsXrVl57FaUFdQYpOU2LiD0j4uiI2AU4BdgS+FqD9XydlJxOiYhJuZ49SYluw3weMzPrBxRdTvGSNgMWAAuBzSNiZVnZ2sBDgIANI+LZHuoZAjwGrATeGBFLy8pWyecYlc/RYytq7NixMWfOnFYvyQ/q2qDQ2/tcjdRbrqfWSa3zt7PF18g5evp7aOQ8fdGy62297a1LN0fE2FrlRWhB7ZK3s8qTE0BOMjcCawHb16lnB2BN4Mby5JTrWQnMym8n9jpiMzPruCIkqC3zdn6N8rvydos+qsfMzAqgCIMkhuXtUzXKS/vX7WQ9kqYCU/PbZyQtBh6vc87+bDi+vv6qENfWwdnbG7q+Rs7frmPa8fmy49p2fa1oZ71V6mr2Z3OTngqLkKDqKf0V9LZHtsd6IuIs4KyXD5bm9NQ32t/5+vqvgXxt4Ovrz9p9bUXo4iu1bIbVKF+n4rhO12NmZgVQhAQ1L29r3Rsanbe17i21ux4zMyuAIiSoa/N2ch4O/rI8zHw8sAyYXaee2fm48flz5fWsAkyuOF89Z9U/pF/z9fVfA/nawNfXn7X12rr+HBSApKtICWRaRJxWtv9k4PPAmRFxcNn+MQARcWdFPWeSBjqcHBFHlu2fBnwXuCoipnTyWszMrD2KkqAqpzqaC2xHemZpPrBj+VRHkgIgIlRRT+VUR38G3grsATya61nQ6esxM7PeK0SCApD0ZuAEYAqwAWkGicuB6RGxpOLYqgkql60PHAfsCbwRWAz8GvjfiLi/k9dgZmbtU5gEZWZmVq4IgyQKo11LfhSNpA0kHSTpMkn/lLRM0lOS/iDpwMrBKQOBpH0lRX4d1O142kHSTpIulfRQ/vl8SNIsSbt2O7bekvShfC3355/PuyX9TNIO3Y6tEZL2lnSapN9Lejr/3J1f5zM7SvqVpCWSnpP0d0lHSHpdX8XdqGauT9JoSUdJukbSIknPS3pE0hWSmppqrj88qNsn2rXkR0F9DPgeqdv0WuA+4A3AR4GzgQ9K+lgMkOZ07i4+DXgGGNrlcNpC0rHAV0hP6f+S9G85HNgGmAD8qmvB9ZKkbwFfInXHX066xreQ7h3vJWm/iOjxy74AjgXeRfqZu5+0zE9NkvYALgWWAxcBS4DdSCs4jCf9ny2SZq7vK8AngDtIP5dLSFPR7Q7sLunwiJjR0Fkjwq/0vXwVaZaJ/6zYf3Le//1ux9iLa9uF9MO/SsX+jUjJKoC9uh1nm65VwO9Is9d/O1/bQd2Oq5fX9LF8Hb8F1q5Svlq3Y+zFtW0EvAQ8TFqxoLxsYr7uu7sdZwPXMZH0rKVIvzAEcH6NY9chDdpaAYwt278G6ZfkAD7Z7WvqxfXtD2xTZf/7gOfzdb+xkfMOuK6dVuQlPyaTlvw4vaL4OOBZYN+8pEe/ExHXRMQv4rWzxT8MfD+/ndDngXXGNFJCPoD079av5e7XbwHPAZ+Kipn6ASLihT4PrH02Id1q+FNEPFpeEBHXAkuBEd0IrBkRcW1E3BX5m7iOvUnXdGFEvLyuT0QsJ7VUAP6jA2G2rJnri4iZEXFLlf3XA9cBqwM7NnJeJ6ikXUt+9EelL7cXuxpFG0h6K/BN4LsRcUO342mTHYFNSV0lT+R7NUdJOry/3J+p4y7Sb9XjJA0vL5C0M7A2qUU8kJS+b35TpewG0i8jO0p6fd+F1Gea+r7xPaikkaU6JpOer7q6TyLqA5JWBfbLb6v9Z+k38rX8mNRl+eUuh9NO78nbR4C/Au8oL5R0A7B3RDzW14G1Q0QskXQUqSv9DkmXk+5FbU66Z/Fb4HNdDLETan7fRMSLku4B3g5sRnomdECQtAkwiZSAG/oF0gkqadeSH/3NN4GtgF9FxFXdDqaX/pc0YOC9EbGs28G00YZ5ezBwD/CvwJ9IXWMnAR8AfkY/7qKNiFMlLQTOAT5bVvRPYGZl198AMOi+b3Jr8ALg9cCXIuKJRj7nLr7GtGvJj8LI0z8dSRqtuG+Xw+kVSeNIraaTIuKP3Y6nzUpDjkVqKV0dEc9ExO3AR0gjqt7Xn7v7JH0JuASYSWo5DQHeDdwNXCDpxO5F1xUD6vsmD5v/MWl04kXAdxr9rBNUMqiW6pB0KGluwjuAiVExU0d/Uta1Nx/4ny6H0wml3zTvjoi/lRfklmKp5TuuT6NqE0kTSINA/l9EfCEi7o6I5yLir6QE/ABwZB7INFAMmu+bnJzOJ41EvRjYp8GBJIATVMmgWapD0hHA/wH/ICWnh7scUm8NJf27vRVYXvZwbpBGYAL8IO87tWtRtq70s/lkjfJSAluzD2LphA/n7WtWGYiI50jzaa5C6r4dKGp+3+RfuDYlDSK4uy+Dard8LT8FPgn8hDQKtanBWL4HlbxqyY/ykXxNLvlRaPlm9DeBW4H3R0TXlw1vgxXAD2uUbUv6YvsD6UuhP3b/3UD6shotafWIeL6ifKu8XdinUbVPaaRaraHkpf2V192fXQP8O2ne0Z9WlO1MGjF8Q0Ss6OvA2kXS6qQW0x7AecABlSOkG9LtB8CK8mIAP6ibr+N/8nXMAdbvdjx9dM3HMzAe1D0/X8dXK/a/H1hJal2t2+04W7y2j+drexh4U0XZB/P1LQM26HasTVzTBOo/qPsY/ehB3Sav7/XAlfmYs6mYIKCZlyeLzZpd8qM/kfRp0g3ol0hTAFXr214YETP7MKyOk3Q8qZvvsxFxdpfDaZmkDUnP4r0F+D2p22sT0j2aIHWd/Kx7EQbifhIAAAazSURBVLYuP4h8FWl04lLgMlKyeiup+0/AERHx3a4F2QBJe5JWUIA0O8YHSF10v8/7Ho+IL1YcfwlpqqMLSdMB7U4agn4J8PEo0JdzM9cn6VzSbBKPA2dQfbDHdRFxXd0TdzsbF+kFvBk4lzTP2fPAvaTBBP26xcErLYmeXtd1O84OXne/bkHla1mf1Jq/J/9sLib9IrV9t2Nrw7WtBhxB6kJ/mtSl+ShpzsHJ3Y6vwWuo939sYZXPjCc/gE1qJd5GWqD1dd2+nt5cH2m2iHrfN8c3cl63oMzMrJA8is/MzArJCcrMzArJCcrMzArJCcrMzArJCcrMzArJCcrMzArJCcrMzArJCcqsAyQtzGscmVmLnKBs0Cib6fxeSWvUOGZhPsYTKZt1mROUDUYjSVPrdNKk/DKzFnmqIxs08hpRT5DmAlsV2DwqlhzJ3XKbAKtFk2vXmFl7uQVlg81zwFdISx4cV+fYV5H0cUk3SHpK0jJJt0n6b0mvr3Lsa+5BSVpd0jRJf5X0hKTn8nFXSPrXKnWMkTRT0iJJKyQ9IuknkrZsIuYJucvyeElbS7pS0pP53NdL2rHG54ZJ+oakeZKW53ivqhFn+TnG5XMsyftGlf99SBoq6ZR8Tcsk3ZpnykbSqpK+LOmufM4Fkg5r9Fpt4HELygaN3IJ6gLRi6VxSV99WETG/7JiFVGlBSfo68N+kJQQuAZ4hrVf0duB60gKQL1TUQ0SMKtv3E+DfSKsZX0OawfpfgPcCP49XL8cwBfg5aabvXwD/BDYGPkpaR2hipGXR613zBNKCnFcCu5AWbbwlX/tepJnRt46IeWWfWZe0vMfbgL/kWIeT1m4aCvxHRJxZ5RyzSGsF/SGfYzjw5Yh4MP99rAbcR5qZ/bfA6vnvYy1gMnAIaYmbX+dr/Bhp+ZtPRsRF9a7VBqBuT+Pul1999SJ17d2f/7x3fv/zimMW5v2rlu3bIe+7D9iobP+qpOQRpC/iynoWlr0fRlp8bw5VllOgbEE+YD1SV+TjwNsqjns7KTn+tcFrnsArSxzsX1H2ubz/jIr9Z+b9Z5J/ic37R5PWElsBjKpxjs/ViKP09/oL4PVl+3fK+5eQkuG6ZWWbkRLoLd3+2fGrOy938dmgFBGXkFoTH5H03jqHfyZvvxoRD5fV8SJwJCnxHFTvlKTF91bk4yvjKV8Mcz9gXeC4iLij4rjbgR8A20h6W51zlrsxXrsg5TmktZfGlXZIWg3Yh5QE/zsiXu5iiYi7gBmkls9+Vc5xa5S1rGo4IsqWMo+I35PWuFoPOCoiniwru5vUknuHpNfVvUIbcDyU1gazI0mrKJ8kafvyL+MK2+btNZUFETFf0v3AppLWLf+CrTjuaUm/AHYDbpV0KWk10j9FxHMVh++Qt+/KqwJX2iJv3wrcUaW8mjlVYnpB0iOk5FAyhtTldmNELKlSzzXAscA2Vcr+XCeGJyNiQZX9D5K6XW+uUvYA8DrSKq4P1KnfBhgnKBu0IuKPki4hdfd9HKh1n2NY3j5Uo/wh0j2dYUDVBJV9AjgK+BQwPe9bnmP4YkQ8kvdtkLefrXMJQ+uUl6sV14ukBFDSyLVCauFVerjKvnJP9RADEVGtvHQfcLU6ddsA5C4+G+yOBl4AviFp9RrHlL44N6pR/saK46qKiGURcXxEbEFKaPuQBhTsQxp4UXm+d0WEenj9qM61taI31+oRV9ZWTlA2qOUupzNIXUz/WeOwW/J2QmWBpLeQRtfdU6t7r8Z5F0XEBcAHgLuA90oqtZxm5+1OjdbXRvNIQ/G3lrRelfKJeVt3BKFZbzlBmcEJpC6wY6jebXZO3h4raURpZ75x/x3S/6Mf9nQCSSMkbVelaAiwNqkr6/m879wcz3GSxlV+QNIqeWh320XE88AFpL+HEyrOuzkwjdTi/HEnzm9WzvegbNCLiCX5OacTa5TfJOlE4EvAP/I9o2dJz0FtReqm+3ad07wJmC1pLqn1sYj0sPCHSd1pMyJiaT7fYkl7A5flz1wN3E4a/TeSNIhiA6DqfIJtcDSp9XaYpPeQnnEqPQe1NnBYRNzToXObvcwJyiyZQXpQdFS1wog4StItwGGkIdarAQtII9pOyi2PniwkzVwxgdRNNpz07M88UkK4sOJ8V0t6J/BFUjfgTqQW1oOkkXSXNnl9DcsJewfSg8kfBb5Aeqj4z8C3I2JWp85tVs4zSZiZWSH5HpSZmRWSE5SZmRWSE5SZmRWSE5SZmRWSE5SZmRWSE5SZmRWSE5SZmRWSE5SZmRWSE5SZmRXS/wcD6r3nY5K5hgAAAABJRU5ErkJggg==\n", 147 | "text/plain": [ 148 | "
" 149 | ] 150 | }, 151 | "metadata": { 152 | "needs_background": "light" 153 | }, 154 | "output_type": "display_data" 155 | }, 156 | { 157 | "data": { 158 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZxcVZn/8c83LCIJhCVBGJEEEIgKChjZIpoEjQyOggKOOoBsAgITUBxhhJFFGZURwaAoqBARFBUUfoozRFkFRA2LIoSAgbAvgbCGJGzP749zihRFVVfd6qqu293f9+tVr0vdc+ve5zSdevqce+45igjMzMzKZkSvAzAzM6vHCcrMzErJCcrMzErJCcrMzErJCcrMzEpp+V4HUEZjxoyJ8ePH9zoMM7Mh64YbbngsIsb2dYwTVB3jx49n9uzZvQ7DzGzIknRPs2PcxWdmZqXkBGVmZqXkBGVmZqXkBGVmZqXkBGVmZqXkBGVmZqXkBGVmZqXkBGVmZqXkBGVmZqXkmSTsVaRl/+21LM2sl9yCMjOzUnKCMjOzUnKCMjOzUnKCMjOzUnKCMjOzUnKCMjOzUnKCMjOzUnKCMjOzUnKCMjOzUnKCMjOzUnKCMjOzUipNgpK0rqSzJD0oaamk+ZJOlbR6P875HkkvSQpJX+lkvGZm1l2lmCxW0obAdcBawMXA7cBWwGHAjpImRcTjBc+5CvAj4DlgVGcjNjOzbitLC+p0UnKaHhG7RMRRETEVOAXYBDixjXN+CxgNfLVzYQ5N0rKXmVlZ9DxBSdoAmAbMB75TU3wssAjYU9LIAufcGdgHmA482JlIzcxsIPU8QQFT83ZWRLxcXRARzwDXAisD27RyMklrAd8HLoqIczsZqJmZDZwyJKhN8vaOBuV35u3GLZ7vTFK9DupPUENRdVeeu/XMrOzKkKBG5+1TDcor+1drdiJJ+wI7AwdHxCNFgpB0gKTZkmYvWLCgyEfNzKwLypCgmqn8jd/nAuSSxgOnAr+IiJ8XvUhEnBkREyNi4tixYwsHORS5lWVmvVSGBFVpIY1uUL5qzXGNnAUsBg7uRFBmZtZbZUhQc/O20T2mjfK20T2qii1JQ9UX5AdzQ1IAZ+fyo/O+i/oXrpmZDYQyPKh7Rd5OkzSieiRffth2EqlldH2T85xDGu1XayPgPcDNwA3ATf2O2MzMuq7nCSoi5kmaRXoW6hDgtKri44GRwBkRsaiyU9KE/Nnbq84zvd75Je1NSlCXRMQxHa+AmZl1Rc8TVHYwaaqjGZJ2AOYAWwNTSF17R9ccPydvffvezGyIKsM9KCJiHjARmElKTEcAGwIzgG2LzsNnZmaDX1laUETEfaTpiVo5tuWWU0TMJCU+MzMbRErRgjIzM6tVmhaUta/6Qdro83FmM7PBwy0oMzMrJScoMzMrJXfxDSLuyjOz4cQtKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyU/B1UCqpn6tvoZp9oyM7Phwi0oMzMrJScoMzMrJScoMzMrpUL3oCStAkwC1gPGAIuBR4GbI+LWzodnZmbDVdMEJen1wCeA/YCtWNbqqty+j3zcY8CFwHcj4pbOh2pmZsNJwwQlaXlgOnA0sDqwBLge+AvwMLAQeD2wJjAB2AY4CDhQ0u+BIyLi712NfojyyD0zs75bULcD6wP/B/wIuDgilvZ1MkkbA3sDewE3SdovIs7pUKxmZjaM9JWgbgN2jYi/tnqyiLgD+KKk44CDSS0sMzOzwhomqIj4cLsnjYjngVPb/byZmZmHmZuZWSn1a6ojSesDb81vb4uIu/sfkvWHl4U3s6GirQSVn4f6AbBbZRcQkn4OHBARz3QoPjMzG6babUF9G5gGHAvcAKwEfBj4FPAc6ZkpMzOztvWZoCStHBHP1Sn6CPCZiDivat+vJK2cy5ygzMysX5oNkrhF0pQ6+5cH6nXjPYOX8DAzsw5olkyuB34v6fvAf1TdW7oM+LakkcBNwOuAD5Ee0P1Nt4I1M7Pho88WVET8G7Az8EHgVkk75qKDgceA84BbgRuBE4C/Aod2LVrrGWnZy8xsIDTtjouI30i6GjgF+K2kc4DDI2JLSe8nzcMn4NaIuKy74ZqZ2XDR0v2iiHga2E/S+cCZwG2SDo6Ii4DfdTNAMzMbngrNJBERvwM2BX4FXCjpfElrdiIQSetKOkvSg5KWSpov6VRJqxc4x39I+m3+7LOSnpZ0i6RvSlq3E3GamdnAaClBSRoj6Z2SxkTEoog4BJgCbAHMkfTx/gQhaUPS81T7AH8mdSfeBRwG/LFAEjwQ+CfgKuB04IfA48BnSffQtuhPnGZmNnCaPQc1ivQlv1vVvguBfSPiaknvAE4Ezs1J6qCIeLiNOE4H1gKmR8RpVdf6Jim5nEhaa6qZTSNiSZ16fJrUNXkisFMb8ZmZ2QBr1oL6KrA7aT2oQ4CZpGT1NYCIWBIRR5CWgd+IdG9q7yIBSNqANCvFfOA7NcXHAouAPfOQ9j7VS07Zz/N2oyKxmZlZ7zRLUDsDF0TEvhHxvYjYj7Ss+87VB0XEn4DNge8CZxSMYWrezoqIl2vO+wxwLbAyacXedn0ob//Wj3OYmdkAajaKbyRwf82++1iWVF4RES8AR+cJY4vYJG/vaFB+J6mFtTHpAeGmJO0PrAuMAjYD3gfcAxxVMLbSauV5JD+zZGaDWSszSeyZ7zv9BXgnsEfeX1eRFXiz0Xn7VIPyyv7VCpxzf2Drqvd/AT4ZEf9o9AFJBwAHAKy33noFLmVmZt3QrIvvMGAxcHXeXgMsAQ7vclzVKu2Allc3iohtIkLAGFLrC+CGqpkw6n3mzIiYGBETx44d2360ZmbWEX22oCLiH5ImkO7hrAfcC/wmIhZ1MIZKC2l0g/JVa45rWUQ8DvxO0l+A24FzJI2LiMXFwzQzs4HUylRHzwE/62IMc/N24wbllZF3je5RNRURT0r6I7AL8DZgdrvnMjOzgVFoJokuuSJvp0l6VTx55d5JpO7Fhve9WvTGvH2xn+cxM7MB0DBBSTpC0krtnljSlpL+udlxETEPmAWMJz1rVe140kjCc6q7FSVNyF2P1dcbl5+pqhfLgcC7SCMQbylSDzMz642+uvj+G/icpBnAuRHxQLOTSRJpUMJBpCXgjwb+t4U4DgauA2ZI2gGYQxqFN4XUtXd0zfFzKpes2rcF8EtJ1+XPPAKsSXp+ajPgWWDPiHiphXjMzKzH+kpQmwHfJM0m8ZX8xX8N6f7NQ8ATwEqkJDCBlAh2ANYmzX93KC0+tBsR8yRNJK0ptSNpOqKHgBnA8RGxsIXT3Eiaw2970vpVa5BGHN4FnAx8KyLuayUeMzPrPUX0PXpb0nakrrddgRWpP9y70pKZS5pN4uyq1XcHnYkTJ8bs2QM3jmKwPVDb5FfGzKwpSTdExMS+jmllFN91wHWSDgLeA7ybNOR8TdLghUdJUwhdGRG39jtqMzMzWlywEF6ZF++S/DIzM+uqMgwzNzMzew0nKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzK6WWE5Skj0harpvBmJmZVRRpQV0I3CPpBEleE93MzLqqSII6HVgZOAaYJ+nXkv4lz2BuZmbWUS0nqIg4FPgnYF/SjOYfBC4mtaq+JOmNfX3ehi5p2cvMrFMKDZKIiCURMTMitgXeTpq5fBRwHHC3pF9J2rHzYZqZ2XDT9ii+iPh7VatqH9ICgR8GLpF0t6TPSxrZoTjNzGyY6dcw85yA9gKmA28krQv1V9JSHCcBt0vavL9BmpnZ8NNWgpK0haTvAQ8C3wM2Bn4AbBkRW5JaVUcBY0ir4pqZmRXS8npQklYGPgEcCLyT1FqaQ0pQP4qIpyvHRsSzwEmS3gTs19GIzcxsWGg5QZFaS6sAL5GeiTo9Iq5s8pkHgJXaC80Gu+pRfV4m3syKKpKgngFOBr4fEQ+3+JnTgZ8WjsrMzIa9IglqXES8XOTkudvv6aYHmpmZ1SgySOL3kvbq6wBJe0i6vJ8xmZmZFUpQk4HxTY4ZB7y33WDMzMwqOr3cxuuBFzt8TjMzG4aK3IMCqDsWK08Yux6wE3Bff4MyMzPrswUl6WVJL0l6Ke86rvK++kVqNd0FbA6c3+WYzcxsGGjWgrqaZa2m9wD3AvPrHPcS8DhwGWlGCTMzs37pM0FFxOTKf0t6GTg7Ik7odlBmZmZF7kGtDzzZrUDMzMyqtZygIuKebgZiQ4MXLTSzTmmYoCR9iXT/6TsRsTC/b0VExJc7Ep2ZmQ1bfbWgjiMlqJ8BC/P7VgTgBGVmZv3SV4Kakrf31rw3MzPruoYJKiKu6ut9p0laFzgB2JG0Iu9DwEXA8RHxRAufHwnsAnwQ2BJ4E/AyMJc0o/ppEfF8d6I3M7NOKzqTRFdI2hC4DlgLuBi4HdgKOAzYUdKkiHi8yWm2B84ldUdeQUpuawAfAr4BfFTSDhGxpDu1MDOzTmp5Lj5J4yXtlFsqlX3LSzpe0l8lXSfpI23GcTopOU2PiF0i4qiImAqcAmwCnNjCOR4G9gDWiYjd8jkOIC1HfyOwHXBIm/GZmdkAKzJZ7LHAj4GlVfuOAf4L2AzYBvi5pG2KBCBpA2AaaYaK79S55iJgz+rEWE9E3BwR59V240VEZaFFSDOym5nZIFAkQW0LXBYRLwJIGgEcTOqOW4/UJbcI+GzBGKbm7azaBRFzcrkWWJmUANv1Qt56pnUzs0GiSIJ6A1D9sO7mwBjSc1L3R8Rs0v2jdxWMYZO8vaNB+Z15u3HB81bbN2//r9EBkg6QNFvS7AULFvTjUq2Rlr3MzOy1iiSoFXj1chuT8vvqFXTvB9YpGMPovH2qQXll/2oFzwuApENJIwNvBs5qdFxEnBkREyNi4tixY9u5lJmZdVCRBHU/8Paq9zsBj0XEnKp9awFPdyKwKpU2Rt21qPr8oPRR4FTSAIpdI+KFJh8xM7OSKDLM/DfAZyV9A1gCvB84u+aYCby6G7AVlRbS6Ablq9Yc1xJJu5DWpnoUmBIRdxWMy8zMeqhIgjqJ9CDs5/L7B0ij7ACQNI40lPuUgjHMzdtG95g2yttG96heQ9LuwE9ILaepEXFnk4+YmVnJFJnN/FFJmwE75F1X5VF2FaNIyevSgjFckbfTJI2oHsknaRXSva7FwPWtnEzSJ4FzSAnULacuGKiBHdXXicIdvGY22BWaSSIiFpO6+uqV3QrcWjSAiJgnaRbpWahDgNOqio8HRgJnRMSiyk5JE/Jnb68+l6RPkQZC3ENKTl4ixMxskCrFVEek56muA2ZI2gGYA2xNmqD2DuDomuMrAzNe+Rtb0hRSchpBapXto9f+qf9kRJza8ejNzKzjCiUoSWuQninaClgdWK7OYRERO9TZ31BuRU1k2WSxO5Emi51Bmix2YQunGceyUYn7NjjmHtKoPjMzK7mWE1TuVrsSGEtVy6WOtu4WRMR9wD4tHvua60fETGBmO9c2M7PyKfIc1DdIzzl9HdgAWCEiRtR51WtVmZmZFVKki2974JKI+GK3gjEzM6so0oIScFu3AjEzM6tWJEHdwLKJXc3MzLqqSII6AdhJ0uQuxWLDkGd1N7NGityDehNpOY1Zkn5KalE9We/AiDinA7GZmdkwViRBzSQNIRewZ37VDilX3ucEZWZm/VIkQbX0jJKZmVknFJks9kfdDMTMzKxakUESZmZmA6bwZLGSxgK7Am8BRkbE/lX71wduybOem5mZta3oZLH7kSZwXYllAyL2z8VvAP4IHAD8sIMxmpnZMNRyF5+k9wNnkpa/+Ajw3eryiPg7aT2oXToZoJmZDU9FWlBHkpbAeG9EPC1pizrH/A3YtiORmZnZsFZkkMRE4DcR8XQfx9wPrN2/kMzMzIq1oFYEFjU5ZjXgpfbDGXqqp/CJtlbKGhr8czCzooq0oOYD72xyzNbA3LajMTMzy4okqIuB7SXtXq9Q0j7A24ELOxGYmZkNb0W6+E4CPg78VNJuwGgASYeSFjP8KHAncFqngzRrhbsRzYaWIlMdPSHpvaSJYKtbUTPy9g/AJyOi2X0qMzOzpgo9qBsR9wKTJb2dNJx8TeAp4PqIuKEL8dkQ5LWfzKwVhac6AoiIv5GeeTIzM+uKdubiGweMJU1ztCC3qszMzDqqpQQlaQzwReATwFo1ZY8A5wFfjYiFHY9wCHHXVt/88zGzak2HmUvaCJgNHEaaEPYl4FFgQf7vtYHPAbMlbdC9UM3MbDjpM0FJGkFqHa0HXAW8DxgVEetExNrAKsA04GpgPHBuV6M1M7Nho1kLahppDr6fAztExOUR8XylMCKWRsTvganABcDWedZzMzOzfmmWoHYFlgL/HtH40cdcdijwArBb58IzM7PhqlmC2hK4NiIWNDtRRDwKXJM/Y2Zm1i/NEtSbSIsQtupWYFz74ZjVJy17mdnw0CxBrQo8WeB8T5IGTpiZmfVLswS1IsXWd3o5f8bMzKxfWlluY0DmhZa0rqSzJD0oaamk+ZJOlbR6gXO8X9LJki6TtFBSSLqmm3GbmVl3tDKTxHGSjutmEJI2BK4jzVJxMXA7sBXp4eAdJU2KiMdbONUhwM7AEuAfQMvJzczMyqWVFpQKvtpxOik5TY+IXSLiqIiYCpwCbAKc2OJ5vg5sCowCPtRmLFZy1QMmGg2e8KAKs8GvzwQVESPaeC1XJIA8PdI00pLy36kpPhZYBOwpaWSzc0XEHyPi1ogoct/MzMxKqMiS790yNW9nRcTL1QUR8QxwLbAysM1AB2ZmZr1ThgS1Sd7e0aD8zrzdeABi6Qh3L5mZ9V8ZEtTovH2qQXll/2rdDELSAZJmS5q9YEHTiTPMzKzLypCgmqm0Q7o63D0izoyIiRExcezYsd28lJmZtaAMCarSQhrdoHzVmuPMzGwYKEOCmpu3je4xbZS3je5RmZnZEFSGBHVF3k7LCyS+QtIqwCRgMXD9QAdmZma90/MEFRHzgFmkFXkPqSk+HhgJnBMRiyo7JU2QNGHAgjQzswHXylRHA+Fg0lRHMyTtAMwBtgamkLr2jq45fk7evmogt6R3A/vnt6PydiNJMyvHRMTenQzczMy6oxQJKiLmSZoInADsCOwEPATMAI6PiIUtnurNwKdq9q1Vs2/v/kVrZmYDoRQJCiAi7gP2afHYuo/ARsRMYGbnorKhoPaB6RiQ+fnNrL96fg/KzMysHicoMzMrJScoMzMrJScoMzMrpdIMkjDrheoBFB48YVYubkGZmVkpOUGZmVkpuYvPrA53/Zn1nltQZmZWSk5QZmZWSu7iM8tqp0Qys95yC8rMzErJCcrMzErJXXw27BTtymtnRJ9HAZr1n1tQZmZWSm5BmXVIo5aZW1Nm7XELyszMSskJyszMSskJyszMSskJyszMSskJyszMSsmj+MwK8HRIZgPHLSgzMyslJygzMysld/GZlZwf9LXhyi0oMzMrJScoMzMrJXfxmZWMRwqaJW5BmZlZKbkFZTaA+jvgoVHryoMnbChyC8rMzErJCcrMzEqpNAlK0rqSzpL0oKSlkuZLOlXS6gXPs0b+3Px8ngfzedftVuxm7ZDqvzp53k7G0sr+duLvb9279XO03ivFPShJGwLXAWsBFwO3A1sBhwE7SpoUEY+3cJ4183k2Bi4HzgcmAPsAH5S0bUTc1Z1amJlZJ5WlBXU6KTlNj4hdIuKoiJgKnAJsApzY4nn+m5ScTomIHfJ5diElurXydczMbBBQ9Hj4j6QNgHnAfGDDiHi5qmwV4CFAwFoRsaiP84wEFgAvA+tExDNVZSPyNcbna/TZipo4cWLMnj273Sq5e8EGXPU/41Z//9r5TNHPNjqulc+3+tXUSuz9/fl0Sy+nsWrn2p2MV9INETGxr2PK0IKamrezqpMTQE4y1wIrA9s0Oc+2wOuBa6uTUz7Py8Cs/HZKvyM2M7OuK0OC2iRv72hQfmfebjxA5zEzsxIowyCJ0Xn7VIPyyv7VunkeSQcAB+S3z0qaC4wBHmty3cFqKNcNhln92h09165WP9vouCafHwM81smu8oH++TRR93ezl7cGOvzzafXf3rhmB5QhQTVT+TH0t4e2z/NExJnAma/6gDS7WR/pYDWU6wau32A2lOsGrl8RZejiq7RsRjcoX7XmuG6fx8zMSqAMCWpu3ja6N7RR3ja6t9Tp85iZWQmUIUFdkbfT8nDwV+Rh5pOAxcD1Tc5zfT5uUv5c9XlGANNqrteKM5sfMmgN5bqB6zeYDeW6gevXsp4/BwUg6VJSApkeEadV7f8m8FngjIg4qGr/BICIuL3mPGeQBjp8MyKOqNo/HfgWcGlE7NjNupiZWWeUJUHVTnU0B9ia9MzSHcB21VMdSQqAiFDNeWqnOvoz8BZgZ+DRfJ553a6PmZn1XykSFICkNwEnADsCa5JmkLgIOD4iFtYcWzdB5bI1gGOBXYB1gMeB/wW+FBH3d7MOZmbWOaVJUGZmZtXKMEiiVDq17EfZSFpT0v6SfiXpH5IWS3pK0jWS9qsdoDIUSNpTUuTX/r2OpxMkbS/pQkkP5d/PhyTNkrRTr2PrD0kfzPW4P/9u3iXpF5K27XVsrZC0m6TTJP1B0tP5d+7cJp/ZTtJvJS2U9Jykv0k6XNJyAxV3q4rUT9JGko6UdLmk+yQ9L+kRSRdLKjTV3GB4UHfAdGrZj5LaHfguqev0CuBe4A3AR4EfAP8safcYIk3q3GV8GvAsMKrH4XSEpGOAL5Oe0v8N6f/lGGALYDLw254F1w+Svg58gdQdfxGpfm8m3TveVdJeEdHnl30JHAO8g/T7dj9pmZ+GJO0MXAgsAX4GLAQ+RFrBYRLp32uZFKnfl4F/BW4j/U4uJE1F92Hgw5IOi4gZLV01IvzKL+BS0kwT/16z/5t5//d6HWM/6jaV9A9gRM3+tUnJKoBdex1nh+oq4PekGez/J9dt/17H1c867Z7r8TtglTrlK/Q6xjbrtTbwEvAwacWC6rIpuc539TrOFuoxhfSspUh/LARwboNjVyUN2loKTKzavxLpD+QAPt7rOvWjfnsDW9TZ/17g+VzvdVq57pDr1mlXXvZjGmnZj+/UFB8LLAL2zMt6DDoRcXlE/DpeO2P8w8D38tvJAx5Yd0wnJeR9SP/fBrXc/fp14Dngk1EzWz9ARLww4IF1xjjSrYY/RcSj1QURcQXwDDC2F4EVERFXRMSdkb+Jm9iNVKfzI+KVdX0iYgmppQLwmS6E2bYi9YuImRFxU539VwFXAisC27VyXSeoZTq17MdgVPlye7GnUXSApLcAXwO+FRFX9zqeDtkOWJ/UXfJEvl9zpKTDBss9mj7cSfqreitJY6oLJL0HWIXUGh5KKt81/1en7GrSHyLbSXrdwIU0YAp91/ge1DKtLNcxjfSM1WUDEtEAkLQ8sFd+W+8fzKCR6/JjUpflF3scTie9K28fAW4ENqsulHQ1sFtELBjowPorIhZKOpLUjX6bpItI96I2JN2z+B1wYA9D7IaG3zUR8aKku4G3ARuQngkdEiSNA3YgJeCW/nh0glqmU8t+DDZfAzYFfhsRl/Y6mH76EmnAwLsjYnGvg+mgtfL2IOBu4H3An0jdYycDHwB+wSDtoo2IUyXNB84CPl1V9A9gZm3X3xAw7L5rcmvwPOB1wBci4olWPucuvtZ1atmP0shTQB1BGq24Z4/D6RdJW5FaTSdHxB97HU+HVYYdi9RSuiwino2IW4GPkEZVvXewdvdJ+gJwATCT1HIaCbwTuAs4T9JJvYuuJ4bUd00eNv9j0ujEnwHfaPWzTlDLDKvlOiQdQpqf8DZgStTM1jGYVHXt3QH8V4/D6YbKX5t3RcRfqwtyS7HS8t1qQKPqAEmTSQNA/l9EfC4i7oqI5yLiRlLyfQA4Ig9iGiqGzXdNTk7nkkah/hzYo8WBJIATVLVhs1yHpMOBbwN/JyWnh3scUn+NIv1/ewuwpOrh3CCNwAT4ft53as+ibF/ld/PJBuWVBPb6AYil0/4lb1+zykBEPEeaT3MEqet2qGj4XZP/2FqfNIjgroEMqtNyXX4KfBz4CWkEaqGBWL4Htcyrlv2oHslXcNmPUss3pL8G3Ay8PyKGwrLoS4EfNijbkvTldg3pi2Ewdv9dTfrC2kjSihHxfE35pnk7f0Cj6ozKSLVGQ8kr+2vrPJhdDvwbad7Rn9aUvYc0WvjqiFg60IF1iqQVSS2mnYFzgH1qR0e3pNcPgJXpxRB+UDfX479yPWYDa/Q6ngGq83EMjQd1z831+ErN/vcDL5NaV6v1Os426vWxXK+HgTfWlP1zrttiYM1ex1qgTpNp/qDuAgbRg7oF6/c64JJ8zA+omRygyMuTxVYpuuzHYCLpU6Sb0C+RpgCq1789PyJmDmBYXSfpOFI336cj4gc9DqdtktYiPYv3ZuAPpK6vcaT7NEHqPvlF7yJsT34I+VLSyMRngF+RktVbSN1/Ag6PiG/1LMgWSNqFtIICpNkxPkDqovtD3vdYRHy+5vgLSFMdnU+aDujDpCHoFwAfixJ9ORepn6SzSbNJPAacTv3BHldGxJVNL9zrbFy2F/Am4GzSPGfPA/eQBhMM6hYHy1oSfb2u7HWcXaz3oG5B5bqsQWrN351/Nx8n/SG1Ta9j62e9VgAOJ3WfP03qznyUNN/gtF7H12Idmv37ml/nM5PID1+TWom3kBZoXa7X9elP/UizRTT7rjmuleu6BWVmZqXkUXxmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmXSBpfl7jyMza5ARlw0bVLOf3SFqpwTHz8zGeSNmsx5ygbDhajzS1TjftkF9m1iZPdWTDRl4f6gnSXGDLAxtGzXIjuVtuHLBCFFy7xsw6yy0oG26eA75MWvLg2CbHvoqkj0m6WtJTkhZLukXSf0p6XZ1jX3MPStKKkqZLulHSE5Key8ddLOl9dc4xQdJMSfdJWirpEUk/kbRJgZgn5y7L4yRtLukSSU/ma18labsGnxst6auS5kpakuO9tEGc1dfYKl9jYd43vvrnIWmUpFNynRZLujnPlI2k5SV9UdKd+ZrzJB3aal1t6HELyusZ/V8AAATESURBVIaN3IJ6gLRi6RxSV9+mEXFH1THzqdOCkvTfwH+SlhC4AHiWtF7R24CrSIs/vlBzHiJifNW+nwCfIK1kfDlpBut/At4N/DJevRzDjsAvSTN9/xr4B7Au8FHSOkJTIi2L3qzOk0mLcV4CTCUt2HhTrvuupFnRN4+IuVWfWY20tMdbgb/kWMeQ1m4aBXwmIs6oc41ZpLWCrsnXGAN8MSIezD+PFYB7SbOy/w5YMf88VgamAQeTlrf531zH3UlL33w8In7WrK42BPV6Gne//BqoF6lr7/7837vl97+sOWZ+3r981b5t8757gbWr9i9PSh5B+iKuPc/8qvejSYvvzabOcgpULcgHrE7qinwMeGvNcW8jJccbW6zzZJYtcbB3TdmBef/pNfvPyPvPIP8Rm/dvRFpHbCkwvsE1DmwQR+Xn+mvgdVX7t8/7F5KS4WpVZRuQEuhNvf7d8as3L3fx2bAUEReQWhMfkfTuJofvm7dfiYiHq87xInAEKfHs3+ySpMX3lubja+OpXghzL2A14NiIuK3muFuB7wNbSHprk2tWuzZeuxjlWaS1l7aq7JC0ArAHKQn+Z0S80sUSEXcCM0gtn73qXOPmqGpZNXB4VC1lHhF/IK1vtTpwZEQ8WVV2F6klt5mk5ZrW0IYcD6W14ewI0grKJ0vapvrLuMaWeXt5bUFE3CHpfmB9SatVf8HWHPe0pF8DHwJulnQhaTXSP0XEczWHb5u378grAtfaOG/fAtxWp7ye2XViekHSI6TkUDGB1OV2bUQsrHOey4FjgC3qlP25SQxPRsS8OvsfJHW73lCn7AFgOdIqrg80Ob8NMU5QNmxFxB8lXUDq7vsY0Og+x+i8fahB+UOkezqjgboJKvtX4Ejgk8Dxed+SHMPnI+KRvG/NvP10kyqMalJerVFcL5ISQEUrdYXUwqv1cJ191Z7qIwYiol555T7gCk3ObUOQu/hsuDsKeAH4qqQVGxxT+eJcu0H5OjXH1RURiyPiuIjYmJTQ9iANKNiDNPCi9nrviAj18fpRk7q1oz919Ygr6ygnKBvWcpfT6aQupn9vcNhNeTu5tkDSm0mj6+5u1L3X4Lr3RcR5wAeAO4F3S6q0nK7P2+1bPV8HzSUNxd9c0up1yqfkbdMRhGb95QRlBieQusCOpn632Vl5e4yksZWd+cb9N0j/jn7Y1wUkjZW0dZ2ikcAqpK6s5/O+s3M8x0raqvYDkkbkod0dFxHPA+eRfg4n1Fx3Q2A6qcX5425c36ya70HZsBcRC/NzTic1KL9O0knAF4C/53tGi0jPQW1K6qb7nyaXeSNwvaQ5pNbHfaSHhf+F1J02IyKeydd7XNJuwK/yZy4DbiWN/luPNIhiTaDufIIdcBSp9XaopHeRnnGqPAe1CnBoRNzdpWubvcIJyiyZQXpQdHy9wog4UtJNwKGkIdYrAPNII9pOzi2PvswnzVwxmdRNNob07M9cUkI4v+Z6l0l6O/B5Ujfg9qQW1oOkkXQXFqxfy3LC3pb0YPJHgc+RHir+M/A/ETGrW9c2q+aZJMzMrJR8D8rMzErJCcrMzErJCcrMzErJCcrMzErJCcrMzErJCcrMzErJCcrMzErJCcrMzErJCcrMzErp/wMsmv/kk9mDzwAAAABJRU5ErkJggg==\n", 159 | "text/plain": [ 160 | "
" 161 | ] 162 | }, 163 | "metadata": { 164 | "needs_background": "light" 165 | }, 166 | "output_type": "display_data" 167 | }, 168 | { 169 | "data": { 170 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZxcVZn/8c83LCIBEpYwOGIIMISIC8u0bBHIopkMyjIs83McUUAGGcCgwoyMOJIw4y6LQRFhBERUZEBRR8UoQdAAahAUMSQQ6ACGJRDZk7Dk+f1xTiWXSlVX3erq7krq+3696nXTdznn1CXph3PuuedRRGBmZtZphg11A8zMzGpxgDIzs47kAGVmZh3JAcrMzDqSA5SZmXUkBygzM+tI6w91A9YVW221VYwZM2aom2Fmtta57bbbHo+IUdX7OyZASdoWOAuYCmwJPAxcC8yIiL+0WOb+wA2knuInI+Ljdc7bF/g4sDewEXAvcAlwfkS83ExdY8aMYe7cua0008ysq0laVGt/RwzxSdoRuA04BvgNcC5wH3AKcIukLVsoc1Pg68DzDc47BLgJ2B/4HvBlYMPchivL1mtmZu3REQEKuADYGpgWEYdGxOkRMYkUJHYGPtlCmV8ERgCfrneCpM2Ai4GXgQkR8f6I+DdgN+AW4AhJ72qhbjMz66chD1CSdgCmAL2k3kvRmcBzwFGShpco8xBSb2wasLiPU48ARgFXRsSq8bmIWE4a8gP412brNTOz9hnyAAVMyttZEbGyeCAingHmABuTng81JGlrUq/o2oi4osm6r6tx7CbS8OC+kl7VTN1mZtY+nRCgds7bBXWO35O3Y5ss7yLS9zqhP3VHxEvA/aSJJDs0WbeZmbVJJwSoEXn7VJ3jlf0jGxUk6VjgEODEiHh0oOuWdLykuZLmLlmypInqzMysWZ0QoBpR3vaZF0TSGOA84H8j4qrBqDsiLoqInojoGTVqjSn8ZmbWD50QoCq9lBF1jm9WdV49lwDLgBOHoG4zM2uzTghQ8/O23jOmnfK23jOqij1IU9WXSIrKB7g0Hz8j77u2mbolrQ9sD7xEeifLzMwGUSesJHFD3k6RNKw4ky+/bDue1DO6tUE5l5Nm+1XbifQS7h2kl4FvLxybDfwzafWKb1ddt38u76aIWNHcV2kvafWfnfjYzLrNkAeoiFgoaRbpXaiTgPMLh2cAw4GvRsRzlZ2SxuVr7y6UM61W+ZKOJgWbH9VY6uhq4LPAuySdX3kXStJGwH/nc77S+rczM7NWDXmAyk4EbgZmSpoMzAP2AiaShvbOqDp/Xt6KfoiIpyX9CylQ/ULSlcBS4GDSFPSrge/0pw4zM2tNJzyDIiIWAj3AZaTAdCqwIzAT2CcinhjAuq8FDiC9mHs48EHgReAjwLsiPLhmZjYU5N+/7dHT0xPtXs3cz6DMrBtIui0ieqr3d0QPyszMrJoDlJmZdSQHKDMz60gOUGZm1pEcoMzMrCM5QJmZWUdygDIzs47kAGVmZh3JAcrMzDqSA5SZmXUkBygzM+tIHROgJG0r6RJJiyWtkNQr6TxJm5co498k/Thf+6ykpyXdKekcSdvWuSb6+DTKQWVmZgOkI9JtSNqRlG5ja+D7wN3AnsApwFRJ45tc0fwDwLPAjcCjwAbA7sCHgfdLmhARt9e4bhFpJfVqD5X8KmZm1iYdEaCAC0jBaVpErEpYKOkcUnD5JHBCE+W8MSKWV+/MOZ8uyuUcWOO63oiY3kK7zcxsgAz5EJ+kHUjZdHuBL1cdPhN4DjhK0vBGZdUKTtlVebtTi800M7NB1gk9qEl5OysiVhYPRMQzkuaQAtjewPUt1nFQ3v6hzvGRko4FtgGeAm6LCD9/MjMbQp0QoHbO2wV1jt9DClBjaTJASToO2BbYBHgT8DbSc6bT61yyK/C1qjJ+DxwVEXc2U6eZmbVXJwSoEXn7VJ3jlf0jS5R5HCl1fMVvgXdHxL01zj0HuIYUIJcD44CPAkcAsyXtFhF/rlWJpOOB4wFGjx5donlmZtbIkD+DakIl8XnTSc8jYu+IELAVqfcFcJukqTXOPTUibo6IxyPi2YiYGxFHkoLWVsBpfdRzUUT0RETPqFGjmv5CZmbWWCcEqEoPaUSd45tVnde0iHgiIn5GClLLgMslvbrJyy/M2/3L1mtmZv3XCQFqft6OrXO8MvOu3jOqhiLiSeAWYBTwhiYvW5K3DWcPmplZ+3VCgLohb6dIekV7JG0KjCf1fvo7q+61eftSk+fvnbf39bNeMzNrwZAHqIhYCMwCxgAnVR2eQerBXB4Rz1V2ShonaVzxREnb5Xeq1iDpA8BbgAeBOwv796j1fpWkN5Ne6gW4oux3MjOz/uuEWXwAJ5KWOpopaTIwjzQLbyJpaO+MqvPn5a0K+3YHvivp5nzNo8CWpJ7Qm0hLIB0VES8XrpkGHCZpNil4rSDN4psKrAdcDHy7Td/RzMxK6IgAFRELJfUAZ5GCw4HAw8BMYEZELG2imN8B5wL7Ae8AtiBNG78POBv4YkQ8WHXNtaRJGG8mvTC8EfAE8BPg4oj4QT+/mpmZtUgRTc/etj709PTE3Llz21qmCv1D/2cys3WVpNsioqd6/5A/gzIzM6vFAcrMzDqSA5SZmXUkBygzM+tIDlBmZtaROmKauTVWnNEHntVnZus+96DMzKwjOUCZmVlHcoAyM7OO5ABlZmYdyQHKzMw6UqlZfIX8TKNJ6dCXAY8Bd0TEXe1vnpmZdauGASqnSP8n4P3AnqzudVUmPkc+73HgGuArEXFndTlN1LMtq1cz35K0mvm1pNXM/9JkGf9GStGxCymArgQWAT8DzomIh+pctwswHZhAWt18EXAl8JmIWFb2u5iZWf/VXc1c0vqkfElnAJuTUlfcBvwWeARYCryaFEzGkfIujSYFrJ8Dp0bEH5tqhLQjKR/U1sD3gbtJwXAiKSX8+Ih4ooly7iXlffo9KR/UBqQ8UQcATwMTIuL2qmv2Ambnc68m5YWaBPQAc4DJEbGiUd0DvZp5Nb8HZWbrinqrmffVg7ob2B64Dvg68P1Gv6gljQWOBt4L3C7p/RFxeRPtu4AUnKZFxPmF8s4BPkzKbntCE+W8MSKW12jXvwAX5XIOLOxfD7gU2Bg4pJL/Kaeevwo4PNf/mSbqNjOzNuqrB/UD4D8j4velC5U2JGXJXRYRX21w7g7AQqAX2DEiVhaObUoa6hOwdTHte8n2jACeBO6NiJ0K+ycB1wM3RcQBddq1CNg+GiTOcg/KzKw1pfNBRcTBrQSnfO0LEXFeo+CUTcrbWcXglMt5hjTMtjFpCLFVB+XtH+rUfV31BRFxHyl1/HbADv2o28zMWtAJa/HtnLcL6hy/B5gCjCX1dhqSdBywLbAJ8CbgbaSe0Okt1D02fxbWqOd44HiA0aNHN9M0MzNrUr8ClKTtSTPmAP4UEfe3UMyIvH2qzvHK/pElyjwO2Kvw82+Bd0fEve2sOyIuIj3boqenx4NuZmZt1NKLupI2lfQd4F7gB8APgXslfTs/N2qnV0xnb0ZE7B0RIk01n5J33yZp6kDXbWZm7dFqD+pLpF/8Z5Kmnm8EHAy8D3ie9M5Usyq9lBF1jm9WdV7T8tT0n0n6LWlW4uWStiu82zRgdZuZWf/0GaAkbRwRz9c49A/Av0bENwv7vidp43ysTICan7dj6xyvzLqr95yooYh4UtItwKHAG4DKdLsBr9vMzFrTaIjvTkkTa+xfH3imxv5nKN8ruyFvp+T3j1YpLK20DLi1ZLnVXpu3LxX2zc7bNYb+8jTzsaTJFff1s24zMyupUYC6Ffi5pAurni1dD3xJ0j9JGidpV0kfJ72g+/MyDYiIhcAsYAxwUtXhGcBw4PLiO1C5znHFEyVtl4PKGiR9AHgLaZWI4jJMNwLzgP0lHVw4fxjw2fzjhY3egTIzs/ar+6LuqhOkdwJfIU0UOD4irpP0OtKSRLuxegKBSM+jDomIxaUaseZSR/NIs/AmkobX9i0udSQpAPJEiMq+Q4Hv5nIWkJY62pL0/tSbSEsgvTMibqyqu3qpoweAyXipIzOzQVHvRd2GASpfvBlwLnAMcDnwofxc5+2kdfgE3BURTb2nVKeO11F/sdilVefWClCjgVOA/Ugv125BWj/wPtJisV+MiAfr1L0Lqbc2EdiUNKz3bUosFusAZWbWmn4FqEIhbye99/Mq4MSIuLZ9TVy7OUCZmbWm9FJHtUTEz4A3At8DrpF0paQt29RGMzOzVZoKUJK2kvS3kraKiOci4iTScNjuwDxJ7xrQVpqZWdfpM0BJ2iSvGPEo8BvgUUlXSdokIm4CdgW+AVwh6VpJ2wx8k83MrBs06kF9GjiSlA/qJOAy4AhyfqSIWB4Rp5LeVdoJ+JOkoweqsWZm1j0aBahDgKsj4tiIuDAi3k9K635I8aSI+DVpyvlXgGZSbJiZmfWpUYAaDjxUte9BUn6mV4iIFyPiDFKqdjMzs35pZiWJoySNl7ShpH2A99DHskOtJjk0MzMrahSgTiGtg3dT3v6K9PLrhwa4XWZm1uX6XNg1Iu7Na94dBIwmLQP0f8V18czMzAZCw5XHc7qN7wxCW8zMzFZpKaOumZnZQKsboCSdKmmjVguWtIekvy9x/raSLpG0WNIKSb2SzpO0eZPXD5f0z5K+JeluSc9JekbS3PxdNqxzXfTx6W8OKjMza1FfQ3yfAj4iaSZwRUT8uVFhkkRKBX8CKQX8GcBPmriuOt3G3aTp6qcAUyWNL6bbqGM/4ApgKSkJ4rWkFc0PAr4AHCZpckQsr3HtItJLyNWqp9ibmdkg6StAvQk4h7SaxH9Lupk0i28uKRXGX4CNSKkxxpHyLk0GtgGeAE6m+Zd2LyAFp2kRcX5lp6RzgA8DnyQFvb48QpoC/78R8UKhjE2BXwD7klbDOLvGtb0RMb3JtnaE4krnXtnczNZFzSQsrPxiPxzYkNUJCl9xWt7OJ60mcWlE1EoJX6v8HYCFQC+wY0SsLBzblBQMBWzd6uxBSe8GvkmagXhQ1bEAboyICa2UXTHY6TaKHKDMbG1WL91GM7P4bgZulnQCsD/wVtKU8y1J70Y9BvwB+EVE3NVC2ybl7axicMp1PyNpDmnYcG9SqvlWvJi3L9U5PlLSsaTe31PAbRHh509mZkOoYYCqyD2iH+VPO+2ctwvqHL+HFKDG0nqAOjZvr6tzfFfga8Udkn4PHBURd7ZYp5mZ9UMnTDMfkbdP1Tle2T+ylcIlnUxKI38HcEmNU84hrcY+ipTu/S3A1aSgNVvSa/so+/g8S3DukiVLWmmemZnV0QkBqpHKk5jST1okHQacR5pAcXhEvFh9TkScGhE3R8TjEfFsRMyNiCNJq7ZvBZxWr/yIuCgieiKiZ9SoUWWbV6fNqz9mZt2sEwJUpYc0os7xzarOa4qkQ4ErSc/IJkTEfSXbdWHe7l/yOjMza4NOCFDz83ZsneM75W29Z1RrkHQk8L+kTMAHRMT8BpfUUhmzG97CtWZm1k+dEKBuyNspkl7RnjzNfDxptmBTs+rylPJvA4tJwemeFtu1d96W7XmZmVkbDHmAioiFwCxgDOl9q6IZpB7M5cV3oCSNy6usv4Kk9wHfIK26vn+jYb28HNMaPSRJbya9HAxpdQozMxtkTU8zH2AnkpY6milpMjAP2AuYSBraO6Pq/Hl5u2oqgaSJpFl6w0i9smO05kyDJyPivMLP00hLIM0mZQpeQVoVYyqwHnAxqTdmZmaDrOkAJekfgB9ExMvtbkRELJTUA5xFCg4HklaQmAnMiIilTRSzHat7hMfWOWcRaVZfxbWkSRhvJr0wvBFpmaafABdHxA9KfhUzM2uThksdrTpRWkl6rnMJ8D8R8cBANmxt066ljlqZXu6ljsxsbVZvqaMyz6AuADYGPg4slPRDSe9UjXE0MzOz/mo6QEXEycBfk4bP5gLvIKXGWCTpE32tuGBmZlZWqVl8EbE8Ii6LiH1Iz22+AmwCTAful/Q9SVPb30wzM+s2LU8zj4g/FnpVx5Beij0Y+JGk+yWdVmsKt5mZWTP69R5UDkDvJU3Xfi1p2vfvSak4PgfcLWm3/jbSzMy6T0sBStLuki4kzeq7kLRM0f8Ae0TEHqRe1emkxVZntqmtZmbWRcq8B7Ux8E/AB4C/JfWW5pEC1Ncj4unKuRHxLPA5Sa8D3t/WFpuZWVcos5LEYlK+pJdJqSguiIhfNLjmz6SXX83MzEopE6CeAc4mrbDwSJPXXICXCjIzsxaUCVDbRcTKMoXnYb+nG55oZmZWpcwkiZ9Lem9fJ0h6T1541czMrF/KBKgJpJQYfdkOOKCVhkjaVtIlkhZLWiGpV9J5kjZv8vrhkv5Z0rck3S3pOUnPSJor6VRJG/Zx7S6SrpL0mKTlkuZLmiHp1a18FzMz6792p9t4NfBS2Ysk7UhKt7E1afmku4E9gVOAqZLGR8QTDYrZj5S7aSkp3ca1wBbAQcAXSGk1JkfE8qq69wJmAxsAV5PSbkwCPgFMztesKPudzMysf8oGqJrrZucFY0eT0mQ82EI7LiAFp2kRcX6h3HOAD5OSB57QoIxHgPcA/xsRLxTK2BT4BbAvKSHi2YVj6wGXkhbBPaSSXiNn9r0KODzX/5kWvpOZmfVDn+k2coqNygmiToAqXgJ8KiI+3nQDpB2AhUAvsGNxIkYOLg/ncrcuZtUtI6eB/ybwfxFxUGH/JOB64KaIOKDqmkq7FgHbR4O8JE63YWbWmnrpNhr1oG5idVDan5RKvbfGeS+TEv1dT1pRooxJeTurepZgRDwjaQ4wBdg7l9+KF/O2evixUvd11RdExH2SFpBWyagEKzMzGyR9BqiImFD5c+5NXRoRZ7W5DTvn7YI6x+8hBaixtB6gKhl2qwNRM3WPzR8HKDOzQVTmGdT2wJMD0IYReftUneOV/SNbKVzSyaQ08neQsgG3rW5JxwPHA4wePbqV5pmZWR1lEhYuioh6v8gHUuWpTOknLZIOA84jTaA4PCJebHBJqboj4qKI6ImInlGjRpVtnpmZ9aFuD0rSJ0i/mL8cEUvzz82IiPivEm2oBL0RdY5vVnVeUyQdClwJPAZMjIj7BqvuwVacWOEJE2a2ruhriG86KUB9h/Ru0fQmywygTICan7dj6xzfKW/rPSdag6QjgW+Rek6TIuKewarbzMzao68ANTFvH6j6ud1uyNspkobVmGY+HlgG3NpMYXlK+eWkldTr9ZwqZgNnkJ5RfbqqnB1IgWsR0FcZZmY2AOoGqIi4sa+f2yUiFkqaRZqpdxJwfuHwDGA48NXiO1CSxuVr7y6WJel9pIkQi0jBaVGD6m8k5bTaX9LBVS/qfjafc2Gjd6DMzKz9+nxRd9AaseZSR/OAvUi9tgXAvsWljiQFQESosG8i8HPSxI9LqL2ixZMRcV5V3dVLHT0ATAZ6gDlAU0sdDeWLukUd8J/TzKyUVl/ULRYwBtgFuLHSm5G0PvCfwKHAc8DnI+J7ZRuXe1E9wFmk4bYDSStIzARmRMTSJorZjtWzEo+tc84i0qy+Yt2/lvQWUm9tCikp46Lcls94HT4zs6HRdA9K0qXAwcBfRcRLed900qKqFS8D+0VEU8+L1iXuQZmZtaZeD6pMuo19gOsLwWkYcCJp5fHRpNXHnyMtrmpmZtYvZQLUX5GGvip2A7YivSf1UETMJT0/eksb22dmZl2qTIDagFeuqDA+/1zMoPsQ8Jo2tMvMzLpcmQD1EPDmws8HAo9HxLzCvq2Bp9vRMDMz625lFov9P+DDkr4ALAfeTkr2VzSOVw4DmpmZtaRMgPocaTr5R/LPfwbOrByUtB0pa+25bWudmZl1raYDVEQ8JulNpJdYIb0P9UzhlE1IweunbWyfmZl1qTI9KCJiGWmor9axu4C72tEoMzOzMpMkzMzMBk2pHpSkLUjLCO0JbA6sV+O0iIjJNfabmZk1rcxafOOAXwCjWJ1pthYvtmNmZv1WZojvC6T3nD4L7ABsEBHDanxq9aoakrStpEskLZa0QlKvpPMkbV6ijLdLOlvS9ZKWSgpJv2pwTfTxWevWFJRWf8zM1mZlhvj2A34UER9rdyNqpNu4mzSMeAowVdL4YrqNPpwEHEJ6T+te0jBkMxYBl9XY/1CT15uZWZuVCVAC/jRA7biAFJymRcSqhIWSziEtPvtJ4IQmyvksKUPu3cDrgPubrL83IqaXabCZmQ2sMkN8twE7t7sBObX6FKAX+HLV4TNJK6QfJWl4o7Ii4paIuCsiXm53O83MbHCVCVBnAQdKmtDmNkzK21kRsbJ4IL8IPAfYGNi7zfUWjZR0rKSPSTpJ0kDWZWZmTSgzxPc60vOhWZK+TepRPVnrxIi4vES5lV7ZgjrH7yH1sMYC15cot4xdga8Vd0j6PXBURNw5QHWamVkfygSoy0hTyAUclT/VU8qV95UJUCPy9qk6xyv7R5Yos4xzgGtIAXI5acHbjwJHALMl7RYRf651oaTjgeMBRo8ePUDNMzPrTmUC1DED1oq+VSZMD8j7VRFxatWuucCRkq4GDgdOo06W4Ii4CLgIUsr3gWifmVm3KrNY7NcHqA2VHtKIOsc3qzpvsFxIClD7D3K9ZmZGZ6zFNz9vx9Y5vlPe1ntGNVCW5G3D2YNmZtZ+pdbiA5A0itSzeD0wPCKOK+zfHrgzr3rerBvydoqkYcWZfJI2JaWWXwYM9qoOlZl89w1yvWZmRskelKT3s/p9pQ/yyudSfwXcAry7TJkRsRCYBYwhrQRRNIPUg7k8Ip4rtGNcXhuwXyTtUev9KklvJr0cDHBFf+sxM7PyyiwW+3bShIA/kF6g/TsKqztExB8l3UXKuvu1moXUdyJpqaOZkiYD84C9gImkob0zqs6fV2lWVRvfChyXf9wkb3eSdFmhnUcXLpkGHCZpNvAgsII0i28qaaX2i4Fvl/wuZmbWBmWG+D4KPAwcEBFPS9q9xjl/APYp24iIWCiph/Qy8FTgwFzXTGBGRCxtsqi/Ad5XtW/rqn1HF/58LWkSxptJLwxvBDwB/AS4OCJ+UO6bmJlZu5QJUD3AlRHxdB/nPARs00pDIuJBmpzKHhE11+qOiMuovehrvXKuJQUpMzPrMGWeQW1IWhevLyMBr4NnZmb9ViZA9QJ/2+CcvVg9bdzMzKxlZQLU94H9JB1Z66CkY0jPcq5pR8PMzKy7lXkG9TngXcC3JR1BXvlB0smkZIaHkRZ2Pb9uCWZmZk0qs9TRXyQdQFoIttiLmpm3vwTeXXxfyczMrFWlVpKIiAeACflF1n2ALUlr5N0aEbcNQPvMzKxLlV7qCCAi/kB658nMzGxAtLIW33bAKFL6iyW5V2UdToU3x8KJQcxsLdDULD5JW0k6R9LDpMVTfw38Brhf0mJJn5e0xUA21MzMukvDACVpJ1ISv1NIC8K+DDxGSkfxMmnliI8AcyXtMHBNNTOzbtJngJI0DPgmMBq4EXgbsElEvCYitgE2BaYAN5FWI/fK32Zm1haNelBTSGvwXQVMjojZEfFC5WBErIiIn5MWWr0a2Cuvem5mZtYvjQLU4aQUFB+MqP9oPR87GXgROKKVhkjaVtIl+ZnWCkm9ks6TtHmJMt4u6WxJ10taKikk/aqJ63aRdJWkxyQtlzRf0gxJr27lu5iZWf81msW3BzAnIpY0OI+IeCwHgz3KNkLSjqR8UFuTllS6G9iT9NxrqqTxEfFEE0WdBBwCLAfuBRoGN0l7AbOBDUi9wAdJPcJPAJMlTY6IFWW/k5mZ9U+jHtTrgLtKlHcXsF0L7biAFJymRcShEXF6REwCzgV2ZnV220Y+C7yRlKzwoEYnS1oPuBTYGDgiIt4dER8lLXp7DSnd/IfLfhkzM+u/RgFqM+DJEuU9SZo40bQ8828Kq1PJF51JSvFxVK3U7NUi4paIuCsimk35cQDweuCmYnLCiFgJ/Hv+8QRJNfNPmZnZwGkUoDakXH6nlfmaMibl7awcGFaJiGeAOaQezt4lyy1T93XVByLiPlK6+e0AT583MxtkzbyoO9DrDuyctwvqHL8nb8d2Wt2Sjpc0V9LcJUsaPqYzM7MSmglQ0yW93MyHNLGgrBF5+1Sd45X9I1soe0DrjoiLIqInInpGjRrV9saZmXWzZtbiK/v8pd09rkr9Q7GC3FDW3W9+cmZma7M+A1RElMm426pKL2VEneObVZ23rtRtZmZ9GIwA1Mj8vK33jGmnvK33nGhtrdvMzPrQCQHqhrydktf+W0XSpqR3kZYBtw5A3bPzdmr1gTz9fSywiLSCu5mZDaIhD1ARsRCYRVps9qSqwzOA4cDlxVTyksZJGteG6m8E5gH7Szq4UP4w0ku/ABf2tcyTmZkNjJYy6g6AE0lLHc2UNJkUNPYCJpKG186oOn9e3r5iGoCktwLH5R83ydudJF1WOSciji78+WVJx5B6UldLuhp4AJhMWiR3Dmk1CzMzG2QdEaAiYqGkHuAs0nDbgcDDwExgRkQsbbKovwHeV7Vv66p9R1fV/WtJbyH11qaQVsJYlNvyGa/DZ2Y2NOTRq/bo6emJuXPn9rucwZga7v/kZtZJJN0WET3V+4f8GZSZmVktDlBmZtaRHKDMzKwjOUCZmVlH6ohZfDa4ihMxPGHCzDqVe1BmZtaRHKDMzKwjOUCZmVlHcoAyM7OO5ABlZmYdyQHKzMw6UscEKEnbSrpE0mJJKyT1SjpP0uYly9kiX9eby1mcy922zvm9kqLO55H2fDszMyurI96DkrQjKd3G1sD3gbuBPYFTgKmSxkfEE02Us2UuZywphcaVwDjgGOAdkvaJiFrJB58Czqux/9kWvs5axe9EmVmn6ogABVxACk7TIuL8yk5J5wAfBj4JnNBEOZ8iBadzI+IjhXKmAV/M9ayRPRd4MiKmt9x6MzNruyFPt5FTqy8EeoEdI2Jl4dimpLxQArYuZtWtUc5wYAmwEnhNRDxTODYs1zEm13Ff4VgvQESM6c/3WJvSbdTjHpSZDYVOTrcxKW9nFYMTQA4yc4CNgb0blLMP8GpgTjE45XJWktLKQ8rSW+1Vkt4j6WOSTpE0UdJ6Zb+ImZm1TycM8e2ctwvqHL+HlOl2LPDmyN0AABAySURBVHB9P8shl1NtG+AbVfvul3RMRNzYR51dx8+szGywdEIPakTePlXneGX/yAEq51JgMilIDQfeBHyVNBz4E0m71qtQ0vGS5kqau2TJkgbNW3tJqz9mZoOlEwJUI5Vfi/39//Wa5UTEjIiYHRGPRsTzEfHHiDgBOIc0ZDi9XoERcVFE9EREz6hRo/rZvKHnQGRmnaQTAlSlZzOizvHNqs4b6HIqLszb/Zs838zM2qgTnkHNz9taz4YAdsrbes+W2l1OxWN5O7zJ89cp7kWZ2VDrhB7UDXk7JU8HXyVPMx8PLANubVDOrfm88fm6YjnDSBMtivU1sk/e1nqx18zMBtiQB6iIWEiaAj4GOKnq8AxSD+by4jtQksZJGldVzrOkmXjDWfO50cm5/J9WvQP1BklbVLdJ0nbAl/KPV5T+UmZm1m+dMMQHcCJpiaKZkiYD84C9SO8sLQDOqDp/Xt5WD0R9DJgAfETSbsBvgNcDh5CG7KoD4JHA6ZJuAO4HngF2BN4BbAT8GPhCP7+bmZm1oCMCVEQslNQDnEVaiuhA0goSM4EZEbG0yXKekLQPcCZwKLAf8ARpKvknIuKhqktuIL0/tTtpSG848CTwK1Jv7Bsx1EttmJl1qSFf6mhdsS4sdVSW/+qYWTt08lJHZmZma3CAMjOzjtQRz6Bs3eW1+8ysVQ5Q1hYORGbWbh7iMzOzjuQelLVsbZpxaGZrHwcoG3IeHjSzWjzEZ2ZmHck9qA6wrg2VrWvfx8yGhgOUrTU8FGjWXTzEZ2ZmHaljApSkbSVdImmxpBWSeiWdJ2nzkuVska/rzeUszuVuO9B1W9+cUt7MyuiIIT5JO5LSbWwNfB+4G9gTOAWYKml8RDzRRDlb5nLGArOBK4FxwDHAOyTtU8wH1c66rT0GahhvoIcHPfxo1n6d0oO6gBQgpkXEoRFxekRMAs4lpcP4ZJPlfIoUnM6NiMm5nENJwWbrXM9A1W0lNNObKp5TfV71sTI9M/fkzNYOQ55uQ9IOwEKgF9gxIlYWjm1KygslYOtiVt0a5QwHlgArgddExDOFY8NyHWNyHfe1s27oX7oN/6Jsn+Jf57L3tT//FNyDMmtdJ6fbmJS3s4oBAiAHmTnAxsDeDcrZB3g1MKcYnHI5K0lp5SFl6W133dYhBrN3NBg9N/f2rJt1QoDaOW8X1Dl+T96OHYBy2lW3rQPqDRuWHU5sZfiyXUOW/QloA92GZodrbWC1614Pxn+zTpgkMSJvn6pzvLJ/5ACU06+6JR0PHJ9/fFbS/AZtrNgKeLzJc7vVOnGP2vmPt0ZZfd6jdtXdn3LqXdsoeLfROvH3aKBI7bk/bfhvtl2tnZ0QoBqpfPX+juy3Uk6f10TERcBFpRsiza013mqr+R415nvUmO9R3zr9/nTCEF+llzKizvHNqs5rZzntqtvMzNqsEwJUZVis3nOenfK23nOi/pTTrrrNzKzNOiFA3ZC3U/J08FXyVO/xwDLg1gbl3JrPG5+vK5YzDJhSVV876y6r9LBgF/I9asz3qDHfo7519P0Z8gAVEQtJU8DHACdVHZ4BDAcuL76HJGmcpHFV5TwLfCOfP72qnJNz+T8triTRSt3tkJ9dWR98jxrzPWrM96hvnX5/hvxFXai53NA8YC/SO0sLgH2Lyw1JCoCIUFU51Usd/QZ4PXAI8FguZ2F/6jYzs8HREQEKQNLrgLOAqcCWpFUcrgVmRMTSqnNrBqh8bAvgTOBQ4DXAE8BPgE9ExEP9rdvMzAZHxwQoMzOzoiF/BtUtnNIDJB0h6XxJv5T0tKSQdEWDa/aV9GNJSyU9L+kPkj4kab3BavdgkbSlpOMkfU/SvZKWSXpK0q8kvb96Ik/huq65RwCSPivpekkP5nu0VNLtks7Mw/y1rumqe1SLpKPyv7mQdFydc94p6Rf5792zkn4t6X2D3dZV7XEPauD1kdJjImmqe1ek9JB0B7Ar8CzwECkVyjcj4j11zj8EuAZYDnwHWAocRFqi6uqIOHIw2j1YJJ0AfIU0xHwD8ADwV8BhpHf1rgGOjMI/2m67RwCSXgB+B/yJ9Gx5OGm9zB5gMbB3RDxYOL/r7lG1/BjjTmA9YBPgXyLif6rOORk4n/RY5DvAC8ARwLbA2RFx2qA2GiAi/BngD/BT0moUH6zaf07ef+FQt3GQ7sNE0rtlAibk735FnXM3I/3yWQH0FPZvRAr2AbxrqL9Tm+/PJNIvzmFV+7chBasADu/me1T5fnX2fzJ/5wu6/R5V3RcBPydlbvh8/s7HVZ0zhhTAnwDGFPZvDtybr9lnsNvuIb4BllN6TCGl9Phy1eEzgeeAo3K6kHVaRNwQEfdE/pvfwBHAKODKiFiVxyQilgMfzz/+6wA0c8hExOyI+GGsubL+I8CF+ccJhUNdd49g1fer5aq83amwryvvUZVppP/5OYb0+6aWY4FXAV+KiN7Kzoj4CynPHsAJA9jGmhygBp5TerSmct+uq3HsJuB5YF9Jrxq8Jg2pF/P2pcI+36NXOihv/1DY19X3SNLrgc8AX4yIm/o4ta/79JOqcwaNA9TAc0qP1tS9bxHxEnA/abHjHQazUUNB0vrAe/OPxV8gXX2PJJ0mabqkcyX9EvgvUnD6TOG0rr1H+e/NN0jDwx9rcHpf9+lhUs9rW0kbt7WRDawNq5mv7dqVTqTb+L6t9hngjcCPI+Knhf3dfo9OI00iqbgOODoilhT2dfM9+gSwO/DWiFjW4Nxm7tPwfN7z7WleY+5BDb12pRPpNl1x3yRNA04lzfw8quzlebtO3qOI2CbSy/rbkGY67gDcLmmPEsWsk/dI0p6kXtPZEXFLO4rM20G9Tw5QA88pPVrT9fdN0knAF0nTqSfGmquadP09AoiIRyPie6TJSFsClxcOd909KgztLQD+s8nLmr1PT/ejaaU5QA08p/RoTd37lv8Bbk+aMHBf9fF1gaQPAV8C/kgKTo/UOK2r71G1iFhECuZvkLRV3t2N92gT0vd9PbC88HJukGYOA1yc952Xf+7rPr2GNLz3UEQM2vAeOEANhqFK6bG2m523U2sc25808/HmiFgxeE0aHJI+CpwL3EEKTo/VObVr71Ef/jpvX87bbrxHK4Cv1fncns/5Vf65MvzX1336+6pzBs9Qv0TWDR/8om6tezKBxi/qLqHLXrAkDckEMBfYosG5XXePSKuPbFNj/zBWv6g7p5vvUYP7N53aL+puTwe+qOuljgaBU3okkg4lrTIP6cH235GGVn6Z9z0eheVU8vlXk/7hXElaouZg8hI1wD/GOvQXOK95dhnp//7Pp/Zzkd6IuKxwTbfdow+RVkO4ibQywhOkmXwHkCZJPAJMjog/Fa7pqnvUF0nTScN8tZY6+iAwEy911H0f4HXApaR11l4AFpEegPf5f8nr0ofV//dW79Nb45rxwI+Bv5CGQu8EPgysN9TfZwjuTwC/6PJ79EbSiix3AI+Tnh89Bfw237+a/5666R41+XfsuDrHDwJuBJ4hvfv0W+B9Q9Ve96DMzKwjeZKEmZl1JAcoMzPrSA5QZmbWkRygzMysIzlAmZlZR3KAMjOzjuQAZWZmHckBymyQSeqV1DvU7TDrdA5QZkBhxedFkjaqc05vPseJPs0GgQOU2SuNBj40wHVMzh8z64OXOjIj9aBI67QFsD6wY0Q8XnVOL7AdsEFEvDTojTTrMu5Bma32PPBfpBQNZzY49xUk/aOkmyQ9JWmZpDsl/YekV9U4d41nUJI2lDRN0u8k/UXS8/m870t6W40yxkm6TNKDklZIelTStyTtXKLNE/KQ5XRJu0n6kaQnc903Stq3znUjJH1a0nxJy3N7f1qnncU69sx1LM37xhTvh6RNJJ2bv9MySXfklciRtL6kj0m6J9e5UNLJzX5XWzu5B2XGqh7Un0l5ceaRhvreGBELCuf0UqMHJelTwH+QVte+GniWlOTtDaSVod8eES9WlUNEjCns+xbwT6QMurNJK27/NfBW4LvxyjQkU4HvAhsAPyTl69kWOIyU92hiRPyuie88gZRQ80fAJFLyutvzdz+ctOr+bhExv3DNSGAOsAtppevZwFbAP5Iyuf5rRHy1Rh2zSDnAfpXr2Ar4WEQszvdjA+ABYAvgZ8CG+X5sTErlfiIpRc1P8nc8kpS+5l0R8Z1G39XWUkO9/Ls//nTChzS091D+8xH55+9WndOb969f2LdP3vcAhUR6pGHCH+ZjH6tRTm/h5xHASlKSwjXSPwBbFv68OWko8nFgl6rz3kAKjr9r8jtPYHUaj6Orjn0g77+gav9X8/6vkv8HN+/fiZT2YgWvTHhXrOMDddpRua8/BF5V2L9f3r+UFAxHFo7tQAqgtw/13x1/Bu7jIT6zKhFxNak38Q+S3trg9GPz9r8j4pFCGS8Bp5ICz3GNqgRE+uW+skZ7isks3wuMBM6MQlK+fN5dwMXA7pJ2aVBn0ZwoJEHMLiHlWtqzskPSBsB7SEHwPyJi1fBLRNxDSna3YW5jtTui0LOq40NRSL0eEb8E7icF5Y9GxJOFY/eRenJvkrRew29oayVPlzWr7VRSFuSzJe1d/GVcZY+8nV19ICIWSHoI2F7SyOIv2Krznpb0Q1KyuDskXUPKMvzriHi+6vR98nbXnB212ti8fT3wpxrHa5lbo00vSnqUFBwqxpGG3OZExNIa5cwGPg7sXuPYbxq04cmIWFhj/2LSsOttNY79GViPlJ35zw3Kt7WQA5RZDRFxi6SrScN9/0hKgV3LiLx9uM7xh0nPdEYANQNU9v+AjwLvBmbkfctzG06LiEfzvi3z9l8afIVNGhwvqteul0gBoKKZ7wqph1ftkRr7imqlt6+0gYiodbzyHHCDBmXbWspDfGb1nQ68CHxa0oZ1zqn84tymzvHXVJ1XU0Qsi4jpETGWFNDeQ5pQ8B7SxIvq+naNCPXx+XqD79aK/nxXz8ay0hygzOrIQ04XkIaYPljntNvzdkL1AUl/Q5pdd3+94b069T4YEd8E/g64B3irpErP6da83a/Z8tpoPmkq/m6SNq9xfGLeNpxBaNYMByizvp1FGgI7g9rDZpfk7ccljarszA/uv0D6N/a1viqQNErSXjUODQc2JQ1lvZD3XZrbc6akPasvkDQsT+1uu4h4Afgm6T6cVVXvjsA0Uo/zGwNRv3UfP4My60NELM3vOX2uzvGbJX0O+Hfgj/mZ0XOk96DeSBqm+3yDal4L3CppHqn38SDpZeF3kobTZkbEM7m+JyQdAXwvX3M9cBdp9t9o0iSKLYGa6wm2wemk3tvJkt5Cesep8h7UpsDJEXH/ANVtXcYByqyxmaQXRcfUOhgRH5V0O3AyaYr1BsBC0oy2s3PPoy+9pJUrJpCGybYivfsznxQQrqyq73pJbwZOIw0D7kfqYS0mzaS7puT3a1oO2PuQXkw+DPgI6aXi3wCfj4hZA1W3dR+vJGFmZh3Jz6DMzKwjOUCZmVlHcoAyM7OO5ABlZmYdyQHKzMw6kgOUmZl1JAcoMzPrSA5QZmbWkRygzMysI/1/OLVlNf2S27EAAAAASUVORK5CYII=\n", 171 | "text/plain": [ 172 | "
" 173 | ] 174 | }, 175 | "metadata": { 176 | "needs_background": "light" 177 | }, 178 | "output_type": "display_data" 179 | }, 180 | { 181 | "data": { 182 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZhdRZ3/8fcnCAIBEiBBHGMIICEoyDItBCJLEslkQGV3HAZUFiM/QFBBYcARwoyKjiyCIoIC4o6gMu5RgqBBxI6AiCHBQIcdQiKQsESW7++PqiuHm3v7Ln27+yT9eT3PfYpb55xa0kl/qXPqVCkiMDMzK5thg90AMzOzWhygzMyslBygzMyslBygzMyslBygzMyslF412A1YXYwaNSrGjRs32M0wM1vlzJ079/GIGF2d7wDVIePGjaO7u3uwm2FmtsqRtKhWvm/xmZlZKTlAmZlZKTlAmZlZKTlAmZlZKTlAmZlZKTlAmZlZKTlAmZlZKTlAmZlZKTlAmZlZKXkliZKRXv5v7yVpZkOZR1BmZlZKDlBmZlZKDlBmZlZKDlBmZlZKDlBmZlZKDlBmZlZKDlBmZlZKDlBmZlZKpQlQksZIukzSQ5JWSOqRdL6kDVso46OSfpqvXS7pKUl3SDpX0pg610Qvn5s710MzM2tFKVaSkLQlcBOwCXAtcBewM3AiMF3SpIhY0kRRHwCWAzcAjwJrAjsCHwaOkrRXRNxa47pFwBU18h9osStmZtYhpQhQwEWk4HRCRFxYyZR0Lim4fBI4polyto2I56ozJb0fuCSXs0+N63oi4sw22m1mZv1k0G/xSdoCmAb0AF+sOnwG8DRwuKThjcqqFZyyq3K6VZvNNDOzAVaGEdSUnM6KiJeKByJimaQ5pAA2EbiuzTrekdM/1Tk+UtKRwKbAk8DciPDzJzOzQVSGALV1ThfUOX43KUCNp8kAJeloYAywHrAd8DbSc6ZT61yyPfDVqjJuBw6PiDuaqdPMzDqrDAFqRE6frHO8kj+yhTKPBnYpfP8DcGhE/LXGuecC15AC5HPABOAU4GBgtqQdIuLBWpVImgHMABg7dmwLzTMzs0YG/RlUEyo7JDW9O1JETIwIAaNIoy+AuZKm1zj3pIi4KSIej4jlEdEdEYeQgtYo4ORe6rkkIroiomv06NFNd8jMzBorQ4CqjJBG1Dm+QdV5TYuIJRHxS1KQeha4UtI6TV5+cU73aLVeMzPruzIEqPk5HV/neGXmXb1nVA1FxBPA74DRwJuavGxxThvOHjQzs84rQ4C6PqfTJL2iPZLWByaRRj99nVX3upy+0OT5E3N6Tx/rNTOzNgx6gIqIhcAsYBxwXNXhmaQRzJUR8XQlU9IESROKJ0raLL9TtRJJHwDeAtwP3FHI36nW+1WS3kx6qRfgG632yczM+q4Ms/gAjiUtdXSBpKnAPNIsvMmkW3unV50/L6cq5O0IfF/STfmaR4GNSSOh7UhLIB0eES8WrjkBOFDSbFLwWkGaxTcdWAO4FPh2h/poZmYtKEWAioiFkrqAs0jBYR/gYeACYGZELG2imD8C5wG7A/sCG5Gmjd8DnAN8PiLur7rmh6RJGG8mvTC8NrAE+BlwaUT8Xx+7ZmZmbVJE07O3rRddXV3R3d3d53JUGBP6R2NmQ4GkuRHRVZ0/6M+gzMzMainFLT6rzaMpMxvKPIIyM7NScoAyM7NScoAyM7NScoAyM7NScoAyM7NScoAyM7NScoAyM7NScoAyM7NScoAyM7NScoAyM7NSKk2AkjRG0mWSHpK0QlKPpPMlbdhCGR+V9NN87XJJT0m6Q9K5ksb0ct0bJV0l6TFJz0maL2lmC9vDm5lZh5ViLT5JW5L2g9oEuBa4C9gZOBGYLmlSRCxpoqgPkPZ9uoG0H9SapH2iPgwcJWmviLi1qu5dgNn53KtJ+0JNAT4BTJU0NSJW9L2XZmbWilIEKOAiUnA6ISIurGRKOpcUXD4JHNNEOdtGxHPVmZLeD1ySy9mnkL8GcDmwLrBfZf+nvPX8VcBBuf6z2+uWmZm1a9Bv8eVt2qcBPcAXqw6fATwNHF5ra/ZqtYJTdlVOt6rK3xPYBrixuDlhRLwEfCx/PUYqrituZmYDYdADFOl2GsCsHBj+ISKWAXNII5yJfajjHTn9U526f159QUTcQ9o6fjNgiz7UbWZmbSjDLb6tc7qgzvG7SSOs8cB1zRQo6WhgDLAesB3wNmARcGobdY/Pn4XN1N0Oj8/MzFZWhgA1IqdP1jleyR/ZQplHA7sUvv8BODQi/trJuiXNAGYAjB07toXmmZlZIy0FKEnrA5OAscAo4FngMeC2iLiz881L1ea06T1lI2IigKSNgZ1IkyPmSvq3iFjpdl67dUfEJaTJF3R1dXnPWzOzDmoYoPK7QP8OHEWa+l15bvWKX96SHgeuAb4UEXe00IbKKGVEneMbVJ3XtDw1/ZeS/kCaun6lpM0i4tn+rtvMzPqmboCS9CrgBOB0YEPgOeBm0u2yR4ClwDrAxsAE0iSGY4APSPoVcFJE/LmJNszP6fg6xysz7+o9J2ooIp6Q9Dtgf+BNQPdA1W1mZu3pbQR1F7A5aYbb14BrG72wKmk88D7gPcCtko6KiCsbtOH6nE6TNKw4k69wS/FZUnDsi9fl9IVC3mxSAJ4OfLp4cp7+Pp40ueKePtZtZmYt6m2a+V+AnSJi34i4qpnVFCJiQUScRpqW/VHSCKvRNQuBWcA44LiqwzOB4cCVEfF0JVPSBEkTiidK2iwHlZVI+gDwFtIqEcXbjzcA84A9JL2zcP4w4DP568UR4edLZmYDTGX43VtjqaN5pFl4k0m313YrLnUkKQAiQoW8/YHv53IWkJY62ph063E70hJIb4+IG6rqrl7q6D5gKtBFegerqaWOurq6oru7u9FpNTUzzbwEPyYzs34haW5EdFXnl+FF3cooqgu4ghSYTgK2BC4Adm1yHb4/AucBawH7AieTJncEcA7wxurglOv+PWl0dS3pfasPkyZNnAXs7XX4zMwGR5/eg5K0OfDG/PUvEXFvu2VFxP3AEU2eu9KYIyLuIwW2dur+C3BIO9eamVn/aCtA5ckLXwEOrmQBIekqYEZeosjMzKxt7Y6gvkC6HXYGMBdYG3gn8F7gGdI7U2ZmZm3rNUBJWjcinqlx6ADg/0XENwt5P5C0bj7mAGVmZn3SaJLEHZIm18h/FVDrNt4yyrG+n5mZreIaBZObgV9JuhT4aOHZ0nXAF/IeTbcCryZtafEe4Mf91VgzMxs6eh1BRcR/APuRpm3fKWl6PnQs8DjwTeBO0hTvs4DbgeP7rbVmZjZkNLwdFxE/lnQj6R2jn0q6EvhQROwkaW/SOnwC7oyIpvZrMjMza6Sp50UR8RRwlKTvkLaX+IukYyPih8Av+7OBZmY2NLW0kkRE/BLYFvgBcI2k7+Q9l8zMzDqqqQAlaZSkf5Y0KiKejojjSOvk7QjMk/Tufm2lmZkNOb0GKEnrSfouaeHVW4BHJV0lab2IuBHYHvg68A1JP5S0af832czMhoJGI6hPk9ao+xppK4wrSMsbnQ0QEc9FxEmkPZu2Ij2bel9/NdbMzIaORgFqP+DqiDgyIi6OiKNI27rvVzwprwi+A/Al4Mv90lIzMxtSGgWo4cADVXn3A+tWnxgRz0fE6cDO7TRE0hhJl0l6SNIKST2Szpe0YZPXD5f0H5K+JekuSU9LWiapW9JJktaqc1308unrLr5mZtamZlaSOFzSNcAfgH8GDqOX7dcj4vZWG1Fjw8K7SIHuRGC6pElN7Am1O/ANYClpG/kfAhuRVrj4HHCgpKkR8VyNaxeRbl9Wqw7OZmY2QBoFqBNJu83eWMh7EPhQh9txESk4nRARF1YyJZ1L2kDwk8AxDcp4hBQ8vxcRfy+UsT7wa2A30nO0c2pc2xMRZ/ah/WZm1mGNljr6K2mliEOBU3O6TUTc3akGSNqCtHVHD/DFqsNnAE+TRnHDG7T1toj4ZjE45fxlvByU9upEm83MrP81s9TRM8B3+7ENU3I6KyJeqqp7maQ5pAA2kbRIbTuez+kLdY6PlHQksCnwJDA3Ivz8ycxsEJVha4ytc7qgzvG7SQFqPO0HqCNz+vM6x7cHvlrMkHQ7cHhE3NFmnWZm1gd1b/HlmW9rt1uwpJ0k/WsTp47I6ZN1jlfyR7bZjuOB6cBtwGU1TjmX9B7XaGB94C3A1aSgNVvS63ope0aeJdi9ePHidppnZmZ19PYM6lPAQkmn9PZLukjJv0j6AWnW3/YdaKNyGi1fKB0InE+aQHFQRDxffU5EnBQRN0XE4xGxPCK6I+IQ0vteo4CT65UfEZdERFdEdI0ePbrV5pmZWS96u8W3HWl08WngfyTdBPwW6AYeBv4GrA1sTJpIMRGYSnqOs4S0L1QzL+1WRkgj6hzfoOq8pkjaH/gO8BgwOSLuaeV64GLgIGCPFq8zM7MOqBugImIB8HZJlenZB5HeNao1kqmMcuYDnwEuL+y+28j8nI6vc3yrnNZ7RrVyY6RDgG+RRk5T2px1WLln1+vsQTMz6x/NzOK7CbhJ0jGk0cRbgbGkkdOzpBHKn4BfR8SdbbTh+pxOkzSsOJMvv8M0KdfT1Kw6SYcCV5Le12pn5FQxMaftXm9mZn3Q9Cy+PCL6Sf50TEQslDSLNFPvOODCwuGZpBHMlyPi6UqmpAn52ruKZUl6L2kixCJScFrUW92SdgLmF8vO+W8mvRwMaXUKMzMbYGWYZg5wLGmpowskTQXmAbuQ9pxaAJxedf68nFZuLSJpMik4DSONyo6QVHUZT0TE+YXvJ5CWQJpNWmNwBel52nRgDeBS4Nt97ZyZmbWuFAEqj6K6gLNIwWEf0kSMC4CZEbG0iWI24+VZiUfWOWcRaVZfxQ9JkzDeTHpheG3SBI+fAZdGxP+12BUzM+sQRbQ8e9tq6Orqiu7u7rauXXmgtzL/mMxsdSVpbkR0Vec3teW7mZnZQHOAMjOzUnKAMjOzUnKAMjOzUmo6QEk6QNIa/dkYMzOzilZGUNcAiySdJWlsfzXIzMwMWgtQFwHrAh8nrXL+I0lvV423Yc3MzPqq6QAVEccD/0R6CbYb2Be4ljSq+kSzW3KYmZk1o6VJEhHxXERcERG7klZf+BKwHnAmcK+kH0ia3vlmmpnZUNP2LL6I+HNhVHUE8CjwTuAnku6VdLIkb1VhZmZt6dM08xyA3kNadPV1pMVbbydtxfFZ4C5JO/S1kWZmNvS0FaAk7SjpYuAh0s6z44GvADtFxE6kUdWppC3TL+hQW83MbAhp5T2odSUdJekW0iSJGcADwInA6yJiRkTcBhARyyPis6SgtdICgHXKHyPpMkkPSVohqUfS+ZI2bPL64ZL+Q9K3JN0l6WlJyyR1SzpJ0lq9XPtGSVdJekzSc5LmS5opaZ1m6h4I0is/Zmaru1a223gIWB94kfRO1EUR8esG1zxI2sKiV5K2JO0HtQlpZuBdwM6k4Ddd0qSIWNKgmN1JmwsuJe0H9UNgI+AdwOdI+z5NjYjnqureBZgNrAlcTdoXagrwCWBqvmZFoz6YmVlntRKglgHnkPZJeqTJay6iuQ3/LiIFpxMi4h876ko6F/gwaXfbYxqU8QhwGPC9iPh7oYz1gV8Du5F27D2ncGwN4HLS+137VfZ/kjQMuAo4KNd/dhN9MDOzDmp6PyhJwyLipY43QNoCWAj0AFsW68jB5WHS5ItNqrdmb6GOQ4FvAj+OiHcU8qcA1wE3RsSeddq1CNg8GvxB9fd+UNW8P5SZrS46sR/UryS9p0Elh+Xt01sxJaezqgNgRCwD5pBGOBNbLLfo+Zy+UKfun1dfEBH3kLab3wzYog91m5lZG1oJUHsB4xqcsxmwZ4Nzqm2d0wV1jt+d0/EtlltU2QK+OhANRN1mZtaGTm+3sQ4rj1IaGZHTJ+scr+SPbKdBko4HpgO3AZd1sm5JM/Iswe7Fixe30zwzM6uj1QBV88mHks2AfUiz4Dqp8oSm5acukg4EzidNoDgoIp5vcElLdUfEJRHRFRFdo0ePbrV5ZmbWi14DlKSXJL0o6cWcdWble/FDGjXdA+wAfKfFNlRGKSPqHN+g6rymSNo/t+UxYK/8TGlA6jYzs75rNM38Rl4ePewB3EeabVftRWAJaUbcV1psw/yc1nvOs1VO6z0nWomkQ4BvkUZOUyLi7jqndrxuMzPrjF4DVETsVflvSS8Bl0fEWR1uw/U5nVY9lT1PM58EPAvc3ExheUr5laSXhCfXGTlVzAZOJz2j+nRVOVuQAtci0ujQzMwGUCvPoDYHPt/pBkTEQmAWaYbgcVWHZwLDgSuL70BJmiBpQnVZkt4LfJ000tujQXACuAGYB+wh6Z2FcoYBn8lfL270DpSZmXVe0y/q9msjVl7qaB6wCzCZdHttt+JSR5ICICJUyJsM/IoUdC+j9mSNJyLi/Kq6q5c6ug+YSlpDcA7Q1FJHflHXzKw99V7UrXuLT9InSM+fvhgRS/P3ZkRE/HcrjYuIhZK6gLNIt9v2Ia0gcQEwMyKWNlHMZrw8IjyyzjmLSLP6inX/XtJbSKO1aaT1BhfltpztdfjMzAZH3RFUfuYUwDYRsSB/b0ZExBqdauCqwiMoM7P2tDyCIt1eg3TLq/jdzMys39UNUBFxQ2/fzczM+lOnlzoyMzPriFZ21B0naR9Jwwt5r8o7z94u6SZJB/RPM83MbKhpZcPCM4B3Aq8p5H0c+K/C96sk7R4RTb1Ua2ZmVk8rt/h2Ba6LiBfgHy+zHkvann0saYv2p0k70JqZmfVJKwHqNaT3gyp2AEaR3pN6ICK6SS/ZvqWD7TMzsyGqlQC1Jq/cdmJS/l7cQfcB4LUdaJeZmQ1xrQSoB4A3F77vAzweEfMKeZsAT3WiYWZmNrS1Mknix8CHJX0OeA7YG7i86pwJvPI2oJmZWVtaCVCfBfYHPpK/P0ia2QdA3lF3N+C8jrXOzMyGrKYDVEQ8Jmk70krfADdExLLCKeuRgtcvOtg+MzMboloZQRERz5Ju9dU6didwZycaZWZmVpqljiSNkXSZpIckrZDUI+l8SRu2UMbeks6RdJ2kpZJC0m8bXBO9fPzCsZnZIGlpBCVpI9JeSzsDGwK1ttWIiJhaI7+3cqs3LLwr13EiMF3SpOKGhb04DtiPNInjr7mNzVgEXFEj/4Emrzczsw5rOkDlLdZ/DYwGetvBqJ2dii4iBacTIuLCQp3nklam+CRwTBPlfAY4nRTgXg/c22T9PRFxZisNNjOz/tXKLb7PkYLIZ4AtgDUjYliNT0ubFUragrSTbQ/wxarDZ5CWTzq8uEhtPRHxu4i4MyJebKUNZmZWPq3c4tsd+ElEnNbhNkzJ6ayIeMWuvRGxTNIcUgCbCFzX4borRko6EtgUeBKY6wVvzcwGVysBSsBf+qENW+d0QZ3jd5MC1Hj6L0BtD3y1mCHpduDwiLij3kWSZgAzAMaOHdtPTTMzG5paucU3l5eDSSeNyOmTdY5X8kf2Q90A55LWFRwNrE9a7PZqUtCaLel19S6MiEsioisiukaPHt1PzTMzG5paCVBnAftI2quf2lJPZUJGO5MvGoqIkyLipoh4PCKWR0R3RBwCXENarf3k/qjXzMx618otvteTpoDPkvRt0ojqiVonRsSVLZRbGSGNqHN8g6rzBsrFwEHAHgNcr5mZ0VqAuoI0ihFweP5Uj2qU81oJUPNzOr7O8a1yWu8ZVX9ZnNOGswfNzKzzWglQR/RTG67P6TRJw4oz+SStT3o+9Cww0LPqJub0ngGu18zMaG2x2K/1RwMiYqGkWaSZescBFxYOzySNYL4cEU9XMvNLw0TEXX2pW9JOwPxi2Tn/zaSXgwG+0Zc6zMysPS0tddSPjiUtdXSBpKnAPGAXYDLp1t7pVedXNkl8xYoWkt4KHJ2/rpfTrSRdUTknIt5XuOQE4EBJs4H7gRWkPa2mk5ZxuhT4dh/6ZWZmbWo5QEkaTZo8sA0wPCKOLuRvDtyRVz1vWh5FdZFmCk4n7db7MHABMDMiljZZ1BuA91blbVKV977Cf/+QNAnjzaQXhtcGlgA/Ay6NiP9rpR9mZtY5imh+9rako0hBY23yhIjK0kaStgVuB2ZExFfrl7J66urqiu7u7rauVW8rG9bRwo/NzKzUJM2NiK7q/Kbfg5K0N3AJ6ZbbAcCXiscj4s+k/aD271tTzczMWrvFdwrpttueEfGUpB1rnPMnYNeOtMzMzIa0VlaS6AJ+HBFP9XLOA6QFV83MzPqklQC1Fmnri96MBLzVhZmZ9VkrAaoH+OcG5+zCyytDmJmZta2VAHUtsLukQ2odlHQEabr2NZ1omJmZDW2tTJL4LPBu4NuSDiYv7irpeNJmhgeS9m66sG4JZmZmTWplqaO/SdqTtBBscRR1QU5/AxxavWyQmZlZO1paSSIi7gP2ymvV7QpsTNoG4+aImNsP7TMzsyGqrbX4IuJPpHeezMzM+kU7a/FtRtoePYDFeVRlZmbWUU0FKEmjgNOAfyctvlo89ijwTeDTLSzqan1UXL/P6/KZ2eqo4TRzSVsB3cCJwGtIL+I+Rtpx9kXSyhEfAbolbdFuQySNkXSZpIckrZDUI+l8SRu2UMbeks6RdJ2kpZJC0m+buO6Nkq6S9Jik5yTNlzRT0jrt9sfMzPqm1wAlaRhpdDQWuAF4G7BeRLw2IjYF1idtNHgjMI42N/eTtCUwl7Rr7y3AeaSdbE8Efidp4yaLOo4ULHcDHmyy7l2AP5AWuf0V8HngKeATwC8lvbr5npiZWac0GkFNI63BdxUwNSJmR8TfKwcjYkVE/Iq0l9LVwC551fNWXUS6dXhCROwfEadGxBRSoNqal3e3beQzwLakzQrf0ehkSWsAlwPrAgdHxKERcQppRYxrSNvNf7jVzpiZWd81ClAHkXaZ/WD0snFUPnY88DxwcCsNyLcFp5GWUvpi1eEzSOv/HS5peKOyIuJ3EXFnRDS7HuCepI0XbyxuThgRLwEfy1+PkdrZscnMzPqiUYDaCZgTEYsbFRQRjwG/zde0YkpOZ+XAUCxzGTCHNMKZ2GK5rdT98+oDEXEPae+rzYC2n62ZmVl7GgWo15M2IWzWnaRf6K3YOqcL6hy/O6fjWyy37HWbmVkvGgWoDYAnWijvCdLEiVaMyOmTdY5X8ke2WG6/1y1phqRuSd2LFzccZJqZWQsaBai1aG1/p5fyNZ1Uef4zGG/79Fp3RFwSEV0R0TV69OgBbJaZ2eqvme02+jswVEYpI+oc36DqvNWlbjMz60UzK0mcKenMfmxDZYPDes95tsppvedEq2rdZmbWi2ZGUGrx06rrczotvxj8csXS+qR3kZ4Fbm6j7EZm53R69YE8/X08sIj00rCZmQ2gXgNURAxr47NGKw2IiIXALNJKFMdVHZ4JDAeuLO4zJWmCpAmt1FPHDcA8YA9J7yyUP4z00i/Axb29A2ZmZv2jre02+sGxwE3ABZKmkoLGLsBk0u2106vOn5fTV4zYJL0VODp/XS+nW0m6onJORLyv8N8v5q3qZwNXS7oauA+YSlpBYw5pNQszMxtgpQhQEbFQUhdwFul22z7Aw6Tdeme2sEr6G4D3VuVtUpX3vqq6fy/pLaTR2jTSNPlFuS1nR8SK1npjZmadIN+96oyurq7o7u5u69q+LqTkH6GZrcokzY2Irur8ZiZJmJmZDTgHKDMzKyUHKDMzKyUHKDMzKyUHKDMzK6VSTDO3vinOAvSMPjNbXXgEZWZmpeQAZWZmpeQAZWZmpeQAZWZmpeQAZWZmpeQAZWZmpeQAZWZmpVSaACVpjKTLJD0kaYWkHknnS9qwxXI2ytf15HIeyuWOqXN+j6So83mkM70zM7NWleJFXUlbkjYs3AS4FrgL2Bk4EZguaVJELGminI1zOeNJmxB+B5gAHAHsK2nXiKi1ffuTwPk18pe30R0zM+uAUgQo4CJScDohIi6sZEo6F/gw8EngmCbK+RQpOJ0XER8plHMC8Plcz/Qa1z0REWe23XozM+u4Qd+wUNIWwEKgB9gyIl4qHFuftLOugE0i4uleyhkOLAZeAl4bEcsKx4blOsblOu4pHOsBiIhxfenHYG5YWI+XPTKzVUGZNyycktNZxeAEkIPMHGBdYGKDcnYF1gHmFINTLuclYFb+OrnGta+WdJik0ySdKGmypDVa7YiZmXVOGW7xbZ3TBXWO3w1MI926u66P5ZDLqbYp8PWqvHslHRERN9SrUNIMYAbA2LFje2mamZm1qgwjqBE5fbLO8Ur+yH4q53JgKilIDQe2A75Muh34M0nb16swIi6JiK6I6Bo9enSD5pmZWSvKMIJqpPKEpq9PVGqWExEzq877M3CMpOXAScCZwAF9rNvMzFpUhhFUZWQzos7xDarO6+9yKi7O6R5Nnm9mZh1UhgA1P6e1ng0BbJXTes+WOl1OxWM5Hd7k+WZm1kFlCFDX53Rang7+D3ma+STgWeDmBuXcnM+blK8rljOMNNGiWF8ju+a01ou9ZmbWzwY9QEXEQtIU8HHAcVWHZ5JGMFcW34GSNEHShKpylpNm4g0nPTcqOj6X/4uqd6DeJGmj6jZJ2gz4Qv76jZY7ZWZmfVaWSRLHkpYoukDSVGAesAvpnaUFwOlV58/LafUrrqcBewEfkbQDcAuwDbAf6ZZddQA8BDhV0vXAvcAyYEtgX2Bt4KfA5/rYNzMza0MpAlRELJTUBZxFWopoH9IKEhcAMyNiaZPlLJG0K3AGsD+wO7CENJX8ExHxQNUl15Pen9qRdEtvOPAE8FvSaOzrMdhLbZiZDVGDvtTR6qKMSx0V+cdsZmVV5qWOzMzMVuIAZWZmpeQAZWZmpVSKSRLW/4rPufw8ysxWBR5BmZlZKXkENQR5NGVmqwKPoMzMrJQ8ghriPJoys7LyCMrMzErJAcrMzErJAcrMzErJAcrMzEqpNAFK0hhJl0l6SNIKST2Szpe0YYvlbJSv68nlPJTLHdPfda/qpJc/ZmaDrRSz+CRtSdoPahPgWuAuYGfgRGC6pEkRsaSJcjbO5YwHZgPfASYARwD7Stq1uGFhJ+seKjzrz8wGSllGUBeRAsQJEbF/RJwaETUW17EAAA8sSURBVFOA80j7NX2yyXI+RQpO50XE1FzO/qRgs0mup7/qXm15ZGVmg2HQ94OStAWwEOgBtoyIlwrH1idtXChgk+K27zXKGQ4sBl4CXhsRywrHhuU6xuU67ulk3VD+/aD6m0dTZtauMu8HNSWns4oBAiAHmTnAusDEBuXsCqwDzCkGp1zOS8Cs/HVyP9RtdXj0ZWbtKkOA2jqnC+ocvzun4/uhnE7VPeQVA1EzQane+b1d22qwa6aO/gigDso2FAzE3/MyTJIYkdMn6xyv5I/sh3L6VLekGcCM/HW5pPkN2lhtFPB4i9eUTdN9aPUvcqMA12GjpM7/LAY4SA2pv08ltzr0o9/+bdewWa3MMgSoRipd7+tTjnbK6fWaiLgEuKTtBkndte67rkpWhz7A6tEP96E8Vod+lKEPZbjFVxmljKhzfIOq8zpZTqfqNjOzDitDgKrcFqv3nGernNZ7TtSXcjpVt5mZdVgZAtT1OZ2Wp4P/Q57qPQl4Fri5QTk35/Mm5euK5QwDplXV18m629X27cESWR36AKtHP9yH8lgd+jHofRj0ABURC0lTwMcBx1UdngkMB64svockaYKkCVXlLAe+ns8/s6qc43P5vyiuJNFO3Z2Un2Gt0laHPsDq0Q/3oTxWh36UoQ+D/qIu1FxuaB6wC+mdpQXAbsXlhiQFQESoqpzqpY5uAbYB9gMey+Us7EvdZmY2MEoRoAAkvR44C5gObExaxeGHwMyIWFp1bs0AlY9tBJwB7A+8FlgC/Az4REQ80Ne6zcxsYJQmQJmZmRUN+jOooWZV2NpD0sGSLpT0G0lPSQpJ32hwzW6SfippqaRnJP1J0ockrTFQ7a5qz8aSjpb0A0l/lfSspCcl/VbSUdWTYgrXlaofuU2fkXSdpPtzP5ZKulXSGfm2dq1rStePapIOz3+3QtLRdc55u6Rf55/dckm/l/TegW5roT09hTZXfx6pc00pfxaSdpd0jaSH8++ihyXNkrRPjXMHpQ8eQQ2gXrb2mEya8l6KrT0k3QZsDywHHiBtWfLNiDiszvn7AdcAzwHfBZYC7yAtJXV1RBwyEO2uatMxwJdIt2uvB+4DXgMcSHrv7RrgkCj8AyhjP3K7/g78EfgL6VnqcNL6kF3AQ8DEiLi/cH4p+1GUb6vfAawBrAe8PyK+UnXO8cCFpNv03wX+DhwMjAHOiYiTB7TRqU09pJVlzq9xeHlEfK7q/FL+LCR9HPhv0koRPyb9OxkF7AhcHxEfK5w7eH2ICH8G6AP8grQqxQer8s/N+RcPdhtzeyaT3gETsFdu2zfqnLsB6ZfmCqCrkL82KRgH8O5B6MOU/I9oWFX+pqRgFcBBZe9HpQ118j+Z23XRqtCPQlsE/Iq0k8D/5jYdXXXOONIvxCXAuEL+hsBf8zW7DkLbe4CeJs8t5c8COCTX/Utg/RrH1yxLHwbtL+lQ+wBb5B/mvTV+aa5PGq08DQwf7LZWta1RgDoyH/9ajWNT8rEbBrsfVe06LbfrwlW8H9tXftGsSv0g7c/2ErAH6ZWQWgHqrJw/s5W/cwPQ9lYCVOl+FqTHOvfk3zWjy96HVWEtvtVFr1t7SJpDepl4InDdQDeuDyr9+nmNYzcCzwC7SXp1RKwYuGb16vmcvlDIWxX78Y6c/qmQV+p+SNoGOBv4fETcKGlKnVN768fPqs4ZaK+WdBgwlvSL/k/AjRHxYtV5ZfxZ7AZsDlwN/E3SvsC2pNHqLRHxu6rzB7UPDlADp5mtPaaR3uFalQJU3X5FxAuS7gXeRBpBzhvIhtUi6VXAe/LX4j+60vdD0smk5zUjSM+f3kr65Xh24bTS9iP/2X+ddIv1tAan99aPhyU9DYyRtG5EPNPZlja0KakfRfdKOiIibijklfFn8ZacPkp6rrld8aCkG4GDI2JxzhrUPngW38Dp1LYiZbOq9ets0v8x/jQiflHIXxX6cTLpHb8PkYLTz4FphV8mUO5+fIL0EP59EfFsg3Ob7Ue9hZ77y+XAVFKQGk76Bf9l0jOzn0navnBuGX8Wm+T0GNIGr28jPWLYlvSMfA/ge4XzB7UPDlDl0altRcqmNP2SdAJwEmn25OGtXp7TQetHRGwa6eX0TUmzEbcAbpW0UwvFDEo/JO1MGjWdU+M2UltF5nRA+xERMyNidkQ8GhHPRMSfI+IY0kSndVh5mbXeDEYfKtPCRRopXRcRyyPiTuAA0qzdPSXt2mR5/doHB6iBs7pu7bFK9EvSccDnSVO1J8fKK4SsEv0AyL8cf0C6JbwxcGXhcOn6Ubi1twD4ryYva7YfT/WhaZ10cU73KOSV7mcB/C2n90TE7cUDeVRbuauwc04HtQ8OUANndd3ao26/8i+mzUmTEe6pPj5QJH0I+ALwZ1JwqvVCZen7US0iFpEC7pskjcrZZezHerk92wDPFV9uJd2yBLg051XeL+qtH68l3V57YBCeP9XzWE6HF/LK+LOotOmJOscrAWydqvMHpQ8OUANnsLf26C+zczq9xrE9gHWBmwZr5pukU4DzgNtIwemxOqeWuh+9+KecVmaQlbEfK4Cv1vncms/5bf5euf3XWz/+teqcMqjcEiv+oi7jz+JGUkDZStJaNY5vm9OenA5uHwZyDv5Q/7CKvKhb1ba9aPyi7mJK9jJibsN/5fq7gY0anFvKfpBW8di0Rv4wXn5Rd07Z+9FL/86k9ntQm1OyF3VJs9VW+nsEbEaahRvAaWX/WQDfyHX/T1X+3qT3054ARpahD17qaADVWOqolFt7SNqftBo8pAfy/0L6P8Pf5LzHo7DMTD7/atIvlO+QlkJ5J3kpFOBdMcB/0fJ6bVeQRhYXUvseeU9EXFG4poz9+BBptYUbSSsvLCEt2bQnaZLEI8DUiPhL4ZrS9aMeSWeSbvPVWurog8AFlGSpo9zWU0l3Q+4FlgFbAvuSfmH/FDggIv5euKZ0PwtJmwBzgDeQ/k3fQgqyB5ACzqER8b3C+YPXh4GO3kP9A7yeNFX1YdI/uEWkh/e9/h/+ALfxzPwXtd6np8Y1k0j/QP9GulV5B/BhYI2S9iGAX68C/dgW+CLpFuXjpNszTwJ/yH2s+fembP1o4ud0dJ3j7wBuIAWDp3O/3ztIbd0T+DZpFugTpBe+F5OWDHoPeW3TVeFnAWxEunNzb/49tIT0P80Ty9QHj6DMzKyUPEnCzMxKyQHKzMxKyQHKzMxKyQHKzMxKyQHKzMxKyQHKzMxKyQHKzMxKyQHKbIBJ6pHUM9jtMCs7BygzoLDC9iJJa9c5pyef452ozQaAA5TZK40l7Vjbn6bmj5n1wksdmZFGUKR1xgJ4FbBlRDxedU4PaVHNNSPihQFvpNkQ4xGU2cueAf6btMXAGQ3OfQVJ75J0o6QnJT0r6Q5J/ynp1TXOXekZlKS1JJ0g6Y+S/ibpmXzetZLeVqOMCZKukHS/pBWSHpX0LUlbt9DmvfItyzMl7SDpJ5KeyHXfIGm3OteNkPRpSfMlPZfb+4s67SzWsXOuY2nOG1f885C0nqTzcp+elXRbXkkbSa+SdJqku3OdCyUd32xfbdXkEZQZ/xhBPUjah2ge6VbfthGxoHBODzVGUJI+BfwnabXxq4HlpE313kRaiXvviHi+qhwiYlwh71vAv5N2/Z1NWjH6n4C3At+PV25vMh34PrAm8CPS/khjgANJ+/ZMjog/NtHnvUhbR/wEmELaLPDW3PeDSKtc7xAR8wvXjCRt1fBG0sris4FRwLtIO+f+v4j4co06ZpH2FvttrmMUae+kh/Kfx5rAfaRVtn8JrJX/PNYlbW1/LGlrmp/lPh5C2rbm3RHx3UZ9tVXUYC337o8/ZfqQbu09kP/74Pz9+1Xn9OT8VxXyds1591HYWJB0m/BHVG1iVyinp/B9BGmjuG5qbF8AbFz47w1JtyIfB95Ydd6bSMHxj032eS9e3nrkfVXHPpDzL6rK/3LO/zKF7SWArUjbgKzglRsMFuv4QJ12VP5cfwS8upC/e85fSgqGIwvHtiAF0FsH+++OP/338S0+syoRcTVpNHGApLc2OP3InP5PRDxSKOMF4CRS4Dm6UZWASL/cX6rRnuImlu8BRgJnRGGTwnzencClwI6S3tigzqI5Udi4MbuMtPfUzpUMSWsCh5GC4H9GxD9uv0TE3aTNBdfKbax2WxRGVnV8KApbh0fEb0j7FW0InBIRTxSO3UMayW0naY2GPbRVkqfLmtV2Emn343MkTSz+Mq6yU05nVx+IiAWSHgA2lzSy+Au26rynJP2ItDnfbZKuIe10+vuIeKbq9F1zun3e4bXa+JxuA/ylxvFaumu06XlJj5KCQ8UE0i23ORGxtEY5s4GPAzvWOHZLgzY8ERELa+Q/RLrtOrfGsQeBNUi7Pj/YoHxbBTlAmdUQEb+TdDXpdt+7SFuO1zIipw/XOf4w6ZnOCNIurPX8G3AKcCgwM+c9l9twckQ8mvM2zun7G3RhvQbHi+q16wVSAKhopq+QRnjVHqmRV/RkL20gImodrzwHXLNB2baK8i0+s/pOJW3r/WlJa9U5p/KLc9M6x19bdV5NEfFsRJwZEeNJAe0w0oSCw0gTL6rr2z4i1Mvnaw361o6+9NWzsaxlDlBmdeRbTheRbjF9sM5pt+Z0r+oDkt5Aml13b73be3XqvT8ivgn8C3A38FZJlZHTzTndvdnyOmg+aSr+DpI2rHF8ck4bziA0a4YDlFnvziLdAjud2rfNLsvpxyWNrmTmB/efI/0b+2pvFUgaLWmXGoeGA+uTbmX9PeddnttzhqSdqy+QNCxP7e64iPg78E3Sn8NZVfVuCZxAGnF+vT/qt6HHz6DMehERS/N7Tp+tc/wmSZ8FPgb8OT8zepr0HtS2pNt0/9ugmtcBN0uaRxp93E96WfjtpNtpF0TEslzfEkkHAz/I11wH3Ema/TeWNIliY6DmeoIdcCpp9Ha8pLeQ3nGqvAe1PnB8RNzbT3XbEOMAZdbYBaQXRcfVOhgRp0i6FTieNMV6TWAhaUbbOXnk0Zse0soVe5Fuk40ivfsznxQQvlNV33WS3gycTLoNuDtphPUQaSbdNS32r2k5YO9KejH5QOAjpJeKbwH+NyJm9VfdNvR4JQkzMyslP4MyM7NScoAyM7NScoAyM7NScoAyM7NScoAyM7NScoAyM7NScoAyM7NScoAyM7NScoAyM7NS+v87fJKVPrVsgQAAAABJRU5ErkJggg==\n", 183 | "text/plain": [ 184 | "
" 185 | ] 186 | }, 187 | "metadata": { 188 | "needs_background": "light" 189 | }, 190 | "output_type": "display_data" 191 | }, 192 | { 193 | "data": { 194 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de7xcVXn/8c+Xm0CAAEkQWwyBlJAqKuIpFyOQkJJGLIKC1lpUEIoWKGqxFcUKQdHWKtBQUdACIlakoPATrEa5Koh6IghCCBgJF8MlELkYIFzy/P5Ya2AymTlzn9nnzPf9es1rZ/Zl7Wdln5wna++111JEYGZmVjTr9DsAMzOzapygzMyskJygzMyskJygzMyskJygzMyskJygzMyskJygzMyskAqToCRtI+kcScskrZK0VNLpkrZoooxrJMUInw27WQczM+uc9fodAICkqcANwFbAZcAdwK7Ah4C5kmZExKNNFDmvxvrn2wrUzMx6phAJCjiTlJyOjYgzSislnQp8BDgF+GCjhUXESZ0O0MzMekv9HupI0vbAEmApMDUiVpdt2xR4ABCwVUSsrFPWNcDeEaGuBWxmZj1RhBbUPnm5oDw5AUTEk5KuB+YAuwNXNlKgpL8BtgOeBRYBV0XEqkYDmjhxYkyZMqXR3c3MrA0LFy58JCImVa4vQoLaMS/vrLH9LlKCmkaDCQq4sOL7w5KOjoiLGzl4ypQpDA8PN3gqMzNrh6R7qq0vQi++8Xn5eI3tpfWbN1DWZcD+wDbARsB04HP52G9LenOtAyUdKWlY0vDy5csbCtzMzLqnCAmqntLzpLoPyyLitIi4PCJ+HxHPRMTiiPgEcByprp8d4dizI2IoIoYmTVqrpWlmZj1WhARVaiGNr7F9s4r9WvE1UhfznXPHCzMzK7giJKjFeTmtxvYd8rLWM6q6IuIZ4Mn8dVyr5ZiZWe8UIUFdnZdzJK0RT27tzACeBm5s9QSSdgS2ICWpR1otx8zMeqfvCSoilgALgCnA0RWb55FaPOeXvwMlabqk6eU7Stpe0p9Wli9pInBu/nphRHg0CTOzUaAI3cwBjiINdTRf0mzSu0u7AbNIt/ZOqNh/UV6Wv5C7F/A1SdeSXvxdAUwG9iM93xoG/qVbFTAzs84qRIKKiCWShoCTgbmkpPIAMB+YFxErGihmIXAB8AZgZ1LniieBW4GLgLMi4tkuhG9mZl1QiAQFEBH3AYc1uO9aQxlFxK3AoR0Oq6NUEXWfR5kyMyu0vj+DMjMzq8YJyszMCskJyszMCskJyszMCskJyszMCskJyszMCskJyszMCskJyszMCskJyszMCskJyszMCskJyszMCskJyszMCskJyszMCskJyszMCskJyszMCqkw80GNVZVzQJmZWWPcgjIzs0JygjIzs0JygjIzs0JygjIzs0JygjIzs0JyL76CKO/tF9G/OMzMisItKDMzKyQnKDMzKyQnKDMzKyQnKDMzKyQnKDMzK6TCJChJ20g6R9IySaskLZV0uqQt2ihzL0kvSApJn+lkvGZm1l2F6GYuaSpwA7AVcBlwB7Ar8CFgrqQZEfFok2VuCnwdeArYpLMRm5lZtxWlBXUmKTkdGxEHRsTxEbEPcBqwI3BKC2X+JzAe+Fznwuws6aWPmZmtqe8JStL2wBxgKfClis0nAiuB90ga10SZBwCHAccCyzoTqZmZ9VLfExSwT14uiIjV5Rsi4kngemBjYPdGCpO0FfBV4NKIuKCTgZqZWe8UIUHtmJd31th+V15Oa7C8s0n1+mA7QZmZWX8VIUGNz8vHa2wvrd+8XkGS3g8cABwVEQ81E4SkIyUNSxpevnx5M4eamVkXFCFB1VPqQjDiEKqSpgCnA/8bERc1e5KIODsihiJiaNKkSU0HaWZmnVWEBFVqIY2vsX2ziv1qOQd4GjiqE0GZmVl/FSFBLc7LWs+YdsjLWs+oSnYhdVVfnl/MDUkBnJu3n5DXXdpeuGZm1gtFeFH36rycI2md8p58+WXbGaSW0Y11yjmf1Nuv0g7AXsDNwELgprYjNjOzrut7goqIJZIWkN6FOho4o2zzPGAccFZErCytlDQ9H3tHWTnHVitf0qGkBHVFRHyy4xUwM7OuaCpBlbVoJgMTSS2bh4GbI+K2NuI4ijTU0XxJs4FFwG7ALNKtvRMq9l9UCqmNc5qZWYHVTVCSNgL+FjicND5e6bnVGr3rJD0CXAJ8OSJubSaI3IoaAk4G5gL7AQ8A84F5EbGimfLMzGz0U0T13tuS1iMNFXQCsAXwDOkZzi+BB4EVwEbABGA6aaSHyaSE9WPguIj4TZfj74qhoaEYHh7uSFmtjLNX45KYmY1JkhZGxFDl+pFaUHcA2wE/II0KfllErKpzkmnAocB7gZskHR4R57cctZmZDayREtTtwEER8etGC4uIO4FPSDqJ9Fxpo/bCMzOzQVUzQUXEW1stNCKeJY3qYGZm1pIivKhrZma2lrbeg5K0HfCq/PX2iLi7/ZDMzMxaTFD5faivAQeXVgEh6SLgyDyPk5mZWctabUH9F2nkhxNJXc83BN4KvA94ivTOlJmZWctGTFCSNo6Ip6psehvwDxHxzbJ135W0cd7mBGVmZm2p10niVkmzqqxfD6h2G+9JCjC+n5mZjX71ksmNwI8lfRX457JnS1cC/yVpHGl08JcB+5Ne0L28W8GamdngGLEFFRF/R5pC/S3AbZLm5k1HAY8A3wRuA35FGkfv18AxXYvWzMwGRt3bcRFxuaTrgNOA70s6H/hwROwiaV/SOHwCbouIK7sbrpmZDYqGnhdFxBPA4ZIuBM4Gbpd0VERcCvyomwGamdlgamokiYj4EbAT8F3gEkkXSprQlcjMzGygNZSgJE2U9AZJEyNiZUQcTZpM8PXAIknv6mqUZmY2cEZMUJI2kfRt4CHgF8BDki6StElEXAe8DvgGcIGkSyVt3f2QzcxsENRrQX0OeAdpPqijgfNIwxv9G0BEPBMRx5Gmgd+B9Gzq0G4Fa2Zmg6NegjoAuDgi3h8RX4mIw0nTuh9QvlNE/BzYGfgycFZXIh0g0ksfM7NBVS9BjQPur1h3H7Bx5Y4R8VxEnADs2qHYzMxsgNVLUDcC75E0Q9IGkvYADsnrq2pmBl4zM7Na6iWoDwFPA9fl5U+BZ4APdzkuMzMbcCO+qBsRv5U0nTTO3mTgXuDyiFjZi+DMzGxwNTLU0VPAt3sQi5mZ2YuaGknCzMysV2omKEnHSdqw1YIl7SLpza0eb2Zmg22kFtRngSWSPibpTxspTMlfSfou8EvSSBNmZmZNG+kZ1GuAU0mjSXxG0g2kXnzDwAPAH4ANgQmkKTd2B2YDWwOPkuaF8ku7ZmbWkpoJKiLuBP5a0htJwxwdBOwJRJXdS2MeLAb+HTi3bPbdhkjahjTp4VxS0nsAuBSYFxF/aLCMfyYNYvsqYCKwGriHNCXIqRFR+dKxmZkVlCKq5ZsqO0qbAnsBbyJ1OZ9AejfqYeAW4JqIuK2lIKSpwA3AVsBlwB2kESlmkZLejIh4tIFyfgv8kTSz70PA+qQR1/cGngBmRsRN9coZGhqK4eHhVqpSJab2jm/w8piZjVqSFkbEUOX6hiYsBMgtoivyp9POJCWnYyPijNJKSacCHwFOAT7YQDk7RcQzlSsl/T1posVTgP06ErGZmXVV37uZS9oemAMsBb5UsflEYCVpuKVx9cqqlpyyi/JyhxbDNDOzHut7ggL2ycsFEbG6fENutV1PGpx29zbOsX9e3tJGGWZm1kMN3+Lroh3z8s4a2+8itbCmAVc2UqCkI4BtgE1IvRH/ktRZ4vi2IjUzs54pQoIan5eP19heWr95E2UeAexW9v2XwLsj4re1DpB0JHAkwOTJk5s4lZmZdUMRbvHVU+oH13B/tojYPSJE6mo+J69eKGnuCMecHRFDETE0adKk1qM1M7OOKEKCKrWQxtfYvlnFfg2LiEcj4kekJPU0cL6kjZoP0czMeq0ICWpxXk6rsb3U867WM6q6IuIx4GfAJODVrZZjZma903CCkvQ2Set2IYar83KOpDXiyS8HzyC1fmrO4tug0niCz7dZjpmZ9UAzLahLgHsknSypY70IImIJsACYQhpSqdw8YBxwfvkkiZKm54kUKVu3bX6nai2SPgD8BXAfcGunYjczs+5pphffmcC7gU8CH5f0A9JgsFdEo+Ml1XYUaaij+ZJmA4tIvfBmkW7tnVCx/6K8LB9I6PXAd/KgtneShjqaQHp/6jWkIZDeExEvtBmrmZn1QMMtqIg4BvgT4P2kEc3fQho37x5Jn2p0So4aZS8BhoDzSInpOGAqMB/Yo5Fx+IBfAacBG+TYPgr8Lan33xeBV0XEta3GaGZmvdXwYLFrHSjtRBof792kd5SeJ43Td1ZE/KBjEfaBB4s1M+udWoPFttyLLyJ+U9aqOox0S+2twBWS7pb00UbGzzMzM6umrW7mOQG9FziW1EtOpKkuJgCfB+6QtHO7QZqZ2eBpKUFJer2krwDLgK+Q3mH6GrBLROxCalUdTxrJYX6HYjUzswHScC8+SRuTOh18AHgDqbW0iJSgvh4RT5T2jYg/Ap+X9Erg8I5GbGZmA6GZbubLgE2BF0jvRJ0ZEdfUOeb3wIathWZmZoOsmQT1JKm79lcj4sEGjzkT+FbTUVlV5T0C3bvPzMa6ZhLUtpUTCtaTb/s9UXdHMzOzCs10kvixpPeOtIOkQyRd1WZMZmZmTSWomaTx8kayLbB3q8GYmZmVdHq6jY3waOFmZtYBzU75XvXRvCQBk4H9SCOGm5mZtWXEFpSk1ZJekFQaAfyk0vfyD6nV9DtgZ+DCLsdsZmYDoF4L6jpeajXtBdwLLK2y3wvAo8CVpBElzMzM2jJigoqImaU/S1oNnBsRJ3c7KDMzs2aeQW0HPNatQMzMzMo1nKAi4p5uBmJmZlauZoKS9CnS86cvRcSK/L0RERGf7kh0ZmY2sEZqQZ1ESlDfBlbk740IwAnKzMzaMlKCmpWX91Z8NzMz67qaCSoirh3pu/VG+QjmZmaDpNNDHZmZmXVEwwlK0hRJ+0kaV7ZuPUnzJP1a0g2S3tadMM3MbNA08x7UicBbgZeXrfsk8K9l3y+StGdE3NiJ4MzMbHA1c4tvD+DKiHgeQNI6wFHAHaSBYncFVgIf6XSQZmY2eJpJUC8Hyl/W3RmYSHpP6v6IGAYuA/6ig/GZmdmAaiZBrc+a023MyN/LZ9C9H3hFB+IyM7MB10yCuh94bdn3/YBHImJR2bqtgCc6EZiZmQ22ZhLU5cC+kr4g6TPAvsD/q9hnOmveBmyYpG0knSNpmaRVkpZKOl3SFg0eP07S30n6H0l3SFop6UlJw5KOk7RBK3GZmVl/NNOL7/PAgcA/5e+/J/XsA0DStsAbgdOaDULSVOAGUgvsMlLHi12BDwFzJc2IiEfrFLMncAFpWKargUuBLYH9gS8Ab5c0OyKeaTY+MzPrvWZGM39Y0muA2XnVtRHxZNkum5CS1w9biONMUnI6NiLOKK2UdCqpV+ApwAfrlPEgcAjwvxHxbFkZmwLXkJLn0cAXW4jPzMx6TBFRf69uBiBtDywhzdQ7NSJWl23bFHgAELBVRKxs8RzvBr4JXB4R+9fbf2hoKIaHh1s5VZVzd6SYtfT5spmZdYykhRExVLm+CEMd7ZOXC8qTE0BuoV0PbAzs3sY5nsvL59sow8zMeqiZZ1BI2hJ4P+n50BbAulV2i4iYXWV9LTvm5Z01tt8FzAGmAVc2UW659+flD1o83szMeqzhBCVpOulZziTSLbdamr35ND4vH6+xvbR+8ybLBUDSMcBc4GbgnBH2OxI4EmDy5MmtnMrMzDqomVt8XyB1ZPh3YHtg/YhYp8qnWquqHaVk2PRTF0lvB04ndaA4KCKeq7VvRJwdEUMRMTRp0qTWIu0hac2PmdlY08wtvj2BKyLiEx2OodRCGl9j+2YV+zVE0oHAhcDDwKyI+F1r4ZmZWT8004IScHsXYlicl9NqbN8hL2s9o1qLpHcA/ws8BOwdEYvrHGJmZgXTTIJayEsdGjrp6ryck0dIf1HuZj4DeBpoaAqP3KX8W8AyUnK6q4OxmplZjzSToE4G9pM0s5MBRMQSYAEwhfQibbl5wDjg/PJ3oCRNz5021iDpfcA3gHuBvXxbz8xs9GrmGdQrScMQLZD0LVKL6rFqO0bE+U3GcRRpqKP5kmYDi4DdgFmkW3snVOxfGqD2xe4BkmaReumtQ2qVHaa1ew88FhGnNxmbmZn1QcMjSUhaTepJV/5bv/Jgkd6Daronn6RXklppc4EJpBEkLgXmRcSKin2DdKLyBHUocG6d09wTEVPqxTIaRpKo5JElzGy0qjWSRDMtqMM6GM9aIuK+Rs9RnpjK1p0HnNfZqMzMrF+aGSz2690MxMzMrFwRxuIzMzNbS1Nj8QFImgQcBPw5MC4ijihbvx1wa0Q83dEozcxs4DQ7WOzhwHxgQ3KHCOCIvPnlwM9I49n9dwdjNDOzAdTwLT5J+wJnk7p9vw34cvn2iPgNcBtp1l0zM7O2NNOC+hip6/feEfGEpNdX2ecWYI+ORGZmZgOtmU4SQ6QZaZ8YYZ/7ga3bC8nMzKy5BLUBUG/K9c2BF1oPx8zMLGkmQS0F3lBnn914aXRyMzOzljWToC4D9sxTWaxF0mHAa4FLOhGYmZkNtmY6SXweeBfwLUkHkycYzFOq7wm8HbgLOKPTQZqZ2eBpZqijP0jaGzgfKG9Fzc/LnwDvLp8Ww8zMrFVNvagbEfcCMyW9ltSdfAJpKvYbI2JhF+IzM7MB1fRQRwARcQvpnSczM7OuaGUsvm2BSaRhjpbnVpWZmVlHNdSLT9JESadKegD4HfBz4BfA3ZKWSfoPSVt2M1AzMxssdROUpB2AYeBDpAFhXwAeBpbnP28N/BMwLGn77oVqZmaDZMQEJWkd4JvAZOBa4C+BTSLiFRGxNbApMAe4DpgCXNDVaM3MbGDUa0HNIY3BdxEwOyKuiohnSxsjYlVE/BjYB7gY2C2Pem5mZtaWegnqIGAV8I8REbV2ytuOAZ4DDu5ceNYo6aWPmdlYUC9B7QJcHxHL6xUUEQ8DP83HmJmZtaVegnolaRLCRt0GbNt6OGZmZkm9BLUZ8FgT5T1G6jhhZmbWlnoJagOam99pdT7GzMysLY28qFuzc4SZmVm3NDLU0UmSTup2IGZmZuUaSVDNdlx2i6tAyrud135RwMyseEa8xRcR67TwWbeVQCRtI+mcPLbfKklLJZ0uaYsmythX0hclXSlphaSQ9NNW4jEzs/5qabqNTpM0FbgB2Io0tfwdwK6k8f/mSpoREY82UNTRwAHAM8BvgYaT21jil3XNbCxoaDTzHjiTlJyOjYgDI+L4iNgHOA3YETilwXL+HdgJ2ATYvyuRmplZT/Q9QeUR0OcAS4EvVWw+EVgJvEfSuHplRcTPIuK2iGima7yZmRVQ3xMUaaBZgAURsbp8Q0Q8CVwPbAzs3uvAzMysf4qQoHbMyztrbL8rL6f1IJaO8MCtZmbtK0KCGp+Xj9fYXlq/eTeDkHSkpGFJw8uX1x0b18zMuqwICaqeUjukq2/xRMTZETEUEUOTJk3q5qnMzKwBRUhQpRbS+BrbN6vYz8zMBkAREtTivKz1jGmHvKz1jMrMzMagIiSoq/NyjqQ14pG0KTADeBq4sdeBmZlZ//Q9QUXEEmABMIU0EkS5ecA44PyIWFlaKWm6pOk9C3KMcO9CMxtNCjHUEXAUaaij+ZJmA4uA3YBZpFt7J1Tsvygv1/hVK+lNwBH56yZ5uYOk80r7RMShnQzczMy6oxAJKiKWSBoCTgbmAvsBDwDzgXkRsaLBov4MeF/Fuq0q1h3aXrRmZtYLhUhQABFxH3BYg/tWvUkVEecB53UuKjMz65e+P4MyMzOrxgnKzMwKqTC3+Kx/Knv1eeZdMysCt6DMzKyQnKDMzKyQfItvQPllXTMrOregzMyskJygzMyskJygzMyskJygzMyskJygzMyskJygzMyskNzN3NZS3gXdo0qYWb84QdmInKzMrF+coKwlTlxm1m1+BmVmZoXkBGVmZoXkW3zWMI/fZ2a95BaUmZkVkhOUmZkVkhOUmZkVkp9BWVfV6o7ubupmVo9bUGZmVkhOUGZmVki+xWdt8+06M+sGJyjrGb9HZWbNcIKyQnFrzMxKCvMMStI2ks6RtEzSKklLJZ0uaYsmy9kyH7c0l7Msl7tNt2K3l0hrfpo9xsyspBAtKElTgRuArYDLgDuAXYEPAXMlzYiIRxsoZ0IuZxpwFXAhMB04DHiLpD0i4nfdqYV1W60EVqv7eq19KvdzS82smIrSgjqTlJyOjYgDI+L4iNgHOA3YETilwXI+S0pOp0XE7FzOgaREt1U+j40SrbTGzGzsUPT5v4+StgeWAEuBqRGxumzbpsADgICtImLlCOWMA5YDq4FXRMSTZdvWyeeYks8xYitqaGgohoeHW62Sf5n2mFtQZqObpIURMVS5vggtqH3yckF5cgLISeZ6YGNg9zrl7AFsBFxfnpxyOauBBfnrrLYjtkJppJXV7rOxkT7NHj8IBq2+1h1FSFA75uWdNbbflZfTelSODZh2E0k7iavRbf2Oq1va/U9DL7R7vtGcrFv5We2kInSSGJ+Xj9fYXlq/eTfLkXQkcGT++kdJi+ucr9xE4JEm9h9LBrXuHal3vYTR7DHtnrPBfZqqezPJp1m9/qUvtXfdR2OSyiZKtevdgXptW21lERJUPaWqt/ukYMRyIuJs4OyWCpaGq90/HQSDWvdBrTe47oNY937Vuwi3+Eotm/E1tm9WsV+3yzEzswIoQoIq3Uqr9Wxoh7ys9Wyp0+WYmVkBFCFBXZ2Xc3J38BflbuYzgKeBG+uUc2Peb0Y+rrycdYA5FefrpJZuDY4Rg1r3Qa03uO6DqC/17nuCioglpC7gU4CjKzbPA8YB55e/AyVpuqTpFeX8EfhG3v+kinKOyeX/sBsjSeTnVwNpUOs+qPUG173fMfRDv+rd9xd1oepQR4uA3UjvLN0JvLF8qCNJARARqiincqijXwB/DhwAPJzLWdLt+piZWfsKkaAAJL0SOBmYC0wgjSBxKTAvIlZU7Fs1QeVtWwInAgcCrwAeBf4P+FRE3N/NOpiZWecUJkGZmZmV6/szqNGqU9ODjDa5nlHj82C/42uXpIMlnSHpJ5KeyPW6oM4xb5T0fUkrJD0l6RZJH5a0bq/iblcz9ZY0ZYSfgZB0Ya/jb5WkCZKOkPRdSb+V9LSkxyX9VNLhlR23yo4bC9e8qbr347qPhhd1C6dT04OMYo8Dp1dZ/8deB9IFnwReR6rL/aTpWmqSdABwCfAM8G1gBbA/aST+GcA7uhlsBzVV7+zXpNvwlX7Twbi67R3Al0mPFK4G7gVeDrwd+BrwZknviLJbTWPomjdd96x31z0i/GnyA/yQNCLFP1asPzWv/0q/Y+xi3ZcCS/sdRxfrN4v0zpyAmfl6XlBj381InW9WAUNl6zck/QcmgHf1u05dqPeUvP28fsfdgXrvQ0ou61Ss35r0CzuAg8boNW+27j2/7r7F16Q8Pcgc0i/qL1VsPhFYCbwnT/9ho0xEXB0Rd0X+F1nHwcAk4MKIeHF+loh4htQiAfiHLoTZcU3We8yIiKsi4nux9kwKDwJfyV9nlm0aS9e82br3nG/xNW/E6UEkXU9KYLsDV/Y6uB55maRDgMmkhHwLcF1EvNDfsHqu9LPwgyrbrgOeAt4o6WURsap3YfXMn0j6AKnX7aPAzyLilj7H1EnP5eXzZesG5ZpXq3tJz667E1TzGpnWYw7pXayxmqC2Jr0UXe5uSYdFxLX9CKhPav4sRMTzku4GXg1sT3q3b6zZN39eJOka4H0RcW9fIuoQSesB781fy5PRmL/mI9S9pGfX3bf4mtep6UFGq3OB2aQkNQ54DXAW6f70/0l6Xf9C67lB/Vl4Cvg08AZgi/zZm/SgfSZw5Ri4xf1vwE7A9yPih2XrB+Ga16p7z6+7E1TndWp6kEKKiHn53vVDEfFURPwmIj5I6iCyEWsPMzXIxuTPQkQ8HBGfiohfRcRj+XMd6c7Bz4E/A47ob5Stk3QscBypd+57mj08L0flNR+p7v247k5QzfO0HtWVHqru1dcoess/C2Ui4nlS92QYpT8Hko4G/hO4HZgVFaPYMIaveQN1r6qb190Jqnme1qO6h/NytN/aaUbNn4V8H3870kPmjg9QXGDL83LU/RxI+jDwX6T3eWbl3myVxuQ1b7DuI+nKdXeCal6npgcZa/bIy1H1D7NNV+Xl3Crb9gI2Bm4Y5b25mrV7Xo6qnwNJHyO9aHsz6Rf0wzV2HXPXvIm6j6Qr190JqknRwvQgY4WkV+fBeCvXb0v63xfAiMMCjTEXA48A75L04nTYkjYEPpO/frkfgXWTpN0kbVBl/T7AR/LXUfNzIOlfSR0DFgKzI+KREXYfU9e8mbr347p7sNgWVBnqaMTpQcYKSScBx5NakXcDTwJTgbeQ3qT/PvC2iHi2XzG2S9KBpJHwIfVU/CvS/wp/ktc9EhEfrdj/YtKwNxeShr15K6k78sXAO0fDy6/N1Dt3KX41cA1pWCSA1/LSO0L/GhGlX9aFJul9wHnAC8AZVH92tDQizis7Zqxc86bq3pfr3qshK8baB3glqcv1A8CzwD2kB4xb9ju2LtZ5b+BbpB4+j5Fe5lsO/Ij03oT6HWMH6ngSqQdWrc/SKsfMICXnP5Bu795K+h/luv2uTzfqDRwOXE4aTeWPpGF/7iWNS7dnv+vS4XoHcM2AXvM16t6P6+4WlJmZFZKfQZmZWSE5QZmZWSE5QZmZWSE5QZmZWSE5QZmZWSE5QZmZWSE5QZmZWSE5QZl1iaSlkpb2Ow6z0coJygaKpMife/L4adX2WZr38YzTZn3kBGWDajLw4S6fY3b+mFkLPNSRDRRJQRo/LYD1gKlRMYJzvi23LbB+pMnYzKwP3IKyQfQU8GnS7KcnNnOgpHdKuk7S45KelnSrpI9LelmVfdd6BiVpA0nHSvqVpD9Ieirvd5mkv6xSxnRJ50m6T9IqSQ9J+h9JOzYR88x8y9Q3a1cAAAUWSURBVPIkSTtLukLSY/nc10p6Y43jxkv6nKTFkp7J8f6wRpzl59g1n2NFXjel/O9D0iaSTst1elrSzXmEcCStJ+kTku7K51wi6ZhG62pji1tQNlByC+r3pJlPF5Fu9e0UEXeW7bOUKi0oSZ8FPk6aD+hi0ojObyZNQXAtsG9EPFdRDhExpWzd/wB/S5q59CrSSNh/ArwJ+E6sOZXHXOA7wPrA94DfAtsAbyeNJD0rIn7VQJ1nkqZIuYI0NcLPgJty3Q8ijca/c0QsLjtmc+B64FXAL3OsE4F3ApsA/xARZ1U5xwJgJvDTfI6JwCciYln++1ifNAL2lqRR8DfIfx8bA3OAo0hT1/xfruM7SNPavCsivl2vrjbG9HvId3/86eWHdGvv/vzng/P371TsszSvX69s3R553b3A1mXr1yMljyD9Iq4sZ2nZ9/HAamCYKtMyABPK/rwF6VbkI8CrKvZ7NSk5/qrBOs/kpekTDq3Y9oG8/syK9Wfl9WdRNo0KsANp3qBVwJQa5/hAjThKf6/fA15Wtn7PvH4FKRluXrZte1ICvanfPzv+9P7jW3w2sCLiYlJr4m2S3lRn9/fn5Wci4sGyMp4HjiMlniPqnRIQ6Zf76irxlE9y+V5gc+DEiLi9Yr/bgK8Cr5f0qjrnLHd9lE28l50DPA/sWlohaX3gEFIS/HhEvHibJSLuAuaTWj7vrXKOm6OsZVXDh6NsSvSI+AlpAswtgI9FxGNl235Hasm9RtK6dWtoY4q70dqgO440O/IXJe1e/su4wi55eVXlhoi4U9L9wHaSNi//BVux3xOSvgfsD9ws6RLSjLU/j4inKnbfIy9fl2cyrjQtL/8cuL3K9mqGq8T0nKSHSMmhZDrpltv1EbGiSjlXAZ8EXl9l2y/qxPBYRCypsn4Z6bbrwirbfg+sS5rp9/d1yrcxxAnKBlpE/EzSxaTbfe8kzQ5azfi8fKDG9gdIz3TGk2YbruVvgI8B7wbm5XXP5Bg+GhEP5XUT8vLv61Rhkzrby9WK63lSAihppK6QWniVHqyyrly1acVLMRAR1baXngOuX6dsG2N8i88MjidNX/85SRvU2Kf0i3PrGttfUbFfVRHxdEScFBHTSAntEFKHgkNIHS8qz/e6iNAIn6/XqVsr2qmre11ZxzhB2cDLt5zOJN1i+scau92UlzMrN0j6M1Lvurtr3d6rcd77IuKbwF8BdwFvklRqOd2Yl3s2Wl4HLSZ1xd9Z0hZVts/Ky7o9CM3a4QRllpxMugV2AtVvm52Tl5+UNKm0Mj+4/wLp39J/j3QCSZMk7VZl0zhgU9KtrGfzunNzPCdK2rXyAEnr5K7dHRcRzwLfJP09nFxx3qnAsaQW5ze6cX6zEj+DMgMiYkV+z+nzNbbfIOnzwL8Av8nPjFaS3oPaiXSb7j/qnOZPgRslLSK1Pu4jvSz816TbafMj4sl8vkclHQx8Nx9zJXAbqfffZFIniglA1fEEO+B4UuvtGEl/QXrHqfQe1KbAMRFxd5fObQY4QZmVm096UXRKtY0R8TFJNwHHkLpYrw8sIfVo+2JueYxkKWnkipmk22QTSe/+LCYlhAsrznelpNcCHyXdBtyT1MJaRupJd0mT9WtYTth7kF5MfjvwT6SXin8B/EdELOjWuc1KPJKEmZkVkp9BmZlZITlBmZlZITlBmZlZITlBmZlZITlBmZlZITlBmZlZITlBmZlZITlBmZlZITlBmZlZIf1/EVduN219N+MAAAAASUVORK5CYII=\n", 195 | "text/plain": [ 196 | "
" 197 | ] 198 | }, 199 | "metadata": { 200 | "needs_background": "light" 201 | }, 202 | "output_type": "display_data" 203 | } 204 | ], 205 | "source": [ 206 | "# lstm plot\n", 207 | "d = read_pickle(\"./acclip-sst-lstm_noise\")\n", 208 | "for i in range(7,13):\n", 209 | " noises=d[i]\n", 210 | " num_bins = len(noises)\n", 211 | " #plt.yticks(np.linspace(0,0.8,3))\n", 212 | " #plt.xticks(np.linspace(0,20,5))\n", 213 | " #plt.xlim(0,20)\n", 214 | " #plt.ylim(0,0.8)\n", 215 | " n, bins, patches = plt.hist(noises, 100, density=1, facecolor='blue')\n", 216 | " plt.ylabel('Density (%)')\n", 217 | " plt.xlabel('Noise norm')\n", 218 | " plt.xticks(fontsize=20)\n", 219 | " plt.yticks(fontsize=20)\n", 220 | " plt.rc('axes', labelsize=20)\n", 221 | " plt.tight_layout()\n", 222 | " \n", 223 | " plt.savefig('sst'+str(i)+'.png',dpi=300)\n", 224 | " plt.show()" 225 | ] 226 | }, 227 | { 228 | "cell_type": "code", 229 | "execution_count": 347, 230 | "metadata": { 231 | "scrolled": true 232 | }, 233 | "outputs": [ 234 | { 235 | "data": { 236 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAc+UlEQVR4nO3debhkVXnv8e8PAZFBQBrFxKHBgGiMIumgDEaQSIiJM3pNrjNEjSJxutdEjSCZjEYxaHCM4pSgD4R4HRKJoKIgaiM4AAJBWkBmWwQZBd77x95Hi2PVqapzqvrs7vp+nqeeTa29au1Vi9PnPWvvNaSqkCSpazZa7gpIktSPAUqS1EkGKElSJxmgJEmdZICSJHXSxstdgS5asWJFrVy5crmrIUnrnTPPPPPaqtp+EmUZoPpYuXIlq1evXu5qSNJ6J8kPJ1WWt/gkSZ1kgJIkdZIBSpLUSQYoSVInGaAkSZ1kgJIkdZIBSpLUSQYoSVInGaAkSZ3kShIzJBkvv3tZSlpO9qAkSZ1kgJIkdZIBSpLUSQYoSVInGaAkSZ1kgJIkdZIBSpLUSQYoSVInGaAkSZ1kgJIkdZIBSpLUSQYoSVInGaAkSZ1kgJIkdZIBSpLUSQYoSVInGaAkSZ1kgJIkdZIBSpLUSQYoSVInGaAkSZ1kgJIkdZIBSpLUSQYoSVInbbzcFdD6JemfXrVu6yFpw2cPSpLUSQYoSVInGaAkSZ1kgJIkdZIBSpLUSQYoSVInGaAkSZ3kPCgNNGjOkyStC/agJEmdZICSJHWSAUqS1EkGKElSJxmgJEmdZICSJHWSAUqS1EkGKElSJxmgJEmdNNZKEkm2AvYGHgCsAG4GrgbOrqpzJl89SdKsGhqgktwD+GPgYGAPftnrmlsIp9p81wInAO+uqu9OvqqSpFkyMEAl2Rg4DHg9sC1wC3AG8E3gSmAtcA9gO2BX4NHAS4AXJ/kC8Oqq+t5Uay9J2mAt1IP6PrAj8F/Ah4FPVdWtCxWWZBfg+cBzgbOSHFxVH5lQXSVJM2ShAHUu8PSq+vaohVXVBcDrkhwBvJSmhyVJ0tgGBqiqetJiC62q24B3LPbzkiQ5zFyS1ElL2rAwyY7AQ9u351bVxUuvkiRJiwxQ7XyoDwAHzSUBleSTwIuq6oYJ1U+SNKMW24N6F3AAcDhwJrAZ8CTgecBNNHOmJElatAUDVJLNq+qmPqeeCvxZVX28J+3EJJu35wxQkqQlGTZI4rtJ9uuTvjHQ7zbeDSzxuZYkSTA8mJwBfCHJ+4H/0/Ns6WTgXUm2AM4C7g48kWaC7memVVlJ0uxYsAdVVf8beDLwh8A5SQ5sT70UuBb4OHAO8C3gSODbwKFTq60kaWYMvR1XVZ9JcipwFPC5JB8BXlFVuyd5PM06fAHOqaqTp1tdSdKsGOl5UVVdDxyc5DjgfcC5SV5aVf8B/Pc0KyhJmk1jrSRRVf8NPAw4ETghyXFJtptKzSRJM22kAJVkRZLfTrKiqm6sqpcB+wGPBM5L8qyp1lKSNHMWDFBJtkzyCeAq4BvAVUk+mWTLqjoVeATwUeBjSf4jyQ7Tr7IkaRYM60H9PfAMmv2gXgYcS7O80ZsBquqWqno1zTbwO9M8m3r+tCorSZodwwLUk4Hjq+qFVfWeqjqYZlv3J/dmqqqvA7sB7wbeO5WaSpJmyrAAtQVw2by0S4HN52esqp9X1euBPSZUN0nSDBsWoM4AnpNk7ySbJtkTeHab3tc4O/BKkjTIsAD158DNwKnt8avALcArplwvSdKMW3CiblX9T5JdadbZewBwCfCZqrpxXVROkjS7Rlnq6CbgE+ugLpIk/cJYK0lIkrSuDAxQSV6dZLPFFpxk9yR/MEb++yX5YJLLk9yaZE2SdyTZdowyHp/kbUlOTrI2SSX56uK+gSRpOS3Ug/o74KIkr03y66MUlsbvJzkR+CbNShOjfO5BNFvHv4BmxYqjgB/QDNL42hjr/b0MeBWwF/CjET8jSeqghZ5B/RbwdprVJP4myek0o/hWA1cAPwE2A7aj2XLj0cD+wA7Aj2n2hRp10u4xwL2Bw6rqnXOJSd4OvBL4W+AlI5TzD8Drge8D9wcuHvH6kqSOSVUtnCHZi6Zn8nRgU6DfB9Iez6dZTeJDPbvvDit/J+AiYA3woKq6s+fcVjTBMMC9xxk9mGQlTYA6rar2GfVzAKtWrarVq1eP85H1QjI8z2IN+TGSNCOSnFlVqyZR1iij+E4HTk/yEuB3gX1ohpxvRzM36mrgO8CXquqcRdThce3xpN7g1F77hiSnAQfQ9NDcEFGSZsRIGxZCEyyAz7avSXpwe7xgwPkLaQLULhigJGlmdGGY+dbt8acDzs+lbzPNSiR5UZLVSVZfc80107yUJGkEXQhQw8w9OZnqU46qel9VraqqVdtvv/00LyVJGsHIt/imaK6HtPWA8/ecl09DTHMwxLjXdPCEpMXqQg/q/Pa4y4DzO7fHQc+oJEkboC4EqC+2xwOS3KU+7TDzvWlGCw7c4kOStOFZ9gBVVRcBJwEraeZb9XoTzaaJH+mdA5Vk13aVdUnSBqoLz6AAXgqcDhydZH/gPOBRwH40t/ZePy//ee3xLk8+kuwDHNK+3bI97pzk2Lk8VfX8SVZckjQdIweoJE8F/l9V3THpSlTVRUlWAUcCBwJPoFlB4mjgTVW1dsSifgN43ry0e89Le/7SatsdyzEYQpLWlXF6UCcAlyf5IPCBqrpkkhWpqktpFosdJW/fX81VdSxw7ORqJUlaLuM8gzoG2Bx4A80q559O8keJf8dLkiZv5ABVVYcCvwa8kGZF8z8EPgX8MMkbR92SQ5KkUYw1iq+qbqmqY6tqT+DhNCuXbwkcAVyc5MQkB06+mpKkWbPoYeZV9b2eXtULgKuAJwGfTXJxktck2WJC9ZQkzZglzYNqA9BzgcOAX6cZ9v1tmq043gJ8P8luS62kJGn2LCpAJXlkkvcAlwPvoVmm6APA7lW1O02v6i+AFTRDxSVJGss486A2B/4YeDHw2zS9pfNoAtSHq+r6ubxV9TPgLUnuDxw80RpLkmbCOPOgLge2Au6gmRN1TFV9achnfgRstriqSZJm2TgB6gbgbcD7q+rKET9zDPBvY9dKkjTzxglQD6yqO8cpvL3td/3QjJIkzTPOIIkvJHnuQhmSPDvJKUuskzYgSf+XJA0zToDal2ZLjIU8EHjsYisjSdKcSe8HdQ/g9gmXKUmaQePuB1X9EtsFYx9As03GpUutlCRJC/agktyZ5I4kc3tAHTH3vvdF02v6AbAbcNyU6yxJmgHDelCn8ste0+8ClwBr+uS7A/gxcDLNihKSJC3JggGqqvad++8kdwIfqqojp10pSZLGeQa1I3DdtCoiSVKvkQNUVf1wmhWRJKnXwACV5I00z5/+uarWtu9HUVX11xOpnSRpZi3UgzqCJkB9Aljbvh9FAQYoSdKSLBSg9muPl8x7L0nS1A0MUFX15YXeS5I0TZNe6kiSpIkYOUAlWZnkCUm26EnbOMmbknw7yelJnjqdakqSZs0486AOB54E3Kcn7Q3AX/W8/2SSx1TVGZOonCRpdo1zi29P4OSquh0gyUbAS4Hv0ywUuwdwI/DKSVdSkjR7xglQ9wF6J+vuBqygmSd1WVWtBj4F/M4E6ydJmlHjBKhNuOt2G3u373t30L0MuO8E6iVJmnHjBKjLgIf3vH8CcG1VndeTdm/g+klUTJI028YZJPEZ4JVJ/hG4BXg88KF5eXblrrcBJUlalHEC1FuApwCvat//iGZkHwBJHgjsBRw1sdpJkmbWOKuZX53kt4D926QvV9UNPVm2pAlen59g/SRJM2qcHhRVdTPNrb5+584BzplEpSRJcqkjSVInjdWDSnIv4IU0k3K3Be7WJ1tV1f590rUEyXLXYLIGfZ+q/umSZs/IASrJrsCXgO2BhX5d+itGkrRk49zi+0eaeU7/AOwEbFJVG/V59etVSZI0lnFu8T0G+GxVvW5alZEkac44ASrAudOqiDSIz6uk2TTOLb4zgQdPqyKSJPUaJ0AdCTwhyb5TqoskSb8wzi2++9Nsp3FSkn+j6VFd1y9jVX1kAnWTJM2wcQLUsTRDyAM8p33NfwqQNs0ApUXZ0OZ7SVq8cQLUC6ZWC0mS5hlnsdgPT7MikiT1ci0+SVInjbUWH0CS7YGnAw8BtqiqQ3rSdwS+2656Lk2V86OkDdu4i8UeDBwNbMYvB0Qc0p6+D/A14EXAv0ywjpKkGTTyLb4kjwfeB1wAPBV4d+/5qvoezX5QT5lkBSVJs2mcHtRrgSuAx1bV9Uke2SfPd4A9J1IzSdJMG2eQxCrgM1V1/QJ5LgN2WFqVJEkaL0BtCtw4JM82wB2Lr44kSY1xAtQa4LeH5HkUcP6iayNJUmucAPUp4DFJntHvZJIXAA8HTphExSRJs22cQRJvAZ4F/FuSg4CtAZIcSrOZ4dOAC4F3TrqSkqTZM85SRz9J8liahWB7e1FHt8evAH9SVcOeU0mSNNRYE3Wr6hJg3yQPpxlOvh3wU+CMqjpzCvWTJM2osZc6Aqiq79DMeZIkaSoWsxbfA4HtaZY5uqbtVUmSNFEjjeJLsiLJ25NcAfwA+DrwDeDiJJcneWuSe02zopKk2TI0QCXZGVgN/DnNgrB3AFcD17T/vQPwKmB1kp2mV1VJ0ixZMEAl2Qj4OPAA4MvA7wFbVtV9q2oHYCvgAOBUYCXwsanWVpI0M4b1oA6gWYPvk8D+VXVKVd02d7Kqbq2qLwCPA44HHtWuei5J0pIMC1BPB24FXl41eBu49tyhwM+BgyZXPUnSrBoWoHYHTquqa4YVVFVXA19tPyNJ0pIMC1D3p9mEcFTnAA9cfHUkSWoMC1D3BK4bo7zraAZOSJK0JMMC1KaMt7/Tne1nJElaklEm6g4cHCFJ0rSMstTREUmOmHZFJEnqNUqAyphl2uOSJC3ZggGqqsbZcVeSpIkxAEmSOskAJUnqJAOUJKmTDFCSpE4yQEmSOskAJUnqJAOUJKmTRpmoK20QMmDK+eCdziQtJ3tQkqROMkBJkjrJACVJ6iQDlCSpkxwkoQ3OoMEQktYv9qAkSZ1kgJIkdZIBSpLUSQYoSVInGaAkSZ1kgJIkdZIBSpLUSQYoSVInGaAkSZ1kgJIkdZIBSpLUSa7FJ02AmyFKk2cPSpLUSQYoSVInGaAkSZ3kMyjNvH7Pj3x2JC0/e1CSpE4yQEmSOskAJUnqJJ9Bdcyg+TRat2ZlXtOsfE+tn+xBSZI6yQAlSeokA5QkqZN8BiUtA+deScPZg5IkdZIBSpLUSQYoSVInGaAkSZ3kIIll4oRcrY+c2Kt1yR6UJKmTOhOgktwvyQeTXJ7k1iRrkrwjybZjlnOv9nNr2nIub8u937TqLkmavE7c4kvyIOB04N7Ap4DvA3sAfw4cmGTvqvrxCOVs15azC3AKcBywK/AC4A+T7FlVP5jOt5AkTVInAhRwDE1wOqyq3jmXmOTtwCuBvwVeMkI5f0cTnI6qqlf1lHMY8E/tdQ6cYL2lBU3iWeO4ZXTpedA4de9SvdUNqWX+qUiyE3ARsAZ4UFXd2XNuK+AKIMC9q+rGBcrZArgGuBO4b1Xd0HNuo/YaK9trLNiLWrVqVa1evXqxX2kkDpJYPw365zKJ/5+TKnucf9LjDnqYVP5xytD6JcmZVbVqEmV14RnU49rjSb3BCaANMqcBmwOPHlLOnsA9gNN6g1Nbzp3ASe3b/ZZcY0nS1HUhQD24PV4w4PyF7XGXdVSOJKkDuvAMauv2+NMB5+fSt5lmOUleBLyofXtrku8NuZ5GtwK4drkrMQnTvDU7RtkLtudyPPdajmtO0Abz89kRDx6eZTRdCFDDzP3YLvUO9YLlVNX7gPcBJFk9qXuosj0nzfacLNtzspJM7AF+F27xzfVsth5w/p7z8k27HElSB3QhQJ3fHgc9G9q5PQ56tjTpciRJHdCFAPXF9nhAOxz8F9ph5nsDNwNnDCnnjDbf3u3nesvZCDhg3vUW8r4R8mh0tudk2Z6TZXtO1sTac9kDVFVdRDMEfCXwsnmn3wRsAXykdw5Ukl2T7DqvnJ8BH23zHzGvnEPb8j8/ykoS7fMoTYjtOVm252TZnpM1yfZc9om60Hepo/OAR9HMWboA2Kt3qaMkBVBVmVfO/KWOvgE8BHgycHVbzkXT/j6SpKXrRIACSHJ/4EiapYi2o1lB4j+AN1XV2nl5+wao9ty9gMOBpwD3BX4M/Cfwxqq6bJrfQZI0OZ0JUJIk9Vr2Z1BdMantPmZFkoOSvDPJV5Jcn6SSfGzIZ/ZK8rkka5PclOQ7SV6R5G7rqt5dlWS7JIckOTHJ/yS5OclPk3w1ycHzBxD1fM42HSDJPyQ5OcmlbXuuTXJWksPbxwH9PmN7jijJc9p/95XkkAF5/ijJl9qf5Z8l+XqS5418DXtQC273sR/N8PWRtvuYJUnOBh4B/Ay4jGZbk49X1bMH5H8ycAJwC/AJYC3wRJpZ58dX1TPWRb27KslLgHfT3Nr+InAJcB/gaTRz+04AnlE9/2Bt04UluQ34FnAuzTPoLWjW9FwFXA48uqou7clve46ofSTzXeBuwJbAn1bVB+blORR4J81jlk8AtwEHAfcD3lZVrxl6oaqa+RfweZoVJl4+L/3tbfp7lruOXXvRBO+daVbo2Ldtp48NyHtPml8QtwKretI3o/nDoIBnLfd3Wub2fBzNL8ON5qXvQBOsCni6bTpWm242IP1v2/Y5xvZcVLsG+ALNDhFvbdvmkHl5VtIE+h8DK3vStwX+p/3MnsOuNfO3+NrtPg6g2e7jn+edPhy4EXhOu52HWlX1xaq6sNqfuiEOArYHjquqXyyDUlW3AG9o3/7ZFKq53qiqU6rq0/WrK/pfCbynfbtvzynbdIi2Lfr5ZHvcuSfN9hzdYTR/UL2A5vdjPy8E7g68q6rWzCVW1U9o9u2DEfb4m/kAxeS2+9Bgc238X33OnQrcBOyV5O7rrkrrlZ+3x9t70mzTxXtie/xOT5rtOYIkDwHeDPxTVZ26QNaF2vM/5+UZyADlNh3rwsA2rqrbgYtpFi7eaV1Wan2QZGPgue3b3n/stumIkrwmyRFJjkryFeCvaYLTm3uy2Z5DtD+LH6W55fy6IdkXas8raHpe90uy+UKFrA+rmU/bpLb70GC28eK9GXgY8Lmq+nxPum06utfQDDiZ81/A86vqmp4023O4NwKPBPapqpuH5B2lPbdo8900qBB7UMNNarsPDWYb95HkMODVNKNKnzPux9vjzLdpVe1QzaT+HWhGRe4EnJVk9zGKmen2TLIHTa/pbVX1tUkU2R4XbE8DlNt0rAu28ZiSvAz4J5oh0vvVvNVUsE3HVlVXVdWJNIOitgM+0nPa9hyg59beBcBfjfixUdvz+oUKMUC5Tce6MLCN2x/+HWkGAAxdyHcWJHkF8C7gezTB6co+2WzTRaqqH9IE/t9MsqJNtj0H25KmXR4C3NIzObdoRjoDvL9Ne0f7fqH2vC/N7b3Lqmrg7T0wQMHktvvQYKe0xwP7nPtdmlGSp1fVreuuSt2U5LXAUcDZNMHp6gFZbdOl+bX2eEd7tD0HuxX4lwGvs9o8X23fz93+W6g9/2BensGWe9JXF144UXep7bcvwyfqXoOTIIe141+1bbEauNeQvLbpwu2zK7BDn/SN+OVE3dNszyW38xH0n6i7IxOYqOtSR4y/3YcgyVNoVoyH5uHz79Pc/vhKm3Zt9Sxl0uY/nuaH9jiaZWSeRLuMDPDMmuEfxnZ9smNp/qJ/J/2fdaypqmN7PmObDtDeJn0rzRymi2h+Ud4HeCzNIIkrgf2r6tyez9ieY0pyBM1tvn5LHb0cOBqXOprIXwL3Bz5EsxbabcAPaR5SL/iX7Ky++OVfToNea/p8Zm/gc8BPaG6bfhd4JXC35f4+y/0aoT0L+JJtOnJ7PoxmZZizgWtpnh/9FPhm29Z9/13bnmO389zP7SEDzj8R+DJwA83cp28Czxu1fHtQkqROcpCEJKmTDFCSpE4yQEmSOskAJUnqJAOUJKmTDFCSpE4yQEmSOskAJS1RkjVJ1ix3PaQNjQFKG6SeFZd/mGSzAXnWtHncuFPqIAOUNnQPAF4x5Wvs374kTZBLHWmD1O5V8xOadcI2Bh5UVdfOy7MGeCCwSVXdvs4rKWlB9qC0IbsJ+GuarRQOH5L3LpI8M8mpSX6a5OYk303yl0nu3ifvrzyDSrJpksOSfCvJT5Lc1Ob7VJLf61PGrkmOTXJpkluTXJXkX5M8eIw679vesjwiyW5JPpvkuvbaX06y14DPbZ3k75Ocn+SWtr6fH1DP3mvs0V5jbZu2src9kmyZ5Kj2O92c5Ox2xXCSbJzkdUkubK95UZJDR/2umg32oLRBantQP6LZl+Y8mlt9D6uqC3ryrKFPDyrJ3wF/SbMK9vHAz2g2WftNmpWZH19VP59XDlW1siftX4E/ptkV9xSalbF/DdgH+Pe661YkBwL/DmwCfJpmv5z7AU+j2Z9ov6r61gjfeV+aDTg/CzyOZvO4s9rv/nSaVfp3q6rzez6zDXAa8FCalaZPAVYAz6TZSfXPquq9fa5xEs0+YF9tr7ECeF1VXd62xybAJcC9gP8GNm3bY3OaLddfSrOlzX+23/EZNNvdPKuqPjHsu2pGLPdy7b58TeNFc2vvsva/D2rf//u8PGva9I170vZs0y6hZ8M7mtuEn27Pva5POWt63m8N3Emz8eCvbNMAbNfz39vS3Iq8FnjovHy/SRMcvzXid96XX27N8fx5517cph8zL/29bfp7af9gbdN3ptme4lbuuuFc7zVePKAec+36aeDuPemPadPX0gTDbXrO7UQTQM9a7p8dX915eYtPG7yqOp6mN/HUJPsMyf7C9vg3VXVlTxm3A6+mCTyHDLskEJpf7nf2qU/v5pfPBbYBDq+ezfPafOcA7wcemeShQ67Z67Tq2diw9UGaPZH2mEtIsgnwbJog+JdV9YvbKVV1Ic1mc5u2dZzv7OrpWQ3wiurZIr2qvgJcTBOUX1tV1/Wc+wFNT+63ktxt6DfUTHB4rWbFq2l2TX5bkkf3/jKeZ/f2eMr8E1V1QZLLgB2TbNP7C3ZevuuTfJpms7azk5xAs9Pw16vqpnnZ92yPj2h3J51vl/b4EODcPuf7Wd2nTj9PchVNcJizK80tt9Oqam2fck4B3gA8ss+5bwypw3VVdVGf9Mtpbrue2efcj4C70ezQ/KMh5WsGGKA0E6rqa0mOp7nd90yaLaj72bo9XjHg/BU0z3S2BvoGqNb/Al4L/AnwpjbtlrYOr6mqq9q07drjnw75ClsOOd9rUL1upwkAc0b5rtD08Oa7sk9ar35b1s/Vgarqd37uOeAmQ8rWjPAWn2bJXwA/B/4+yaYD8sz94txhwPn7zsvXV1XdXFVHVNUuNAHt2TQDCp5NM/Bi/vUeUVVZ4PXhId9tMZbyXR1dpakzQGlmtLecjqG5xfTyAdnOao/7zj+R5DdoRtddPOj23oDrXlpVHwd+H7gQ2CfJXM/pjPb4mFHLm6DzaYbi75Zk2z7n92uPQ0cQStNggNKsOZLmFtjr6X/b7IPt8Q1Jtp9LbB/c/yPNv5l/WegCSbZP8qg+p7YAtqK5lXVbm/ahtj6HJ9lj/geSbNQO7Z64qroN+DhNOxw577oPAg6j6XF+dBrXl4bxGZRmSlWtbec5vWXA+dOTvAX4v8D32mdGN9LMg3oYzW26tw65zK8DZyQ5j6b3cSnNZOE/ormddnRV3dBe78dJDgJObD9zMnAOzei/B9AMotgO6Lue4AT8BU3v7dAkv0Mzx2luHtRWwKFVdfGUri0tyAClWXQ0zUTRlf1OVtVrk5wFHEozxHoT4CKaEW1va3seC1lDs3LFvjS3yVbQzP05nyYgHDfveicneTjwGprbgI+h6WFdTjOS7oQxv9/I2oC9J83E5KcBr6KZVPwN4K1VddK0ri0N40oSkqRO8hmUJKmTDFCSpE4yQEmSOskAJUnqJAOUJKmTDFCSpE4yQEmSOskAJUnqJAOUJKmT/j93Y42NXhPaXwAAAABJRU5ErkJggg==\n", 237 | "text/plain": [ 238 | "
" 239 | ] 240 | }, 241 | "metadata": { 242 | "needs_background": "light" 243 | }, 244 | "output_type": "display_data" 245 | } 246 | ], 247 | "source": [ 248 | "# lstm plot\n", 249 | "d = read_pickle(\"./adam-mrpc-bert_noise\")\n", 250 | "for i in [1]:\n", 251 | " noises=d[i]\n", 252 | " num_bins = len(noises)\n", 253 | " plt.yticks(np.linspace(0,0.5,6))\n", 254 | " plt.xticks(np.linspace(0,40,5))\n", 255 | " plt.xlim(0,40)\n", 256 | " #lt.ylim(0,0.8)\n", 257 | " n, bins, patches = plt.hist(noises, 100, density=1, facecolor='blue')\n", 258 | " plt.ylabel('Density (%)')\n", 259 | " plt.xlabel('Noise norm')\n", 260 | " plt.xticks(fontsize=20)\n", 261 | " plt.yticks(fontsize=20)\n", 262 | " plt.rc('axes', labelsize=20)\n", 263 | " plt.tight_layout()\n", 264 | " \n", 265 | " #plt.savefig('bert'+str(i)+'.png',dpi=300)\n", 266 | " plt.show()" 267 | ] 268 | } 269 | ], 270 | "metadata": { 271 | "kernelspec": { 272 | "display_name": "Python 3", 273 | "language": "python", 274 | "name": "python3" 275 | }, 276 | "language_info": { 277 | "codemirror_mode": { 278 | "name": "ipython", 279 | "version": 3 280 | }, 281 | "file_extension": ".py", 282 | "mimetype": "text/x-python", 283 | "name": "python", 284 | "nbconvert_exporter": "python", 285 | "pygments_lexer": "ipython3", 286 | "version": "3.7.4" 287 | }, 288 | "pycharm": { 289 | "stem_cell": { 290 | "cell_type": "raw", 291 | "metadata": { 292 | "collapsed": false 293 | }, 294 | "source": [] 295 | } 296 | } 297 | }, 298 | "nbformat": 4, 299 | "nbformat_minor": 2 300 | } 301 | -------------------------------------------------------------------------------- /optimizers/ACClip.py: -------------------------------------------------------------------------------- 1 | import torch 2 | from torch.optim import Optimizer 3 | import math 4 | 5 | class ACClip(Optimizer): 6 | 7 | 8 | def __init__(self, params, lr=1e-4, betas=(0.9, 0.99), eps=1e-5, 9 | weight_decay=1e-5, alpha=2, mod=1): 10 | if not 0.0 <= lr: 11 | raise ValueError("Invalid learning rate: {}".format(lr)) 12 | if not 0.0 <= eps: 13 | raise ValueError("Invalid epsilon value: {}".format(eps)) 14 | if not 0.0 <= betas[0] < 1.0: 15 | raise ValueError("Invalid beta parameter at index 0: {}".format(betas[0])) 16 | if not 0.0 <= betas[1] < 1.0: 17 | raise ValueError("Invalid beta parameter at index 1: {}".format(betas[1])) 18 | if not 1.0 <= alpha <= 2.0: 19 | raise ValueError("Invalid alpha parameter: {}".format(alpha)) 20 | 21 | defaults = dict(lr=lr, betas=betas, eps=eps, 22 | weight_decay=weight_decay, alpha=alpha, mod=mod) 23 | super(ACClip, self).__init__(params, defaults) 24 | 25 | def __setstate__(self, state): 26 | super(ACClip, self).__setstate__(state) 27 | 28 | def step(self, closure=None): 29 | 30 | loss = None 31 | if closure is not None: 32 | loss = closure() 33 | 34 | for group in self.param_groups: 35 | for p in group['params']: 36 | if p.grad is None: 37 | continue 38 | grad = p.grad.data 39 | if grad.is_sparse: 40 | raise RuntimeError('ACClip does not support sparse gradients') 41 | 42 | state = self.state[p] 43 | 44 | # State initialization 45 | if len(state) == 0: 46 | # the momentum term, i.e., m_0 47 | state['momentum'] = torch.zeros_like(p.data) 48 | # the clipping value, i.e., \tao_0^{\alpha} 49 | state['clip'] = torch.zeros_like(p.data) 50 | # second-order momentum, i.e., v_t 51 | state['second_moment'] = torch.zeros_like(p.data) 52 | # the number of step in total 53 | state['step'] = 0 54 | 55 | state['step'] += 1 56 | momentum, clip, second_moment = state['momentum'], state['clip'], state['second_moment'] 57 | beta1, beta2 = group['betas'] 58 | # bias_decay1 = 1 - beta1 ** state['step'] 59 | # bias_decay2 = 1 - beta2 ** state['step'] 60 | 61 | alpha = group['alpha'] 62 | 63 | if group['weight_decay'] != 0: 64 | grad.add_(group['weight_decay'], p.data) 65 | 66 | # update momentum and clip 67 | momentum.mul_(beta1).add_(1 - beta1, grad) 68 | clip.mul_(beta2).add_(1 - beta2, grad.abs().pow(alpha)) 69 | second_moment.mul_(beta2).addcmul_(1-beta2, grad, grad) 70 | 71 | # truncate large gradient 72 | denom = clip.pow(1/alpha).div(momentum.abs().add(group['eps'])).clamp(min=0.0, max=1.0) 73 | 74 | # calculate eta_t 75 | if group['mod'] == 1: 76 | denom.div_((second_moment.mul(beta2).sqrt()).add(group['eps'])) 77 | step_size = group['lr'] 78 | p.data.addcmul_(-step_size, denom, momentum) 79 | 80 | return loss 81 | 82 | -------------------------------------------------------------------------------- /run_mrpc.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team. 3 | # Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | """ Finetuning the library models for sequence classification on GLUE (Bert, XLM, XLNet, RoBERTa).""" 17 | 18 | from __future__ import absolute_import, division, print_function 19 | 20 | import argparse 21 | import glob 22 | import logging 23 | import os 24 | import random 25 | import pickle 26 | 27 | import numpy as np 28 | import torch 29 | from torch.optim import Adam, SGD 30 | from torch.utils.data import (DataLoader, RandomSampler, SequentialSampler, 31 | TensorDataset) 32 | from torch.utils.data.distributed import DistributedSampler 33 | 34 | try: 35 | from torch.utils.tensorboard import SummaryWriter 36 | except: 37 | from tensorboardX import SummaryWriter 38 | 39 | from tqdm import tqdm, trange 40 | 41 | from transformers import (WEIGHTS_NAME, BertConfig, 42 | BertForSequenceClassification, BertTokenizer, 43 | RobertaConfig, 44 | RobertaForSequenceClassification, 45 | RobertaTokenizer, 46 | XLMConfig, XLMForSequenceClassification, 47 | XLMTokenizer, XLNetConfig, 48 | XLNetForSequenceClassification, 49 | XLNetTokenizer, 50 | DistilBertConfig, 51 | DistilBertForSequenceClassification, 52 | DistilBertTokenizer) 53 | 54 | from transformers import AdamW, WarmupLinearSchedule 55 | 56 | from transformers import glue_compute_metrics as compute_metrics 57 | from transformers import glue_output_modes as output_modes 58 | from transformers import glue_processors as processors 59 | from transformers import glue_convert_examples_to_features as convert_examples_to_features 60 | from optimizers.ACClip import ACClip 61 | 62 | logger = logging.getLogger(__name__) 63 | 64 | ALL_MODELS = sum((tuple(conf.pretrained_config_archive_map.keys()) for conf in (BertConfig, XLNetConfig, XLMConfig, 65 | RobertaConfig, DistilBertConfig)), ()) 66 | 67 | MODEL_CLASSES = { 68 | 'bert': (BertConfig, BertForSequenceClassification, BertTokenizer), 69 | 'xlnet': (XLNetConfig, XLNetForSequenceClassification, XLNetTokenizer), 70 | 'xlm': (XLMConfig, XLMForSequenceClassification, XLMTokenizer), 71 | 'roberta': (RobertaConfig, RobertaForSequenceClassification, RobertaTokenizer), 72 | 'distilbert': (DistilBertConfig, DistilBertForSequenceClassification, DistilBertTokenizer) 73 | } 74 | 75 | 76 | def set_seed(args): 77 | random.seed(args.seed) 78 | np.random.seed(args.seed) 79 | torch.manual_seed(args.seed) 80 | if args.n_gpu > 0: 81 | torch.cuda.manual_seed_all(args.seed) 82 | 83 | 84 | def train(args, train_dataset, model, tokenizer): 85 | """ Train the model """ 86 | train_loss, test_loss, test_f1, test_acc = [], [], [], [] 87 | model_name = "{}-{}-{}".format(args.optimizer.lower(), args.task_name, args.model_type) 88 | if args.local_rank in [-1, 0]: 89 | tb_writer = SummaryWriter() 90 | 91 | args.train_batch_size = args.per_gpu_train_batch_size * max(1, args.n_gpu) 92 | train_sampler = RandomSampler(train_dataset) if args.local_rank == -1 else DistributedSampler(train_dataset) 93 | train_dataloader = DataLoader(train_dataset, sampler=train_sampler, batch_size=args.train_batch_size) 94 | 95 | if args.max_steps > 0: 96 | t_total = args.max_steps 97 | args.num_train_epochs = args.max_steps // (len(train_dataloader) // args.gradient_accumulation_steps) + 1 98 | else: 99 | t_total = len(train_dataloader) // args.gradient_accumulation_steps * args.num_train_epochs 100 | 101 | # Prepare optimizer and schedule (linear warmup and decay) 102 | no_decay = ['bias', 'LayerNorm.weight'] 103 | optimizer_grouped_parameters = [ 104 | {'params': [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)], 105 | 'weight_decay': args.weight_decay}, 106 | {'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], 'weight_decay': 0.0} 107 | ] 108 | if args.optimizer.lower() == "adamw": 109 | print ("We use AdamW optimizer!") 110 | optimizer = AdamW(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon) 111 | elif args.optimizer.lower() == "adam": 112 | print("We use Adam optimizer!") 113 | optimizer = Adam(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon) 114 | elif args.optimizer.lower() == "sgd": 115 | print("We use SGD optimizer!") 116 | optimizer = SGD(optimizer_grouped_parameters, lr=args.learning_rate, momentum=0.9) 117 | elif "acclip" in args.optimizer.lower(): 118 | print ("We use ACClip optimizer!") 119 | optimizer = ACClip(optimizer_grouped_parameters, lr=args.learning_rate) 120 | scheduler = WarmupLinearSchedule(optimizer, warmup_steps=args.warmup_steps, t_total=t_total) 121 | 122 | # multi-gpu training (should be after apex fp16 initialization) 123 | if args.n_gpu > 1: 124 | model = torch.nn.DataParallel(model) 125 | 126 | # Distributed training (should be after apex fp16 initialization) 127 | if args.local_rank != -1: 128 | model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.local_rank], 129 | output_device=args.local_rank, 130 | find_unused_parameters=True) 131 | # Train! 132 | logger.info("***** Running training *****") 133 | logger.info(" Num examples = %d", len(train_dataset)) 134 | logger.info(" Num Epochs = %d", args.num_train_epochs) 135 | logger.info(" Instantaneous batch size per GPU = %d", args.per_gpu_train_batch_size) 136 | logger.info(" Total train batch size (w. parallel, distributed & accumulation) = %d", 137 | args.train_batch_size * args.gradient_accumulation_steps * ( 138 | torch.distributed.get_world_size() if args.local_rank != -1 else 1)) 139 | logger.info(" Gradient Accumulation steps = %d", args.gradient_accumulation_steps) 140 | logger.info(" Total optimization steps = %d", t_total) 141 | 142 | global_step = 0 143 | tr_loss, logging_loss = 0.0, 0.0 144 | model.zero_grad() 145 | train_iterator = trange(int(args.num_train_epochs), desc="Epoch", disable=args.local_rank not in [-1, 0]) 146 | set_seed(args) # Added here for reproductibility (even between python 2 and 3) 147 | for _ in train_iterator: 148 | epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0]) 149 | for step, batch in enumerate(epoch_iterator): 150 | model.train() 151 | batch = tuple(t.to(args.device) for t in batch) 152 | inputs = {'input_ids': batch[0], 153 | 'attention_mask': batch[1], 154 | 'labels': batch[3]} 155 | if args.model_type != 'distilbert': 156 | inputs['token_type_ids'] = batch[2] if args.model_type in ['bert', 157 | 'xlnet'] else None # XLM, DistilBERT and RoBERTa don't use segment_ids 158 | outputs = model(**inputs) 159 | loss = outputs[0] # model outputs are always tuple in transformers (see doc) 160 | 161 | if args.n_gpu > 1: 162 | loss = loss.mean() # mean() to average on multi-gpu parallel training 163 | if args.gradient_accumulation_steps > 1: 164 | loss = loss / args.gradient_accumulation_steps 165 | 166 | loss.backward() 167 | 168 | tr_loss += loss.item() 169 | if (step + 1) % args.gradient_accumulation_steps == 0: 170 | if args.optimizer.lower() != "acclip": # make sure we don't clip for acclip 171 | torch.nn.utils.clip_grad_norm_(model.parameters(), args.max_grad_norm) # 172 | optimizer.step() 173 | scheduler.step() # Update learning rate schedule 174 | model.zero_grad() 175 | global_step += 1 176 | 177 | if args.local_rank in [-1, 0] and args.logging_steps > 0 and global_step % args.logging_steps == 0: 178 | # Log metrics 179 | if args.local_rank == -1 and args.evaluate_during_training: # Only evaluate when single GPU otherwise metrics may not average well 180 | results, eval_loss = evaluate(args, model, tokenizer) 181 | for key, value in results.items(): 182 | tb_writer.add_scalar('eval_{}'.format(key), value, global_step) 183 | print ('eval_{}'.format(key), value, global_step) 184 | print ('eval_loss', eval_loss, global_step) 185 | test_f1.append(results['f1']*100) 186 | test_acc.append(results['acc']*100) 187 | test_loss.append(eval_loss) 188 | tb_writer.add_scalar('lr', scheduler.get_lr()[0], global_step) 189 | iter_train_loss = (tr_loss - logging_loss) / args.logging_steps 190 | tb_writer.add_scalar('loss', iter_train_loss, global_step) 191 | print ("eval_Training_Loss_{}".format(iter_train_loss), global_step) 192 | logging_loss = tr_loss 193 | train_loss.append(iter_train_loss) 194 | 195 | if args.local_rank in [-1, 0] and args.save_steps > 0 and global_step % args.save_steps == 0: 196 | # Save model checkpoint 197 | output_dir = os.path.join(args.output_dir, 'checkpoint-{}'.format(global_step)) 198 | if not os.path.exists(output_dir): 199 | os.makedirs(output_dir) 200 | model_to_save = model.module if hasattr(model, 201 | 'module') else model # Take care of distributed/parallel training 202 | model_to_save.save_pretrained(output_dir) 203 | torch.save(args, os.path.join(output_dir, 'training_args.bin')) 204 | logger.info("Saving model checkpoint to %s", output_dir) 205 | 206 | if args.max_steps > 0 and global_step > args.max_steps: 207 | epoch_iterator.close() 208 | break 209 | if args.max_steps > 0 and global_step > args.max_steps: 210 | train_iterator.close() 211 | break 212 | 213 | if args.local_rank in [-1, 0]: 214 | tb_writer.close() 215 | 216 | if not os.path.exists("./curves"): 217 | os.mkdir("./curves") 218 | with open(os.path.join('./curves', model_name), "wb") as f: 219 | pickle.dump({'train_loss': train_loss, 'test_loss': test_loss, 220 | 'test_f1': test_f1, 'test_acc': test_acc}, f) 221 | return global_step, tr_loss / global_step 222 | 223 | 224 | def plot_noise(args, train_dataset, model, tokenizer): 225 | """ Train the model + Plot the noise""" 226 | # train_loss, test_loss, test_f1, test_acc = [], [], [], [] 227 | model_name = "{}-{}-{}".format(args.optimizer.lower(), args.task_name, args.model_type) 228 | # if args.local_rank in [-1, 0]: 229 | # tb_writer = SummaryWriter() 230 | 231 | args.train_batch_size = args.per_gpu_train_batch_size * max(1, args.n_gpu) 232 | train_sampler = RandomSampler(train_dataset) if args.local_rank == -1 else DistributedSampler(train_dataset) 233 | train_dataloader = DataLoader(train_dataset, sampler=train_sampler, batch_size=args.train_batch_size) 234 | 235 | if args.max_steps > 0: 236 | t_total = args.max_steps 237 | args.num_train_epochs = args.max_steps // (len(train_dataloader) // args.gradient_accumulation_steps) + 1 238 | else: 239 | t_total = len(train_dataloader) // args.gradient_accumulation_steps * args.num_train_epochs 240 | 241 | # Prepare optimizer and schedule (linear warmup and decay) 242 | no_decay = ['bias', 'LayerNorm.weight'] 243 | optimizer_grouped_parameters = [ 244 | {'params': [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)], 245 | 'weight_decay': args.weight_decay}, 246 | {'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], 'weight_decay': 0.0} 247 | ] 248 | if args.optimizer.lower() == "adamw": 249 | print("We use AdamW optimizer!") 250 | optimizer = AdamW(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon) 251 | elif args.optimizer.lower() == "adam": 252 | print("We use Adam optimizer!") 253 | optimizer = Adam(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon) 254 | elif args.optimizer.lower() == "sgd": 255 | print("We use SGD optimizer!") 256 | optimizer = SGD(optimizer_grouped_parameters, lr=args.learning_rate, momentum=0.9) 257 | elif args.optimizer.lower() == "acclip": 258 | print("We use ACClip optimizer!") 259 | optimizer = ACClip(optimizer_grouped_parameters, lr=args.learning_rate) 260 | scheduler = WarmupLinearSchedule(optimizer, warmup_steps=args.warmup_steps, t_total=t_total) 261 | 262 | # multi-gpu training (should be after apex fp16 initialization) 263 | if args.n_gpu > 1: 264 | model = torch.nn.DataParallel(model) 265 | 266 | # Distributed training (should be after apex fp16 initialization) 267 | if args.local_rank != -1: 268 | model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.local_rank], 269 | output_device=args.local_rank, 270 | find_unused_parameters=True) 271 | # Train! 272 | logger.info("***** Running training *****") 273 | logger.info(" Num examples = %d", len(train_dataset)) 274 | logger.info(" Num Epochs = %d", args.num_train_epochs) 275 | logger.info(" Instantaneous batch size per GPU = %d", args.per_gpu_train_batch_size) 276 | logger.info(" Total train batch size (w. parallel, distributed & accumulation) = %d", 277 | args.train_batch_size * args.gradient_accumulation_steps * ( 278 | torch.distributed.get_world_size() if args.local_rank != -1 else 1)) 279 | logger.info(" Gradient Accumulation steps = %d", args.gradient_accumulation_steps) 280 | logger.info(" Total optimization steps = %d", t_total) 281 | 282 | global_step = 0 283 | tr_loss, logging_loss = 0.0, 0.0 284 | model.zero_grad() 285 | train_iterator = trange(int(args.num_train_epochs), desc="Epoch", disable=args.local_rank not in [-1, 0]) 286 | set_seed(args) # Added here for reproductibility (even between python 2 and 3) 287 | 288 | epoch_num = 0 289 | diction = {} 290 | 291 | for _ in train_iterator: 292 | epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0]) 293 | # i = 0 --> calculate average gradient 294 | # i = 1 --> calculate noises 295 | # i = 2 --> optimizer.step() 296 | avg_grad = None 297 | noise_sample = list() 298 | for i in range(3): 299 | num = 0 300 | for step, batch in enumerate(epoch_iterator): 301 | model.train() 302 | batch = tuple(t.to(args.device) for t in batch) 303 | inputs = {'input_ids': batch[0], 304 | 'attention_mask': batch[1], 305 | 'labels': batch[3]} 306 | if args.model_type != 'distilbert': 307 | inputs['token_type_ids'] = batch[2] if args.model_type in ['bert', 308 | 'xlnet'] else None # XLM, DistilBERT and RoBERTa don't use segment_ids 309 | outputs = model(**inputs) 310 | loss = outputs[0] # model outputs are always tuple in transformers (see doc) 311 | 312 | if args.n_gpu > 1: 313 | loss = loss.mean() # mean() to average on multi-gpu parallel training 314 | if args.gradient_accumulation_steps > 1: 315 | loss = loss / args.gradient_accumulation_steps 316 | 317 | loss.backward() 318 | 319 | tr_loss += loss.item() 320 | if (step + 1) % args.gradient_accumulation_steps == 0: 321 | # if args.optimizer.lower() != "acclip": # make sure we don't clip for acclip 322 | # torch.nn.utils.clip_grad_norm_(model.parameters(), args.max_grad_norm) # 323 | if (i <= 1): 324 | emb_grad = None 325 | for j in model.parameters(): 326 | if (emb_grad is None): 327 | emb_grad = j.grad.data.view(-1) 328 | else: 329 | emb_grad = torch.cat((emb_grad, j.grad.data.view(-1))) 330 | if (i == 0): 331 | if (avg_grad is None): 332 | avg_grad = emb_grad 333 | else: 334 | avg_grad += emb_grad 335 | num += 1 336 | if (i == 1): 337 | diff = emb_grad - avg_grad 338 | l2_norm = torch.norm(diff, p=2) 339 | noise_sample.append(l2_norm.item()) 340 | if (i == 2): 341 | optimizer.step() 342 | scheduler.step() # Update learning rate schedule 343 | global_step += 1 344 | if args.local_rank in [-1, 345 | 0] and args.logging_steps > 0 and global_step % args.logging_steps == 0: 346 | # Log metrics 347 | if args.local_rank == -1 and args.evaluate_during_training: # Only evaluate when single GPU otherwise metrics may not average well 348 | results, eval_loss = evaluate(args, model, tokenizer) 349 | for key, value in results.items(): 350 | print('eval_{}'.format(key), value, global_step) 351 | print('eval_loss', eval_loss, global_step) 352 | model.zero_grad() 353 | 354 | if args.max_steps > 0 and global_step > args.max_steps: 355 | epoch_iterator.close() 356 | break 357 | if (i == 0): 358 | avg_grad /= num 359 | if (i == 1): 360 | diction[epoch_num] = noise_sample 361 | epoch_num += 1 362 | 363 | if args.max_steps > 0 and global_step > args.max_steps: 364 | train_iterator.close() 365 | break 366 | if not os.path.exists("./noises"): 367 | os.mkdir("./noises") 368 | with open(os.path.join('./noises', model_name + '_noise'), "wb") as f: 369 | pickle.dump(diction, f) 370 | 371 | return global_step, tr_loss / global_step 372 | 373 | 374 | def evaluate(args, model, tokenizer, prefix=""): 375 | # Loop to handle MNLI double evaluation (matched, mis-matched) 376 | eval_task_names = ("mnli", "mnli-mm") if args.task_name == "mnli" else (args.task_name,) 377 | eval_outputs_dirs = (args.output_dir, args.output_dir + '-MM') if args.task_name == "mnli" else (args.output_dir,) 378 | 379 | results = {} 380 | for eval_task, eval_output_dir in zip(eval_task_names, eval_outputs_dirs): 381 | eval_dataset = load_and_cache_examples(args, eval_task, tokenizer, evaluate=True) 382 | 383 | if not os.path.exists(eval_output_dir) and args.local_rank in [-1, 0]: 384 | os.makedirs(eval_output_dir) 385 | 386 | args.eval_batch_size = args.per_gpu_eval_batch_size * max(1, args.n_gpu) 387 | # Note that DistributedSampler samples randomly 388 | eval_sampler = SequentialSampler(eval_dataset) if args.local_rank == -1 else DistributedSampler(eval_dataset) 389 | eval_dataloader = DataLoader(eval_dataset, sampler=eval_sampler, batch_size=args.eval_batch_size) 390 | 391 | # Eval! 392 | logger.info("***** Running evaluation {} *****".format(prefix)) 393 | logger.info(" Num examples = %d", len(eval_dataset)) 394 | logger.info(" Batch size = %d", args.eval_batch_size) 395 | eval_loss = 0.0 396 | nb_eval_steps = 0 397 | preds = None 398 | out_label_ids = None 399 | for batch in tqdm(eval_dataloader, desc="Evaluating"): 400 | model.eval() 401 | batch = tuple(t.to(args.device) for t in batch) 402 | 403 | with torch.no_grad(): 404 | inputs = {'input_ids': batch[0], 405 | 'attention_mask': batch[1], 406 | 'labels': batch[3]} 407 | if args.model_type != 'distilbert': 408 | inputs['token_type_ids'] = batch[2] if args.model_type in ['bert', 409 | 'xlnet'] else None # XLM, DistilBERT and RoBERTa don't use segment_ids 410 | outputs = model(**inputs) 411 | tmp_eval_loss, logits = outputs[:2] 412 | 413 | eval_loss += tmp_eval_loss.mean().item() 414 | nb_eval_steps += 1 415 | if preds is None: 416 | preds = logits.detach().cpu().numpy() 417 | out_label_ids = inputs['labels'].detach().cpu().numpy() 418 | else: 419 | preds = np.append(preds, logits.detach().cpu().numpy(), axis=0) 420 | out_label_ids = np.append(out_label_ids, inputs['labels'].detach().cpu().numpy(), axis=0) 421 | 422 | eval_loss = eval_loss / nb_eval_steps 423 | if args.output_mode == "classification": 424 | preds = np.argmax(preds, axis=1) 425 | elif args.output_mode == "regression": 426 | preds = np.squeeze(preds) 427 | result = compute_metrics(eval_task, preds, out_label_ids) 428 | results.update(result) 429 | 430 | output_eval_file = os.path.join(eval_output_dir, prefix, "eval_results.txt") 431 | with open(output_eval_file, "w") as writer: 432 | logger.info("***** Eval results {} *****".format(prefix)) 433 | for key in sorted(result.keys()): 434 | logger.info(" %s = %s", key, str(result[key])) 435 | writer.write("%s = %s\n" % (key, str(result[key]))) 436 | 437 | return results, eval_loss 438 | 439 | 440 | def load_and_cache_examples(args, task, tokenizer, evaluate=False): 441 | if args.local_rank not in [-1, 0] and not evaluate: 442 | torch.distributed.barrier() # Make sure only the first process in distributed training process the dataset, and the others will use the cache 443 | 444 | processor = processors[task]() 445 | output_mode = output_modes[task] 446 | # Load data features from cache or dataset file 447 | cached_features_file = os.path.join(args.data_dir, 'cached_{}_{}_{}_{}'.format( 448 | 'dev' if evaluate else 'train', 449 | list(filter(None, args.model_name_or_path.split('/'))).pop(), 450 | str(args.max_seq_length), 451 | str(task))) 452 | if os.path.exists(cached_features_file) and not args.overwrite_cache: 453 | logger.info("Loading features from cached file %s", cached_features_file) 454 | features = torch.load(cached_features_file) 455 | else: 456 | logger.info("Creating features from dataset file at %s", args.data_dir) 457 | label_list = processor.get_labels() 458 | if task in ['mnli', 'mnli-mm'] and args.model_type in ['roberta']: 459 | # HACK(label indices are swapped in RoBERTa pretrained model) 460 | label_list[1], label_list[2] = label_list[2], label_list[1] 461 | examples = processor.get_dev_examples(args.data_dir) if evaluate else processor.get_train_examples( 462 | args.data_dir) 463 | features = convert_examples_to_features(examples, 464 | tokenizer, 465 | label_list=label_list, 466 | max_length=args.max_seq_length, 467 | output_mode=output_mode, 468 | pad_on_left=bool(args.model_type in ['xlnet']), 469 | # pad on the left for xlnet 470 | pad_token=tokenizer.convert_tokens_to_ids([tokenizer.pad_token])[0], 471 | pad_token_segment_id=4 if args.model_type in ['xlnet'] else 0, 472 | ) 473 | if args.local_rank in [-1, 0]: 474 | logger.info("Saving features into cached file %s", cached_features_file) 475 | torch.save(features, cached_features_file) 476 | 477 | if args.local_rank == 0 and not evaluate: 478 | torch.distributed.barrier() # Make sure only the first process in distributed training process the dataset, and the others will use the cache 479 | 480 | # Convert to Tensors and build dataset 481 | all_input_ids = torch.tensor([f.input_ids for f in features], dtype=torch.long) 482 | all_attention_mask = torch.tensor([f.attention_mask for f in features], dtype=torch.long) 483 | all_token_type_ids = torch.tensor([f.token_type_ids for f in features], dtype=torch.long) 484 | if output_mode == "classification": 485 | all_labels = torch.tensor([f.label for f in features], dtype=torch.long) 486 | elif output_mode == "regression": 487 | all_labels = torch.tensor([f.label for f in features], dtype=torch.float) 488 | 489 | dataset = TensorDataset(all_input_ids, all_attention_mask, all_token_type_ids, all_labels) 490 | return dataset 491 | 492 | 493 | def main(): 494 | parser = argparse.ArgumentParser() 495 | 496 | ## Required parameters 497 | parser.add_argument("--data_dir", default=None, type=str, required=True, 498 | help="The input data dir. Should contain the .tsv files (or other data files) for the task.") 499 | parser.add_argument("--model_type", default=None, type=str, required=True, 500 | help="Model type selected in the list: " + ", ".join(MODEL_CLASSES.keys())) 501 | parser.add_argument("--model_name_or_path", default=None, type=str, required=True, 502 | help="Path to pre-trained model or shortcut name selected in the list: " + ", ".join( 503 | ALL_MODELS)) 504 | parser.add_argument("--task_name", default=None, type=str, required=True, 505 | help="The name of the task to train selected in the list: " + ", ".join(processors.keys())) 506 | parser.add_argument("--output_dir", default=None, type=str, required=True, 507 | help="The output directory where the model predictions and checkpoints will be written.") 508 | parser.add_argument("--optimizer", default=AdamW, type=str, required=True, 509 | help="The optimizer") 510 | 511 | ## Other parameters 512 | parser.add_argument("--config_name", default="", type=str, 513 | help="Pretrained config name or path if not the same as model_name") 514 | parser.add_argument("--tokenizer_name", default="", type=str, 515 | help="Pretrained tokenizer name or path if not the same as model_name") 516 | parser.add_argument("--cache_dir", default="", type=str, 517 | help="Where do you want to store the pre-trained models downloaded from s3") 518 | parser.add_argument("--max_seq_length", default=128, type=int, 519 | help="The maximum total input sequence length after tokenization. Sequences longer " 520 | "than this will be truncated, sequences shorter will be padded.") 521 | parser.add_argument("--do_train", action='store_true', 522 | help="Whether to run training.") 523 | parser.add_argument("--do_eval", action='store_true', 524 | help="Whether to run eval on the dev set.") 525 | parser.add_argument("--evaluate_during_training", action='store_true', 526 | help="Rul evaluation during training at each logging step.") 527 | parser.add_argument("--do_lower_case", action='store_true', 528 | help="Set this flag if you are using an uncased model.") 529 | parser.add_argument("--do_plot", action='store_true', help="Whether to run plot.") 530 | parser.add_argument("--per_gpu_train_batch_size", default=8, type=int, 531 | help="Batch size per GPU/CPU for training.") 532 | parser.add_argument("--per_gpu_eval_batch_size", default=8, type=int, 533 | help="Batch size per GPU/CPU for evaluation.") 534 | parser.add_argument('--gradient_accumulation_steps', type=int, default=1, 535 | help="Number of updates steps to accumulate before performing a backward/update pass.") 536 | parser.add_argument("--learning_rate", default=5e-5, type=float, 537 | help="The initial learning rate for Adam.") 538 | parser.add_argument("--weight_decay", default=0.0, type=float, 539 | help="Weight deay if we apply some.") 540 | parser.add_argument("--adam_epsilon", default=1e-8, type=float, 541 | help="Epsilon for Adam optimizer.") 542 | parser.add_argument("--max_grad_norm", default=1.0, type=float, 543 | help="Max gradient norm.") 544 | parser.add_argument("--num_train_epochs", default=3.0, type=float, 545 | help="Total number of training epochs to perform.") 546 | parser.add_argument("--max_steps", default=-1, type=int, 547 | help="If > 0: set total number of training steps to perform. Override num_train_epochs.") 548 | parser.add_argument("--warmup_steps", default=0, type=int, 549 | help="Linear warmup over warmup_steps.") 550 | parser.add_argument('--logging_steps', type=int, default=10, 551 | help="Log every X updates steps.") 552 | parser.add_argument('--save_steps', type=int, default=100, 553 | help="Save checkpoint every X updates steps.") 554 | parser.add_argument("--eval_all_checkpoints", action='store_true', 555 | help="Evaluate all checkpoints starting with the same prefix as model_name ending and ending with step number") 556 | parser.add_argument("--no_cuda", action='store_true', 557 | help="Avoid using CUDA when available") 558 | parser.add_argument('--overwrite_output_dir', action='store_true', 559 | help="Overwrite the content of the output directory") 560 | parser.add_argument('--overwrite_cache', action='store_true', 561 | help="Overwrite the cached training and evaluation sets") 562 | parser.add_argument('--seed', type=int, default=42, 563 | help="random seed for initialization") 564 | parser.add_argument("--local_rank", type=int, default=-1, 565 | help="For distributed training: local_rank") 566 | args = parser.parse_args() 567 | 568 | if os.path.exists(args.output_dir) and os.listdir( 569 | args.output_dir) and args.do_train and not args.overwrite_output_dir: 570 | raise ValueError( 571 | "Output directory ({}) already exists and is not empty. Use --overwrite_output_dir to overcome.".format( 572 | args.output_dir)) 573 | 574 | # Setup CUDA, GPU & distributed training 575 | if args.local_rank == -1 or args.no_cuda: 576 | device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu") 577 | args.n_gpu = torch.cuda.device_count() 578 | else: # Initializes the distributed backend which will take care of sychronizing nodes/GPUs 579 | torch.cuda.set_device(args.local_rank) 580 | device = torch.device("cuda", args.local_rank) 581 | torch.distributed.init_process_group(backend='nccl') 582 | args.n_gpu = 1 583 | args.device = device 584 | 585 | # Setup logging 586 | logging.basicConfig(format='%(asctime)s - %(levelname)s - %(name)s - %(message)s', 587 | datefmt='%m/%d/%Y %H:%M:%S', 588 | level=logging.INFO if args.local_rank in [-1, 0] else logging.WARN) 589 | logger.warning("Process rank: %s, device: %s, n_gpu: %s, distributed training: %s", 590 | args.local_rank, device, args.n_gpu, bool(args.local_rank != -1)) 591 | 592 | # Set seed 593 | set_seed(args) 594 | 595 | # Prepare GLUE task 596 | args.task_name = args.task_name.lower() 597 | if args.task_name not in processors: 598 | raise ValueError("Task not found: %s" % (args.task_name)) 599 | processor = processors[args.task_name]() 600 | args.output_mode = output_modes[args.task_name] 601 | label_list = processor.get_labels() 602 | num_labels = len(label_list) 603 | 604 | # Load pretrained model and tokenizer 605 | if args.local_rank not in [-1, 0]: 606 | torch.distributed.barrier() # Make sure only the first process in distributed training will download model & vocab 607 | 608 | args.model_type = args.model_type.lower() 609 | config_class, model_class, tokenizer_class = MODEL_CLASSES[args.model_type] 610 | config = config_class.from_pretrained(args.config_name if args.config_name else args.model_name_or_path, 611 | num_labels=num_labels, 612 | finetuning_task=args.task_name, 613 | cache_dir=args.cache_dir if args.cache_dir else None) 614 | tokenizer = tokenizer_class.from_pretrained(args.tokenizer_name if args.tokenizer_name else args.model_name_or_path, 615 | do_lower_case=args.do_lower_case, 616 | cache_dir=args.cache_dir if args.cache_dir else None) 617 | model = model_class.from_pretrained(args.model_name_or_path, 618 | from_tf=bool('.ckpt' in args.model_name_or_path), 619 | config=config, 620 | cache_dir=args.cache_dir if args.cache_dir else None) 621 | 622 | if args.local_rank == 0: 623 | torch.distributed.barrier() # Make sure only the first process in distributed training will download model & vocab 624 | 625 | model.to(args.device) 626 | 627 | logger.info("Training/evaluation parameters %s", args) 628 | 629 | # Plotting 630 | if args.do_plot: 631 | if args.do_train: 632 | print ("Conflict Arguments!") 633 | print ("Do Plot and Do Train") 634 | raise 635 | else: 636 | print ("======= Plotting Noise Start =======\n") 637 | print ("====================================\n") 638 | train_dataset = load_and_cache_examples(args, args.task_name, tokenizer, evaluate=False) 639 | global_step, tr_loss = plot_noise(args, train_dataset, model, tokenizer) 640 | logger.info(" global_step = %s, average loss = %s", global_step, tr_loss) 641 | 642 | # Training 643 | if args.do_train: 644 | train_dataset = load_and_cache_examples(args, args.task_name, tokenizer, evaluate=False) 645 | global_step, tr_loss = train(args, train_dataset, model, tokenizer) 646 | logger.info(" global_step = %s, average loss = %s", global_step, tr_loss) 647 | 648 | # Saving best-practices: if you use defaults names for the model, you can reload it using from_pretrained() 649 | if args.do_train and (args.local_rank == -1 or torch.distributed.get_rank() == 0): 650 | # Create output directory if needed 651 | if not os.path.exists(args.output_dir) and args.local_rank in [-1, 0]: 652 | os.makedirs(args.output_dir) 653 | 654 | logger.info("Saving model checkpoint to %s", args.output_dir) 655 | # Save a trained model, configuration and tokenizer using `save_pretrained()`. 656 | # They can then be reloaded using `from_pretrained()` 657 | model_to_save = model.module if hasattr(model, 658 | 'module') else model # Take care of distributed/parallel training 659 | model_to_save.save_pretrained(args.output_dir) 660 | tokenizer.save_pretrained(args.output_dir) 661 | 662 | # Good practice: save your training arguments together with the trained model 663 | torch.save(args, os.path.join(args.output_dir, 'training_args.bin')) 664 | 665 | # Load a trained model and vocabulary that you have fine-tuned 666 | model = model_class.from_pretrained(args.output_dir) 667 | tokenizer = tokenizer_class.from_pretrained(args.output_dir) 668 | model.to(args.device) 669 | 670 | # Evaluation 671 | results = {} 672 | if args.do_eval and args.local_rank in [-1, 0]: 673 | tokenizer = tokenizer_class.from_pretrained(args.output_dir, do_lower_case=args.do_lower_case) 674 | checkpoints = [args.output_dir] 675 | if args.eval_all_checkpoints: 676 | checkpoints = list( 677 | os.path.dirname(c) for c in sorted(glob.glob(args.output_dir + '/**/' + WEIGHTS_NAME, recursive=True))) 678 | logging.getLogger("transformers.modeling_utils").setLevel(logging.WARN) # Reduce logging 679 | logger.info("Evaluate the following checkpoints: %s", checkpoints) 680 | for checkpoint in checkpoints: 681 | global_step = checkpoint.split('-')[-1] if len(checkpoints) > 1 else "" 682 | prefix = checkpoint.split('/')[-1] if checkpoint.find('checkpoint') != -1 else "" 683 | 684 | model = model_class.from_pretrained(checkpoint) 685 | model.to(args.device) 686 | result, eval_loss = evaluate(args, model, tokenizer, prefix=prefix) 687 | result = dict((k + '_{}'.format(global_step), v) for k, v in result.items()) 688 | results.update(result) 689 | return results 690 | 691 | 692 | if __name__ == "__main__": 693 | main() 694 | -------------------------------------------------------------------------------- /run_mrpc.sh: -------------------------------------------------------------------------------- 1 | export GLUE_DIR=./glue_data 2 | 3 | python run_mrpc.py \ 4 | --model_type bert \ 5 | --model_name_or_path bert-base-cased \ 6 | --task_name MRPC \ 7 | --optimizer acclip \ 8 | --do_plot \ 9 | --do_eval \ 10 | --do_lower_case \ 11 | --data_dir $GLUE_DIR/MRPC/ \ 12 | --max_seq_length 128 \ 13 | --per_gpu_train_batch_size 32 \ 14 | --learning_rate 2e-5 \ 15 | --num_train_epochs 5.0 \ 16 | --evaluate_during_training \ 17 | --overwrite_output_dir \ 18 | --output_dir ./outputs 19 | -------------------------------------------------------------------------------- /run_text_classifier.py: -------------------------------------------------------------------------------- 1 | from models.lstm_attn import LSTM_Attn 2 | import argparse 3 | import torch 4 | import torch.nn.functional as F 5 | from utils import load_dataset 6 | import sys, os, pickle 7 | from optimizers.ACClip import ACClip 8 | 9 | parser = argparse.ArgumentParser() 10 | parser.add_argument('--mode', type=str, default='train', help='mode=train or mode=plot') 11 | parser.add_argument('--model', type=str, default='lstm', help='name of the model') 12 | parser.add_argument('--checkpoint', type=str, default='', help='load existing checkpoint') 13 | parser.add_argument('--dataset', type=str, default='imdb', help='the dataset used for text classification') 14 | parser.add_argument('--epoch', type=int, default='20', help='The number of epochs') 15 | parser.add_argument('--optimizer', type=str, default='sgd', help='Type of otpimizer') 16 | parser.add_argument('--lr', type=float, default=0.001) 17 | parser.add_argument('--momentum', type=float, default=0.9) 18 | parser.add_argument('--batch_size', type=int, default=32) 19 | parser.add_argument('--hidden_size', type=int, default=256) 20 | parser.add_argument('--emb_dim', type=int, default=300) 21 | args = parser.parse_args() 22 | 23 | 24 | def print_noise(model, optimizer, train_batches, eval_batches, total_epoch, model_name): 25 | step = 0 26 | model.train() 27 | train_loss, test_loss, test_acc = [], [], [] 28 | diction = {} 29 | for epoch in range(total_epoch): 30 | # i = 0 -- calc average gradient 31 | # i = 1 -- calc noise 32 | # i = 2 -- update 33 | noise_sample = list() 34 | for i in range(3): 35 | local_step = 0 36 | total_loss = 0.0 37 | for batch in train_batches: 38 | optimizer.zero_grad() 39 | inputs, target = batch.text, batch.label 40 | if torch.cuda.is_available(): 41 | inputs, target = inputs.cuda(), target.cuda() 42 | pred = model(inputs) 43 | loss = F.cross_entropy(pred, target) 44 | total_loss += loss.item() 45 | loss.backward() 46 | params = list(model.parameters()) 47 | if (i <= 1): 48 | emb_grad = params[0].grad.data.view(-1) 49 | for j in range(2, 8): # params[1] is none and we need to skip 50 | emb_grad = torch.cat((emb_grad, params[j].grad.data.view(-1))) 51 | # the 0 th round -- calc average gradient 52 | if (i == 0): 53 | if (local_step == 0): 54 | avg_grad = emb_grad 55 | else: 56 | avg_grad += emb_grad 57 | # the 1 th round -- output noise 58 | if (i == 1): 59 | diff = emb_grad - avg_grad 60 | l2_norm = torch.norm(diff, p=2) 61 | # print(noise_sample,l2_norm) 62 | noise_sample.append(l2_norm.item()) 63 | # print(emb_grad.size()) 64 | if (i == 2): 65 | optimizer.step() 66 | step += 1 67 | local_step += 1 68 | if (i == 0): 69 | avg_grad /= local_step 70 | diction[epoch] = noise_sample 71 | eval_loss, eval_acc = eval(model, eval_batches) 72 | print ("validation loss {:.4f} and acc {:.4f}".format(eval_loss, eval_acc)) 73 | 74 | if not os.path.exists("./noises"): 75 | os.mkdir("./noises") 76 | with open(os.path.join('./noises', model_name + '_noise'), "wb") as f: 77 | pickle.dump(diction, f) 78 | 79 | def train(model, optimizer, train_batches, eval_batches, total_epoch, model_name): 80 | step = 0 81 | model.train() 82 | train_loss, test_loss, test_acc = [], [], [] 83 | for epoch in range(total_epoch): 84 | total_loss = 0.0 85 | for batch in train_batches: 86 | optimizer.zero_grad() 87 | inputs, target = batch.text, batch.label 88 | if torch.cuda.is_available(): 89 | inputs, target = inputs.cuda(), target.cuda() 90 | pred = model(inputs) 91 | loss = F.cross_entropy(pred, target) 92 | total_loss += loss.item() 93 | loss.backward() 94 | optimizer.step() 95 | step += 1 96 | if step % 100 == 0: 97 | sys.stdout.write('\rBatch[{}] - loss: {:.4f}\n'.format(step, loss.item())) 98 | train_loss.append(total_loss/len(train_batches)) 99 | eval_loss, eval_acc = eval(model, eval_batches) 100 | test_loss.append(eval_loss) 101 | test_acc.append(eval_acc) 102 | print ("validation loss {:.4f} and acc {:.4f}".format(eval_loss, eval_acc)) 103 | 104 | if not os.path.exists("./curves"): 105 | os.mkdir("./curves") 106 | with open(os.path.join('./curves', model_name), "wb") as f: 107 | pickle.dump({'train_loss': train_loss, 'test_loss': test_loss, 'test_acc': test_acc}, f) 108 | 109 | def eval(model, eval_batches): 110 | model.eval() 111 | total_epoch_loss = 0 112 | total_epoch_acc = 0 113 | model.eval() 114 | with torch.no_grad(): 115 | for idx, batch in enumerate(eval_batches): 116 | inputs = batch.text 117 | target = batch.label 118 | if torch.cuda.is_available(): 119 | inputs, target = inputs.cuda(), target.cuda() 120 | target = torch.autograd.Variable(target).long() 121 | prediction = model(inputs) 122 | loss = F.cross_entropy(prediction, target) 123 | num_corrects = (torch.max(prediction, 1)[1].view(target.size()).data == target.data).sum() 124 | acc = 100.0 * num_corrects/len(batch) 125 | total_epoch_loss += loss.item() 126 | total_epoch_acc += acc.item() 127 | model.train() 128 | return total_epoch_loss/len(eval_batches), total_epoch_acc/len(eval_batches) 129 | 130 | if __name__ == "__main__": 131 | print ("Start loading and processing dataset") 132 | train_batches, dev_batches, test_batches, class_num, vocab_size, word_embeds \ 133 | = load_dataset(dataset=args.dataset, batch_size=args.batch_size, word_dim=args.emb_dim) 134 | 135 | if args.model == "lstm": 136 | model = LSTM_Attn(args.batch_size, class_num, args.hidden_size, vocab_size, args.emb_dim, word_embeds) 137 | if torch.cuda.is_available(): 138 | model.cuda() 139 | 140 | if "sgd" in args.optimizer.lower(): 141 | optimizer = torch.optim.SGD(model.parameters(), lr=args.lr, momentum=args.momentum) 142 | elif args.optimizer == "adam": 143 | optimizer = torch.optim.Adam(model.parameters(), lr=args.lr) 144 | elif "acclip" in args.optimizer.lower(): 145 | optimizer = ACClip(model.parameters(), lr=args.lr, weight_decay=0) 146 | 147 | model_name = "{}-{}-{}".format(args.optimizer, args.dataset, args.model) 148 | if args.mode.lower() == 'plot': 149 | print ("Start Plotting Noise Norm!") 150 | print_noise(model, optimizer, train_batches, dev_batches, args.epoch, model_name) 151 | else: 152 | print ("Start training models!") 153 | train(model, optimizer, train_batches, dev_batches, args.epoch, model_name) -------------------------------------------------------------------------------- /utils.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import torch 4 | from torchtext import data 5 | from torchtext import datasets 6 | from torchtext.vocab import Vectors, GloVe 7 | 8 | 9 | def load_dataset(dataset="imdb", batch_size=32, word_dim=300): 10 | """ 11 | tokenizer : Breaks sentences into a list of words. If sequential=False, no tokenization is applied 12 | Field : A class that stores information about the way of preprocessing 13 | fix_length : An important property of TorchText is that we can let the input to be variable length, and TorchText will 14 | dynamically pad each sequence to the longest sequence in that "batch". But here we are using fi_length which 15 | will pad each sequence to have a fix length of 200. 16 | 17 | build_vocab : It will first make a vocabulary or dictionary mapping all the unique words present in the train_data to an 18 | idx and then after it will use GloVe word embedding to map the index to the corresponding word embedding. 19 | 20 | vocab.vectors : This returns a torch tensor of shape (vocab_size x embedding_dim) containing the pre-trained word embeddings. 21 | BucketIterator : Defines an iterator that batches examples of similar lengths together to minimize the amount of padding needed. 22 | """ 23 | 24 | tokenize = lambda x: x.split() 25 | TEXT = data.Field(sequential=True, tokenize=tokenize, lower=True, include_lengths=False, batch_first=True, 26 | fix_length=200) 27 | LABEL = data.LabelField(sequential=False) 28 | if dataset == "imdb": 29 | train_data, test_data = datasets.IMDB.splits(TEXT, LABEL) 30 | elif dataset == "sst": 31 | train_data, dev_data, test_data = datasets.SST.splits(TEXT, LABEL, 32 | fine_grained=False) 33 | print ("after split, we start to build vocab") 34 | TEXT.build_vocab(train_data, vectors=GloVe(name='6B', dim=word_dim), min_freq=10) 35 | LABEL.build_vocab(train_data) 36 | 37 | word_embeddings = TEXT.vocab.vectors 38 | print ("Length of Text Vocabulary: " + str(len(TEXT.vocab))) 39 | print ("Vector size of Text Vocabulary: ", TEXT.vocab.vectors.size()) 40 | print ("Label Length: " + str(len(LABEL.vocab))) 41 | 42 | train_data, valid_data = train_data.split() # Further splitting of train & validation 43 | print ("train: {}; dev: {}; test: {}".format(len(train_data), len(valid_data), len(test_data))) 44 | train_iter, valid_iter, test_iter = data.BucketIterator.splits((train_data, valid_data, test_data), 45 | batch_size=batch_size, 46 | sort_key=lambda x: len(x.text), repeat=False, 47 | shuffle=True) 48 | 49 | '''Alternatively we can also use the default configurations''' 50 | # train_iter, test_iter = datasets.IMDB.iters(batch_size=32) 51 | vocab_size = len(TEXT.vocab) 52 | class_num = len(LABEL.vocab) 53 | 54 | return train_iter, valid_iter, test_iter, class_num, vocab_size, word_embeddings --------------------------------------------------------------------------------