├── models └── .keep ├── images └── fhchat.png ├── floyd.yml ├── README.md ├── support.py └── language-identification.ipynb /models/.keep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /images/fhchat.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/floydhub/language-identification-template/master/images/fhchat.png -------------------------------------------------------------------------------- /floyd.yml: -------------------------------------------------------------------------------- 1 | env: tensorflow-1.7 2 | machine: cpu 3 | data: 4 | - source: floydhub/datasets/language-identification/1 5 | destination: languageidentification -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Language Identification 2 | 3 | [Language identification](https://en.wikipedia.org/wiki/Language_identification) is one of the most common feature of every Social Network or Web application, this is commonly paired with [Machine Translation](https://en.wikipedia.org/wiki/Machine_translation) to improve the user experience and content accesibility(a must have in the 2.0 society). *What can you use it for?* This is a foundation for other features such as Machine Translation (as mentioned before) and post/tweets/articles and documents analysis. 4 | 5 | ### Try it now 6 | 7 | [![Run on FloydHub](https://static.floydhub.com/button/button.svg)](https://floydhub.com/run?template=https://github.com/floydhub/language-identification-template) 8 | 9 | Click this button to open a Workspace on FloydHub that will train this model. 10 | 11 | ### Language identification of short pieces of text from Wikipedia 12 | 13 | In this notebook we will build a deep learning model able to [detect the languages from short piceces of text (140 characters, old Tweets lenght) with high accuracy using neural networks](http://machinelearningexp.com/deep-learning-language-identification-using-keras-tensorflow/). The task is commonly solved using hard-coded rules or NLP library, but we will attack the problem using Deep Learning. 14 | 15 | ![fhChat](https://raw.githubusercontent.com/floydhub/language-identification-template/master/images/fhchat.png) 16 | 17 | *Made with [Sketch Group Chat](https://www.sketchappsources.com/free-source/1558-group-chat-sketch-freebie-resource.html)* 18 | 19 | We have [already gathered and extract the raw dataset](https://floydhub.com/floydhub/datasets/language-identification/1) from https://dumps.wikimedia.org for 7 languages: Italian, Spanish and French which are considered to be in Latin language group, English and German have also common roots. Czech and Slovakian are extremely similar and are considered to be one of major challenged in the language recognition. 20 | 21 | iso-code | language | example 22 | ---------|----------|-------- 23 | en | English | Hello world! 24 | fr | French | Bonjour tout le monde! 25 | es | Spanish | Hola mundo! 26 | it | Italian | Ciao mondo! 27 | de | German | Hallo welt! 28 | cz | Czech | Ahoj světe! 29 | sk | Slovakian | Dobrý deň svet! 30 | 31 | We will: 32 | 33 | - Preprocess text data for NLP 34 | - Build and train Deep Neural Network using Keras and Tensorflow 35 | - Evaluate our model on the test set 36 | - Run the model on your own text! 37 | -------------------------------------------------------------------------------- /support.py: -------------------------------------------------------------------------------- 1 | import pandas as pd 2 | import matplotlib.pyplot as plt 3 | import seaborn as sns 4 | import re 5 | 6 | ############ 7 | # Alphabet # 8 | ############ 9 | 10 | # we will use alphabet for text cleaning and letter counting 11 | def define_alphabet(): 12 | base_en = 'abcdefghijklmnopqrstuvwxyz' 13 | special_chars = ' !?¿¡' 14 | german = 'äöüß' 15 | italian = 'àèéìíòóùú' 16 | french = 'àâæçéèêêîïôœùûüÿ' 17 | spanish = 'áéíóúüñ' 18 | czech = 'áčďéěíjňóřšťúůýž' 19 | slovak = 'áäčďdzdžéíĺľňóôŕšťúýž' 20 | all_lang_chars = base_en + german + italian + french + spanish + czech + slovak 21 | small_chars = list(set(list(all_lang_chars))) 22 | small_chars.sort() 23 | big_chars = list(set(list(all_lang_chars.upper()))) 24 | big_chars.sort() 25 | small_chars += special_chars 26 | letters_string = '' 27 | letters = small_chars + big_chars 28 | for letter in letters: 29 | letters_string += letter 30 | return small_chars,big_chars,letters_string 31 | 32 | ######## 33 | # Plot # 34 | ######## 35 | 36 | def print_confusion_matrix(confusion_matrix, class_names, figsize = (10,7), fontsize=14): 37 | """Prints a confusion matrix, as returned by sklearn.metrics.confusion_matrix, as a heatmap. 38 | 39 | Arguments 40 | --------- 41 | confusion_matrix: numpy.ndarray 42 | The numpy.ndarray object returned from a call to sklearn.metrics.confusion_matrix. 43 | Similarly constructed ndarrays can also be used. 44 | class_names: list 45 | An ordered list of class names, in the order they index the given confusion matrix. 46 | figsize: tuple 47 | A 2-long tuple, the first value determining the horizontal size of the ouputted figure, 48 | the second determining the vertical size. Defaults to (10,7). 49 | fontsize: int 50 | Font size for axes labels. Defaults to 14. 51 | 52 | Returns 53 | ------- 54 | matplotlib.figure.Figure 55 | The resulting confusion matrix figure 56 | 57 | FROM: https://gist.github.com/shaypal5/94c53d765083101efc0240d776a23823 58 | """ 59 | df_cm = pd.DataFrame( 60 | confusion_matrix, index=class_names, columns=class_names, 61 | ) 62 | fig = plt.figure(figsize=figsize) 63 | try: 64 | heatmap = sns.heatmap(df_cm, annot=True, fmt="d") 65 | except ValueError: 66 | raise ValueError("Confusion matrix values must be integers.") 67 | heatmap.yaxis.set_ticklabels(heatmap.yaxis.get_ticklabels(), rotation=0, ha='right', fontsize=fontsize) 68 | heatmap.xaxis.set_ticklabels(heatmap.xaxis.get_ticklabels(), rotation=45, ha='right', fontsize=fontsize) 69 | plt.title('Confusion Matrix') 70 | plt.ylabel('True label') 71 | plt.xlabel('Predicted label') 72 | return fig 73 | 74 | ################################### 75 | # Data cleaning utility functions # 76 | ################################### 77 | 78 | # we will create here several text-cleaning procedures. 79 | # These procedure will help us to clean the data we have for training, 80 | # but also will be useful in cleaning the text we want to classify, before the classification by trained DNN 81 | 82 | # remove XML tags procedure 83 | # for example, Wikipedia Extractor creates tags like this below, we need to remove them 84 | # ... 85 | def remove_xml(text): 86 | return re.sub(r'<[^<]+?>', '', text) 87 | 88 | # remove new lines - we need dense data 89 | def remove_newlines(text): 90 | return text.replace('\n', ' ') 91 | 92 | # replace many spaces in text with one space - too many spaces is unnecesary 93 | # we want to keep single spaces between words 94 | # as this can tell DNN about average length of the word and this may be useful feature 95 | def remove_manyspaces(text): 96 | return re.sub(r'\s+', ' ', text) 97 | 98 | # and here the whole procedure together 99 | def clean_text(text): 100 | text = remove_xml(text) 101 | text = remove_newlines(text) 102 | text = remove_manyspaces(text) 103 | return text 104 | 105 | ################# 106 | # Preprocessing # 107 | ################# 108 | 109 | # this function will get sample of texh from each cleaned language file. 110 | # It will try to preserve complete words - if word is to be sliced, sample will be shortened to full word 111 | def get_sample_text(file_content,start_index,sample_size): 112 | # we want to start from full first word 113 | # if the firts character is not space, move to next ones 114 | while not (file_content[start_index].isspace()): 115 | start_index += 1 116 | #now we look for first non-space character - beginning of any word 117 | while file_content[start_index].isspace(): 118 | start_index += 1 119 | end_index = start_index+sample_size 120 | # we also want full words at the end 121 | while not (file_content[end_index].isspace()): 122 | end_index -= 1 123 | return file_content[start_index:end_index] 124 | 125 | # we need only alpha characters and some (very limited) special characters 126 | # exactly the ones defined in the alphabet 127 | # no numbers, most of special characters also bring no value for our classification task 128 | # (like dot or comma - they are the same in all of our languages so does not bring additional informational value) 129 | 130 | # count number of chars in text based on given alphabet 131 | def count_chars(text, alphabet): 132 | alphabet_counts = [] 133 | for letter in alphabet: 134 | count = text.count(letter) 135 | alphabet_counts.append(count) 136 | return alphabet_counts 137 | 138 | # process text and return sample input row for DNN 139 | # note that we are counting separatey: 140 | # a) counts of all letters regardless of their size (whole text turned to lowercase letter) 141 | # b) counts of big letters only 142 | # this is because German uses big letters for beginning of nouns so this feature is meaningful 143 | def get_input_row(content,start_index,sample_size, alphabet): 144 | sample_text = get_sample_text(content,start_index,sample_size) 145 | counted_chars_all = count_chars(sample_text.lower(), alphabet[0]) 146 | counted_chars_big = count_chars(sample_text, alphabet[1]) 147 | all_parts = counted_chars_all + counted_chars_big 148 | return all_parts 149 | -------------------------------------------------------------------------------- /language-identification.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Language Identification\n", 8 | "\n", 9 | "[Language identification](https://en.wikipedia.org/wiki/Language_identification) is one of the most common feature of every Social Network or Web application, this is commonly paired with [Machine Translation](https://en.wikipedia.org/wiki/Machine_translation) to improve the user experience and content accesibility(a must have in the 2.0 society). The goal of the task is to detect the natural language of a given piece of text. *What can you use it for?* This is a foundation for other features such as Machine Translation (as mentioned before) and post/tweets/articles and documents analysis.\n", 10 | "\n", 11 | "### Language identification of short pieces of text from Wikipedia\n", 12 | "\n", 13 | "In this notebook we will build a deep learning model able to [detect the languages from short piceces of text (140 characters, old Tweets lenght) with high accuracy using neural networks](http://machinelearningexp.com/deep-learning-language-identification-using-keras-tensorflow/). The task is commonly solved using hard-coded rules or NLP library, but we will attack the problem using Deep Learning. \n", 14 | "\n", 15 | "\n", 16 | "\n", 17 | "*Made with [Sketch Group Chat](https://www.sketchappsources.com/free-source/1558-group-chat-sketch-freebie-resource.html)*\n", 18 | "\n", 19 | "We have [already gathered and extract the raw dataset](https://floydhub.com/floydhub/datasets/language-identification/1) from https://dumps.wikimedia.org for 7 languages: *Italian*, *Spanish* and *French* which are considered to be in Latin language group, *English* and *German* have also common roots. *Czech* and *Slovakian* are extremely similar and are considered to be one of major challenged in the language recognition.\n", 20 | "\n", 21 | "iso-code | language | example\n", 22 | "---------|----------|--------\n", 23 | "en | English | Hello world!\n", 24 | "fr | French | Bonjour tout le monde!\n", 25 | "es | Spanish | Hola mundo!\n", 26 | "it | Italian | Ciao mondo!\n", 27 | "de | German | Hallo welt!\n", 28 | "cz | Czech | Ahoj světe!\n", 29 | "sk | Slovakian | Dobrý deň svet!\n", 30 | "\n", 31 | "We will:\n", 32 | "\n", 33 | "- Preprocess text data for NLP\n", 34 | "- Build and train Deep Neural Network using Keras and Tensorflow\n", 35 | "- Evaluate our model on the test set\n", 36 | "- Run the model on your own text!\n", 37 | "\n", 38 | "\n", 39 | "### Instructions\n", 40 | "- To execute a code cell, click on the cell and press `Shift + Enter` (shortcut for Run).\n", 41 | "- To learn more about Workspaces, check out the [Getting Started Notebook](get_started_workspace.ipynb).\n", 42 | "- **Tip**: *Feel free to try this Notebook with your own data and on your own super awesome regression task.*\n", 43 | "\n", 44 | "Now, let's get started! 🚀" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": {}, 50 | "source": [ 51 | "## Initial Setup\n", 52 | "Let's start by importing some packages." 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": 1, 58 | "metadata": {}, 59 | "outputs": [ 60 | { 61 | "name": "stderr", 62 | "output_type": "stream", 63 | "text": [ 64 | "Using TensorFlow backend.\n" 65 | ] 66 | } 67 | ], 68 | "source": [ 69 | "import os\n", 70 | "import random\n", 71 | "import numpy as np\n", 72 | "import tensorflow as tf\n", 73 | "import time\n", 74 | "\n", 75 | "from sklearn import preprocessing\n", 76 | "from sklearn.metrics import classification_report\n", 77 | "from sklearn.model_selection import train_test_split\n", 78 | "\n", 79 | "import keras\n", 80 | "from keras.models import Sequential\n", 81 | "from keras.layers import Dense\n", 82 | "from keras.layers import Dropout\n", 83 | "import keras.optimizers" 84 | ] 85 | }, 86 | { 87 | "cell_type": "markdown", 88 | "metadata": {}, 89 | "source": [ 90 | "## Training Parameters\n", 91 | "\n", 92 | "We'll set the hyperparameters for training our model. If you understand what they mean, feel free to play around - otherwise, we recommend keeping the defaults for your first run 🙂" 93 | ] 94 | }, 95 | { 96 | "cell_type": "code", 97 | "execution_count": 2, 98 | "metadata": {}, 99 | "outputs": [], 100 | "source": [ 101 | "# Hyperparams if GPU is available\n", 102 | "if tf.test.is_gpu_available():\n", 103 | " # GPU\n", 104 | " BATCH_SIZE = 512 # Number of images used in each iteration\n", 105 | " EPOCHS = 12 # Number of passes through entire dataset\n", 106 | " \n", 107 | "# Hyperparams for CPU training\n", 108 | "else:\n", 109 | " # CPU\n", 110 | " BATCH_SIZE = 64\n", 111 | " EPOCHS = 12" 112 | ] 113 | }, 114 | { 115 | "cell_type": "markdown", 116 | "metadata": {}, 117 | "source": [ 118 | "## Data preparation\n", 119 | "\n", 120 | "*WARNING*\n", 121 | "\n", 122 | "Make sure that the dataset has been mounted before running the next Code Cells. The data mounting should take about 3 minutes." 123 | ] 124 | }, 125 | { 126 | "cell_type": "code", 127 | "execution_count": 3, 128 | "metadata": {}, 129 | "outputs": [ 130 | { 131 | "name": "stdout", 132 | "output_type": "stream", 133 | "text": [ 134 | "ALPHABET:\n", 135 | "abcdefghijklmnopqrstuvwxyzßàáâäæçèéêìíîïñòóôöùúûüýÿčďěĺľňœŕřšťůž !?¿¡ABCDEFGHIJKLMNOPQRSTUVWXYZÀÁÂÄÆÇÈÉÊÌÍÎÏÑÒÓÔÖÙÚÛÜÝČĎĚĹĽŇŒŔŘŠŤŮŸŽ\n", 136 | "ALPHABET LEN(VOCAB SIZE): 132\n" 137 | ] 138 | } 139 | ], 140 | "source": [ 141 | "#################\n", 142 | "# Configuration #\n", 143 | "#################\n", 144 | "\n", 145 | "# dictionary of languages that our classifier will cover\n", 146 | "LANGUAGES_DICT = {'en':0,'fr':1,'es':2,'it':3,'de':4,'sk':5,'cs':6}\n", 147 | "\n", 148 | "# Length of cleaned text used for training and prediction - 140 chars\n", 149 | "MAX_LEN = 140\n", 150 | "\n", 151 | "# number of language samples per language that we will extract from source files\n", 152 | "NUM_SAMPLES = 250000\n", 153 | "\n", 154 | "# For reproducibility\n", 155 | "SEED = 42\n", 156 | "\n", 157 | "from support import define_alphabet\n", 158 | "# Load the Alphabet\n", 159 | "alphabet = define_alphabet()\n", 160 | "print('ALPHABET:')\n", 161 | "print(alphabet[2])\n", 162 | "\n", 163 | "VOCAB_SIZE = len(alphabet[2])\n", 164 | "print('ALPHABET LEN(VOCAB SIZE):', VOCAB_SIZE)\n", 165 | "\n", 166 | "# Folders from where load / store the raw, source, cleaned, samples and train_test data\n", 167 | "data_directory = \"/floyd/input/languageidentification/data\"\n", 168 | "source_directory = os.path.join(data_directory, 'source')\n", 169 | "cleaned_directory = os.path.join(data_directory, 'cleaned')\n", 170 | "samples_directory = os.path.join('/tmp', 'samples')\n", 171 | "train_test_directory = os.path.join('/tmp', 'train_test')" 172 | ] 173 | }, 174 | { 175 | "cell_type": "markdown", 176 | "metadata": {}, 177 | "source": [ 178 | "Before feeding the data into the model, we have to preprocess the text.\n", 179 | "\n", 180 | "We will use the characters frequency as features to our model. This representation is similar to the [Bag of Words](https://en.wikipedia.org/wiki/Bag-of-words_model) model, with the exception that we are using characters and not words for defining the Vocabulary. You can see an example below:" 181 | ] 182 | }, 183 | { 184 | "cell_type": "code", 185 | "execution_count": 4, 186 | "metadata": {}, 187 | "outputs": [ 188 | { 189 | "name": "stdout", 190 | "output_type": "stream", 191 | "text": [ 192 | "1. SAMPLE TEXT: \n", 193 | " die Fähre \"Ibn Batouta\", und den Mondkrater Ibn Battuta. Liste der Gemeinden in Österreich Dies ist eine Zusammenstellung von Listen der\n", 194 | "\n", 195 | "2. REFERENCE ALPHABET: \n", 196 | " ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', 'ß', 'à', 'á', 'â', 'ä', 'æ', 'ç', 'è', 'é', 'ê', 'ì', 'í', 'î', 'ï', 'ñ', 'ò', 'ó', 'ô', 'ö', 'ù', 'ú', 'û', 'ü', 'ý', 'ÿ', 'č', 'ď', 'ě', 'ĺ', 'ľ', 'ň', 'œ', 'ŕ', 'ř', 'š', 'ť', 'ů', 'ž', ' ', '!', '?', '¿', '¡', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', 'À', 'Á', 'Â', 'Ä', 'Æ', 'Ç', 'È', 'É', 'Ê', 'Ì', 'Í', 'Î', 'Ï', 'Ñ', 'Ò', 'Ó', 'Ô', 'Ö', 'Ù', 'Ú', 'Û', 'Ü', 'Ý', 'Č', 'Ď', 'Ě', 'Ĺ', 'Ľ', 'Ň', 'Œ', 'Ŕ', 'Ř', 'Š', 'Ť', 'Ů', 'Ÿ', 'Ž']\n", 197 | "\n", 198 | "3. SAMPLE INPUT ROW: \n", 199 | " [6, 4, 1, 8, 18, 1, 2, 2, 11, 0, 1, 4, 4, 13, 3, 0, 0, 7, 7, 11, 5, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 20, 0, 0, 0, 0, 0, 2, 0, 1, 0, 1, 1, 0, 2, 0, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n", 200 | "\n", 201 | "4. INPUT SIZE (VOCAB SIZE): 132\n" 202 | ] 203 | } 204 | ], 205 | "source": [ 206 | "from support import get_sample_text, get_input_row\n", 207 | " \n", 208 | "# let's see if our processing is returning counts\n", 209 | "# last part calculates also input_size for DNN so this code must be run before DNN is trained\n", 210 | "path = os.path.join(cleaned_directory, \"de_cleaned.txt\")\n", 211 | "with open(path, 'r') as f:\n", 212 | " content = f.read()\n", 213 | " random_index = random.randrange(0,len(content)-2*MAX_LEN)\n", 214 | " sample_text = get_sample_text(content,random_index,MAX_LEN)\n", 215 | " print (\"1. SAMPLE TEXT: \\n\", sample_text)\n", 216 | " print (\"\\n2. REFERENCE ALPHABET: \\n\", alphabet[0]+alphabet[1])\n", 217 | " \n", 218 | " sample_input_row = get_input_row(content, random_index, MAX_LEN, alphabet)\n", 219 | " print (\"\\n3. SAMPLE INPUT ROW: \\n\",sample_input_row)\n", 220 | " \n", 221 | " input_size = len(sample_input_row)\n", 222 | " if input_size != VOCAB_SIZE:\n", 223 | " print(\"Something strange happened!\")\n", 224 | " \n", 225 | " print (\"\\n4. INPUT SIZE (VOCAB SIZE): \", input_size)\n", 226 | " del content" 227 | ] 228 | }, 229 | { 230 | "cell_type": "markdown", 231 | "metadata": {}, 232 | "source": [ 233 | "Now we will apply the transformation from raw text to Bag of Characters representation for all the data we have collected. At the end of the proprocessing, we will have 250k samples per language where every sample will be piece of text 140 characters long, represented using the Bag of Characters model.\n", 234 | "\n", 235 | "Dataset dimension (1750k, 133):\n", 236 | "- rows: 1750k (250k * 7) or (NUM_SAMPLES * num_languages)\n", 237 | "- columns: 133 (132 + 1) or (VOCAB_SIZE + language_index)" 238 | ] 239 | }, 240 | { 241 | "cell_type": "code", 242 | "execution_count": 5, 243 | "metadata": {}, 244 | "outputs": [ 245 | { 246 | "name": "stdout", 247 | "output_type": "stream", 248 | "text": [ 249 | "Processing file : /floyd/input/languageidentification/data/cleaned/en_cleaned.txt\n", 250 | "File size : 101.42 MB | # possible samples : 768340 | # skip chars : 199\n", 251 | "----------------------------------------------------------------------------------------------------\n", 252 | "Processing file : /floyd/input/languageidentification/data/cleaned/fr_cleaned.txt\n", 253 | "File size : 98.72 MB | # possible samples : 747899 | # skip chars : 191\n", 254 | "----------------------------------------------------------------------------------------------------\n", 255 | "Processing file : /floyd/input/languageidentification/data/cleaned/es_cleaned.txt\n", 256 | "File size : 97.56 MB | # possible samples : 739111 | # skip chars : 187\n", 257 | "----------------------------------------------------------------------------------------------------\n", 258 | "Processing file : /floyd/input/languageidentification/data/cleaned/it_cleaned.txt\n", 259 | "File size : 101.89 MB | # possible samples : 771891 | # skip chars : 200\n", 260 | "----------------------------------------------------------------------------------------------------\n", 261 | "Processing file : /floyd/input/languageidentification/data/cleaned/de_cleaned.txt\n", 262 | "File size : 101.22 MB | # possible samples : 766797 | # skip chars : 198\n", 263 | "----------------------------------------------------------------------------------------------------\n", 264 | "Processing file : /floyd/input/languageidentification/data/cleaned/sk_cleaned.txt\n", 265 | "File size : 85.65 MB | # possible samples : 648847 | # skip chars : 151\n", 266 | "----------------------------------------------------------------------------------------------------\n", 267 | "Processing file : /floyd/input/languageidentification/data/cleaned/cs_cleaned.txt\n", 268 | "File size : 90.56 MB | # possible samples : 686098 | # skip chars : 166\n", 269 | "----------------------------------------------------------------------------------------------------\n", 270 | "Vocab Size : 132\n", 271 | "----------------------------------------------------------------------------------------------------\n", 272 | "Samples array size : (1750000, 133)\n", 273 | "/tmp/samples/lang_samples_132.npz size : 57.21 MB\n" 274 | ] 275 | } 276 | ], 277 | "source": [ 278 | "# Utility function to return file Bytes size in MB\n", 279 | "def size_mb(size):\n", 280 | " size_mb = '{:.2f}'.format(size/(1000*1000.0))\n", 281 | " return size_mb + \" MB\"\n", 282 | "\n", 283 | "# Now we have preprocessing utility functions ready. Let's use them to process each cleaned language file\n", 284 | "# and turn text data into numerical data samples for our neural network\n", 285 | "# prepare numpy array\n", 286 | "sample_data = np.empty((NUM_SAMPLES*len(LANGUAGES_DICT),input_size+1),dtype = np.uint16)\n", 287 | "lang_seq = 0 # offset for each language data\n", 288 | "jump_reduce = 0.2 # part of characters removed from jump to avoid passing the end of file\n", 289 | "\n", 290 | "for lang_code in LANGUAGES_DICT:\n", 291 | " start_index = 0\n", 292 | " path = os.path.join(cleaned_directory, lang_code+\"_cleaned.txt\")\n", 293 | " with open(path, 'r') as f:\n", 294 | " print (\"Processing file : \" + path)\n", 295 | " file_content = f.read()\n", 296 | " content_length = len(file_content)\n", 297 | " remaining = content_length - MAX_LEN*NUM_SAMPLES\n", 298 | " jump = int(((remaining/NUM_SAMPLES)*3)/4)\n", 299 | " print (\"File size : \",size_mb(content_length),\\\n", 300 | " \" | # possible samples : \",int(content_length/VOCAB_SIZE),\\\n", 301 | " \"| # skip chars : \" + str(jump))\n", 302 | " for idx in range(NUM_SAMPLES):\n", 303 | " input_row = get_input_row(file_content, start_index, MAX_LEN, alphabet)\n", 304 | " sample_data[NUM_SAMPLES*lang_seq+idx,] = input_row + [LANGUAGES_DICT[lang_code]]\n", 305 | " start_index += MAX_LEN + jump\n", 306 | " del file_content\n", 307 | " lang_seq += 1\n", 308 | " print (100*\"-\")\n", 309 | " \n", 310 | "# Let's randomy shuffle the data\n", 311 | "np.random.shuffle(sample_data)\n", 312 | "# reference input size\n", 313 | "print (\"Vocab Size : \",VOCAB_SIZE )\n", 314 | "print (100*\"-\")\n", 315 | "print (\"Samples array size : \",sample_data.shape )\n", 316 | "\n", 317 | "# Create the the sample dirctory if not exists\n", 318 | "if not os.path.exists(samples_directory):\n", 319 | " os.makedirs(samples_directory)\n", 320 | "\n", 321 | "# Save compressed sample data to disk\n", 322 | "path_smpl = os.path.join(samples_directory,\"lang_samples_\"+str(VOCAB_SIZE)+\".npz\")\n", 323 | "np.savez_compressed(path_smpl,data=sample_data)\n", 324 | "print(path_smpl, \"size : \", size_mb(os.path.getsize(path_smpl)))\n", 325 | "del sample_data" 326 | ] 327 | }, 328 | { 329 | "cell_type": "markdown", 330 | "metadata": {}, 331 | "source": [ 332 | "Sanity check." 333 | ] 334 | }, 335 | { 336 | "cell_type": "code", 337 | "execution_count": 6, 338 | "metadata": {}, 339 | "outputs": [ 340 | { 341 | "name": "stdout", 342 | "output_type": "stream", 343 | "text": [ 344 | "Sample record : \n", 345 | " [10 2 3 6 13 4 4 2 10 0 0 8 3 6 5 3 0 4 10 6 4 0 0 1\n", 346 | " 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", 347 | " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 1\n", 348 | " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", 349 | " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", 350 | " 0 0 0 0 0 0 0 0 0 0 0 0 0]\n", 351 | "\n", 352 | "Sample language : en\n", 353 | "\n", 354 | "Dataset shape (Total_samples, Alphabet): (1750000, 133)\n", 355 | "Language bins count (samples per language): \n", 356 | "en 250000\n", 357 | "fr 250000\n", 358 | "es 250000\n", 359 | "it 250000\n", 360 | "de 250000\n", 361 | "sk 250000\n", 362 | "cs 250000\n" 363 | ] 364 | } 365 | ], 366 | "source": [ 367 | "# utility function to turn language id into language code\n", 368 | "def decode_langid(langid): \n", 369 | " for dname, did in LANGUAGES_DICT.items():\n", 370 | " if did == langid:\n", 371 | " return dname\n", 372 | "\n", 373 | "# Loading the data\n", 374 | "path_smpl = os.path.join(samples_directory,\"lang_samples_\"+str(VOCAB_SIZE)+\".npz\")\n", 375 | "dt = np.load(path_smpl)['data']\n", 376 | "\n", 377 | "# Sanity chech on a random sample\n", 378 | "random_index = random.randrange(0,dt.shape[0])\n", 379 | "print (\"Sample record : \\n\",dt[random_index,])\n", 380 | "print (\"\\nSample language : \",decode_langid(dt[random_index,][VOCAB_SIZE]))\n", 381 | "\n", 382 | "# Check if the data have equal share of different languages\n", 383 | "print (\"\\nDataset shape (Total_samples, Alphabet):\", dt.shape)\n", 384 | "bins = np.bincount(dt[:,input_size])\n", 385 | "\n", 386 | "print (\"Language bins count (samples per language): \") \n", 387 | "for lang_code in LANGUAGES_DICT: \n", 388 | " print (lang_code, bins[LANGUAGES_DICT[lang_code]])" 389 | ] 390 | }, 391 | { 392 | "cell_type": "markdown", 393 | "metadata": {}, 394 | "source": [ 395 | "### Data preprocessing\n", 396 | "\n", 397 | "Even if our data is ready for the training, we have to [Standardize](https://en.wikipedia.org/wiki/Feature_scaling#Standardization) the dataset. This will help our model to converge faster since it makes the data more \"computational friendly\"." 398 | ] 399 | }, 400 | { 401 | "cell_type": "code", 402 | "execution_count": 7, 403 | "metadata": {}, 404 | "outputs": [ 405 | { 406 | "name": "stdout", 407 | "output_type": "stream", 408 | "text": [ 409 | "Example data before processing:\n", 410 | "X : \n", 411 | " [ 8. 0. 5. 5. 21. 2. 1. 7. 6. 0. 0. 6. 1. 8. 5. 0. 0. 7.\n", 412 | " 10. 10. 4. 3. 2. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n", 413 | " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n", 414 | " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 24. 0. 0. 0. 0. 1. 0. 0.\n", 415 | " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 1. 0.\n", 416 | " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n", 417 | " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n", 418 | " 0. 0. 0. 0. 0. 0.]\n", 419 | "Y : \n", 420 | " 0.0\n", 421 | "X preprocessed shape : (1750000, 132)\n", 422 | "\n", 423 | "Example data after processing:\n", 424 | "X : \n", 425 | " [-4.17050153e-01 -1.13270748e+00 6.38392270e-01 1.75331384e-01\n", 426 | " 1.73380041e+00 5.33405364e-01 -3.80145878e-01 1.91697395e+00\n", 427 | " -5.14976740e-01 -6.19032502e-01 -7.30089128e-01 1.92491740e-01\n", 428 | " -1.09850657e+00 -2.04314422e-02 -7.94259489e-01 -1.46870685e+00\n", 429 | " -4.61331427e-01 5.33611178e-02 1.16716337e+00 1.14771831e+00\n", 430 | " 3.73478569e-02 2.73657739e-01 1.30994463e+00 -3.63413811e-01\n", 431 | " -1.49887251e-02 -7.82141924e-01 -1.45357341e-01 -2.73461759e-01\n", 432 | " -5.23208618e-01 -8.40839669e-02 -2.57696390e-01 -1.61323845e-02\n", 433 | " -9.63160172e-02 -2.67638355e-01 -5.72866321e-01 -1.24845751e-01\n", 434 | " -8.58770609e-02 -4.83961701e-01 -7.90864602e-02 -4.38546538e-02\n", 435 | " -1.56157464e-01 -1.38136834e-01 -3.22708428e-01 -1.61213249e-01\n", 436 | " -1.93664357e-01 -1.26288012e-01 -3.17232043e-01 -4.67012003e-02\n", 437 | " -2.45021537e-01 -3.98998171e-01 -6.17011636e-03 -3.92622024e-01\n", 438 | " -1.21239848e-01 -3.00596684e-01 -4.27031331e-02 -2.16784000e-01\n", 439 | " -1.39083400e-01 -5.39894253e-02 -2.65944358e-02 -2.88298607e-01\n", 440 | " -3.61458272e-01 -1.90843359e-01 -2.18461171e-01 -3.85120064e-01\n", 441 | " 1.30634761e+00 -3.31691280e-02 -3.66677120e-02 -1.20478580e-02\n", 442 | " -9.79928020e-03 9.66669083e-01 -4.30910617e-01 -4.42056417e-01\n", 443 | " -4.23938990e-01 -4.10972983e-01 -3.52509290e-01 -3.54364634e-01\n", 444 | " -3.37542146e-01 -3.59003991e-01 -3.15173328e-01 -3.17866415e-01\n", 445 | " -4.55488026e-01 -4.60290730e-01 -3.92121077e-01 -3.03513795e-01\n", 446 | " -4.86154377e-01 -1.12909846e-01 1.68673170e+00 -5.44467688e-01\n", 447 | " 1.40097451e+00 -2.54374862e-01 -3.80121171e-01 -2.62794465e-01\n", 448 | " -9.93090570e-02 -1.18509911e-01 -2.20897570e-01 -5.71699217e-02\n", 449 | " -4.62426692e-02 -1.81627274e-02 -3.68494987e-02 -6.64461171e-03\n", 450 | " -8.24078172e-03 -4.20710333e-02 -8.09805691e-02 -2.85832421e-03\n", 451 | " -7.55929330e-04 -2.11150870e-02 -2.74087414e-02 -1.60357123e-03\n", 452 | " -6.56590983e-03 -2.61863228e-03 -1.99632142e-02 -4.02917853e-03\n", 453 | " -4.37377803e-02 -1.51186134e-03 -5.64606264e-02 0.00000000e+00\n", 454 | " -5.21061346e-02 -5.66610694e-03 -1.08200289e-01 -2.99859717e-02\n", 455 | " -2.85716611e-03 -1.06904586e-03 -2.65672822e-02 -4.56597749e-03\n", 456 | " -6.93865819e-03 -1.01418595e-03 -4.24020365e-02 -8.98558497e-02\n", 457 | " -1.53618287e-02 -1.76383916e-03 0.00000000e+00 -7.48927444e-02]\n", 458 | "Y : \n", 459 | " [1. 0. 0. 0. 0. 0. 0.]\n", 460 | "/tmp/train_test/train_test_data_132.npz size : 94.82 MB\n" 461 | ] 462 | } 463 | ], 464 | "source": [ 465 | "# we need to preprocess data for DNN yet again - scale it \n", 466 | "# scaling will ensure that our optimization algorithm (variation of gradient descent) will converge well\n", 467 | "# we need also ensure one-hot econding of target classes for softmax output layer\n", 468 | "# let's convert datatype before processing to float\n", 469 | "dt = dt.astype(np.float32)\n", 470 | "# X and Y split\n", 471 | "X = dt[:, 0:input_size] # Samples\n", 472 | "Y = dt[:, input_size] # The last element is the label\n", 473 | "del dt\n", 474 | "\n", 475 | "# Random index to check random sample\n", 476 | "random_index = random.randrange(0,X.shape[0])\n", 477 | "print(\"Example data before processing:\")\n", 478 | "print(\"X : \\n\", X[random_index,])\n", 479 | "print(\"Y : \\n\", Y[random_index])\n", 480 | "\n", 481 | "# X PREPROCESSING\n", 482 | "# Feature Standardization - Standar scaler will be useful later during DNN prediction\n", 483 | "standard_scaler = preprocessing.StandardScaler().fit(X)\n", 484 | "X = standard_scaler.transform(X) \n", 485 | "print (\"X preprocessed shape :\", X.shape)\n", 486 | "\n", 487 | "# Y PREPROCESSINGY \n", 488 | "# One-hot encoding\n", 489 | "Y = keras.utils.to_categorical(Y, num_classes=len(LANGUAGES_DICT))\n", 490 | "\n", 491 | "# See the sample data\n", 492 | "print(\"\\nExample data after processing:\")\n", 493 | "print(\"X : \\n\", X[random_index,])\n", 494 | "print(\"Y : \\n\", Y[random_index])\n", 495 | "\n", 496 | "# Train/test split. Static seed to have comparable results for different runs\n", 497 | "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.20, random_state=SEED)\n", 498 | "del X, Y\n", 499 | "\n", 500 | "# Create the train / test directory if not extists\n", 501 | "if not os.path.exists(train_test_directory):\n", 502 | " os.makedirs(train_test_directory)\n", 503 | "\n", 504 | "# Save compressed train_test data to disk\n", 505 | "path_tt = os.path.join(train_test_directory,\"train_test_data_\"+str(VOCAB_SIZE)+\".npz\")\n", 506 | "np.savez_compressed(path_tt,X_train=X_train,Y_train=Y_train,X_test=X_test,Y_test=Y_test)\n", 507 | "print(path_tt, \"size : \",size_mb(os.path.getsize(path_tt)))\n", 508 | "del X_train,Y_train,X_test,Y_test" 509 | ] 510 | }, 511 | { 512 | "cell_type": "markdown", 513 | "metadata": {}, 514 | "source": [ 515 | "## Train - Test Split\n", 516 | "\n", 517 | "Split: 80% for Train and 20% for Test" 518 | ] 519 | }, 520 | { 521 | "cell_type": "code", 522 | "execution_count": 8, 523 | "metadata": {}, 524 | "outputs": [ 525 | { 526 | "name": "stdout", 527 | "output_type": "stream", 528 | "text": [ 529 | "X_train: (1400000, 132)\n", 530 | "Y_train: (1400000, 7)\n", 531 | "X_test: (350000, 132)\n", 532 | "Y_test: (350000, 7)\n" 533 | ] 534 | } 535 | ], 536 | "source": [ 537 | "# Load train data first from file\n", 538 | "path_tt = os.path.join(train_test_directory, \"train_test_data_\"+str(VOCAB_SIZE)+\".npz\")\n", 539 | "train_test_data = np.load(path_tt)\n", 540 | "\n", 541 | "# Train Set\n", 542 | "X_train = train_test_data['X_train']\n", 543 | "print (\"X_train: \",X_train.shape)\n", 544 | "Y_train = train_test_data['Y_train']\n", 545 | "print (\"Y_train: \",Y_train.shape)\n", 546 | "\n", 547 | "# Test Set\n", 548 | "X_test = train_test_data['X_test']\n", 549 | "print (\"X_test: \",X_test.shape)\n", 550 | "Y_test = train_test_data['Y_test']\n", 551 | "print (\"Y_test: \",Y_test.shape)\n", 552 | "\n", 553 | "del train_test_data" 554 | ] 555 | }, 556 | { 557 | "cell_type": "markdown", 558 | "metadata": {}, 559 | "source": [ 560 | "### Model\n", 561 | "\n", 562 | "We will implement a really simple 3 layers Neural Network with [Droput](https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf) for preventing Overfitting. We will also use the [Xavier initializer](https://www.quora.com/What-is-an-intuitive-explanation-of-the-Xavier-Initialization-for-Deep-Neural-Networks)(one of the best initialization scheme, this will improve the chance to converge in a better \"place\": a point with better accuracy).\n", 563 | "\n", 564 | "![nn](http://neuralnetworksanddeeplearning.com/images/tikz40.png)\n", 565 | "\n", 566 | "*From http://neuralnetworksanddeeplearning.com/*" 567 | ] 568 | }, 569 | { 570 | "cell_type": "code", 571 | "execution_count": 9, 572 | "metadata": {}, 573 | "outputs": [ 574 | { 575 | "name": "stdout", 576 | "output_type": "stream", 577 | "text": [ 578 | "_________________________________________________________________\n", 579 | "Layer (type) Output Shape Param # \n", 580 | "=================================================================\n", 581 | "dense_1 (Dense) (None, 500) 66500 \n", 582 | "_________________________________________________________________\n", 583 | "dropout_1 (Dropout) (None, 500) 0 \n", 584 | "_________________________________________________________________\n", 585 | "dense_2 (Dense) (None, 300) 150300 \n", 586 | "_________________________________________________________________\n", 587 | "dropout_2 (Dropout) (None, 300) 0 \n", 588 | "_________________________________________________________________\n", 589 | "dense_3 (Dense) (None, 100) 30100 \n", 590 | "_________________________________________________________________\n", 591 | "dropout_3 (Dropout) (None, 100) 0 \n", 592 | "_________________________________________________________________\n", 593 | "dense_4 (Dense) (None, 7) 707 \n", 594 | "=================================================================\n", 595 | "Total params: 247,607\n", 596 | "Trainable params: 247,607\n", 597 | "Non-trainable params: 0\n", 598 | "_________________________________________________________________\n" 599 | ] 600 | } 601 | ], 602 | "source": [ 603 | "model = Sequential()\n", 604 | "# Note: glorot_uniform is the Xavier uniform initializer.\n", 605 | "\n", 606 | "model.add(Dense(500,input_dim=input_size, kernel_initializer=\"glorot_uniform\", activation=\"sigmoid\"))\n", 607 | "model.add(Dropout(0.5))\n", 608 | "model.add(Dense(300, kernel_initializer=\"glorot_uniform\", activation=\"sigmoid\"))\n", 609 | "model.add(Dropout(0.5))\n", 610 | "model.add(Dense(100, kernel_initializer=\"glorot_uniform\", activation=\"sigmoid\"))\n", 611 | "model.add(Dropout(0.5))\n", 612 | "model.add(Dense(len(LANGUAGES_DICT), kernel_initializer=\"glorot_uniform\", activation=\"softmax\"))\n", 613 | "model_optimizer = keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)\n", 614 | "model.compile(loss='categorical_crossentropy',\n", 615 | " optimizer=model_optimizer,\n", 616 | " metrics=['accuracy'])\n", 617 | "\n", 618 | "model.summary()" 619 | ] 620 | }, 621 | { 622 | "cell_type": "markdown", 623 | "metadata": {}, 624 | "source": [ 625 | "### Train & Evaluate\n", 626 | "\n", 627 | "If you left the default hyperpameters in the Notebook untouched, your training should take approximately:\n", 628 | "\n", 629 | "- On CPU machine: 24 minutes for 12 epochs.\n", 630 | "- On GPU machine: 3 minute for 12 epochs.\n", 631 | "\n", 632 | "*Note*: You can follow the execution on Tensorboard." 633 | ] 634 | }, 635 | { 636 | "cell_type": "code", 637 | "execution_count": null, 638 | "metadata": {}, 639 | "outputs": [], 640 | "source": [ 641 | "from keras.callbacks import TensorBoard\n", 642 | "\n", 643 | "# Tensorboard\n", 644 | "tensorboard = TensorBoard(log_dir=\"run\")\n", 645 | "\n", 646 | "# let's fit the data\n", 647 | "# history variable will help us to plot results later\n", 648 | "history = model.fit(X_train,Y_train,\n", 649 | " epochs=EPOCHS,\n", 650 | " validation_split=0.1,\n", 651 | " batch_size=BATCH_SIZE,\n", 652 | " callbacks=[tensorboard],\n", 653 | " shuffle=True,\n", 654 | " verbose=2)" 655 | ] 656 | }, 657 | { 658 | "cell_type": "code", 659 | "execution_count": 11, 660 | "metadata": {}, 661 | "outputs": [ 662 | { 663 | "name": "stdout", 664 | "output_type": "stream", 665 | "text": [ 666 | "350000/350000 [==============================] - 15s 42us/step\n", 667 | "acc: 97.60%\n" 668 | ] 669 | } 670 | ], 671 | "source": [ 672 | "# Evaluation on Test set\n", 673 | "scores = model.evaluate(X_test, Y_test, verbose=1)\n", 674 | "print(\"%s: %.2f%%\" % (model.metrics_names[1], scores[1]*100))" 675 | ] 676 | }, 677 | { 678 | "cell_type": "code", 679 | "execution_count": 12, 680 | "metadata": {}, 681 | "outputs": [], 682 | "source": [ 683 | "# and now we will prepare data for scikit-learn confusion matrix and classification report\n", 684 | "Y_pred = model.predict_classes(X_test)\n", 685 | "Y_pred = keras.utils.to_categorical(Y_pred, num_classes=len(LANGUAGES_DICT))\n", 686 | "LABELS = list(LANGUAGES_DICT.keys())" 687 | ] 688 | }, 689 | { 690 | "cell_type": "code", 691 | "execution_count": 13, 692 | "metadata": {}, 693 | "outputs": [ 694 | { 695 | "data": { 696 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkgAAAHGCAYAAACPTHSaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd4FFXbx/HvSaFDqNKlgwpKkdBBIBh6E0R9BBVBfGjy0ESKCgiIClixgAgWOgoq0psUKQFC7x0CSA8QSkhy3j92E0NCe2ULSX6f69qL7Jkzs/cwO5M7p8wYay0iIiIi8g8fbwcgIiIi8qBRgiQiIiKSgBIkERERkQSUIImIiIgkoARJREREJAElSCIiIiIJKEESSQaMMWmNMb8bY8KNMdPvYzsvGmMWuDI2bzDGzDXGvOztOEQk6VKCJOJBxpj/GGPWG2MuG2NOOH+RV3PBplsCOYFs1tpn/+1GrLUTrbXBLojnJsaYmsYYa4yZmaC8tLN82T1uZ6Ax5qe71bPW1rfWfv8vwxURUYIk4inGmB7AJ8AwHMnMw8CXQFMXbL4AsMdaG+WCbbnLaaCyMSZbvLKXgT2u+gDjoOuaiNw3XUhEPMAYEwAMBjpba3+x1kZYa29Ya3+31vZ21kltjPnEGHPc+frEGJPauaymMeaYMaanMeaUs/WprXPZIOAd4Dlny1S7hC0txpiCzpYaP+f7V4wxB4wxl4wxB40xL8YrXxlvvSrGmBBn112IMaZKvGXLjDHvGWNWObezwBiT/Q7/DZHALOB55/q+wHPAxAT/V58aY44aYy4aYzYYY6o7y+sB/eLt5+Z4cQw1xqwCrgCFnWXtncu/Msb8HG/7HxhjFhtjzD0fQBFJcZQgiXhGZSANMPMOdfoDlYAyQGmgAjAg3vJcQACQF2gHjDbGZLHWvoujVWqqtTaDtXbcnQIxxqQHPgPqW2szAlWATbeolxX4w1k3GzAK+CNBC9B/gLbAQ0AqoNedPhv4AXjJ+XNdYBtwPEGdEBz/B1mBScB0Y0waa+28BPtZOt46bYAOQEbgcILt9QQedyZ/1XH8371s9ZwlEbkDJUginpENOHOXLrAXgcHW2lPW2tPAIBy/+GPdcC6/Ya2dA1wGSvzLeGKAUsaYtNbaE9ba7beo0xDYa6390VobZa2dDOwCGserM95au8daexWYhiOxuS1r7V9AVmNMCRyJ0g+3qPOTtfas8zNHAqm5+35OsNZud65zI8H2ruD4fxwF/AR0tdYeu8v2RCSFU4Ik4hlngeyxXVy3kYebWz8OO8vitpEgwboCZPj/BmKtjcDRtfVf4IQx5g9jzCP3EE9sTHnjvT/5L+L5EegC1OIWLWrGmF7GmJ3Obr0LOFrN7tR1B3D0TguttWuBA4DBkciJiNyREiQRz1gNXAea3aHOcRyDrWM9TOLup3sVAaSL9z5X/IXW2vnW2qeB3DhahcbeQzyxMYX9y5hi/Qh0AuY4W3fiOLvA3gRaAVmstZmBcByJDcDtusXu2F1mjOmMoyXquHP7IiJ3pARJxAOsteE4BlKPNsY0M8akM8b4G2PqG2M+dFabDAwwxuRwDnZ+B0eX0L+xCahhjHnYOUC8b+wCY0xOY0xT51ik6zi66mJusY05QHHnrQn8jDHPAY8Bs/9lTABYaw8CT+EYc5VQRiAKx4w3P2PMO0CmeMv/Bgr+f2aqGWOKA0OA1ji62t40xtyxK1BERAmSiIc4x9P0wDHw+jSObqEuOGZ2geOX+HpgC7AV2Ogs+zeftRCY6tzWBm5OanyccRwHzuFIVjreYhtngUY4BjmfxdHy0shae+bfxJRg2yuttbdqHZsPzMMx9f8wcI2bu89ib4J51hiz8W6f4+zS/An4wFq72Vq7F8dMuB9jZwiKiNyK0UQOERERkZupBUlEREQkASVIIiIiIgkoQRIRERFJQAmSiIiISAJKkEREREQSuNNdfR9oN07sTLHT79IWqOPtELwqpT9hVM9YTbliNOs4RYuKDPPoyX/jzAGXfeH8sxdOchcutSCJiIiIJJBkW5BERETEjWKivR2BVylBEhERkcTsrZ5AlHKoi01EREQkAbUgiYiISGIxKbsFSQmSiIiIJGLVxSYiIiIi8akFSURERBJTF5uIiIhIAupiExEREZH41IIkIiIiielGkSIiIiIJqItNREREROJTC5KIiIgkpllsIiIiIjfTjSJFRERE5CZqQRIREZHE1MUmIiIikoC62EREREQkPrUgiYiISGK6UaSIiIhIAim8iy3FJ0jR0dE893ovHsqejS+HD2DNhs2M/Pp7YmJiSJc2LUPfeoOH8+Xmgy/GsS50KwDXrkdy7vwFVv8xiV17D/Dex99w+coVfHx86ND6WerXrgbAsRN/03vwCC6EX+KxEkUY3u9/+Pv7e3N378nYMSNp2KAOp06foUzZIAAmTfyK4sWLAJA5IBMXwi9SPjAYgMcff5SvRn9AxkwZiImJoVLlhly/ft1r8d+P1KlTs3TJz6ROnRpfP19++eUPBg8eSc2aVfnwg7fxT+VP6MatvNahJ9HRjr+uPh41mHr1anP16lXatetO6KZtXt6L++fj48Oa1XMIO36S5s1f4duxo6heoxIXwy8B0L59dzZv2UGmTBn5fsJn5M+fFz8/X0Z9/A0//DDNy9Hfnz27V3P5cgTR0dFERUVRuUpDBr7bi8aN6xITE8Op02do374HJ078DcCo2ON/5Srt2ndnUzI4/rG6vfEar776AtZatm3bRbv2PWj36gu80bU9RYsWImfuUpw9e97bYbrNvj1ruHT5MtHRMURFRVGpcgM+eH8ADRs9TWRkJAcOHKZd+x6Eh1/0dqjiBsZa6+0Y/pUbJ3a6JPDvp/3K9t37uBxxlS+HD6Bh6058NrQvRQrkZ8qsOWzduZehfbvdtM7EX2azc+9BhvTpyqGjYRhjKJAvD6fOnKNVh5789v3nZMqYgZ4DPySoemUaBFVn0MivKFG0IM83rX/fMactUOe+t3En1atV5PLlCMaP/zQuQYrvow/eIfziRYYM/QRfX19C1s3jlbbd2LJlB1mzZuHChXBi3Dj7wbhtyw7p06cjIuIKfn5+/LlsJr16DWLixK+oW+859u49wLvv9uLI4WOMnzCFevVq07lTWxo3aUPFCuUYNWoQVas1dmt8xrj7fwC6dXuNJ8uVJmOmDHEJ0pw5i/ll5h831evzZhcCAjLRr/8wsmfPyraty8n/cFlu3Ljh9hjdZc/u1VSu0uCmX/wZM2bg0qXLAHTu/CqPPlqMLl36Uq9ebTp1akuTJm2oUKEco0YOolp19x3/GA9er/PkycWfS2fyeOlaXLt2jcmTvmbu3CVs2bqd8+fDWbxwBhUr10/2CVLCfXy6Tg2WLF1FdHQ07w/rB0DffsM8Ek9UZJj7T/54rm9f7LIvXOqSQR6N3RVS9CDtk6fOsHzNelo0fDquzBiIiLgKwKWIK+TInjXRenMWr6BBUHUACubPS4F8eQB4KHtWsmYJ4Hz4Ray1rN24leCnqgDQtF4tlqxc6+5dcokVK9dy7vyF2y5v2bIxU6b+CkDw00+xdetOtmzZAcC5c+fdmhx5QkTEFQD8/f3w9/cnOjqayMhI9u49AMCiRctp3rwBAE0a1+WniTMAWLtuIwGZA8iV6yHvBO4iefPmpn79IL4bP+muda21ZMiYHoAMGdJz7vwFoqKi3B2ix8UmRwDp06Ul9g/Lxo2DmfiT4/ivW7eRzJkzJfnjH5+fnx9p06bB19eXdGnTcuLESTZt2s7hw8e8HZrXLFy0PK71eM3ajeTNm9vLEbmRjXHdKwlya4JkHN40xuw3xlw1xmw1xrR2LitojLHGmBbGmIXGmCvGmB3GmKfvtl1X+eCLcfR4/eWb/iIf1LszHd96j6CW7fh9wTLa/6fFTescP3mKsBOnqFj28UTb27pzDzduRJE/Ty4uhF8iY4b0+Pn5ApAzRzZOnT7n3h3ygOrVKvL3qdPs23cQgGLFCmMtzJk9kXVr59GrZ0cvR3j/fHx8WB+ygONhW1i0eDnrQkLx8/PjyXJPANDimYbkz+9IivPkycWxo8fj1g07doK8eXJ5JW5XGTliIH37DiUm5uY/HgcPfpMN6xfy0UfvkipVKgC+/GoCj5QoxuFDG9i4YRE9e75DUm2VjmWxzPljEmtWz6FduxfjygcPepP9+9bxwgvNGTRoBOA4/keP/XP8j4WdIE8SP/6xjh8/yaiPv+bg/nUcOxJK+MWLLFy03NtheZS1lrlzJrN2zVzax/suxGr7yvPMm7/UC5GJJ7i7BWkI0A7oDDwGvA98Y4xpGK/OUOAzoDQQAkwxxmRwc1ws+yuErFkCKFmi6E3lP0z/na+Gv83iGeNoVj+ID0d/d9PyuUtWEvxUZXx9fW8qP332HH2HfcKQPl3x8Um+DXPPPdeMqc7WIwA/P1+qVgmkzctdeKpmM5o1rU/tWtW8GOH9i4mJoXxgMAULlSewfFlKlixB69adGDFiIH+tms2lyxFERyfNv4jupkGDIE6dPkOoc7xdrAFvD6fU409RuUpDsmbJTO9enQAIfromm7dsp0DBJwmsUJdPPhlCxoxuP33dqlatZ6hYqT6Nm7Sh439fplq1igC88+6HFClagcmTZ9KpY1svR+l+mTMH0KRxXYoWr0T+AuVInz4d//nPM94Oy6OeqtWcChXr0ahxazp2fIXqzu8CQN+33iAqKopJk37xYoRuFhPjulcS5Lbf5MaY9EAPoL21dp619qC1dhIwFkfCFOtja+3v1tq9QD8gK1DmNtvsYIxZb4xZ/+1P9zcQNHTbLpatCiH4udfoPXgk60K30PGt99i9/yBPPFYcgPq1qrFp+66b1pu7ZAX1g2rcVHY54gqd3hrCG+1aU7pkCQAyB2Tk0uUIoqIcTbF/nz7LQzkSd9clJb6+vjRvVp9p03+LKzsWdoIVK9dy9ux5rl69xtx5SyhbtpQXo3Sd8PCLLPtzFcHBNVmzdgO1aj9DlaqNWLFiDXuc3W3Hj58kn7M1CSBvvtyEHT/prZDvW5XKgTRqGMye3av56cfR1KpZlQnjP+PkyVMAREZG8v0P0ygf6DhFX3q5FbNmzQVg//5DHDp4lBIJ/uhIao47j9/p02f59dd5BAbefDmaPGUmzZvXj6ubP98/xz9f3txx6yd1QUHVOXjoCGfOnCMqKoqZs+ZSuVJ5b4flUTd/F+bGfRdeatOKhg3q0OalLt4Mz+2sjXbZKylyZ1PHY0AaYJ4x5nLsC+gIFIlXb0u8n2Pbqm/ZiW+tHWOtLW+tLd++dav7Cq57hzYsnjGOBVPH8tE7PalQ9gk+H9KPy5evcOhoGAB/rd9E4QL54tY5cPgYFy9dpowzCQK4ceMG3d5+nybBNQmuWSWu3BhDhbKPs+DPvwD4dd5SaletcF8xe1udoOrs3r2PsLATcWULFvxJqVKPxI1TqFG9Ejt37vVilPcne/asBARkAiBNmjTUCarB7t37yZEjGwCpUqWid6/OjBnzIwC/z15A6xdbAlCxQjkuhl+MSyaSogFvD6dwkUCKl6hM6zadWbpsFa+0feOmcTVNmtRlx/bdABw9GhbXYvjQQ9kpXrwIBw8e9krsrpAuXVoyZEgf93OdOjXYvn03RYsWiqvTuHFddu/eD8Ds2Qt4sbXj+FeoUI7w8EtJ+vjHd/RIGBUrliNt2jQA1K5VjV27ku65/f+V8LvwdJ2n2L59N3WDa9KrV0eaPfMKV69e83KU4k7unOYfm3w1Bo4kWHaDfyYjxU13sdZa53ggr/RR+fn5MrB3Z7q/8wHGx4dMGdLzXp+uccvnLllB/drVbxqzNG/pKjZs3sGF8EvMmrcEgKFvvcEjxQrT/fWX6D14JJ+Pm8ijxQrzTAOPDa+6Lz/9OJqnalQme/asHDqwnkGDRzB+whRatWoaNzg71oUL4Xzy6RjWrJ6DtZZ585YwZ+5iL0V+/3Lnzsl34z7B19cH4+PDjBm/M2fOIoa/P4AGDevg4+PDmG9+YNmyVQDMnbuY+vVqs2vnKq5evUr79j28vAfu8f2Ez8mRIxvGwObNO+jc5S0Ahg37lG+/HcXGDYswBvr3H5akZzXlzJmD6dO+BRzXgylTZrFgwTKmThlD8eKFiYmxHDlyjM5d+gIwd+4S6tWrzc6dK7l65RrtX0s+x39dSCi//PIHIevmExUVxaZN2xn77US6dH6VXj07kStXDkI3LGLuvCW8/t/e3g7X5XLmzMGM6eOAf74L8xcsY9eOlaROnZp5c6cAsHbtxrjzIdlJooOrXcVt0/yNMRmB00BHa+34WywvCBwEAq216+OVW+BZa+2MO23fVdP8kyJ3T/N/0CW5uaIu5olp/vJg8uQ0f3nweHqa/7WNv7nsC5emXJMkd+FyWwuStfaSMWYEMMI4rujLgQxAJSAGWOCuzxYRERG5H+6+k/bbwN9AL+Ar4CKwCfjQzZ8rIiIi9yOFd7G5NUGyjv67z52vW0nU5GatTXLNcCIiIslOCn9YbfK9YY+IiIjIv5TiH1YrIiIit6AuNhEREZEEkugdsF1FXWwiIiIiCagFSURERBJTF5uIiIhIAupiExEREZH41IIkIiIiiaXwFiQlSCIiIpKItbpRpIiIiIjEoxYkERERSUxdbCIiIiIJpPBp/upiExEREUlALUgiIiKSmLrYRERERBJQF5uIiIiIxKcWJBEREUlMXWwiIiIiCaiLTURERETiUwuSiIiIJKYutqQpbYE63g7Ba64eXeLtELwqXf7a3g5BvCTGWm+HIJJypPAESV1sIiIiIgkk2RYkERERcaMUPkhbCZKIiIgklsK72JQgiYiISGIpvAVJY5BEREREElALkoiIiCSmLjYRERGRBNTFJiIiIiLxqQVJREREElMXm4iIiEgCKTxBUhebiIiISAJqQRIREZHEUvizD5UgiYiISGLqYhMRERGR+JQgiYiISGIxMa573QNjjK8xJtQYM9v5vpAxZq0xZp8xZqoxJpWzPLXz/T7n8oLxttHXWb7bGFM3Xnk9Z9k+Y8xb9xKPEiQRERFJzMa47nVvugE7473/APjYWlsUOA+0c5a3A847yz921sMY8xjwPFASqAd86Uy6fIHRQH3gMeAFZ907UoIkIiIiXmWMyQc0BL51vjdAbWCGs8r3QDPnz02d73EuD3LWbwpMsdZet9YeBPYBFZyvfdbaA9baSGCKs+4dKUESERGRxFzYxWaM6WCMWR/v1SHBp30CvAnENjdlAy5Ya6Oc748BeZ0/5wWOAjiXhzvrx5UnWOd25XekWWwiIiKSmAun+VtrxwBjbrXMGNMIOGWt3WCMqemyD71PSpBERETEm6oCTYwxDYA0QCbgUyCzMcbP2UqUDwhz1g8D8gPHjDF+QABwNl55rPjr3K78ttTFJiIiIol5aBabtbavtTaftbYgjkHWS6y1LwJLgZbOai8Dvzp//s35HufyJdZa6yx/3jnLrRBQDFgHhADFnLPiUjk/47e77b5XEiRjjI8x5htjzFljjH2QmtREREQEj0/zv4U+QA9jzD4cY4zGOcvHAdmc5T2AtwCstduBacAOYB7Q2Vob7WyB6gLMxzFLbpqz7h15qwWpAdAWaAzkBv7yUhy3NHbMSI4f28ym0MWJlnX/3+tERYaRLVuWuLKPRw1m146VbNywkLJlSnky1PsWHR1Ny3bd6NRnMABrNmzm2Xb/o8Wr3WjTuQ9Hjh0H4MTfp2nbrT8t23Wj+StdWb56PQA3oqLoN/Rjmr/clcatOzH2p+lx2x4w/FNqNGlDs5e7eH7H/qXUqVPz16rZbFi/kE2blvDOOz0BWLrkF9aHLGB9yAIOH9rAjBnj4tb5eNRgdibR4387Pj4+rFs7j5kzJwDw/YTP2bb1T0I3LmLMNyPw83P0zteoUZnTp3YQsm4+Ievm07/f/7wYtWsVL14k7pivD1nAuTO7eKNrewA6d2rLtq1/snnTEoa/39/LkbpHvnx5WLRgOls2L2XzpiV07eKYYV26dElWrfid9SELWLN6DoHly3g5Uve41e+BFi0asXnTEiKvHeXJck94Mbrky1q7zFrbyPnzAWttBWttUWvts9ba687ya873RZ3LD8Rbf6i1toi1toS1dm688jnW2uLOZUPvJRZvJUhFgRPW2r+stSed0+7ixN4Mylt++GEaDRu9mKg8X748PF2nBocPH4srq1+vNsWKFuKRx6rRsWMfRn/xvidDvW8/zfidwgX+6Zp9b+RXDH+7Jz9/9ykN6zzFNz9MA+CbH6ZSt1ZVZoz7lBEDezPk468BWLB0FZE3opj5/edM+/Zjpv82n7ATfwPQrF4QX3800OP7dD+uX7/O08GteLL805QvH0zd4JpUrFCOWrWfoXxgMOUDg1mzdgOzZjnOu3r1alO0aCEedR7/L5LY8b+drl3bsWvXvrj3k6fMpNTjT1G2XB3Spk3Dq6++ELds5ap1BFaoS2CFugwd9ok3wnWLPXv2xx3zChXrceXKVWb9OpeaT1WhSeO6lHvyaUqXqc3IUV97O1S3iIqKovebg3iidC2qVmtMx46v8OijxRg+rD/vDRlF+cBgBg0akWwTxFv9Hti+fRfPtnqNFSvWeCkqD/P8fZAeKB5PkIwxE3Dc2OlhZ/faIWPMMmPMV8aYEcaY08AqT8cV34qVazl3/kKi8pEjBvJWv6HYeCP7Gzeuy48THbdpWLtuIwGZA8iV6yGPxXo/Tp46w/LV62nR8Om4MmMMEVeuAHApIoIc2bM6yjFERFx1lF++Qo5sznIDV69dIyoqmuvXr+Pv50eG9OkAKF+mFAGZMnhyl1wiIsKx//7+fvj7+990vDNmzECtmlX59dd5ADRpXJefkujxv528eXNTv34Q342fFFc2b96SuJ9D1m8iX97c3gjNa4JqV+PAgcMcORLG66+/xIcfjSYy0vF33enTZ70cnXucPHmK0E3bALh8OYJdu/aSN08urLVkzJQRgEwBGTnu/IMoubnV74Fdu/axZ89+L0XkeTbGuuyVFHmjBakbMBjHfQhyA4HO8taAAaoDL3khrjtq3DiYsLATbNmy46byvHlycezo8bj3YcdOkDdPLk+H96988Pm39Oj4Csbnn6/BoDe70PHNwQS1aMvv85fR/kXH+LhObV9g9oJlBLVoS6c3B9Hvf45bWDxdsypp06ShVvOXefrZdrzyfDMCnBfPpMrHx4f1IQs4HraFRYuXsy4kNG5Z06b1WLJ0FZcuXQYgTxI+/rczcsRA+vYdSswtLmp+fn68+J8WzF+wLK6sUsUnWR+ygN9++5HHHi3uwUg9p1WrpkyZOguAYsUKU61aBf5a+TtLFs2g/JOlvRyd+xUokI8ypUuxdl0oPXq9ywfvD+Dg/hA+HP42/Qckj1ZTkYQ8niBZa8OBS0C0s3vttHPRQWttT2vtLmvtzjtswuPSpk1D3z5dGThohLdDcZllf4WQNUsAJUsUvan8h2m/8tWH77D45/E0axDEh184xtrMWbycpvVrs/jn8Xz54bv0HfIxMTExbN25B18fH5bMnMC8qWP5fuqvHD1+0hu75DIxMTGUDwymYKHyBJYvS8mSJeKWPdeqKVOdvyiTowYNgjh1+gyhoVtvufzzz4axYuVaVq1aB0Bo6FaKFqtI+cBgvvxyPNPjjc1KLvz9/WncKJgZP88GwM/PlyxZMlOlWmP6vDWEyZOSZxdbrPTp0zFt6lh69HqXS5cu83qHl+jZeyCFigTSs/cgxn4z0tshirt4f5C2Vz1I0/w33K1C/DtxxsREeCImAIoUKUjBgg+zcf1C9u1ZQ758uQlZO5+cOXMQdvwk+fLniaubN19uwpJAghC6dQfLVq0juFV7eg/6iHUbt9DxzcHs3n+IJx5zJAT1a1dn07ZdAPzyx0Lq1qoGQJlSjxAZGcn58IvMWbicqhXL4e/nR7YsmSnz+CNsjzd2JSkLD7/Isj9XERxcE4Bs2bIQGFiWOXP+GbR5PIke/9upUjmQRg2D2bN7NT/9OJpaNasyYfxnAAzo350cObLSu/eguPqXLl2O65KcN2+J43sQbwJDclCvXi1CQ7dy6tQZwNFKGDsGLWT9JmJiYsju7IpObvz8/Jg+dSyTJ8+M2+eX2jzLzJlzAJgx43cCA5PnIG1BY5C8HUA8d814rLVjrLXlrbXlfXzSeyImALZt20WefKUpWrwSRYtX4tixEwRWrMvff59m9uwFtHF2Q1WsUI6L4Rc5efKUx2L7t7q//jKLfx7Pgmnf8tG7valQ7gk+H9afyxERHDrquH/WXyGhFC6QD4DcOXOwduMWAPYfOsr1yBtkzRxA7pw5WOcsv3L1Glu276FQgbvewf2BlT17VgICMgGQJk0a6gTVYPdux5iDFs80Ys6cRVy/fj2u/u+zF9A6CR7/2xnw9nAKFwmkeInKtG7TmaXLVvFK2zdo2/YFnn76KVq36XLTmKycOXPE/Vy+fBl8fHw4e/a8N0J3m+efaxbXvQbw62/zqVmzCuDobkuVKhVnzpzzVnhuNXbMSHbu2scnn/5zA+TjJ/7mqRqVAahdqxp79x30VngibqU7ad/CTz+O5qkalcmePSuHDqxn0OARjJ8w5ZZ158xdTL16tdm9cxVXrl6lffseHo7Wdfz8fBnYuwvdBwzH+BgyZczAe2+9AUDvzq/y7odf8MO0XzHGMKRvN4wxvNC8AQOGf0rTlzpjLTRrEESJIoUc6wz6iJDQbVwIv+gYu9T2BVo0CvbmLt5V7tw5+W7cJ/j6+mB8fJgx43fmzFkEQKtWTfjwo9E31Z87dzH169Vm185VXE3ix/9ORn/xPoePHGPFcsd92mbNmsvQYZ/wzDMNeb1DG6Kiorl69Rqt23TycqSulS5dWuoE1aBjpz5xZeMnTOHbsSPZFLqYyMgbvNou+dzaIL6qVQJp07olW7buYH3IAgDefns4//1vb0aNGoyfnx/Xr12jY8c3vRype9zq98C58xf49OMh5MiRld9+/YHNm7fT4BYznpONJDq42lWMdeGzVu75Q43pBXRx3jUTY8wyYJu19p5vmOOXKm+KPXJXjy65e6VkLF3+2t4OwascD61OmWK8cL20+9ZRAAAgAElEQVQSeVBERYZ59OS/8nknl51w6bp+meQuXA9SF5uIiIjIA8ErXWzW2hHAiHjva3ojDhEREbmNJDr7zFU0BklEREQSS+Fd2upiExEREUlALUgiIiKSmLrYRERERBJI4dP81cUmIiIikoBakERERCSxJPqIEFdRgiQiIiKJqYtNREREROJTC5KIiIgkYjWLTURERCQBdbGJiIiISHxqQRIREZHENItNREREJAF1sYmIiIhIfGpBEhERkcQ0i01EREQkAXWxiYiIiEh8akESERGRxDSLTURERCQBdbGJiIiISHxJtgXJeDsAL0qXv7a3Q/CqiAPzvB2CV2UoUt/bIYhICqBnsYmIiIgklMK72JQgiYiISGIpPEHSGCQRERGRBNSCJCIiIolpmr+IiIhIAupiExEREZH41IIkIiIiidgU3oKkBElEREQSS+EJkrrYRERERBJQC5KIiIgkpjtpi4iIiCSgLjYRERERiU8tSCIiIpJYCm9BUoIkIiIiiVibshMkdbGJiIiIJKAWJBEREUlMXWwiIiIiCaTwBEldbCIiIiIJqAVJREREEtGz2EREREQSSuEJkrrYRERERBJQC5KIiIgklrIfxaYE6Vby5cvD+O8+5aGc2bHWMu7biXz+xTgmTvyKEsWLABAQkInw8IuUDwwmKKg6w4b2I1UqfyIjb9DnrSEsW7bKy3vx76VOnZqlS34mderU+Pr58ssvfzB48EjGffsx1atX4uLFSwC0a9+dzZu3A1CjRmVGjRyEn78fZ8+cI6hOS2/uwj2Ljo7m+U59eChbVkYP68eajVsY9c2PxFhLurRpGPJmZx7Om5tpv89n8q/z8fXxIV3aNLzb/XWKFMwPwO79hxj88RgirlzB+Pgw5cvhpE6Vis/GTeK3hX9y8VIE6/74yct7+v/j4+PDmtVzCDt+kubNX4krHzVqMK+8/BxZs5UA4OGH8zJmzEhyZM/GuXMXeKXtG4SFnfBS1K6VOnVqli35mVSpU+PnPA8GOc+DGtUrEX6L8yA5KV68CJMmfhX3vnChhxk4aATZsmWhceNgYmIsp0+d4dX23Tlx4m8vRuoetzv+tWpW5YMP3iZVKn82btzKax16Eh0d7e1w3SKlj0EySfVOmf6p8rot8Fy5HiJ3rocI3bSNDBnSs3btPFq2fJWdO/fG1fnwg3cIv3iRoUM/oUyZkvz99xlOnPibkiVL8MfsiRQsVN5d4XlE+vTpiIi4gp+fH38um0mPHu/SoUMb/piziF9++eOmugEBmVi+/FcaNXqRo0ePkyNHNk6fPuu22CIOzHPZtr6f/jvb9+wnIuIKo4f1o9FLXfnsvT4ULpCPKb/OY+uufQzt04XLEVfIkD4dAEv/CmHqb/P5evgAoqKjafV6b97v+wYlihTkQvglMmZIh6+vL5t37CFPzhw0fKmrSxOkDEXqu2xbt9Ot22s8Wa40GTNliEuQypV7gq5d2tG0ab24BGnypK+ZM2cRP/40g5o1q/DyS8/R9tVubosrxsPXq/jnwfJlM+l+h/MgOfPx8eHIoQ1UqdaI8+fDuXTpMgBdOr/Ko48Wp3OXt7wcoXskPP49ew1i0sSvCK73HHv3HmDgu704fPgY4ydM8Ug8UZFhxiMf5HThxdouO+EyT1zi0dhdQWOQbuHkyVOEbtoGwOXLEezatZc8eXLdVKdly8ZMnforAJs2bY/7C2r79t2kTZuGVKlSeTZoF4uIuAKAv78f/v7+d7zl/AvPN2fWrLkcPXocwK3JkSudPH2WFWs30KJBUFyZMYbLVxz7fjniCg9lywIQlxwBXL12Pe7nv9ZvpnjhApQoUhCAzAEZ8fX1BaD0Y8XJ4Vw/KcmbNzf16wfx3fhJcWU+Pj4Mf38AffsNvanuo48WY6mztXTZsr9o3DjYo7G6W/zzwO8u50FyFlS7GgcOHObIkbC45AgcCURy/j9JePyjo6OJjIxk794DACxatJxnmjfwZojuFWNd90qC3J4gGYc3jTH7jTFXjTFbjTGt4y1/xxhz2Bhz3Rhz0hjzg7tj+v8oUCAfZUqXYt260LiyatUqcurUafbtO5io/jPPNCQ0dBuRkZGeDNPlfHx8WB+ygONhW1i0eDnrQhz7P3hwHzZuWMiIjwbGJYHFihUmS+YAFi2czto1c2ndOml0r304ejzdO7TBx/zzh83Anv+lU99hBD3Xgd8XLqfdC83jlk2eNZf6rTszasyP9O3SDoDDx45jjOH1Pu/R6vXefDdllsf3w9VGjhhI375DiYl3UevUqS2z/1jAyZOnbqq7ZctOmjVz/IJo1rQ+mTJlJGvWzB6N151iz4MTYVtYHO88eM95HoyMdx4kZ61aNWXK1H++2+8N7sPB/SG88EJzBg76yIuRudetjr+fnx9PlnsCcFzv8+XP4+Uo3SjGha8kyBMtSEOAdkBn4DHgfeAbY0xDY0wLoBfQCSgGNALWeSCme5I+fTqmTR1Lz17v3vRX0/PPNWOKs/UovsceK86wof3o1LmPJ8N0i5iYGMoHBlOwUHkCy5elZMkS9B/wPqVK1aBS5YZkzZqZ3r07AeDn50u5ck/QpOlLNGj4H/r1/R/FihX28h7c2Z+r15M1SwAlnWPKYv3482y+fL8fi6eOoVm9Wnz01fdxy15oVp+5P42m+2utGfPTDMAxhil02y6G9+vG958OYfHKdazZuMWj++JKDRoEcer0GUJDt8aV5c6dkxbPNGT06PGJ6vd56z1qVK/EurXzqF6jEseOnSA6OoleDW8h9jwokOA8KOk8D7JkzcybzvMgufL396dxo2Bm/Dw7ruztdz6gUJFAJk+eSedObb0YnXvd6vi/2LoTI0cMZPWq2Vy+HJGsvu9yM7cmSMaY9EAPoL21dp619qC1dhIwFkfCVAA4ASyw1h6x1q631n5xh+11MMasN8asj4mJcGfo+Pn5MW3qWCZPnsmsWXPjyn19fWnWrD7Tp/92U/28eXMzffo4Xn21GwcOHHZrbJ4UHn6RZX+uIji4ZlzrQWRkJBO+n0pg+bIAHAs7wYKFy7hy5Spnz55n5co1PPHEY94M+65Ct+9m6V8h1P1PR3oP+YR1m7bRqd8wdu8/zBOPFgegXs0qbNq+O9G69WtVZclfIQDkzJ6NJx9/lCwBmUibJjXVK5Zl597ELYtJRZXKgTRqGMye3av56cfR1KpZlU2hiylSpCA7d6xkz+7VpEuXlh07VgJw4sTftHruNSpUrMc773wAOL4zyU3seVA3wXnwfbzzILmqV68WoaFbOXXqTKJlkyb/QvPk3MXkFP/4r1m7gZq1n6Fy1UasWLEmrrstObIx1mWvpMjdLUiPAWmAecaYy7EvoCNQBJjuXH7QGDPOGPOsMSb17TZmrR1jrS1vrS3v45PerYGPHTOSXbv28cmnY24qDwqqzu7d+26aqRMQkInffv2B/v2H8dfq9W6NyxOyZ89KQEAmANKkSUOdoBrs3r2fXLkeiqvTtEk9tu/YBcDvv8+napUK+Pr6kjZtGgIrlGXXrr233PaD4n/tX2Tx1DHMn/QVHw34HxXKlOKz9/pwOeIKh5xjqVZv2ELhAnkBOHzsn+O9fM1GHs7rGJNWJbAMew8e4eq160RFR7N+yw6KFMjn+R1ykQFvD6dwkUCKl6hM6zadWbpsFTlzleLhAuUoXqIyxUtU5sqVqzz2WDUAsmXLgnF2UfZ5swvffz/Vm+G71L2cB03inQfJlaPF/J/utaJFC8X93KRxXXbv3u+NsNzudsc/R45sAKRKlYrevTozZsyP3gzTvVJ4F5u7p/nHJmCNgSMJlt2w1h41xpQAgoA6wEjgXWNMRWute5uI7qBqlUBat27J1q07WB+yAHD84pg3bwnPtWoaNzg7VqdObSlSpCAD+ndnQP/uANRv8EKSGaycUO7cOflu3Cf4+vpgfHyYMeN35sxZxIL508iRIysYw5bN2+nU2TFzZdeufcxfsJSNGxcRExPD+O8ms/0WLS8POj9fXwb2/C/dB43AxxgyZUzP4F6dAcf4ozUbt+Dn50emDOkZ2qcrAAEZM9CmZWNe6NQHYwzVK5SjRqUnARj1zY/8sWQF165fJ+i5DrRoEESnl5/z2v65w1M1qvDekLfAWlasWMsb3fp7OySXiX8e+DjPgz/mLGLh/Glkz5EVYwyb450HyVG6dGmpE1SDjp3+GTYwbGhfihcvQkxMDEeOhCXb/b/d8f/g/QE0aFgHHx8fvvnmh7hJCpL8uHWavzEmI3Aa6GitTTyAIXH9nMBJoK61dsGd6rpzmr882Fw5zT8p8sQ0/weVp6f5izxIPD3N/1zzp1x2wmWd+WeSm+bv1hYka+0lY8wIYIRxtMMvBzIAlXA0ukU6Y1gLXAaeA24AD3b/jIiISHKXRLvGXMUTd9J+G/gbx2y1r4CLwCbgQyA90AcYAfgDO4BnrLVJd5SriIiIJHluT5Csow/vc+frVpL+jWNERESSGasWJBEREZEEUniCpEeNiIiIiCSgFiQRERFJJKV3sakFSURERBLz4I0ijTFpjDHrjDGbjTHbjTGDnOWFjDFrjTH7jDFTjTGpnOWpne/3OZcXjLetvs7y3caYuvHK6znL9hlj7noDLyVIIiIi4m3XgdrW2tJAGaCeMaYS8AHwsbW2KHAex7Ndcf573ln+sbMexpjHgOeBkkA94EtjjK8xxhcYDdTH8ZSPF5x1b0sJkoiIiCRiY1z3uutnOcQ+Fd7f+bJAbWCGs/x7oJnz56bO9ziXBznvt9gUmGKtve68ZdA+oILztc9ae8BaGwlMcda9LSVIIiIikogrE6T4D5t3vjok/DxnS88m4BSwENgPXLDWRjmrHAPyOn/OCxwFcC4PB7LFL0+wzu3Kb0uDtEVERMStrLVjgDF3qRMNlDHGZAZmAo94IrbbUYIkIiIiiXhrFpu19oIxZilQGchsjPFzthLlA8Kc1cKA/MAxY4wfEACcjVceK/46tyu/JXWxiYiISGLWuO51F8aYHM6WI4wxaYGngZ3AUqCls9rLwK/On39zvse5fInzyR2/Ac87Z7kVAooB64AQoJhzVlwqHAO5f7tTTGpBEhEREW/LDXzvnG3mA0yz1s42xuwAphhjhgChwDhn/XHAj8aYfcA5HAkP1trtxphpOJ7tGgV0dnbdYYzpAswHfIHvrLXb7xSQcSRcSY9/qrxJM3C5bxEH5nk7BK/KUKS+t0Pwmpgker0ScYWoyLC7N8W40MkaNV12wuVavsyjsbuCWpBEREQkERuT5HIal9IYJBEREZEE1IIkIiIiiaT0Z7EpQRIREZFE7D3MPkvO1MUmIiIikoBakERERCQRdbGJiIiIJKBZbCIiIiJyE7UgJUHGpOysPmPRBt4OwasuH5zv7RC8Jl3BYG+HIJJipPT7sipBEhERkURSehebEiQRERFJRAnSbRhjMt1pRWvtRdeHIyIiIuJ9d2pB2g5YIH4KGfveAg+7MS4RERHxIo1Bug1rbX5PBiIiIiIPjpTexXZP0/yNMc8bY/o5f85njHnSvWGJiIiIeM9dEyRjzBdALaCNs+gK8LU7gxIRERHvsta47JUU3csstirW2nLGmFAAa+05Y0wqN8clIiIiXpTSHzVyL11sN4wxPjgGZmOMyQak8P82ERERSc7upQVpNPAzkMMYMwhoBQxya1QiIiLiVTFJtGvMVe6aIFlrfzDGbADqOIuetdZuc29YIiIi4k1JdeyQq9zrnbR9gRs4utn0gFsRERFJ1u5lFlt/YDKQB8gHTDLG9HV3YCIiIuI9Nsa47JUU3UsL0ktAWWvtFQBjzFAgFHjfnYGJiIiI96T0O2nfS3fZCW5OpPycZSIiIiLJ0p0eVvsxjjFH54Dtxpj5zvfBQIhnwhMRERFvSKpdY65ypy622Jlq24E/4pWvcV84IiIi8iDQNP/bsNaO82QgIiIiIg+Kuw7SNsYUAYYCjwFpYsuttcXdGJeIiIh4UUq/D9K9DNKeAIwHDFAfmAZMdWNMIiIi4mXWuu6VFN1LgpTOWjsfwFq731o7AEeiJCIiIpIs3UuCdN35sNr9xpj/GmMaAxld8eHGmAnGmNmu2JYrpU6dmr9WzWbD+oVs2rSEd97pedPyj0cN5vy5PXHvO7zWhtCNi1gfsoBlS2fy6KPFPB2yy/n4+LBu7TxmzpwAQMeOr7Bjx0oirx8jW7YscfUyZcrIzF/Gsz5kAZtCF/PSS628FLHrBARkYvKkr9myeSmbNy2hYsVyccv+160D168djfs/aNwomPUhC1i3dh5/rfqDKlUCvRX2vxIdHc2zr/Wkc9+hAKzZsIVWHXrSsn0PXurajyNhjjt6zJq3hBrNXqFl+x60bN+Dn/9YGLeN0kEt48q79h8WV75241ZadehJ87bd6P/+Z0RFR3t25+5T6tSpWe28DmzetIR3ndeBH77/nO3blrMpdDFjx4zEz+9eH0iQtOTLl4dFC6bHnQddu7QDYNDA3mzcsJD1IQuY+8ckcufO6eVIPaNrl3ZsCl3M5k1LeKNre2+H4xEx1rjslRQZe5e2L2NMRWAHkAXHWKQA4ANr7ar7/nBjApwxXDDGLAO2WWu73Mu6/qnyurXRLn36dEREXMHPz48/l82kR493WbtuI0+We4KuXdvTtGk9smR1DMPKmDEDly5dBqBRo6f57+sv06hxa7fFZoz7v2zdur3Gk+VKkzFTBpo3f4UypUty/kI4CxdMp3KVBpw9ex6APm92ISAgE/36DyN79qxs27qc/A+X5caNG26Lzd37/+23o1i1ah3jx0/B39+fdOnSEh5+kXz5cvP1Vx9RvEQRKld2/B/Efk8ASpV6hEkTv+KJ0rXcGt+lA/Nctq3vp/3G9t37ibhyhdHv96dRm858NqQvhQvkY8qsuWzdtY+hb3Vl1rwlbN+9n/7dXku0jQr1/8O6uZNuKouJiSH4+df5duQgCubPwxffTSZPzhw807BOovX/P9IVDL6v9f+/4l8Hli+bSfce75I1a2bmzlsCwE8/jmbFirV8M+YHj8blCblyPUTuXA8RumkbGTKkZ93aebRo+SrHjp2Iu9516fwqjz5anM5d3vJytO5VsmQJJv70JZWrNCQy8gZzZk+kU5e32L//kEfjiIoM82imEfpwU5f9ni175NcklyXdtQXJWrvWWnvJWnvEWtvGWtvEFcmRc9vh1toLrtiWq8X+0vP398Pf3x9rLT4+Pgwf/jZv9R1yU93YiwU4Lqh3SzofdHnz5qZ+/SC+G//PL71Nm7dz+PCxRHWttWTImB6ADBnSc+78BaKiojwWq6tlypSR6tUqMn78FABu3LhBePhFAD768F369ht60/GN/Z5A0jv2J0+fYcWaDbSIl7QYY7js3KfLEVd4KF5r4f/HhYuX8Pf3o2D+PABULl+ahSuS3h1C4l8H/JzXgdjkCCAkZBP58uX2VnhudfLkKUI3Oe72cvlyBLt27SVvnlzJ7np3Lx55pBjr1oVy9eo1oqOjWb5iDc2baaRJcnenG0XOxHFjyFuy1j5zvx9ujJkAZAfOAE8BTxljOjsXF7LWHrrfz/i3YruYihQpyFdfT2BdSChdu7Rj9uwFnDx5KlH9jv99mW7dOpAqVSqC6ybtbqaRIwbSt+9QMmbMcNe6X341gV9+Hs/hQxvImDEDL7bumKQvmAUL5uf06XOMHTuKJx5/lI2hW+nZ812Calfn+PGTbN26M9E6TZrUY8h7fciRIzvNmr/shaj/nQ+/+I7ur7/ElatX48oG9upEp75DSJ0qFRnSp2Pi6OFxyxYtX82GLTsomC83b3Z+lVwPZQcgMjKS517vjZ+vD6/+5xmCqlUkS0AmoqOj2b57HyVLFGXhn6s5eeqMx/fxfsVeB4rGuw7E8vPz48UXW9CjxztejNAzChTIR5nSpVi7zrH/7w3uQ+sXWxJ+8SJ1nn7Wy9G53/btu3hvcB+yZs3C1atXqV+vNus3bPZ2WG6XhC/lLnGnFqQvgNF3eLlSN2A1jtlyuZ2voy7+jP+XmJgYygcGU7BQeQLLl6VatYq0aNGIL0Z/d8v6X339PY88WpV+/YfSr283D0frOg0aBHHq9BlCQ7feU/3gp2uyect2ChR8ksAKdfnkkyH3lFg9qPz8/ChbthRjxvxAxUr1uRJxhbcH9ODNN7swaPDIW67z22/zeKJ0LZ5t1Z6B7/bycMT/zp+r15M1cwAlSxS5qfzHGb/z5fsDWDz9W5rVq81HX44HoGblQOZP/oZfxn1MpSdL03/4Z3HrzJ/yDVO/+YjhA7rz4RffcTTsJMYYPny7Jx+OHs8LHd8kXbq0+Prcy5DHB0vsdaCA8zpQsmSJuGVffD6MFSvWsnLVOi9G6H7p06dj2tSx9Oj1blzr0dvvfEChIoFMnjyTzp3aejlC99u1ax8ffTSauXMmMWf2RDZt3k50dIy3w3K7lD4G6bZXLGvt4ju9XBmEtTYciASuWGtPOl+JRnQaYzoYY9YbY9bHxES4MoTbCg+/yLI/V1GzZhWKFCnIrp2r2LtnDenSpWXnjpWJ6k+d+itNmtT1SGzuUKVyII0aBrNn92p++nE0tWpWZcL4z25b/6WXWzFr1lwA9u8/xKGDRylRoqinwnW5sLATHAs7QUjIJgB+mTmHsmVLUbBgfkJC5rN791/ky5ubNWvmkjNnjpvWXblyLYUKPXzTIPYHVei2XSz9K4S6z79O78GjWBe6lU5vDWH3/kM88ZhjbF29WlXZtH03AJkDMpIqlT8ALRrWYceeA3HbypkjGwD58+SifJlS7NznWFamZAm+/2wok7/6kPJPPEaBfHk8uYsuFXsdqBtcE4C3B3QnR45s9Oo90KtxuZufnx/Tp45l8uSZced5fJMm/0Lz5g28EJnnjZ8whYqV6lMrqAUXLoSzd++Bu68kSVqS+pPOWjvGWlveWlvexye92z4ne/asBARkAiBNmjTUCarBxo1byf9wWYoVr0Sx4pW4cuUqjz5WDYCiRQvFrdugQR327TvottjcbcDbwylcJJDiJSrTuk1nli5bxStt37ht/aNHw6hdy/H/8NBD2SlevAgHDx72VLgu9/ffpzl27ATFixUGoFatqoSGbiP/w2UpUaIKJUpU4VjYCSpVqs/ff5+mSOGCceuWKVOKVKlSxw1gf5D977XWLJ7+LfOnfMNH7/SgQtnH+WxoXy5fvsKho8cBWL1+M4UfzgfA6bPn4tZd9lcIhR/OC0D4pctERjoG5J8Pv8imbbsoUiA/AGfPO4YXRkbe4LvJM2mVxP5wuNV1YPfu/bza9gWCn67Ji607J+nu5HsxdsxIdu7axyefjokri3+9a9K4Lrt37/dGaB6XI/YPgfx5aNasPpOnzPRyRO5nrXHZKylKnvNT71Pu3Dn5btwn+Pr6YHx8mDHjd+bMWXTb+p06vkLtoOpE3Yji/PlwXm33Pw9G6xmdO79Kzx4dyZUrBxvWL2TevKX8t2Nvhg37lG+/HcXGDYswBvr3H5YkEoQ76d79bSZM+JxUqfw5ePAIr3Xoedu6zZrXp/WLLbhxI4qrV6/Ruk0nD0bqWn6+vgzs1ZHu736IjzFkypiBwW86hgRO/GUOy1aF4OvrQ0CmjLz3VlcADh4+xqBRX+NjDDHW0u6F5hQp6EiQJkz9lT9Xr8daS6smdalY7nGv7du/Ef864OO8DvwxZxHXrhzm8OFjrFzxGwCzZs1hyNBPvByt61WtEkib1i3ZsnUH60MWAPD228Np2/Z5ihcvQkxMDEeOhNGpc/KewRZr+tSxZM2WhRs3onjjjf5xkzeSs6TaNeYqd53mH1fRmNTW2usu/XDnIG1rbSNjzAJgv7W2472s6+5p/g8yT0zzf5Cl9P135TT/pMbT0/xFHiSenua/Ns8zLvs9W/H4L0nuwn3XLjZjTAVjzFZgr/N9aWPM526I5RBQwRhT0BiT3XlzShEREfEC68JXUnQvSchnQCPgLIC1djPgjjvhjcAxUHsHcBp42A2fISIiIvcgpc9iu5cxSD7W2sMJujVc8swAa+0r8X7eA1R2xXZFRERE7se9JEhHjTEVAGuM8QW6Anvuso6IiIgkYUl19pmr3EuC1BFHN9vDwN/AImeZiIiIJFPJ/1aYd3bXBMlaewp43gOxiIiIiDwQ7pogGWPGcotB6NbaDm6JSERERLzOoi62u4l/h8Q0QHO8/Jw0ERERca+YpDo/30XupYttavz3xpgfgcQPIRMRERFJJv7No0YKATldHYiIiIg8OGLUxXZnxpjz/DMGyQc4B6SMh++IiIikUBqDdAfGcXfI0kCYsyjGJvfHV4uIiEiKd8cEyVprjTFzrLWlPBWQiIiIeF9Kvw/SvTyLbZMxpqzbIxEREZEHhsW47JUU3bYFyRjjZ62NAsoCIcaY/UAEYHA0LpXzUIwiIiIiHnWnLrZ1QDmgiYdiERERkQdESu9iu1OCZACstfs9FIuIiIg8IJQg3V4OY0yP2y201o5yQzwiIiIiXnenBMkXyABJdHSViIiI/GtJdXC1q9wpQTphrR3ssUhERETkgRGTsvOj/2vvzuNtqv4/jr8+XGQeUgkNxgZ9KWMkyRgiMjR8M30VIURE86C+KWmeEKHBXIYyT6EM1zyLkCGSzEOmu35/7O3+7mTqe8/Z7j3vp8d5uHftfc75rHvuPftzPmutvc+5zD/CfzQiIiISqc5VQaoatihERETkkqJrsZ2Fc25vOAMRERGRS0ekX1fsvBervVR5l4mLTDGRfjm8CO9/putrBB1CYI79PifoEAKVMe+dQYcgEjFSbIIkIiIioaPzIImIiIgkEBPBIzWgBElERESSENmTGc69zF9EREQkIqmCJCIiIolE+hwkVZBEREQkkRhLvtv5mNk1ZjbTzNaY2Woz6+S35zKzqWa2wf8/p99uZvaBmW00sxVmVjLOYzX3999gZs3jtJcys5X+fT6w87pOaMkAACAASURBVCyHV4IkIiIiQTsFPOWcuxm4HWhvZjcDPYDpzrkiwHT/e4BaQBH/1hr4FLyECngJKAeUBV46k1T5+zwW5373nCsgJUgiIiKSSAyWbLfzcc7tdM4t8b8+BKwF8gH3AYP93QYD9f2v7wOGOM98IIeZXQ3UBKY65/Y65/YBU4F7/G3ZnHPznXMOGBLnsZKkBElEREQSccl4M7PWZrYozq312Z7XzK4HbgMWAFc553b6m3YBV/lf5wO2xbnbdr/tXO3bk2g/K03SFhERkZByzvUD+p1vPzPLAowGnnTOHYw7Tcg558wsbGcfUAVJREREEgnnJG0AM0uHlxx97Zz71m/+wx8ew/9/t9++A7gmzt3z+23nas+fRPtZKUESERGRRGKS8XY+/oqyAcBa59w7cTaNA86sRGsOjI3T3sxfzXY7cMAfipsM1DCznP7k7BrAZH/bQTO73X+uZnEeK0kaYhMREZGg3QE0BVaa2TK/7VmgFzDCzFoBvwFN/G0TgNrARuAo0BLAObfXzHoC0f5+rzrn9vpftwMGARmBif7trJQgiYiISCLhvNSIc24unHW5W9Uk9ndA+7M81kBgYBLti4BbLjQmJUgiIiKSyIXOHUqtNAdJREREJAFVkERERCSRSL8WmxIkERERSSTSEyQNsYmIiIgkoAqSiIiIJOIifJK2EiQRERFJRENsYWZm35vZoHA/r4iIiMiF0hykc0iTJg0LF0ziu+8GATBj+miiF04meuFktmxexKiRnwPw0IMNWLxoKksWT+PHWWMo/q+bAow6eRUtWohF0VNib3v3rKNjh0fJmTMHkyYMZe3quUyaMJQcObIHHWrIZM+ejeHD+rFq5Y+sXDGL28uVAqB9u5asWvkjy5fNoNcbzwUcZWgk1fcSJYrx05zxLIqewvx5EyhT+tagw/xHTp8+TaMW7WnX7SUAFixeRuOWT1D/kcd5tufbnDp1Ot7+K9eup0SlOkyZOSe2beyEqdR+oBW1H2jF2AlTY9vf7zuIqg2aUqZag/B0JgQyZMjAvJ++Z/GiqSxfNoOXXnwqdlvPV7uzZvUcVq6YxRPt/xNglKGTP39epk0ZyYrlM1m+bAYdnmgFQPHiNzN39jiWLpnGmO8GkTVrloAjDZ1wXmrkUqQhtnPo0KEV69ZtJGs27w+gStWGsduGD+vH+PGTAdi8ZStVqzVi//4D1Kx5N5988hYV76wbSMzJ7ZdffqV0mRqAlzBu3bKYMWMn0v3p9syYOZe3en/M093a0/3p9jzz7H8DjjY03n3nVSZPnskDD7YmXbp0ZMqUkcp3VaBe3ZqULFWdEydOcMUVlwcdZkgk1fdh33xGz9feYdLkmdS6pwq93niOqtUbBx3qRftq5FgKXn8th48cJSYmhmdf68OA99/g+mvz81H/IYydOI2GdWsCXjL17idfUKFMydj7Hzh4iE+/+IbhAz4A4IFWHalc8XayZ8tK5TvK8XDDetR+sFUgfUsOx48fp1qNJhw5cpSoqChmz/qOSZNmcuONhcmfPy/FbqmEcy7V/u6fOnWKbk+/wtJlq8iSJTMLF0xi2vTZ9P2sN92792T2nPm0aP4AXZ9qy0sv9w463JAI55m0L0UhrSCZWSYzG2Rmh83sDzN7NsH29Gb2ppltN7OjZhZtZjVDGdOFypfvamrVqsrAL75JtC1r1ixUrlyBseO8BGn+/MXs338AgAULlpAv39VhjTVcqlapyKZNv7F16w7q1q3JkC9HAjDky5HUq3dPwNGFRrZsWbmzYjkGfjEUgJMnT3LgwEHatGnGW70/5sSJEwD8+edfQYYZEmfru3OOrNmyevtkz8rvO/8IMsx/ZNfuP5n988LYBGj/gYOki4ri+mu9i32XL1OSabPmxu7/zahxVK98B7ly5oht+2nBYsqXuY3s2bKSPVtWype5jZ8WLAagxC03cUXuXGHsUWgcOXIUgHTpoohKlw7nHI+3acZrr7+Ld6WH1Pm7D7Br126WLlsFwOHDR1i3bgP58uahaJGCzJ4zH4Bp0+fQoEHtIMOUEAr1ENvbQHWgId61VG4DKsXZ/gVwF/Aw3vVRBgPjzaxEiOM6rz5vv8wzz7xOTEziHPq+ejWZOfMnDh06nGhby5YPMnnyzHCEGHZNmtzHsOFjALjqytzs2rUb8N5Irroyd5ChhUyBAteyZ89fDPj8XaIXTqbvZ73JlCkjRYoUpGLFsvw8dzwzpo2idKnAf2WT3dn63qXrS7z5xvNs/jWat3q9wHPPvxF0qBftzff70qVdK8y8t8CcObJz+nQMq9b+AsCUWXPZtXsPAH/8uYfps3/mgQZ14j3GH3/uIc+VV8R+f9UVufnjzz1h6kF4pEmThkXRU9i5YwXTp89mYfRSCha8niaN6zF/3gS+H/clhQsXCDrMkLvuuvzcWuIWFixcypo1v1CvnpdYN2p4L9fkzxtwdKETY8l3S4lCliCZWRagFfC0c26yc24V3tV2Y/zthYCHgCbOudnOuU3OuY/wrtDbJlRxXYjatauy+889LF26MsntTR6oz/DhYxO133VXBVq2eJBnn3s91CGGXbp06ah7bw1Gjf4+ye1nPk2mNlFp03Lbbf+ib98hlClbkyNHjtL96SeIikpLzpw5qFCxLt17vMbQbz4LOtRkd7a+t2ndjKe6vUyBQmV4qtsr9O/bJ+hQL8qsnxaQK2cOit1YJLbNzOj9ag/e+qAfDz7aicyZMpImjff2+Ob7fenc9j+x30eSmJgYSpepwXUFSlOm9G0UK3YDGTKk5++/j3N7+dp8PvAbPu+Xsl7/i5U5cyZGDO9Pl64vcejQYR5t3YW2bZqzYP5EsmbNzIkTJ4MOMWQ0Byl0CgHpgXlnGpxzh83sTNZREu/KvWvM4qWXGYAZST2gmbUGWgOkTZuDNGkzhyBsqFC+DPfWqcE9Natw2WUZyJYtK4O++IAWLTty+eU5KVP6Vho3fjTeff51y0189tlb1KvXlL1794ckriDdc8/dLF26kt1nPlXv3kOePFeya9du8uS5kt2ptMy+fcdOtm/fycLopQB8++0PPN3tCXZs38mYMRMBiF60jJiYGHLnzsWePXuDDDdZna3vd9xRhs5dXgRg1Kjx9PssZc2/WLpiDbPmzmfOvGiOnzjpJX6vvMWbLz3NkE/fBrzhs9+27QBg9boNdHupFwD7Dhxkzrxo0qZNy1VX5CZ66YrYx/3jzz2Uua14+DsUBgcOHGTWjz9Rs0Zltu/YyXdjJgAwZsxEBvR/J+DoQicqKoqRw/szdOh3sX/v69f/Sq06DwNQpEhBatdKdKF5SSWC/EiUBm8OWBng1ji3m4Akl0U45/o550o750qHKjkCeP6FXhQsVIaiN5TnkabtmTnrJ1q07AjA/ffXYcKEaRw/fjx2/2uuycvwEf1p2bITGzZsDllcQXrwgfqxw2sA34+fQrOm3sTcZk0bx05YT23++ONPtm//naJFCwFQpUpF1q79hbHjJlO5cgXAe5NMnz59qkqO4Ox9/33nH9xVqbzXdndFNmxMWb/zndu2ZPqYr5gyejC9X+lB2VIlePOlp/lrn/fB5sSJEwz8eiRN6ntzSyaPGsSU0YOZMnowNSpX5Pmu7alaqQJ3lCvFzwuXcODgIQ4cPMTPC5dwh7/CMTXInTsX2bNnA+Cyyy6jWtVKrF//K+PGTaLyXd7v/l2VyvPLhk1BhhlS/fv1Ye26jbz3fr/YtjOT0s2MZ5/pRN9+XwYVXsipghQ6vwIngduBTQBmlhlvrtGvwFK8ClIe51yKmbTTpPF99H7743htzz3bmctz5eDDD7xVXKdOnaJ8hTpJ3T1FypQpI9WqVqJtu+6xbW/2/phh33xGyxYPsXXrdh58+PEAIwytTp1fYMjgD0mfPh2bN2+l1aNdOHLkKJ/378OypdM5ceIk/2n1ZNBhhkRSfR83fjLvvPMqUVFRHP/7b9q2fTroMJPFF1+P4sefF+JiYnigQR3KlTr36QuyZ8tKmxYP8eCjnQB4vOXDZPcnr/f5eAATps7k77+PU7X+I9xf9x7at3ok5H1ITldffRUDB7xH2rRpSJMmDaNGjeeHCdOY+9NCvhz8EZ06PcaRw0dp83i3oEMNiTsqlKHpI41YsXINi6KnAPDCC70oXLgAbdu2AGDMmAkMGjw8wChDK3VOnLhwFsq5I2b2KXAvXkXod+BFoBbwrXOuhZl9BdwJPAUsAXIBlYFNzrlvz/XY6TPkj9jXLiaVzvcROZ9jv885/06pWMa8dwYdggTo1IkdYZ3u/Pa1jyTbwabr1q9S3FTtUJ8HqSuQGfgOOAp86H9/RkvgOeAtID+wF1gIpJiKkoiISGqUUlefJZeQJkjOuSNAM/+W1PaTwMv+TURERC4RKXXuUHKJvHWrIiIiIuehS42IiIhIIpE+21UJkoiIiCQSE+EpkobYRERERBJQBUlEREQSifRJ2kqQREREJJHIHmDTEJuIiIhIIqogiYiISCIaYhMRERFJINLPpK0hNhEREZEEVEESERGRRCL9PEhKkERERCSRyE6PNMQmIiIikogqSCIiIpKIVrGJiIiIJBDpc5A0xCYiIiKSgCpIIiIikkhk14+UIImIiEgSIn0OkobYRERERBJQBUlEREQSifRJ2ik2QYpxkf3CiUSijHnvDDqEQB3bNiPoEAKVo0DNoEOIKJF+lNUQm4iIiEgCKbaCJCIiIqET6ZO0lSCJiIhIIi7CB9mUIImIiEgikV5B0hwkERERkQRUQRIREZFEtMxfREREJIHITo80xCYiIiKSiCpIIiIikoiG2EREREQS0Co2EREREYlHFSQRERFJRCeKFBEREUlAQ2wiIiIiEo8qSCIiIpKIhthEREREEtAQm4iIiIjEowqSiIiIJBLjNMQmIiIiEk9kp0caYhMRERFJRBUkERERSUTXYhMRERFJINKX+Qc6xGZmLczscJAxXKg0adIQvXAyY78bDMCAz99lw/p5LIqewqLoKZQoUSzgCMOjwxOtWLZ0OsuXzaBjh0eDDicsEr72Ve6uyMIFk1gUPYUfZ35HoULXBxtgCPXv14ffty9n2dLpsW2vvNyNJYunsih6ChN/+Iarr74qwAhDJ3/+vEybMpIVy2eyfNkMOjzRCoA333ieVSt/ZMniqYwa+TnZs2cLONJ/5vTp0zRq1Yl23V8FYP7i5TRu9SQN/9OJpu27s3X77wD8vms3rZ58ngYtOtCi47Ps2r0HgHUbNvHvtt24r1l7GrTowMTpc2If+5vR31ProdbcUqke+/YfDH/nLsKnn73Fli2LiI6eHNvWoEFtohdN4dDhTdxW8l+x7Q88cB/z5k+IvR06vInixW8GoHHjeixcOIkFCyYyZuxgLr88Z9j7IslLc5AuUMcOj7Ju3YZ4bd2feY3SZWpQukwNli9fHVBk4VOs2A20avUw5SvUoWSp6tSpXS1VJwdnJHztP/roDZo1f4LSZWowdNgYnn2mU4DRhdaQISOoc++/47W93edTSpaqTukyNfhhwjSef65zQNGF1qlTp+j29CsUL3E3d1SsS9u2LbjppiJMmz6bErdWoWSp6mzYsIke3Z8IOtR/5KtR4yl43TWx3/fs8ym9XniK0QPfp061u+g7ZAQAb38ykHo17+a7QR/StvkDvNdvCACXXZaB/z7bmbFDPqbv2y/z5oefc/CQ93n3tn/dxOfv9CRvnivD37GL9NWXo6hfv3m8tjVr1vPwQ48zd+7CeO3Dh4+l/O21KX97bR5t1ZktW7axYsUa0qZNS+/eL1Kr1kOUK1eLVSvX0ubx+I+ZEsUk4y0lUoJ0AfLlu5rataoycODQoEMJ1I03FmHhwqUcO/Y3p0+fZvac+TSoXyvosEIqqdfeOUe2rFkByJ49Kzt3/hFUeCE3Z+4C9u7bH6/t0KH/L/pmzpwJl0qXAu/atZuly1YBcPjwEdat20C+vHmYOm02p0+fBmD+giXky3d1kGH+I7t272H2vEU0rFM9ts3MOHL0KACHjhzhity5APh1yzbKliwOQNmSxZk5dwEA11+Tj+uuyQvAlbkvJ1fO7LHVopuKFiJfCqks/vTTQvbuPRCvbf36X9mwYdM579e4ST1GjRoPeD87zMiUKRMAWbOljveFGFyy3VKisCRIZlbJzOab2WEzO2BmC83sliT2y2lmP5nZZDPLHI7YLsQ7fV6hxzOvERMTPw/u+Wp3liyeSp/eL5M+ffqAoguf1avXUbFiOXLlyknGjJdR654q5M+fN+iwQiqp175Nm66MH/clWzYt4t//bsibb30UYITB6Plqdzb/Gs1DDzXg5Vd6Bx1OyF13XX5uLXELCxYujdfessWDTJo8M6Co/rk3P/ycLm1bYGn+/xDwytNP0PbpV6nasCXjJ8/i0X83AuCGwgWYNnseANNmz+PI0WPsPxB/2Gzlml84efIU1+TLE75OBKxhw3sZOWIc4FUbn+z0PAujJ/HrpoXceGNhBg8aHnCE8r8KeYJkZlHAWGAuUAIoB7wHnE6wX15gNrAdqOucOxLq2C5EndrV2L17D0uWrozX/tzzb1DslkrcXr4OOXPl4Olu7QKKMHzWrdtI794fM3HCN0z4/muWLV/N6dMptXh6fmd77Tt1eoy69ZpyfcHSDB48nLd7vxRQhMF54cU3KVCoDEOHfkf7di2DDiekMmfOxIjh/enS9aV41bNnenTk1KlTfPPNtwFGd/Fm/RxNrpzZKXZD4XjtQ0aM5dO3XmT66C+oX7sqb300AICu7VqyaNkqGrXqxKJlq7nqistJEyex+nPPXp55/V1ee6ZjvPbUrHSZWzl29Bhr1vwCQFRUFI899ggVytehUMGyrFq1jq6p4JjgkvFfShSOVWzZgBzAeOfcr37bOgAzK+f/XxiYCkwG2jnnkjzqmllroDWApc1OmjShLzJVqFCauvfWoNY9Vbjssgxky5aVwYM+oHmLjgCcOHGCwYOH06Xz4yGP5VLwxaBhfDFoGACv9ezB9u07A44odJJ67ceNGcINNxRiYbRXSRgxchw/fP91wJEG55uh3zJ+3Je88mqfoEMJiaioKEYO78/Qod8xZszE2PZmTZtQp3Y1qtdsEmB0/8zSlWuY9dNC5sxfzPETJzhy5Chtn36VzVu3U/zmGwCoVeVO2nR9GfCGz95//VkAjh49xrTZP5MtaxYADh85Srvur9LxsUcoUezGQPoThMaN6jJi5LjY74uX8CZqb968FYBvR//AU13bBhJbckq9H38vTMjTfefcXmAQMNnMfjCzLmZ2bZxd0uNVlyY45x4/W3LkP1Y/51xp51zpcCRHAM8934vrC5amcNHb+fcj7Zg58yeat+hInjiTD+vVu4fVa9aFJZ6gXXHF5QBcc01e6tevxdBh3wUcUegk9do3aNiS7NmzUaRIQQCqVa2UaPJ+ale4cIHYr+vVrcn69b+eY++UrX+/Pqxdt5H33u8X21azRmW6dm1L/ftbcOzY3wFG9890btOc6aO/YMqIz+n9UjfKlizOh/99jsNHjrBl2w4Afo5eSsHr8gOwb//B2CHm/l+PokHtagCcPHmSTs/9l3o176ZG5TuC6UwAzIz7G9Zh1MjxsW2//76LG28qQm5/3laVqhVZv25jUCGmSGY20Mx2m9mqOG25zGyqmW3w/8/pt5uZfWBmG81shZmVjHOf5v7+G8yseZz2Uma20r/PB2Zm54spLOdBcs61NLP3gHuAesDrZlbf33wSmALUNrPrnHO/hSOm/9WXgz8i9xW5MDOWL19Nu/Y9gg4pLEYO70+uy3Ny8uQpOnZ8jgMHLu0lvMnt9OnTtGnbjRHD+xET49i/bz+Ptn4q6LBC5qsvP+auSuXJnTsXWzYt4pVX36ZWrSoULVqImJgYtm7dkWp/9++oUIamjzRixco1LIqeAsALL/Ti3XdeJUOGDEya6FVSFyxYQvsnUvbPICoqLS93e4LOz/fC0hjZsmahZw+vSh69bCXv9R2CmVGqRDGe96vlk2bOZfHy1ew/eIgxk2YA8PoznbixSEG+GjWeL4Z+y569+7i/ZUfuvL0Ur3bvEFj/zmXQoA+4s9LtXH55Tn7ZMI/XXnuXffsO0KfPy+TOnYtvRw9kxYq13HdfMwAqVizH9u072bJlW+xj7Nq5m//+930mTxnBqZMn2bptB21adw2qS8kmzAswBgEfAUPitPUApjvneplZD//77kAtoIh/Kwd8CpQzs1zAS0BpvCulLDazcc65ff4+jwELgAl4+chEzsGCWIFiZhOBfXiJ0Ud4w3CDgTuAys65red7jKj0+VLmoKaIyD90bNuMoEMIVI4CNYMOIVBHjm45b9UjOd137b3Jdpwdu/X788ZuZtcD3zvnbvG/X4+XE+w0s6uBWc65G8ysr//10Lj7nbk559r47X2BWf5tpnPuRr/9obj7nU04JmkXMLNeZlbBzK4zs7uB4sCaM/v4w2rNgZ+BWQmG4ERERCTyXOWcOzPRdRdw5twR+YBtcfbb7redq317Eu3nFI4lB0eBosBI4Be8StHXwJtxd0qQJM1UkiQiIhKc5DxRpJm1NrNFcW6tLyYW5w13hXXkKORzkJxzfwD3n2XzIP92Zt/TwCOhjklERETOLTmX5zvn+gH9zrtjfH+Y2dVxhth2++07gGvi7Jffb9uBN8wWt32W354/if3PKTJOWiEiIiIpzTi8kSX8/8fGaW/mr2a7HTjgD8VNBmr4J53OCdQAJvvbDprZ7f7qtWZxHuuswrKKTURERFKWcF4ixMyG4lV/cpvZdrzVaL2AEWbWCvgNOHPisQlAbWAj3jSeluCdVsjMegLR/n6v+qcaAmiHN2KVEW/12jlXsIESJBEREUlCOFe5O+ceOsumqkns64D2Z3mcgcDAJNoXAYkucXYuGmITERERSUAVJBEREUkk0i81ogRJREREEkmpF5lNLhpiExEREUlAFSQRERFJJJyr2C5FSpBEREQkkSCu1Xop0RCbiIiISAKqIImIiEgiGmITERERSUCr2EREREQkHlWQREREJJGYCJ+krQRJREREEons9EhDbCIiIiKJqIIkIiIiiWgVm4iIiEgCkZ4gaYhNREREJAFVkERERCSRSL/UiBIkEZEUIut11YMOIVCHNk8OOoSIoiE2EREREYlHFSQRERFJJNIvNaIESURERBLRHCQRERGRBDQHSURERETiUQVJREREEtEQm4iIiEgCGmITERERkXhUQRIREZFEtMxfREREJIGYCJ+DpCE2ERERkQRUQRIREZFENMQmIiIikoCG2EREREQkHlWQREREJBENsYmIiIgkoCE2EREREYlHFSQRERFJRENsIiIiIgloiE1ERERE4lEFSURERBLREJuIiIhIAs7FBB1CoDTEdhGKFi3Eougpsbe9e9bRscOjQYcVNpHY//79+vD79uUsWzo9tq1hw3tZvmwGJ/7eRqmSxQOMLvw2/jKfpUumsSh6CvPnTQg6nJBK6rV/8YUu/LZ5UezfQK17qgQYYWh06NCKJUumsXjxVIYM+ZAMGTJw9913MG/eDyxYMJEZM0ZTsOB1ADRt2oht25ayYMFEFiyYSMuWDwYc/YU7ffo0jR7tTLserwEwf/FyGj/WhYatnqTpE8+wdftOAN78aAANWz1Jw1ZPUueRdpSv83C8xzl85ChVG7Xi9ff6AXDs7+O07dGTuk3bc1+LDrzbd0h4OybJRhWki/DLL79SukwNANKkScPWLYsZM3ZiwFGFTyT2f8iQEXzyyRd88cX7sW2rV6+jcZPH+PTjXgFGFpxq1Rvz11/7gg4j5JJ67QHe/6A/77zbN6CoQitv3qto374lt95alb//Ps5XX31CkyZ1efrpJ2jU6FHWr99I69ZNeeaZjjz22FMAjBo1ns6dXww48ov31ejvKXhdfg4fOQZAz3f78sHrz1DoumsYNmYCfb8cwevPdKL7E61i7/P1t9+zdsPmeI/z4cBvKFXi5nhtLR+oT9nb/sXJkydp1eVF5ixYzJ3lSoW+U8ksJsKH2MJSQTLPU2a2wcyOm9l2M3vD3/aimf3mt+8ysxSRbletUpFNm35j69YdQYcSiEjp/5y5C9i7b3+8tnXrNvLLL78GFJGES1KvfSSIiooiY8bLSJs2LZkyZWTnzj9wzpEtWxYAsmfPxs6dfwQc5f9m1+49zJ6/iIZ1qse2mcERP1k6dOQoV+TOleh+E6bPoXbVO2O/X71+I3/t3U+F0rfGtmW8LANlb/sXAOnSpeOmooX448+/QtWVkHLOJdstJQrXENt/gReAN4BiQGNgm5k1BLoC7YAiwL3AwjDF9D9p0uQ+hg0fE3QYgYn0/kcq5xwTJwxlwfyJPNrq30GHE4h2bVuyZPFU+vfrQ44c2YMOJ1n9/vsfvPtuPzZsmM+WLYs4ePAg06bNoW3b7owZM5iNGxfw8MP307v3J7H3qV+/NtHRk/nmm8/In//qAKO/cG9+NIAubZpjZrFtr3RrT9sePanaqBXjp8zi0YcbxrvP77t2s2Pnbsr5yU9MTAy9P/mCrm1bnPV5Dh46zI8/R1MuwobiU4uQJ0hmlgXoDPRwzg10zm10zs1zzn0CXAfsBKY457Y65xY55z46x2O1NrNFZrYoJuZIqEM/q3Tp0lH33hqMGv19YDEEKdL7H8nuursBZcvdw711H6Ft2xbcWbFc0CGF1Wd9h1D0xgqUKl2DXbt20/utlDe0dC45cmSnbt3q3HjjHRQoUIZMmTLx0EMN6NChFfXrN6dw4XIMGTKCt956AYAffpjGDTdUoEyZmsyYMYfPP38n4B6c36yfo8mVMzvFbigcr33IyPF82usFpo8aQP1aVXnr44Hxtk+cMZcad5Unbdq0AAwbM5FKt5ciz5W5k3yeU6dO83TPd/j3/XW4Jm+e0HQmxGJwyXZLicIxB+lmIAMwPYltI4FOwGYzmwxMAsY5fxPi8wAAEQdJREFU544n9UDOuX5AP4Co9PkC+4nfc8/dLF26kt279wQVQqAivf+R7PffdwHw559/MXbsRMqUuZU5cxcEHFX4xP2d/3zA14wdMzjAaJJflSoV2bJlG3v27AVg7NhJlC9fmuLFbyY6ehngzTkaN+5LAPbu/f8hyIEDh/L668+EP+iLtHTVOmb9FM2c+Ys5fuIkR44epW2Pnmzeup3iNxcFoNbdFWnz9Cvx7jdxxhyee7JN7PfL16xn8Yo1DBszkaPH/ubkqVNkyngZnds0A+DlPp9wbf6radq4Xvg6l8xS6tBYcgl0FZtzbhtwA9AGOAj0ARabWeYg4zqfBx+oH9HDS5He/0iVKVNGsmTJHPt19Wp3sXr1+oCjCq88ea6M/br+fbVSXf+3bdtB2bIlyZjxMgDuvvsO1q7dQLZsWSlcuAAAVaveybp1G4D4P497763OunUbwx/0RercuinTRw1gyvD+9H7xKcreVpwPX3uWw4ePsmWbN6fy50XLKHhd/tj7bPptOwcPHebWYjfEtr35fBemjficKcP707VtC+rVuDs2Ofrg8685fOQIPeJM8JaUJxwVpLXAcaAqsCHhRufc38APwA9m1gvYBdwBTAlDbBctU6aMVKtaibbtugcdSiAirf9fffkxd1UqT+7cudiyaRGvvPo2e/ft5/13X+OKK3IxbuwQli9fTe17U/98nKuuuoJRIwcAEBWVlmHDxjB5yqxggwqhpF77u+6qQIkSN+Oc47fftqe6v4Po6GV8990E5s+fwKlTp1m+fDUDBnzDjh07GTasLzExMezff4A2bboB0L59S+rUqc6pU6fYt29/7Mq2lCYqKi0vd2tP5xffxNKkIVuWzPTs3iF2+8QZc6hV5c54c5bOZtfuPfT7aiQFrs1P48e6APBQgzo0urf6ee556Yn0S41YOEpoZvYm0BroAswGLgdKAcfwkrQFwGHgAeBV4Abn3OakH80T5BCbiEgQotKkDTqEQB3aPDnoEAKV7uqbzp+hJaM8OW5KtuPsrv1rwxp7cgjXeZCeAfbhrWTLD/wBDAGige7A20A6YA1w//mSIxEREZFQCkuC5LzzlffybwlpMouIiMglJtInaetM2iIiIpJISl2en1x0LTYRERGRBFRBEhERkUQ0xCYiIiKSQKQv89cQm4iIiEgCqiCJiIhIIhpiExEREUlAq9hEREREJB5VkERERCQRDbGJiIiIJKBVbCIiIiISjypIIiIikoiL8EnaSpBEREQkEQ2xiYiIiEg8qiCJiIhIIlrFJiIiIpJApM9B0hCbiIiISAKqIImIiEgiGmITERERSSDSEyQNsYmIiIgkoAqSiIiIJBLZ9SOwSC+h/VNm1to51y/oOIIQyX0H9V/9j9z+R3LfQf2PNBpi++daBx1AgCK576D+q/+RK5L7Dup/RFGCJCIiIpKAEiQRERGRBJQg/XORPA4dyX0H9V/9j1yR3HdQ/yOKJmmLiIiIJKAKkoiIiEgCSpBEREREElCCJEkyMws6BhERkaAoQRLM5399M4DT5LSIYmZ6L4hQcf72rww6lqDpg6HEpTfFs4iEPxQzu9LMzPnMrC4w18zKBx1bEMwsS9AxhMuZhMjMMgM452LM7JZgo5JwOvMe5//t3w2MMLNMkZgsm1khM0unD4YSV8T9IZyLmeUxs+sh9VdQzKwv8B5wmf/9tUAT4Bnn3LwgYwuCmdUEvjCzEkHHEg5+QnQtMMDMbjez+4EVZnZr0LGFU5zqyVVmVtjMcprZmb+JVPn+aGZdzay+nxid6eOtwAHn3NEgYwuCmdUDvgU6pNbXXP4Z/TL4zKwB8BPwo5ktNLMyZpYqL+ZrZg8A9YG3nHPHzKwk8DJQAPjR3yfVV9DOMLOGwChgNf+fMKba/ptZOzO7HbgCKAR8BHwNNHPOLYuUg8SZ6qmZ1Qem+LfpwIdmVtA5FxNshMnPzHIAZYBhZlYrTh+vBo6DlzwHFV+4+a/9cLzzG30fSX2X84uIN8LzMbPiwCdAf6ALcAwYA1RLpUnStcBf/sHwHmAQUAEoDVwPqb+Cdob/2n8EdHHOveycW+Bvui7AsELGzK4AqgN7nHOL8Q4MJYFfgd8gtrqUahNE8KpDfnJUFfgKGAj8CxgLNAfuDDK+UHHO7QeeBQbjDanVibP5GICZZQgitnAzs7zAC0A359zHwBYzy25mD/rVxNT43i8XIeITJDMrhVc56e+c6+WcG+2cuwtYBXxB6kySZgNRZjYD+AHoDDyJd5DsaGZlgwwuHOIkAEWB3c65/maW2cxamtlUYJWZ9TWz9AGGmeycc38CDzrnNppZGaAsXkKwB3jWzGr5+7nUliSZWXMzew5ik8B0eMPK/Z1z7wNZgJb+94P9+2QMLOBkdqYy6Jz7FXgLGIpXSboT2AL85fc385n9zaxoQOGGlP93fRjIiZcYpcVLHH8A+gKLgTuCi1AuBRGbIPkLtzLgvUl8BxSKu5rLOVcTWIH3Cbt2akqS/CrJDKAyEO2cm+6cmwT0BK4EnjSz0gGGGHJxKmTbgOxmNhhveOU+vOT4EeAxoFowEYaOc+64Pzn7JaAYMAd4HMiKlyDX9PdzZnZXcJEmH//AXw+ob2YdAZxzJ4E8QLRfWVsCTAae8O9TF6iaioYcHYD/t30lXuV0GDAReBHv57MGWGtmG/GSph/M7PJAog0RM6uB1/dyeB8W3wN2AiWA0c657HjD7c0CC1IuCRF/qZEzE1WBIkBN59x6v/we42+fD+QASjnnjgQYarLxDxbf470BVgCWOuce9rc9jDfMuAb4NDVO2DazCnjzMHICm4FMQFW8ZGkAsNY5d9rMZgM9nXNTAws2hMzsRuAdIAPQHu8A+jneJ+uRQH68uWn5nHM7Awoz2Zi3jP0NvKrhOOdcbzMbBNyMNx9rAtDJOXfK/xvpD6wDejnnTgUUdrKIM9+qAd5r/AHwKV7VrBPwKF5/38GrqKcFYoDNzrnfgok6+fmLEb4CXsN7vU/gTS3IhPdh+ZBfXRwOrHLO9QwsWAlcRCZIcQ6QVwBz8ZKB4UB6oJFzbnOCJOla59zWwAIOATPLhDfn4DHgKWBxnCTpQbw3kJlAB+fc34EFmsz8N8iBwHi8ialZgE1Ac7+icGa/nnifIO9wzm0PItZwMLMiwMd4B8T2wGm8g2Rhv+0hf65SiuVXhaOccyfNO8/Xy3hJ0rvAArwJ6nmcc/ni7P8aXhWxmnNuQyCBJzO/cjIaLyEa7Zw74LffCLTBS5LqOedmBhdl6Pi/65OAd/w5R0ntkxvv59MWqOicWxfGEOUSE3EJUoIDZB7gKrzS+gt45fXDQBPn3Ja4SVJqZWZZgQeBrsRPkhr5328OMr7k5B8IJuJVBPr63y/Em3PylL9PZaAFUBuvorg0oHDDJkGS1AFYjzdJ/Yhz7o8gY0sOcaonTYCHgFzAbcABvKGWHcB/gf3ARv9ulYDqqeX194cJPwVOOuee8D8g3YSXBO4CduNV0p7C+71PdVVTM6uCN7+ohv8h2PCOgTFxtjcH7gbuSy2vvfxzqWVs/YL4B8Q+QHfnXFO8uQbX463o2gbUwDtITPerRqk6OQJwzh3CKy2/DZQws/F++6jUlBz58gN7/eSoAF5CPDROclQT76BxAqgcKW+QfoWkPd4y7yFAWefcptSQHEHsXKqyeIsuvsdLgIsBy4G6eB+SqgJTgYPAMqB8Knv90+AlvXn9KtqHeMONNfDmHtXCO9XFJ3hDzalRFiAjELv4IE5yVAlvFeMM4O5U9trLPxRRCRKJD5BTiHOABC4H7sf7RJU2oBjDzjl3GC9J+gzIY2b5Ag4pVDICe82sEN75niYB7QD8A2hFvHNhPemcWxNYlAHwk6QuePPSdgQbTUgUxzvwD3fObfY/ELXGqxh3Bio4555yzrVwzr2aWobVzvDnUPXCq47MwZuQ3885Vwz4BrgGb+VWh1Q8rLQSr3rYGhKdyuQ+IC8wyl/lJ0KqWZl1geIeIGfiDbfEPUA+iFdhuiulT8q8WM65w2Y2EPjSP1dKarQGb+nuBuBj51yHONseBm7Bm58QcWcTBnDOrTOzh51zJ4KOJQSO473fZQMOm3dZid/9FW0LgRfNLLtz7oMzQ3KBRhsCzrlZ5p0pPo9zbmGc0zhcj1c5u8z/sJQq+cNq7YG+/qrkL/Hm3DUDWuFVDVPFQhxJHpGWIF3IAfJYpCVHZ6T2Nwfn3K9m9h+8VTx/+0Ou6YGm+CcHdM7tCzLGoKXS5AhgHl6FoD3wXJwJ+ZfhVU424Z0cNlWfJNVfbHJmwUlJ884i3xrvdz/VJkdxDAFO4Q0xPoKXGJ7CG1ZbG2RgcumJqATpAg+Qe4OMUUJuNN5r/iHehN2DwN94b5CrggxMQsd5J8ZsjXftubR47wH7gMZ4E5S7nlnVFQnM7AagO95qvkrOuRUBhxQWzrnTwJdmNgvvMjtHgS3Oud2BBiaXpEhcxZYO78D4IXCI/z9A/sc5tyzI2CR8zCw/3tDCIWCHc25PsBFJqPlDSg/infx1L97wSja8VU1Lgowt3PyfRTG8OZm/Bx2PyKUo4hKkM3SAFIlMZnY93oqljMBC59yWIOMRkUtTxCZIIiIiImcTacv8RURERM5LCZKIiIhIAkqQRERERBJQgiQiIiKSgBIkERERkQSUIImIiIgkoARJ5BJnZqfNbJmZrTKzkWaW6X94rMpm9r3/dT0z63GOfXOYWbt/8Bwvm1nXC21PsM8gM2t0Ec91vZnpDOgikuyUIIlc+o455251zt0CnAAej7vRPBf9t+ycG+ec63WOXXLgX8xZRCTSKEESSVnmAIX9ysl6MxsCrAKuMbMaZjbPzJb4laYsAGZ2j5mtM7MlwP1nHsjMWpjZR/7XV5nZd2a23L9VAHoBhfzqVW9/v25mFm1mK8zslTiP9ZyZ/WJmc4EbztcJM3vMf5zlZjY6QVWsmpkt8h/vXn//tGbWO85zt/lff5AiIueiBEkkhTCzKKAWsNJvKgJ84pwrBhwBngeqOedKAouALmZ2GdAfqAuUAvKc5eE/AH50zpUASgKrgR7Ar371qpuZ1fCfsyxwK1DKzCqZWSm8a5zdCtQGylxAd751zpXxn28t0CrOtuv956gDfOb3oRVwwDlXxn/8x8yswAU8j4jIPxIVdAAicl4ZzezMhZTnAAOAvMBvzrn5fvvtwM3AT951SEkPzANuBDY75zYAmNlXQOsknqMK0Axir3h+wMxyJtinhn9b6n+fBS9hygp855w76j/HuAvo0y1m9hreMF4WYHKcbSOcczHABjPb5PehBlA8zvyk7P5z/3IBzyUictGUIIlc+o45526N2+AnQUfiNgFTnXMPJdgv3v3+Rwa84Zzrm+A5nvwHjzUIqO+cW25mLYDKcbYlvECk85+7g3MubiJ15sKzIiLJTkNsIqnDfOAOMysMYGaZzawosA643swK+fs9dJb7Twfa+vdNa2bZgUN41aEzJgP/iTO3KZ+ZXQnMBuqbWUYzy4o3nHc+WYGdZpYO+HeCbY3NLI0fc0Fgvf/cbf39MbOiZpb5Ap5HROQfUQVJJBVwzv3pV2KGmlkGv/l559wvZtYa+MHMjuIN0WVN4iE6Af3MrBVwGmjrnJtnZj/5y+gn+vOQbgLm+RWsw8AjzrklZjYcWA7sBqIvIOQXgAXAn/7/cWPaCiwEsgGPO+f+NrPP8eYmLTHvyf8E6l/YT0dE5OKZcwmr2SIiIiKRTUNsIiIiIgkoQRIRERFJQAmSiIiISAJKkEREREQSUIIkIiIikoASJBEREZEElCCJiIiIJKAESURERCSB/wNEVQCgeou1VgAAAABJRU5ErkJggg==\n", 697 | "text/plain": [ 698 | "
" 699 | ] 700 | }, 701 | "metadata": {}, 702 | "output_type": "display_data" 703 | } 704 | ], 705 | "source": [ 706 | "# Plot confusion matrix \n", 707 | "from sklearn.metrics import confusion_matrix\n", 708 | "from support import print_confusion_matrix\n", 709 | "\n", 710 | "cnf_matrix = confusion_matrix(np.argmax(Y_pred,axis=1), np.argmax(Y_test,axis=1))\n", 711 | "_ = print_confusion_matrix(cnf_matrix, LABELS)" 712 | ] 713 | }, 714 | { 715 | "cell_type": "code", 716 | "execution_count": 14, 717 | "metadata": {}, 718 | "outputs": [ 719 | { 720 | "name": "stdout", 721 | "output_type": "stream", 722 | "text": [ 723 | " precision recall f1-score support\n", 724 | "\n", 725 | " en 0.97 0.97 0.97 49999\n", 726 | " fr 0.98 0.98 0.98 49917\n", 727 | " es 0.98 0.97 0.97 49889\n", 728 | " it 0.97 0.97 0.97 49977\n", 729 | " de 0.99 0.99 0.99 50111\n", 730 | " sk 0.97 0.98 0.98 50047\n", 731 | " cs 0.98 0.97 0.98 50060\n", 732 | "\n", 733 | "avg / total 0.98 0.98 0.98 350000\n", 734 | "\n" 735 | ] 736 | } 737 | ], 738 | "source": [ 739 | "# Classification Report\n", 740 | "print(classification_report(Y_test, Y_pred, target_names=LABELS))" 741 | ] 742 | }, 743 | { 744 | "cell_type": "markdown", 745 | "metadata": {}, 746 | "source": [ 747 | "## It's your turn\n", 748 | "\n", 749 | "Test out the model you just trained. Run the code Cell below and type your text in the widget, Have fun!🎉\n", 750 | "\n", 751 | "Here are some inspirations:\n", 752 | "\n", 753 | "##### EN - Frank Baum, The Wonderful Wizard of Oz, Project Gutenberg, public domain\n", 754 | "You are welcome, most noble Sorceress, to the land of the Munchkins. We are so grateful to you for having killed the Wicked Witch of the East, and for setting our people free from bondage.\n", 755 | "\n", 756 | "##### DE - Johann Wolfgang von Goethe, Faust: Der Tragödie erster Teil, Project Gutenberg, public domain\n", 757 | "Habe nun, ach! Philosophie, Juristerei und Medizin, Und leider auch Theologie Durchaus studiert, mit heißem Bemühn. Da steh ich nun, ich armer Tor! Und bin so klug als wie zuvor.\n", 758 | "\n", 759 | "##### FR - Pierre Benoît, L'Atlantide, \n", 760 | "Voilà cinq mois que j'en faisais fonction, et, ma foi, je supportais bien cette responsabilité et goûtais fort cette indépendance. Je puis même affirmer, sans me flatter.\n", 761 | "\n", 762 | "##### IT - Alberto Boccardi, Il peccato di Loreta, Project Gutenberg, public domain\n", 763 | "Giovanni Sant'Angelo, che negli anni passati a Padova in mezzo alla baraonda tanto gioconda degli studenti, aveva appreso ad amare con foga di giovane qualche alto ideale, tornato in famiglia dovette fare uno sforzo.\n", 764 | "\n", 765 | "##### ES - Fernando Callejo Ferrer, Música y Músicos Portorriqueños, Project Gutenberg, public domain\n", 766 | "Dedicada esta sección a la reseña de los compositores nativos y obras que han producido, con ligeros comentarios propios a cada uno, parécenos oportuno dar ligeras noticias sobre el origen de la composición.\n", 767 | "\n", 768 | "##### CS - František Omelka, Blesky nad Beskydami, Project Gutenberg, public domain\n", 769 | "A Slávek, jsa povzbuzen, se ptal a otec odpovídal. Přestože byl prostým venkovským listonošem, nepřivedla jej žádná synova otázka do rozpaků. Od mládí se zajímal o dějepis a literaturu.\n", 770 | "\n", 771 | "##### SK - Janko Matúška, Nad Tatrou sa blýska, national anthem of Slovakia, https://en.wikipedia.org/wiki/Nad_Tatrou_sa_blýska\n", 772 | "Nad Tatrou sa blýska Hromy divo bijú Zastavme ich, bratia Veď sa ony stratia Slováci ožijú To Slovensko naše Posiaľ tvrdo spalo Ale blesky hromu Vzbudzujú ho k tomu Aby sa prebralo.\n", 773 | "\n", 774 | "Can you do better? Play around with the model hyperparameters!" 775 | ] 776 | }, 777 | { 778 | "cell_type": "code", 779 | "execution_count": 15, 780 | "metadata": {}, 781 | "outputs": [ 782 | { 783 | "data": { 784 | "application/vnd.jupyter.widget-view+json": { 785 | "model_id": "51a58e6a7b0f497aa907a3df96d293fb", 786 | "version_major": 2, 787 | "version_minor": 0 788 | }, 789 | "text/plain": [ 790 | "interactive(children=(Textarea(value='', description='TEXT', placeholder='Type the text to identify here'), Bu…" 791 | ] 792 | }, 793 | "metadata": {}, 794 | "output_type": "display_data" 795 | } 796 | ], 797 | "source": [ 798 | "# and now we will have some fun. Seeing is believing!\n", 799 | "# We will take some texts and try to predict the text's language using our trained neural network.\n", 800 | "\n", 801 | "from ipywidgets import interact_manual\n", 802 | "from ipywidgets import widgets\n", 803 | "from support import clean_text\n", 804 | "\n", 805 | "\n", 806 | "def get_prediction(TEXT):\n", 807 | " if len(TEXT) < MAX_LEN:\n", 808 | " print(\"Text has to be at least {} chars long, but it is {}/{}\".format(MAX_LEN, len(TEXT), MAX_LEN))\n", 809 | " return(-1)\n", 810 | " # Data cleaning\n", 811 | " cleaned_text = clean_text(TEXT)\n", 812 | " \n", 813 | " # Get the MAX_LEN char\n", 814 | " input_row = get_input_row(cleaned_text, 0, MAX_LEN, alphabet)\n", 815 | " \n", 816 | " # Data preprocessing (Standardization)\n", 817 | " test_array = standard_scaler.transform([input_row])\n", 818 | " \n", 819 | " raw_score = model.predict(test_array)\n", 820 | " pred_idx= np.argmax(raw_score, axis=1)[0]\n", 821 | " score = raw_score[0][pred_idx]*100\n", 822 | " \n", 823 | " # Prediction\n", 824 | " prediction = LABELS[model.predict_classes(test_array)[0]]\n", 825 | " print('TEXT:', TEXT, '\\nPREDICTION:', prediction.upper(), '\\nSCORE:', score)\n", 826 | "\n", 827 | "interact_manual(get_prediction, TEXT=widgets.Textarea(placeholder='Type the text to identify here'));" 828 | ] 829 | }, 830 | { 831 | "cell_type": "markdown", 832 | "metadata": {}, 833 | "source": [ 834 | "## Save your model" 835 | ] 836 | }, 837 | { 838 | "cell_type": "code", 839 | "execution_count": 16, 840 | "metadata": {}, 841 | "outputs": [], 842 | "source": [ 843 | "# Saving Model Weight\n", 844 | "model.save_weights('models/lang_identification_weights.h5')" 845 | ] 846 | }, 847 | { 848 | "cell_type": "markdown", 849 | "metadata": {}, 850 | "source": [ 851 | "##### That's all folks - don't forget to shutdown your workspace once you're done 🙂" 852 | ] 853 | } 854 | ], 855 | "metadata": { 856 | "kernelspec": { 857 | "display_name": "Python 2", 858 | "language": "python", 859 | "name": "python2" 860 | }, 861 | "language_info": { 862 | "codemirror_mode": { 863 | "name": "ipython", 864 | "version": 2 865 | }, 866 | "file_extension": ".py", 867 | "mimetype": "text/x-python", 868 | "name": "python", 869 | "nbconvert_exporter": "python", 870 | "pygments_lexer": "ipython2", 871 | "version": "2.7.10" 872 | } 873 | }, 874 | "nbformat": 4, 875 | "nbformat_minor": 2 876 | } 877 | --------------------------------------------------------------------------------