├── LICENSE
├── README.md
├── char_rnns
├── README.md
├── dino_names.txt
├── notebooks
│ ├── Dino_Names_GRU.ipynb
│ ├── Dino_Names_LSTM.ipynb
│ ├── Dino_Names_RNN.ipynb
│ └── Dino_Names_Scratch.ipynb
└── plots
│ ├── Dino_Names_GRU.jpeg
│ ├── Dino_Names_LSTM.jpeg
│ ├── Dino_Names_RNN.jpeg
│ └── Dino_Names_Scratch.jpeg
├── neural_machine_translation
├── README.md
├── notebooks
│ ├── Attention_Is_All_You_Need.ipynb
│ ├── Conv_Seq2Seq.ipynb
│ ├── Seq2Seq.ipynb
│ └── Seq2Seq_with_Attention.ipynb
└── plots
│ ├── Conv_Seq2Seq.jpeg
│ ├── Seq2Seq.jpeg
│ ├── Seq2Seq_with_Attention.jpeg
│ └── Transformer.jpeg
├── text_classification
├── README.md
├── notebooks
│ ├── BERT.ipynb
│ ├── CNN.ipynb
│ ├── FastText.ipynb
│ └── LSTM.ipynb
└── plots
│ ├── BERT.png
│ ├── CNN.png
│ ├── FastText.png
│ └── LSTM.png
└── word_rnn
├── Book 1 - The Philosopher's Stone.txt
├── Main_Paragraph_generation_on_larger_dataset_lstm (1).ipynb
├── README.md
├── ezgif.com-gif-maker.gif
├── paragraph generation loss gru.png
├── paragraph generation loss lstm.png
├── paragraph generation loss rnn.png
├── wordRNN_paragraph_gru2.pth
├── wordRNN_paragraph_lstm2.pth
└── wordRNN_paragraph_rnn2.pth
/LICENSE:
--------------------------------------------------------------------------------
1 | MIT License
2 |
3 | Copyright (c) 2020 IvLabs
4 |
5 | Permission is hereby granted, free of charge, to any person obtaining a copy
6 | of this software and associated documentation files (the "Software"), to deal
7 | in the Software without restriction, including without limitation the rights
8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 |
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 |
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Natural Language Processing
2 | This is the Natural Language Processing repository of IvLabs and contains implementation of various architectures, starting from Character Level RNN(s) built from scratch, up to and including the Transformer architecture.
3 | Further, we have also included a rough roadmap for enthusiasts with basic knowledge of Machine/Deep Learning.
4 |
5 | We have implemented and compared the following architectures:
6 | - [x] [Character Level RNN (Char RNN)](char_rnns)
7 | - [x] From scratch
8 | - [x] Vanilla RNN
9 | - [x] LSTM
10 | - [x] GRU
11 |
12 | - [x] [Language Models (Word RNN)](word_rnn)
13 | - [x] Vanilla RNN
14 | - [x] LSTM
15 | - [x] GRU
16 |
17 | - [x] [Neural Machine Translation](neural_machine_translation)\
18 | For Neural Machine Translation, we have implemented the following papers.
19 | - [x] [Sequence to Sequence Learning with Neural Networks](https://arxiv.org/abs/1409.3215)
20 | - [x] [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473)
21 | - [x] [Convolutional Sequence to Sequence Learning](https://arxiv.org/abs/1705.03122)
22 | - [x] [Attention Is All You Need](https://arxiv.org/abs/1706.03762)
23 |
24 | -4 |
14 | | Hidden Dimensions | 64 |
15 | | Loss Function | Cross Entropy Loss |
16 | | Optimizer | AdamW |
17 |
18 |
60 |
65 |
66 | Note: ** A few changes have been made in order to improve performance.
24 | 1. Unlike the paper, reversing the input sequences resulted in a lower BLEU Score. Hence, the input sequences have not been reversed.
25 | 2. Further an additional parameter called ```teacher_forcing_ratio```, which is the probability of using the ground truth tokens as inputs while decoding has been introduced. It is usually set to 1 while training and 0 while sampling. However, setting it to 0.5 while training resulted in a better BLEU Score than setting it to 1.
26 |
27 | ### 2. [Neural Machine Translation by Jointly Learning to Align and Translate](https://github.com/IvLabs/Natural-Language-Processing/blob/master/neural_machine_translation/notebooks/Seq2Seq_with_Attention.ipynb) [](https://colab.research.google.com/drive/1hOd2JFafWgOvdbeXWoSm1gIiMEQ64KbM?usp=sharing)
28 | This paper presents a remarkable improvement in the Sequence to Sequence architecture by introducing a (soft)alignment metric called "attention". This metric induces a sense of similarity between tokens of the source and decoded sentences, which increases the BLEU score by almost 1.5 times and is much more robust with regards to the length of source and target sentences.\
29 | **Note: ** In this implementation, setting the ```teacher_forcing_ratio``` to 1 resulted in a better BLEU score (even with higher validation and test perplexities) than setting it to 0.5.
30 |
31 | ### 3. [Convolutional Sequence to Sequence Learning](https://github.com/IvLabs/Natural-Language-Processing/blob/master/neural_machine_translation/notebooks/Conv_Seq2Seq.ipynb) [](https://colab.research.google.com/drive/18uxa0ZMlck4f5fzTUdkze_cuKz-Hgx_o?usp=sharing)
32 | Unlike the previous papers, this paper is the first paper to tackle language modelling without using Recurrent Neural Networks. However, it does provide an attention mechanism and outperforms the sequential attention based architecture while having almost similar training times. The validation and test perplexities are quite low and the BLEU score is much better than that of the previous paper.
33 |
34 | ### 4. [Attention Is All You Need](https://github.com/IvLabs/Natural-Language-Processing/blob/master/neural_machine_translation/notebooks/Attention_Is_All_You_Need.ipynb) [](https://colab.research.google.com/drive/1RlDhclIlJWzcFC0iPqwbiFEuiCbYIJBG?usp=sharing)
35 | This paper can be said to have revolutionized almost all of Natural Language Processing, all owing to its tremendous simplicity and efficiency which makes it the backbone of almost all present State Of The Art architectures. The proposed *Transformer* architecture introduces a novel metric called *Self-Attention* which induces a sense of (soft)alignment amongst the source sentence tokens too. Additionally architecture uses simple feed-forward layers which makes it almost 4 times lighter than the Convolutional Sequence to Sequence architecture, while attaining the highest BLEU score amongst all the above mentioned architectures.
36 |
37 | ## Summary
38 | Below is a table, summarising the number of parameters and the BLEU scores achieved by each architecture.
39 |
40 | | Architecture | No. of Trainable Parameters | BLEU Score |
41 | | ----------------------------------- |:---------------------------:|:----------:|
42 | | Sequence to Sequence | 13,899,013 | 18.94 |
43 | | Sequence to Sequence with Attention | 20,518,917 | 31.24 |
44 | | Convolutional Sequence to Sequence | 37,351,685 | 36.53 |
45 | | Attention Is All You Need | 9,038,853 | 37.50 |
46 |
47 | **Note:**
48 | 1. The above BLEU scores may vary slightly upon training the models (even with fixed SEED).
49 | 2. The research paper notes for the above mentioned papers can be found [here](https://github.com/IvLabs/ResearchPaperNotes/tree/master/natural_language_processing).
50 |
51 |
52 | ### Reference(s):
53 | * [PyTorch Seq2Seq by Ben Trevett](https://github.com/bentrevett/pytorch-seq2seq)
54 |
--------------------------------------------------------------------------------
/neural_machine_translation/plots/Conv_Seq2Seq.jpeg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IvLabs/Natural-Language-Processing/dcfd38d00a8d5137b122c778a67445d7e95cf53e/neural_machine_translation/plots/Conv_Seq2Seq.jpeg
--------------------------------------------------------------------------------
/neural_machine_translation/plots/Seq2Seq.jpeg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IvLabs/Natural-Language-Processing/dcfd38d00a8d5137b122c778a67445d7e95cf53e/neural_machine_translation/plots/Seq2Seq.jpeg
--------------------------------------------------------------------------------
/neural_machine_translation/plots/Seq2Seq_with_Attention.jpeg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IvLabs/Natural-Language-Processing/dcfd38d00a8d5137b122c778a67445d7e95cf53e/neural_machine_translation/plots/Seq2Seq_with_Attention.jpeg
--------------------------------------------------------------------------------
/neural_machine_translation/plots/Transformer.jpeg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IvLabs/Natural-Language-Processing/dcfd38d00a8d5137b122c778a67445d7e95cf53e/neural_machine_translation/plots/Transformer.jpeg
--------------------------------------------------------------------------------
/text_classification/README.md:
--------------------------------------------------------------------------------
1 | # Sentiment Analysis and Text Classification
2 |
3 | With the advent rise of digitization, the need to automate the analysis of the predominant method of interacting with the digital world, i.e. text, has exponentially risen for a variety of reasons and purposes.
4 |
5 | Text classification is heavily used in tasks such as spam detection, news categorization, customer query tagging, and of course, sentiment analysis.
6 |
7 | Sentiment analysis is the automated process of determining the polarity of a topic, given an input sentence.
8 | It is used in tasks such as analyzing a product/company's public sentiment/word of mouth, customer feedback analysis, political analysis, etc.
9 |
10 | We experiment with 4 architectures on two datasets. The IMDb dataset was to used to train the model for predicting the polarity of movie reviews. For text classification, the TREC dataset was used to train the model to predict one of six classes - `['ENTY', 'HUM', 'DESC', 'NUM', 'LOC', 'ABBR']`.
11 |
12 |
13 | Below is a table addressing some common data and optimization related parameters.
14 |
15 | | Parameter | Value |
16 | | -------------- |:------------------:|
17 | | Training Set | 17500 |
18 | | Testing Set | 25000 |
19 | | Validation Set | 7500 |
20 | | Loss Function | Cross Entropy Loss |
21 | | Optimizer | Adam |
22 |
23 | ## Architectures
24 |
25 | ### [1. Long Short-Term Memory Networks](https://github.com/IvLabs/Natural-Language-Processing/blob/master/text_classification/notebooks/LSTM.ipynb) [](https://colab.research.google.com/github/IvLabs/Natural-Language-Processing/blob/master/text_classification/notebooks/LSTM.ipynb)
26 | RNNs were one of the first architectures for capturing sequential data on various tasks such as predicting parts of speech, sentiment analysis, text classification, etc. However, RNNs suffers from a vanishing gradient problem that make them unable to propagate useful gradient information, thereby causing the model not to learn from larger contexts. To overcome this problem, LSTMs were introduced, which are a modified form of the RNN.
27 |
28 | ### [2. Bag of Tricks for Efficient Text Classification](https://github.com/IvLabs/Natural-Language-Processing/blob/master/text_classification/notebooks/FastText.ipynb) [](https://colab.research.google.com/github/IvLabs/Natural-Language-Processing/blob/master/text_classification/notebooks/FastText.ipynb)
29 | This paper shows that tweaking baseline linear classifiers enables us to learn an efficient and accurate text classifier for a very large corpus with a large output space (whereas deep networks would be very slow). Sentences being represented by BoW and trained by improved linear models (like logistic regression or SVM) with a rank constraint and a fast loss approximation are shown to achieve performance on par with the state-of-the-art.
30 |
31 | ### [3. Convolutional Neural Networks for Sentence Classification](https://github.com/IvLabs/Natural-Language-Processing/blob/master/text_classification/notebooks/CNN.ipynb) [](https://colab.research.google.com/github/IvLabs/Natural-Language-Processing/blob/master/text_classification/notebooks/CNN.ipynb)
32 | CNNs trained on top of pre-trained word vectors for sentence-level classification tasks are shown to achieve excellent results on multiple benchmarks.
33 | The underlying idea strengthened here is that feature extractors obtained from a pre-
34 | trained deep learning model perform well on a variety of tasks—including tasks that are very different from the original task for which the feature extractors were trained.
35 |
36 | ### [4. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://github.com/IvLabs/Natural-Language-Processing/blob/master/text_classification/notebooks/BERT.ipynb) [](https://colab.research.google.com/github/IvLabs/Natural-Language-Processing/blob/master/text_classification/notebooks/BERT.ipynb)
37 | BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, without substantial task-specific architecture modifications.
38 |
39 | ## Summary
40 | Below is a table, summarising the number of parameters and the train, test, validation accuracies.
41 |
42 | | Architecture | No. of Learnable Parameters | Train Acc. (%) | Valid. Acc. (%) | Test Acc. (%) |
43 | | ------------ | --------------------------- | -------------- | --------------- | ------------- |
44 | | LSTM | 4,810,857 | 89.10 | 88.37 | 87.62 |
45 | | FastText | 2,500,301 | 88.35 | 86.67 | 86.28 |
46 | | CNN | 843,906 | 94.95 | 79.89 | 84.90 |
47 | | BERT | 110,468,609 | 91.04 | 91.04 | 91.44 |
48 |
49 | ## Plots
50 |
51 |
56 |
57 | ## Reference(s)
58 | * [PyTorch Sentiment Analysis by Ben Trevett](https://github.com/bentrevett/)
59 |
60 |
--------------------------------------------------------------------------------
/text_classification/notebooks/CNN.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {
6 | "id": "-B_MqscXWGI4"
7 | },
8 | "source": [
9 | "# Installing Packages"
10 | ]
11 | },
12 | {
13 | "cell_type": "code",
14 | "execution_count": null,
15 | "metadata": {},
16 | "outputs": [],
17 | "source": [
18 | "!pip install -U torchtext==0.6.0"
19 | ]
20 | },
21 | {
22 | "cell_type": "markdown",
23 | "metadata": {
24 | "id": "bbMNpP192mb2"
25 | },
26 | "source": [
27 | "# Importing Required Libraries"
28 | ]
29 | },
30 | {
31 | "cell_type": "code",
32 | "execution_count": 1,
33 | "metadata": {
34 | "id": "slw2Y5t3taWQ"
35 | },
36 | "outputs": [],
37 | "source": [
38 | "import torch\n",
39 | "from torchtext import data, datasets\n",
40 | "import torch.nn as nn\n",
41 | "import torch.optim as optim\n",
42 | "from torchtext.data import Field, LabelField, BucketIterator\n",
43 | "import torch.nn.functional as F\n",
44 | "\n",
45 | "import random\n",
46 | "\n",
47 | "import matplotlib.pyplot as plt"
48 | ]
49 | },
50 | {
51 | "cell_type": "code",
52 | "execution_count": 2,
53 | "metadata": {
54 | "colab": {
55 | "base_uri": "https://localhost:8080/"
56 | },
57 | "id": "7kAMYpKr2tJV",
58 | "outputId": "3ecc3577-55f7-4711-b775-7f5471892d0d"
59 | },
60 | "outputs": [
61 | {
62 | "name": "stdout",
63 | "output_type": "stream",
64 | "text": [
65 | "Notebook is running on cuda\n"
66 | ]
67 | }
68 | ],
69 | "source": [
70 | "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
71 | "print(\"Notebook is running on\", device)"
72 | ]
73 | },
74 | {
75 | "cell_type": "markdown",
76 | "metadata": {
77 | "id": "8ZOcRV2w2vpH"
78 | },
79 | "source": [
80 | "Fixing SEED for reproducibility of results"
81 | ]
82 | },
83 | {
84 | "cell_type": "code",
85 | "execution_count": 3,
86 | "metadata": {
87 | "id": "N99mW-8XtnWr"
88 | },
89 | "outputs": [],
90 | "source": [
91 | "SEED = 4444\n",
92 | "\n",
93 | "random.seed(SEED)\n",
94 | "torch.manual_seed(SEED)\n",
95 | "torch.cuda.manual_seed(SEED)\n",
96 | "torch.backends.cudnn.deterministic = True"
97 | ]
98 | },
99 | {
100 | "cell_type": "code",
101 | "execution_count": 4,
102 | "metadata": {
103 | "id": "O5NaQke7touV"
104 | },
105 | "outputs": [],
106 | "source": [
107 | "FIELD = Field(tokenize = 'spacy', tokenizer_language = 'en_core_web_sm')\n",
108 | "\n",
109 | "LABEL = LabelField()"
110 | ]
111 | },
112 | {
113 | "cell_type": "markdown",
114 | "metadata": {
115 | "id": "RWXCd0V73YK7"
116 | },
117 | "source": [
118 | "# Splitting the data"
119 | ]
120 | },
121 | {
122 | "cell_type": "code",
123 | "execution_count": 5,
124 | "metadata": {
125 | "colab": {
126 | "base_uri": "https://localhost:8080/"
127 | },
128 | "id": "79ih2w2sttNq",
129 | "outputId": "c4358b46-cc35-4382-a74c-bdc0118e067b"
130 | },
131 | "outputs": [],
132 | "source": [
133 | "train_data, test_data = datasets.TREC.splits(FIELD, LABEL, fine_grained=False)"
134 | ]
135 | },
136 | {
137 | "cell_type": "code",
138 | "execution_count": 6,
139 | "metadata": {
140 | "id": "nxpdowpRtwNQ"
141 | },
142 | "outputs": [],
143 | "source": [
144 | "train_data, valid_data = train_data.split(random_state = random.seed(SEED))"
145 | ]
146 | },
147 | {
148 | "cell_type": "code",
149 | "execution_count": 7,
150 | "metadata": {
151 | "colab": {
152 | "base_uri": "https://localhost:8080/"
153 | },
154 | "id": "BSWt_wDwtzLt",
155 | "outputId": "548f8358-857e-454d-c80a-f8e320606108"
156 | },
157 | "outputs": [
158 | {
159 | "name": "stdout",
160 | "output_type": "stream",
161 | "text": [
162 | "Number of training examples: 3816\n",
163 | "Number of validation examples: 1636\n",
164 | "Number of testing examples: 500\n"
165 | ]
166 | }
167 | ],
168 | "source": [
169 | "print(f'Number of training examples: {len(train_data)}')\n",
170 | "print(f'Number of validation examples: {len(valid_data)}')\n",
171 | "print(f'Number of testing examples: {len(test_data)}')"
172 | ]
173 | },
174 | {
175 | "cell_type": "code",
176 | "execution_count": 8,
177 | "metadata": {
178 | "colab": {
179 | "base_uri": "https://localhost:8080/"
180 | },
181 | "id": "trFz-SmNt0fO",
182 | "outputId": "4d2dcf36-d394-40f5-c1b1-42bb09e00c6b"
183 | },
184 | "outputs": [],
185 | "source": [
186 | "MAX_VOCAB_SIZE = 25000 # excluding and token\n",
187 | "FIELD.build_vocab(train_data, max_size = MAX_VOCAB_SIZE, vectors=\"glove.6B.100d\", unk_init = torch.Tensor.normal_)\n",
188 | "LABEL.build_vocab(train_data)"
189 | ]
190 | },
191 | {
192 | "cell_type": "code",
193 | "execution_count": 9,
194 | "metadata": {
195 | "colab": {
196 | "base_uri": "https://localhost:8080/"
197 | },
198 | "id": "PQUUTTgat5hg",
199 | "outputId": "3703b174-a45b-4bb2-b5a7-65c7d123899e"
200 | },
201 | "outputs": [
202 | {
203 | "name": "stdout",
204 | "output_type": "stream",
205 | "text": [
206 | "Unique tokens in FIELD vocabulary: 7518\n",
207 | "Unique tokens in LABEL vocabulary: 6\n"
208 | ]
209 | }
210 | ],
211 | "source": [
212 | "print(f\"Unique tokens in FIELD vocabulary: {len(FIELD.vocab)}\")\n",
213 | "print(f\"Unique tokens in LABEL vocabulary: {len(LABEL.vocab)}\")"
214 | ]
215 | },
216 | {
217 | "cell_type": "markdown",
218 | "metadata": {
219 | "id": "BIZMVoIA4S2K"
220 | },
221 | "source": [
222 | "# Model Definition"
223 | ]
224 | },
225 | {
226 | "cell_type": "code",
227 | "execution_count": 10,
228 | "metadata": {
229 | "id": "egyx2GtzuEdP"
230 | },
231 | "outputs": [],
232 | "source": [
233 | "class CNN(nn.Module):\n",
234 | " def __init__(self, vocab_size, emb_dim, num_filters, filter_sizes, output_dim, pad_idx):\n",
235 | " super().__init__()\n",
236 | " self.embedding = nn.Embedding(vocab_size, emb_dim)\n",
237 | " self.convs = nn.ModuleList([nn.Conv2d(in_channels = 1, \n",
238 | " out_channels = num_filters, \n",
239 | " kernel_size = (fs, emb_dim)) \n",
240 | " for fs in filter_sizes])\n",
241 | " self.fc = nn.Linear(len(filter_sizes) * num_filters, output_dim)\n",
242 | " self.dropout = nn.Dropout(0.5)\n",
243 | " \n",
244 | " def forward(self, text): # [input] = [seq_len, batch_size]\n",
245 | " text = text.permute(1, 0) # [input] = [batch_size, seq_len]\n",
246 | " embedded = self.embedding(text) # [embedded] = [batch_size, seq_len, emb_dim]\n",
247 | " embedded = embedded.unsqueeze(1) # [embedded] = [batch_size, 1, seq_len, emb_dim]\n",
248 | " conved = [F.relu(conv(embedded)).squeeze(3) for conv in self.convs] # [conv] = [batch_size, num_filters, seq_len - filter_sizes[n]]\n",
249 | " pooled = [F.max_pool1d(conv, conv.shape[2]).squeeze(2) for conv in conved] # [pooled] = [batch_size, num_filters]\n",
250 | " output = self.dropout(torch.cat(pooled, dim = 1)) # [output] = [batch_size, num_filters * len(filter_sizes)]\n",
251 | " return self.fc(output)"
252 | ]
253 | },
254 | {
255 | "cell_type": "code",
256 | "execution_count": 11,
257 | "metadata": {
258 | "id": "2PDdgoXiufHr"
259 | },
260 | "outputs": [],
261 | "source": [
262 | "def batch_accuracy(preds, y):\n",
263 | " top_pred = preds.argmax(1, keepdim = True)\n",
264 | " correct = top_pred.eq(y.view_as(top_pred)).sum()\n",
265 | " acc = correct.float() / y.shape[0]\n",
266 | " return acc"
267 | ]
268 | },
269 | {
270 | "cell_type": "markdown",
271 | "metadata": {
272 | "id": "OpnqXXA1uvuD"
273 | },
274 | "source": [
275 | "# Training and Evaluation Functions"
276 | ]
277 | },
278 | {
279 | "cell_type": "code",
280 | "execution_count": 12,
281 | "metadata": {
282 | "id": "gKSFO3hRutOc"
283 | },
284 | "outputs": [],
285 | "source": [
286 | "def Train(model, iterator, optimizer, criterion): \n",
287 | " epoch_loss = 0\n",
288 | " epoch_acc = 0\n",
289 | " model.train()\n",
290 | " for batch in iterator:\n",
291 | " optimizer.zero_grad()\n",
292 | " inp = batch.text\n",
293 | " label = batch.label \n",
294 | " predictions = model(inp)\n",
295 | " loss = criterion(predictions, label)\n",
296 | " acc = batch_accuracy(predictions, label)\n",
297 | " loss.backward()\n",
298 | " optimizer.step()\n",
299 | " epoch_loss += loss.item()\n",
300 | " epoch_acc += acc.item()\n",
301 | " return epoch_loss / len(iterator), epoch_acc / len(iterator)"
302 | ]
303 | },
304 | {
305 | "cell_type": "code",
306 | "execution_count": 13,
307 | "metadata": {
308 | "id": "YqTp93Aruwoe"
309 | },
310 | "outputs": [],
311 | "source": [
312 | "def Evaluate(model, iterator, criterion):\n",
313 | " epoch_loss = 0\n",
314 | " epoch_acc = 0\n",
315 | " model.eval()\n",
316 | " with torch.no_grad():\n",
317 | " for batch in iterator:\n",
318 | " inp = batch.text\n",
319 | " label = batch.label \n",
320 | " predictions = model(inp).squeeze(1)\n",
321 | " loss = criterion(predictions, label)\n",
322 | " acc = batch_accuracy(predictions, label)\n",
323 | " epoch_loss += loss.item()\n",
324 | " epoch_acc += acc.item()\n",
325 | " return epoch_loss / len(iterator), epoch_acc / len(iterator)"
326 | ]
327 | },
328 | {
329 | "cell_type": "markdown",
330 | "metadata": {
331 | "id": "v1_3yX3-42vO"
332 | },
333 | "source": [
334 | "# Data Iterators, Hyperparameters and Model Initialization"
335 | ]
336 | },
337 | {
338 | "cell_type": "code",
339 | "execution_count": 14,
340 | "metadata": {
341 | "id": "yIHX-jbs5ClS"
342 | },
343 | "outputs": [],
344 | "source": [
345 | "BATCH_SIZE = 64\n",
346 | "\n",
347 | "train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits((train_data, valid_data, test_data), batch_size = BATCH_SIZE, device = device)"
348 | ]
349 | },
350 | {
351 | "cell_type": "code",
352 | "execution_count": 15,
353 | "metadata": {
354 | "id": "o6c7ld9wuYh4"
355 | },
356 | "outputs": [],
357 | "source": [
358 | "VOCAB_SIZE = len(FIELD.vocab) # dimension of one-hot vector / vocabulary\n",
359 | "EMB_DIM = 100 # dimensions of word embeddings\n",
360 | "NUM_FILTERS = 100 # number of filters in CNN\n",
361 | "FILTER_SIZES = [2,3,4] # filter sizes of CNN\n",
362 | "OUTPUT_DIM = len(LABEL.vocab) # dimensions of output\n",
363 | "DROPOUT = 0.5 \n",
364 | "\n",
365 | "NUM_EPOCHS = 10\n",
366 | "LR = 0.001"
367 | ]
368 | },
369 | {
370 | "cell_type": "code",
371 | "execution_count": 16,
372 | "metadata": {
373 | "id": "j5U3vbZcwEgr"
374 | },
375 | "outputs": [],
376 | "source": [
377 | "model = CNN(VOCAB_SIZE, EMB_DIM, NUM_FILTERS, FILTER_SIZES, OUTPUT_DIM, FIELD.vocab.stoi[FIELD.pad_token])"
378 | ]
379 | },
380 | {
381 | "cell_type": "code",
382 | "execution_count": 17,
383 | "metadata": {
384 | "id": "ziglTF1D5voa"
385 | },
386 | "outputs": [],
387 | "source": [
388 | "optimizer = optim.Adam(model.parameters(), lr=LR)\n",
389 | "\n",
390 | "criterion = nn.CrossEntropyLoss()\n",
391 | "\n",
392 | "model = model.to(device)\n",
393 | "criterion = criterion.to(device)"
394 | ]
395 | },
396 | {
397 | "cell_type": "code",
398 | "execution_count": 18,
399 | "metadata": {
400 | "colab": {
401 | "base_uri": "https://localhost:8080/"
402 | },
403 | "id": "X7Mm3BnuuZ2F",
404 | "outputId": "e195ceb6-9a27-4359-8aae-c8777a5f7b1a"
405 | },
406 | "outputs": [
407 | {
408 | "name": "stdout",
409 | "output_type": "stream",
410 | "text": [
411 | "The model has 843,906 trainable parameters\n"
412 | ]
413 | }
414 | ],
415 | "source": [
416 | "def count_parameters(model):\n",
417 | " return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
418 | "\n",
419 | "print(f'The model has {count_parameters(model):,} trainable parameters')"
420 | ]
421 | },
422 | {
423 | "cell_type": "markdown",
424 | "metadata": {
425 | "id": "99qvK1qw5aAV"
426 | },
427 | "source": [
428 | "# Training"
429 | ]
430 | },
431 | {
432 | "cell_type": "code",
433 | "execution_count": 19,
434 | "metadata": {
435 | "id": "WT1J71IjuySn"
436 | },
437 | "outputs": [],
438 | "source": [
439 | "import time\n",
440 | "\n",
441 | "def Epoch_time(start_time, end_time):\n",
442 | " elapsed_time = end_time - start_time\n",
443 | " elapsed_mins = int(elapsed_time / 60)\n",
444 | " elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
445 | " return elapsed_mins, elapsed_secs"
446 | ]
447 | },
448 | {
449 | "cell_type": "code",
450 | "execution_count": 21,
451 | "metadata": {
452 | "colab": {
453 | "base_uri": "https://localhost:8080/"
454 | },
455 | "id": "TKDOAVH5uz2s",
456 | "outputId": "846b4443-f6d6-4519-fe6d-a70d4911f35d"
457 | },
458 | "outputs": [
459 | {
460 | "name": "stdout",
461 | "output_type": "stream",
462 | "text": [
463 | "Learning Rate: 0.001, Number of Filters: 100, Filter Sizes: [2, 3, 4]\n",
464 | "Time taken for epoch 1: 0m 17s\n",
465 | "Training Loss: 1.4261 | Validation Loss: 1.0630\n",
466 | "Training Accuracy: 43.17 %| Validation Accuracy: 62.21 %\n",
467 | "Time taken for epoch 2: 0m 14s\n",
468 | "Training Loss: 0.9428 | Validation Loss: 0.8337\n",
469 | "Training Accuracy: 64.54 %| Validation Accuracy: 70.10 %\n",
470 | "Time taken for epoch 3: 0m 14s\n",
471 | "Training Loss: 0.7243 | Validation Loss: 0.7207\n",
472 | "Training Accuracy: 73.87 %| Validation Accuracy: 74.49 %\n",
473 | "Time taken for epoch 4: 0m 14s\n",
474 | "Training Loss: 0.5866 | Validation Loss: 0.6712\n",
475 | "Training Accuracy: 79.45 %| Validation Accuracy: 74.23 %\n",
476 | "Time taken for epoch 5: 0m 15s\n",
477 | "Training Loss: 0.4986 | Validation Loss: 0.6235\n",
478 | "Training Accuracy: 83.14 %| Validation Accuracy: 76.77 %\n",
479 | "Time taken for epoch 6: 0m 14s\n",
480 | "Training Loss: 0.3997 | Validation Loss: 0.5953\n",
481 | "Training Accuracy: 87.04 %| Validation Accuracy: 77.98 %\n",
482 | "Time taken for epoch 7: 0m 14s\n",
483 | "Training Loss: 0.3328 | Validation Loss: 0.5735\n",
484 | "Training Accuracy: 89.69 %| Validation Accuracy: 78.20 %\n",
485 | "Time taken for epoch 8: 0m 14s\n",
486 | "Training Loss: 0.2581 | Validation Loss: 0.5676\n",
487 | "Training Accuracy: 92.07 %| Validation Accuracy: 79.17 %\n",
488 | "Time taken for epoch 9: 0m 15s\n",
489 | "Training Loss: 0.2212 | Validation Loss: 0.5526\n",
490 | "Training Accuracy: 93.41 %| Validation Accuracy: 79.92 %\n",
491 | "Time taken for epoch 10: 0m 14s\n",
492 | "Training Loss: 0.1761 | Validation Loss: 0.5503\n",
493 | "Training Accuracy: 94.95 %| Validation Accuracy: 79.89 %\n",
494 | "Model with Train Loss 0.1761, Validation Loss: 0.5503 was saved.\n"
495 | ]
496 | }
497 | ],
498 | "source": [
499 | "print(f\"Learning Rate: {LR}, Number of Filters: {NUM_FILTERS}, Filter Sizes: {FILTER_SIZES}\")\n",
500 | "train_losses = []\n",
501 | "valid_losses = []\n",
502 | "min_losses = [float('inf'), float('inf')]\n",
503 | "\n",
504 | "start_time = time.time()\n",
505 | "for epoch in range(1, NUM_EPOCHS+1):\n",
506 | " \n",
507 | " train_loss, train_acc = Train(model, train_iterator, optimizer, criterion)\n",
508 | " train_losses.append(train_loss)\n",
509 | " valid_loss, valid_acc = Evaluate(model, valid_iterator, criterion)\n",
510 | " valid_losses.append(valid_loss)\n",
511 | "\n",
512 | " if valid_loss < min_losses[0]:\n",
513 | " min_losses[0] = valid_loss\n",
514 | " min_losses[1] = train_loss\n",
515 | " torch.save(model.state_dict(), 'CNN.pt')\n",
516 | "\n",
517 | " elapsed_time = Epoch_time(start_time, time.time())\n",
518 | " print(f\"Time taken for epoch {epoch}: {elapsed_time[0]}m {elapsed_time[1]}s\")\n",
519 | " start_time = time.time()\n",
520 | " print(f\"Training Loss: {train_loss:.4f} | Validation Loss: {valid_loss:.4f}\")\n",
521 | " print(f\"Training Accuracy: {train_acc*100:.2f} %| Validation Accuracy: {valid_acc*100:.2f} %\")\n",
522 | "\n",
523 | "print(f\"Model with Train Loss {min_losses[1]:.4f}, Validation Loss: {min_losses[0]:.4f} was saved.\")"
524 | ]
525 | },
526 | {
527 | "cell_type": "code",
528 | "execution_count": 27,
529 | "metadata": {
530 | "id": "OkjBpvPt8BmJ"
531 | },
532 | "outputs": [
533 | {
534 | "data": {
535 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABFBElEQVR4nO3dd3xV9f348dc7gwSSEEYgEJJAQPZI2EOU4EAcgAOrCChubautvzr7tdU6vuq3trW2VauCCwWpq7hpVUBFtuwlI0CYYQQSICHj/fvjnIRLSMJNcm9uwn0/H4/7yL1nfM77niT3fT/jfI6oKsYYY4JXSKADMMYYE1iWCIwxJshZIjDGmCBnicAYY4KcJQJjjAlylgiMMSbIWSIIEiKSLiKZfiz/JRH5ncfrO0Vkj4jkikhz92d7Pxx3tYik+7pct+zbReQ5f5RdyTEjRGSdiLSozeP6gj9/F8a/LBH4gfuhV/IoFpFjHq/HV6M8rz7ERWSAiHwmItkickBEForIjdV7F1Wjqneo6uNuHOHAn4ERqhqtqvvdn5trcgwReV1Enihz3O6qOrsm5VZwrAbAw8AfPZeJyKMi8pOIHBGRDBGZIiLt3PWzRSRPRJI89rlARDI8XmeIyF4RifJYdouIzHbfTz4wBXiwCrE+KiJTq/1mfcRfvwsAEWktIpNFZJeI5LjJ8g+e59FUnyUCP3A/9KJVNRrYBozyWPa2P44pIoOBr4E5wFlAc+BO4GJ/HO804oFIYHUAju0rY4B1qrrDY9l7wGjgOiAWSAWWAOd7bHME+B2VCwV+Vcn6d4AbRCSiqkH7i4iEBfDYzYAfgIbAYFWNAS4EmgAdqlFewN5LnaWq9vDjA8gALnCfh+B809sE7AdmAM3cdS8C73vs9wzwFRAFHAOKgVz3kVDOcb4D/lFJHOlApsfrkjhygDXAFR7rzsJJKIeAfcC77nIB/gLsBQ4DK4Ee7rrXgSeATjgfhurG+rW7XoGz3OcNgT8BW91jfAc0dNf9C9jtLp8LdHeX3wYUAMfdcj8u5/xGAM8BO93Hc0CE5/sHfuPGvwu4sZLzNQV42OP1Be7vIamSfWYDj7jntIPHfhll/h4eBA4ATdxltwCzy5T1EzDMy7+xR4GpFawbBMwDsoHlQLrHuhuBtW68m4Hby/69AA+4v4+33OPMAN5091kN9Kvgb/102/YBfnTX/Qt4F3iigvfwBM7fWkgF69u5f19hZX4Xt7jPJwHf4/zt7geecs9HD4/tW7i/35bu68uAZe5284BeHts+AOxwY18PnB/oz5maPqxGULvuAi4HhgEJwEHgH+663wA9RWSSiJwD3AzcoKpHcL7V79QTtYqdnoWKSCNgMM43Vm9tAs7B+Wb7B2CqiLR21z0OzAKaAonA39zlI4BzcT7sY4Gf4fxjlVLVDUB392UTVT2vnGM/C/QFhgDNgPtxEh3A50BHoCWwFHjbLfdl9/n/uedgVDnl/g/OB18azrf1ATjNOyVauXG3wTm//xCRpuWdHKAnzj95iQuAhaq6vYLtS+wAXsE5pxVZjPNBdW8l26zFeQ+ISLLb3Jd8mmOfRETaAJ/ifJA2c4/3vkf/w16cD7zGOEnhLyLSx6OIVu5+bXESMTg1ouk438ZnAn+vJIRyt3Wb3T7E+fLQDJgGXFFJORcAH6hqcSXbnM5AnGQXDzwGfACM81j/M2COqu4Vkd44XwRux6lZ/xOY6fbfdAZ+CfRXp2ZyEU4CrNcsEdSuO4D/UdVMddqCHwXGikiYqh4FJuK0rU8F7lJVbzt3m+L8Lnd5G4iq/ktVd6pqsaq+i/MNdIC7ugDnnz9BVfNU9TuP5TFAF0BUda2qen1MABEJAW4CfqWqO1S1SFXnuecDVZ2iqjke5ydVRGK9LH488Jiq7lXVLJwP44ke6wvc9QWq+hlOzaJzBWU1wfnGV6I53p/fp4BRItK9km1+D9xVSadwjhsDqrpNVZuo6jYvj19iAvCZqn7m/p7/g5OELnHL/VRVN6ljDk7yP8dj/2LgEVXNV9Vj7rLv3PKKcGoJqZUcv6JtBwFhwPPu7+IDYGEl5VTl3Fdkp6r+TVUL3ffyDnCtx/rr3GXgJL1/quoC9+/zDSDfjbsIp+bZTUTCVTVDVTfVMLaAs0RQu9oCH7rf7rJxvvUV4XxLQVUX4HxrEZxqtbcO4vzTtj7dhiVE5HoRWeYRSw8gzl19vxvDQnckyE1ufF/jfKv7B7BXRF4WkcZViBP3GJE4NZKyMYWKyNMisklEDnPim1Zc2W0rkIDT3FRiq7usxH5VLfR4fRSIrqCsgzhJr3RfvDy/bhL6O843z4q2WQV8QsWdwjE4zRI10Ra4uuR37P6eh+K+DxG5WETmuwMLsnEShOe5zlLVvDJl7vZ4fhSIrKTNvaJtE4Ad6razuCqraXl97itRtvxvgEYiMtDt7E/DqaWAc95+U+a8JeF8MdoI/BrnS8peEZkuIgnUc5YIatd24GL3213JI1LdDkkR+QXOt42dOB/GJSqdItatTfwAXOVNECLSFqf54pdAc1VtAqzC+fBHVXer6q2qmoBTPX5BRM5y1z2vqn2BbjhNRPd599ZL7QPyKL+T7zqcTtoLcJpw2pWE7P483VS5O3H+iUsku8uqYwXO+yvxX2CAiCR6uf8fgeE4TWAVeQS4FaepqqyuOG36NbEdeKvM31uUqj7tdkS/j9NMF+/+DXzGiXMNpz/f1bULaCMinsdKqmhjnHN/hVubLM8R92cjj2Wtymxz0ntxaykzcJqHxgGfqGpJDXA78GSZ89ZIVae5+76jqkNx/tYUpz+vXrNEULteAp50P4gRkRYiMsZ93gmnLXcCTnPG/SKS5u63B2h+miaS+4FJInKfiDR3y0wVkenlbBuF8wec5W53I06NAPf11R4feAfdbYtFpL/7DSoc558vjxNt+15x23mnAH8WkQS3FjDY/WCKwamC78f5p/7fMrvvASq7FmEa8LB7XuNwml+qO6zyM5y+nJK4/wv8B6dG11dEwkQkRkTuKKkxlXmf2Tgd4veXXeexzUacTtK7PZe7bfvNgPlViDdERCI9HhE4732UiFzknudIcYYiJwINcL50ZAGFInIxTh9QbfgBpyb8S/c8juFEs2R5/ozTj/GGx/9OGxH5s4j0cmtgO4AJ7vu8Ce9GE70DXIPTpPiOx/JXgDvcv3URkSgRudT9fXcWkfPc85vHiYEc9Zolgtr1V5xOs1kikoPzjz7QrS5PBZ5R1eWq+hPwW+AtEYlQ1XU4H3Kb3arqKVVRVZ0HnOc+NovIAeBlnA+0stuuwfmQ+gHnw7UnzqiKEv2BBSKS68b7K3WuAWiM809yEKfZZT8e4+yr4F6cUSCLcEbPPIPzt/imW+4OnJFMZT8IJ+O0zWaLyEfllPsEThv4Crf8pe6y6vgY6FLmXI/FOZ/v4oxqWgX0w/nGWp6/4nzgVeYxnMTs6TrgjZJ+E7ezOPc0ncXjcD6USh6b3I7tMTh/S1k433Tvwxl9k4OTgGbg/D6vw/ld+52qHgeuxOmwz8b58vMJzpeA8rY/gDOwoADn7zIHZ0TdIWCju9mtOO9tP85ghXlexLEA5wtNAs4ghZLli93y/o5zbjbijDwCJ3k+jVOz3Y0zqOEhL952nSYnN9MZY0qIyG1AN1X9dS0eMwKnSehcVd1bW8cNNBFZALykqq8FOpZgZInAGFPrRGQYzvDcfThNMy8B7as6Cs34hl1hZ4wJhM44zVJROCPlxloSCByrERhjTJCzzmJjjAly9a5pKC4uTtu1axfoMIwxpl5ZsmTJPlUt90r2epcI2rVrx+LFiwMdhjHG1CsisrWiddY0ZIwxQc4SgTHGBDlLBMYYE+TqXR+BMaZ2FBQUkJmZSV5e2QlITV0WGRlJYmIi4eHhXu9jicAYU67MzExiYmJo164dJ08UauoqVWX//v1kZmaSkpLi9X7WNGSMKVdeXh7Nmze3JFCPiAjNmzevci3OEoExpkKWBOqf6vzOgiYRbMrK5Q8fr6agqN5PHW6MMT4VNIlg2/6jvPZ9Bp+ttHmtjKkP9u/fT1paGmlpabRq1Yo2bdqUvj5+/Hil+y5evJi777670m0AhgwZ4pNYZ8+ezWWXXeaTsgLBb53FIjIFuAzYq6o9KtmuP84NUq5V1ff8Fc+wTi1o3yKKyd9tYXRqglV5janjmjdvzrJlywB49NFHiY6O5t577y1dX1hYSFhY+R9h/fr1o1+/fqc9xrx5p71/TVDwZ43gdWBkZRuISCjO3alm+TEOAEJChJvOTmFF5iEWZRz09+GMMX4wadIk7rjjDgYOHMj999/PwoULGTx4ML1792bIkCGsX78eOPkb+qOPPspNN91Eeno67du35/nnny8tLzo6unT79PR0xo4dS5cuXRg/fjwlMzN/9tlndOnShb59+3L33XdX6Zv/tGnT6NmzJz169OCBBx4AoKioiEmTJtGjRw969uzJX/7yFwCef/55unXrRq9evbj22mtrfrKqwG81AlWdKyLtTrPZXTg30O7vrzg8XdUnkWdnrWfyd5sZkNKsNg5pzBnhDx+vZs3Owz4ts1tCYx4Z1b3K+2VmZjJv3jxCQ0M5fPgw3377LWFhYfz3v//lt7/9Le+///4p+6xbt45vvvmGnJwcOnfuzJ133nnKOPsff/yR1atXk5CQwNlnn833339Pv379uP3225k7dy4pKSmMGzfO6zh37tzJAw88wJIlS2jatCkjRozgo48+IikpiR07drBq1SoAsrOzAXj66afZsmULERERpctqS8D6CNwbdF8BvOjFtreJyGIRWZyVlVXtYzZsEMr4gcnMWrOHbfuPVrscY0zgXH311YSGhgJw6NAhrr76anr06ME999zD6tWry93n0ksvJSIigri4OFq2bMmePXtO2WbAgAEkJiYSEhJCWloaGRkZrFu3jvbt25eOya9KIli0aBHp6em0aNGCsLAwxo8fz9y5c2nfvj2bN2/mrrvu4osvvqBx48YA9OrVi/HjxzN16tQKm7z8JZAXlD0HPKCqxadrr1fVl3FuxE6/fv1qdCed6we34+W5m3lt3pZqfRsxJhjVpf+VqKio0ue/+93vGD58OB9++CEZGRmkp6eXu09ERETp89DQUAoLC6u1jS80bdqU5cuX8+WXX/LSSy8xY8YMpkyZwqeffsrcuXP5+OOPefLJJ1m5cmWtJYRAjhrqB0wXkQxgLPCCiFzu74PGN47ksl4JzFi0ncN5Bf4+nDHGjw4dOkSbNm0AeP31131efufOndm8eTMZGRkAvPvuu17vO2DAAObMmcO+ffsoKipi2rRpDBs2jH379lFcXMxVV13FE088wdKlSykuLmb79u0MHz6cZ555hkOHDpGbm+vz91ORgCUCVU1R1Xaq2g54D/i5qn5UG8e+eWgKR44X8e7C7bVxOGOMn9x///089NBD9O7d2y/f4Bs2bMgLL7zAyJEj6du3LzExMcTGxpa77VdffUViYmLpIyMjg6effprhw4eTmppK3759GTNmDDt27CA9PZ20tDQmTJjAU089RVFRERMmTKBnz5707t2bu+++myZNmvj8/VTEb/csFpFpQDoQB+wBHgHCAVT1pTLbvg584s3w0X79+qkvbkxzzT9/IPPgMebcl05YaNBcTmGM19auXUvXrl0DHUbA5ebmEh0djaryi1/8go4dO3LPPfcEOqxKlfe7E5ElqlrumFq/fQKq6jhVba2q4aqaqKqTVfWlsknA3XaSP68hKM/NQ1PYkX2ML1bvrs3DGmPqmVdeeYW0tDS6d+/OoUOHuP322wMdks8F7eyj53eNp13zRkz+bguX9UoIdDjGmDrqnnvuqfM1gJoK2jaR0BDhxrNT+HFbNku32QVmxpjgFbSJAGBs30QaR4Yx+bstgQ7FGGMCJqgTQVREGOMGJPP5yl1kHrQLzIwxwSmoEwHADUOcuy+9MS8j0KEYY0xABH0iSGjSkEt6tmb6wu3k5vvnSkJjTNUNHz6cL7/88qRlzz33HHfeeWeF+6Snp1MyvPySSy4pd86eRx99lGeffbbSY3/00UesWbOm9PXvf/97/vvf/1Yh+vLV1emqgz4RgDOUNCe/kBmL7AIzY+qKcePGMX369JOWTZ8+3ev5fj777LNqX5RVNhE89thjXHDBBdUqqz6wRACkJTWhb9umvDZvC0XF/rnAzhhTNWPHjuXTTz8tvQlNRkYGO3fu5JxzzuHOO++kX79+dO/enUceeaTc/du1a8e+ffsAePLJJ+nUqRNDhw4tnaoanGsE+vfvT2pqKldddRVHjx5l3rx5zJw5k/vuu4+0tDQ2bdrEpEmTeO8951Knr776it69e9OzZ09uuukm8vPzS4/3yCOP0KdPH3r27Mm6deu8fq+Bnq46aK8jKOuWoSnc+fZS/rNmDyN7tAp0OMbULZ8/CLtX+rbMVj3h4qcrXN2sWTMGDBjA559/zpgxY5g+fTo/+9nPEBGefPJJmjVrRlFREeeffz4rVqygV69e5ZazZMkSpk+fzrJlyygsLKRPnz707dsXgCuvvJJbb70VgIcffpjJkydz1113MXr0aC677DLGjh17Ull5eXlMmjSJr776ik6dOnH99dfz4osv8utf/xqAuLg4li5dygsvvMCzzz7Lq6++etrTUBemq7YagWtE91YkNm3IFBtKakyd4dk85NksNGPGDPr06UPv3r1ZvXr1Sc04ZX377bdcccUVNGrUiMaNGzN69OjSdatWreKcc86hZ8+evP322xVOY11i/fr1pKSk0KlTJwBuuOEG5s6dW7r+yiuvBKBv376lE9WdTl2YrtpqBK7QEGHSkHY88elaVmRm0yuxSaBDMqbuqOSbuz+NGTOGe+65h6VLl3L06FH69u3Lli1bePbZZ1m0aBFNmzZl0qRJ5OXlVav8SZMm8dFHH5Gamsrrr7/O7NmzaxRvyVTWvpjGujanq7YagYdr+icRHWEXmBlTV0RHRzN8+HBuuumm0trA4cOHiYqKIjY2lj179vD5559XWsa5557LRx99xLFjx8jJyeHjjz8uXZeTk0Pr1q0pKCjg7bffLl0eExNDTk7OKWV17tyZjIwMNm7cCMBbb73FsGHDavQe68J01VYj8BATGc41/ZN4Y14GD17chdaxDQMdkjFBb9y4cVxxxRWlTUSpqan07t2bLl26kJSUxNlnn13p/n369OGaa64hNTWVli1b0r//iTvjPv744wwcOJAWLVowcODA0g//a6+9lltvvZXnn3++tJMYIDIyktdee42rr76awsJC+vfvzx133FGl91MyXXWJf/3rX6XTVasql156KWPGjGH58uXceOONFBcXA5w0XfWhQ4dQVZ9NV+23aaj9xVfTUFdk+4GjDPvjN9x2bgcevLiL345jTF1n01DXX3VmGur6KqlZIy7q3oppC7dx9LhdYGaMOfNZIijHLeekcOhYAe8vyQx0KMYY43eWCMrRJ7kpqUlNmPJ9BsV2gZkJYvWt6dhU73dmiaAcIsLNQ1PYsu8IX6/bG+hwjAmIyMhI9u/fb8mgHlFV9u/fT2RkZJX2s1FDFbi4RysSYiOZ/N0WLugWH+hwjKl1iYmJZGZmkpWVFehQTBVERkaeNCrJG5YIKhAeGsINQ9rx1OfrWL3zEN0TYgMdkjG1Kjw8nJSUlECHYWqBNQ1V4toByTRqEGoXmBljzmiWCCoR2zCcq/sm8vHynew9XL1L2I0xpq6zRHAaN56dQmGx8tb8rYEOxRhj/MJviUBEpojIXhFZVcH68SKyQkRWisg8EUn1Vyw10S4uigu6xjN1/lbyCooCHY4xxvicP2sErwMjK1m/BRimqj2Bx4GX/RhLjdw8NIWDRwv4YOmOQIdijDE+57dEoKpzgQOVrJ+nqgfdl/OBqo13qkUDU5rRo01jpny/xcZUG2POOHWlj+BmoMK5ZEXkNhFZLCKLAzGmueQCs417c5mzwcZUG2POLKdNBCJytYjEuM8fFpEPRKSPrwIQkeE4ieCBirZR1ZdVtZ+q9mvRooWvDl0ll/ZMoGVMhA0lNcaccbypEfxOVXNEZChwATAZeNEXBxeRXsCrwBhV3e+LMv2lQZhzgdm3P+1j/e5Tb1hhjDH1lTeJoGSozKXAy6r6KdCgpgcWkWTgA2Ciqm6oaXm14boByUSGh9h9jY0xZxRvEsEOEfkncA3wmYhEeLOfiEwDfgA6i0imiNwsIneISMntfH4PNAdeEJFlIuK/u834SNOoBlzVJ5EPl+1gX25+oMMxxhifOO0dykSkEc4w0JWq+pOItAZ6quqs2giwLH/foex0NmXlcv6f5vDrCzry6ws6BSwOY4ypipreoaw18KmbBNKBq4GFvguvfunQIprhnVvYBWbGmDOGN4ngfaBIRM7CuegrCXjHr1HVcbec0559uceZuXxnoEMxxpga8yYRFKtqIXAl8DdVvQ+nlhC0hnRoTpdWMUz5zi4wM8bUf94kggIRGQdcD3ziLgv3X0h1n4hw09AU1u3O4fuNdXrUqzHGnJY3ieBGYDDwpKpuEZEU4C3/hlX3jUlLIC46gsnfbQ50KMYYUyOnTQSquga4F1gpIj2ATFV9xu+R1XERYaFMHNSWb9ZnsXFvbqDDMcaYavPmeoB04CfgH8ALwAYROde/YdUP4wcl0yAshCnf2wVmxpj6y5umoT8BI1R1mKqeC1wE/MW/YdUPcdERXJHWhg+WZnLwyPFAh2OMMdXiTSIIV9X1JS/c6SCCurPY083npJBXUMw7C7cFOhRjjKkWbxLBYhF5VUTS3ccrQJ2fDqK2dIqP4ZyOcbwxL4PjhcWBDscYY6rMm0RwJ7AGuNt9rAHuqHSPIHPz0BT25uTzyQq7wMwYU/94M2ooX1X/rKpXuo+/AN/UQmz1xrBOLejYMprJdoGZMaYequ4dypJ9GkU9V3KB2eqdh1mwpcK7cxpjTJ1U3URgX3vLuKJ3G5pFNeDVb20oqTGmfgmraIWIXFnRKqChf8Lxs6z10KKzX4qODA9l/MBk/v7NRjL2HaFdXJRfjmOMMb5WWY1gVAWPyzgx51D9sewdeGEwrPVf6BMHtyU8JITX7AIzY0w9UmGNQFVvrM1A/K7raFg0Gd67Eca/B+2H+fwQLWMiGZWawL+WZPL/LuxMbCO73MIYU/dVt4+g/omIhvH/guZnwbRxkLnEL4e5eWgKR48XMW2RXWBmjKkfgicRADRqBhM+gOgW8PZVsGeNzw/RLaExQzo05415GRQU2QVmxpi6L7gSAUDj1jDxIwiNgLeugIMZPj/EzUNT2HUoj89X7fZ52cYY42vezD66RER+ISJNayOgWtEsBSZ+CIV58OYYyPHtB/bwzi1pHxfF5G832wVmxpg6z5sawTVAArBIRKaLyEUiIn6Oy//iu8GE9yE3y6kZHPXdhWAhIcKNZ7djeeYhlmw96LNyjTHGH7yZYmKjqv4P0AnnpvVTgK0i8gcRaVbRfiIyRUT2isiqCtaLiDwvIhtFZIWI9Knum6i2xH4w7h3YvxHe+Rnk++4GM1f1TSS2YTiTv7OhpMaYus2rPgIR6YVzX4I/Au8DVwOHga8r2e11YGQl6y8GOrqP24AXvYnF59qnw9gpsGMJvDsBCvN9UmyjBmFcNzCZL1fvZvuBoz4p0xhj/MGrPgKcG9EsAnqp6t2qukBV/wRUeMNeVZ0LVNbeMgZ4Ux3zgSYi0rpq4ftI11Ew5h+w+Rt4/2YoKvRJsTcMbkeICK99n+GT8owxxh+8qRFcrarnq+o7qnrS12VVrWgaCm+0AbZ7vM50lwVG2nUw8mlY+zF8/CsorvnQz1axkVzWqzUzFm8nJ6/AB0EaY4zveZMIDrlt+UvdEUR/FZHmfo/Mg4jcJiKLRWRxVlaW/w406E4Y9gAsmwqzHgYfjPi5eWh7cvMLeXfR9tNvbIwxAeBNIpgOZAFXAWPd5+/64Ng7gCSP14nuslOo6suq2k9V+7Vo0cIHh65E+kMw4HaY/w+Y+2yNi+uZGMuAds147fsMCu0CM2NMHeRNImitqo+r6hb38QQQ74NjzwSud0cPDQIOqeouH5RbMyJOE1Gva+GbJ2DByzUu8qahKezIPsasNXt8EKAxxvhWhZPOeZglItcCM9zXY4EvT7eTiEwD0oE4EckEHsG96b2qvgR8BlwCbASOAnVnkruQEBjzd8g/DJ/fB5GxkHpNtYu7sFs8yc0aMfm7LVzSMzD94cYYUxE53ZWvIpIDRAEl7RohwBH3uapqY/+Fd6p+/frp4sWLa+dgBXnw9ljYOg+ufRs6X1ztol77fgt/+HgNH/58CL2Tz5yLtI0x9YOILFHVfuWt8+aCshhVDVHVMPcR4i6Lqe0kUOvCI2HcNGidCjNugC3fVruoq/slERMRZheYGWPqHG8vKBstIs+6j8v8HVSdEhHjTEXRLAWmXQs7llarmOiIMK4dkMTnq3azI/uYj4M0xpjq8+aCsqeBXwFr3MevROQpfwdWpzRq5kxS16gZTL0K9q6rVjE3DGkHwJvzMnwXmzHG1JA3NYJLgAtVdYqqTsGZNuJS/4ZVBzVOcKavDglzp6/eWuUiEps2YmSPVkydv5VVOw75PkZjjKkGb+9H0MTjeawf4qgfmndwagYFR+CtyyGn6sNBf3tJV5o0asCEyQtYu+uw72M0xpgq8iYR/C/wo4i8LiJvAEuAJ/0bVh3Wqodzz+Oc3TD1SjhWtWmm2zRpyLRbB9EwPJTxry5gw54cPwVqjDHeqTQRiEgIzrDRQcAHODOPDlZVX1xZXH8lDXCGk2ath3eugeNHTr+Ph+TmjXjn1kGEhQjXvbKAjXt9N/21McZUVaWJQFWLgftVdZeqznQfdv9FgA7nwdjJkLkI3p0IhcertHtKXBTv3DoIgOtemc+WfVVLJsYY4yveNA39V0TuFZEkEWlW8vB7ZPVBtzEw6nnY9BV8cCsUF1Vp97NaRvPOrQMpLFaue2U+2/bbfQuMMbXP21tV/gKYi9M/sASopUt764E+E2HEE7DmI2f66irOWNopPoapNw/kWEER416ZT+ZBSwbGmNrlTSLoqqopng+gm78Dq1eG3AXn3As/vgX/+V2Vk0G3hMZMvXkgOXkFjHtlPrsO2QVnxpja400imOflsuB23sPQ/1aY9zf47s9V3r1Hm1jeunkg2UcKGPfyfPYczvNDkMYYc6oKE4GItBKRvkBDEektIn3cRzrQqLYCrDdE4OL/g55Xw1ePwaJXq1xEalITXr9pAFk5+Yx7ZT5ZOb65f7IxxlSmshrBRcCzODeM+TPOzev/BPw/4Lf+D60eCgmBy1+ETiPh03th5XtVLqJv26a8ftMAdmXncd0r89mfa8nAGONf3kxDfZWqvl9L8ZxWrU5DXV0Fx2DqWNg+H659BzpdVOUifti0nxtfX0i75lFMu3UQTaMa+CFQY0ywqNE01MAnInKdiPxWRH5f8vBxjGeW8IbO9NXx3WHG9ZDxfZWLGNyhOa9e35/N+44wYfICDh0t8EOgxhjjXSL4NzAGKMS5IU3Jw1QmsjFM+ACaJDvTV+9cVuUihnaM4+WJfflpTy7XT1nA4TxLBsYY3/OmaWiVqvaopXhOq140DXk6lAlTRkLBUbjxC2jRqcpFfLV2D3dMXVI6sig6wps7jBpjzAk1bRqaJyI9fRxT8IhNhOv/DRLizFiavb3KRZzfNZ6/jevDisxD3PjaQo7kF/o+TmNM0PImEQwFlojIehFZISIrRWSFvwM7ozTv4DQT5ec6ySA3q8pFjOzRiuev7c2SrQe5+Y1FHDteteksjDGmIt4kgouBjsAIYBRwmfvTVEXrXjB+BhzaAW9cBpvnVLmIS3u15i/XpLFwywFufXMxeQWWDIwxNVfZBWXnAajqViBEVbeWPIC+tRXgGSV5kDOaKD8H3hwNb46BHUuqVMSYtDb8cWwq32/ax+1vLSG/0JKBMaZmKqsRPOvxvOx1BA/7IZbg0GE43LUULvpf2LUCXjnPmcY6a4PXRVzVN5Gnr+zJnA1Z/HzqUo4XFvsxYGPMma6yRCAVPC/vdfkFiIx0+xY2isiD5axPFpFvRORHt//hEm/KrffCI2HwL+BXy2HYg7Dpa3hhIHz0C687k6/pn8wTl/fgq3V7uWvaUgqKLBkYY6qnskSgFTwv7/UpRCQU+AdOH0M3YJyIlJ219GFghqr2Bq4FXjhtxGeSyMYw/CEnIQy8A1bOgL/1gS8egiP7Trv7hEFteXRUN75cvYdfT19GoSUDY0w1VDYgvb2IzMT59l/yHPd1ihdlDwA2qupmABGZjnNh2hqPbRRo7D6PBXZWIfYzR1QcjHwKBv0c5jwNC16CpW86tYbBv3QSRgUmnZ1CYbHyxKdrCQsV/vyzNEJDvKqwGWMMUMkFZSIyrLIdVbXSYS8iMhYYqaq3uK8nAgNV9Zce27QGZgFNgSjgAlU9pfdURG4DbgNITk7uu3Xr1soOXf9lrYevn4C1M6FhMzjnN9D/FqdJqQIvzt7EM1+s48o+bXh2bCohlgyMMR4qu6CswhrB6T7ofWQc8Lqq/klEBgNviUgP917JnrG8DLwMzpXFtRBXYLXoDNe8BTuWOlNaz/ofmP8CpD8IqddB6Km/tjvTO1BQVMyf/7OB8JAQnrqypyUDY4xXvLmOoLp2AEkerxPdZZ5uBmYAqOoPQCQQ58eY6pc2feD6j+D6mRDTCmbeBS8MgtUfQvGp/QF3n9+Ru887i3cXb+d3/17F6aYPMcYY8G8iWAR0FJEUEWmA0xk8s8w224DzAUSkK04iqPplt2e69sPglq/gmrchJBT+NQleGQ4bvzrltpj3XNiJO9M78PaCbfzh4zWWDIwxp1WlRCAiISJScc+lB1UtBH4JfAmsxRkdtFpEHhOR0e5mvwFuFZHlwDRgktonV/lEoOtlcOc85+Y3Rw/A1CvhjVGwfZHHZsL9F3XmlqEpvD4vgyc/XWvJwBhTKW9mH30HuAMowvmW3xj4q6r+0f/hnarezT7qL4X5sOR1mPN/cHQfdL7UuW9yvDNCV1X5w8dreH1eBncM68ADIzsjYn0GxgSrms4+2k1VDwOXA5/jDB2d6LvwTLWERcDA251rEIY/DBnfwotD4IPb4WAGIsIjo7oxfmAyL83ZxF/+4/2Vy8aY4OLNxPbhIhKOkwj+rqoFImJtDXVFRDQMuw/63wzf/QUWvgyr3od+NyLn3sfjY3pQWKQ8//VGwkJDuPv8joGO2BhTx3iTCP4JZADLgbki0hY47M+gTDU0agYjHodBd8KcZ2DRZPhxKiGDfs5Tl/ySgmJ3aGloCHemdwh0tMaYOuS0fQTl7iQS5nYG1zrrI/DS/k3wzZNO7SCyCcVn38P92wfy3ooDPHxpV245p32gIzTG1KIa9RGIyK9EpLE4JovIUuA8n0dpfKt5Bxg7BW6fC4n9CfnqEf6460aeSl7E05+u5PXvtwQ6QmNMHeFNZ/FNbmfxCJypICYCT/s1KuM7rVNhwnsw6TOkSTLj9v6F72MeYumnr/Lq3I02tNQY41UiKBlzeAnwlqquxstpqE0d0u5suOlLGPcuLZrG8nyDvzPyq4v4/v+uZP83L8DuVVBsN7kxJhh501m8RERm4QwbfUhEYgCb77g+EoHOIwnpOILiVe9TPO9dOu1aTPM5X8Mc0IjGSNIASBoEyQOhTV9oEBXoqI0xfubNBWUhQBqwWVWzRaQ50EZVA3IDe+ss9q3d2cf46/v/IW/TPEbEZDC80WYiD24AFELCoFUv5xabSQOdnzGtAh2yMaYaKuss9mrUkDslxLnuyzmq+rEP46sSSwS+p6p8vmo3v//3Kg4eLeDuIXHc2eEADXYuhG0LnPsqFx5zNm7SFpIHOzWGpEHQoguE+HPKKmOML9QoEYjI00B/4G130Thgkar+1qdReskSgf9kHz3Ok5+u5V9LMmkfF8VTV/ZkYPvmUHgcdq+EbT/A9vlOcjiy19kpMhYSBzi1heRBkNAHGjQK7BsxxpyipolgBZBWco8A9xaUP6pqL59H6gVLBP733U/7eOjDFWw/cIzrBibz4MVdaBwZfmIDVTi4xUkI236A7Qsga52zLiTMGalU0s+QNAhi4gPzRowxpXyRCNJV9YD7uhkw2xLBme3o8UL+PGsDU77fQsuYSB6/vAcXdqvkA/3oAchcBNvmO4+dS6Ewz1nXNOXkfoa4ztacZEwtq2kiuBZ4BvgGZ9joucCDqvqurwP1hiWC2rVsezYPvr+CdbtzuLRXax4d1Z0WMRGn37HwOOxa7jYluY+j+5x1kU3cpODWGNr0gfCGfn0fxgS7aicCd8TQWOBbnH4CgIWqutvnUXrJEkHtO15YzD/nbOJvX2+kYYNQfndZN67q06Zq01qrwoHNTkIo6WfYt95ZVzI6KWkAJPZ3fsYmOcNdjTE+UdMaweKKdg4ESwSBs3FvDg++v5LFWw9yTsc4/veKniQ1q0HH8NEDTv/C9oVOs9KOJVBw1FkX3QqS+jsd0UkDoHUahEf65H0YE4x8MWpoH/AucKRkeUmfQW2zRBBYxcXK1AVbeebzdRQr/GZEJ248O4XQEB98ey8qhD2rnKSwfSFkLoSDGc66kHBo3ctpUiqtNSTW/JjGBImaJoLyZidTVQ3I9JWWCOqGHdnHePjDlXyzPovUpCY8c1VPurTy6i6mVZO7100MC5xbcu788cQ1DTEJZWoNqc4Ne4wxp6jxBWV1iSWCukNVmbl8J3/4eA2HjxXw8/QO/OK8s4gIC/XfQYsKnGsaPGsN2ducdaEN3KGrHrWGxgn+i8WYeqRaiUBEJrjr3yqzfCJQpKrv+DxSL1giqHsOHDnO45+s4cMfd3BWy2ieuaonfds2q70AcnafSAoltYaifGdd48STaw2tekFYg9qLzZg6orqJYAFwvqrmllkeBcxV1b4+j9QLlgjqrm/W7+XhD1ex89Axrh/UlvtGdiE6wpt5DX2s5ErozIVOgti+EA5nOutCIyAhza0xDHSSg82fZIJAdRPBUlXtU8G6FXZBmSlPbn4hz365njd+yKB140ievKInw7u0DHRYcHjnidFJ2xfCrmVQdNxZF5vs1BriOkPTdice0S1tCKs5Y1Q3EawF+qnqkTLLY3DmGuri80i9YImgfliy9SAPvL+CjXtzGZOWwO8v60bz6DrUkVuYD7tWuLWGBZC55EStoURYQ4/E0PbkJNGkrc2pZOqV6iaCe4HzgTtUdau7rB3wD5wpJv7oxYFHAn8FQoFXVfWUO5uJyM+ARwEFlqvqdZWVaYmg/sgvLOKFbzbxwuyNREeE8cio7oxJS6jahWi1qeAYZG93hqx6PrK3Oj+P5568fXS8kxA8E0TJI6a1TaNh6pSaXFl8B/AQEO0uygWeVtUXvThoKLABuBDIBBYB41R1jcc2HYEZwHmqelBEWqrq3srKtURQ/6zfncMD769g2fZs0ju34InLe5DYtJ59m1aFo/tPTRIHM+DgVqc2oR73awptAE2ST00QJYkj0g9DbY2phC/uRxADoKo5VTjoYOBRVb3Iff2QW8ZTHtv8H7BBVV/1tlxLBPVTUbHy+rwMnv1yPSJw/0WduX5wO0J8cSFaXVB4HA5tP1F7KJso8rJP3r5hszJJou2JRNE4wa6HMD5XWSLwakhHVRKAhzbAdo/XmcDAMtt0cgP8Hqf56FFV/aJsQSJyG3AbQHJycjVCMYEWGiLcPDSFEd3i+e2HK3n04zXMXL6TZ67qRcf4mECHV3NhDaB5B+dRnmMHnYRQtrlp1zJYOxOKC0/evmEzp3kpppXHz5LnraFxa4hqCaEBGJVlzjh+u6BMRMYCI1X1Fvf1RGCgqv7SY5tPgALgZ0AiMBfoqarZFZVrNYL6T1X5YOkOHv90DUfzixg/KJmJg9rSvkX06Xc+ExUXweEdJxJFzm7I2XXyz9w9oEVldhRnZNNJyaKcn43irL/C1LxGUE07gCSP14nuMk+ZwAJVLQC2iMgGoCNOf4I5Q4kIV/VN5NxOLXjq87W89cNWXvs+g7PPas7EQW25oGs8YaFB9MEVEur0JzRJhpRzyt+muAiO7Ds1QZT8PLwTdiyFI1k44y48yw9zOrbLrV20cqbqiGkFDZvacNkg5c1cQ0uAKcA7qnrQ64JFwnA6i8/HSQCLgOtUdbXHNiNxOpBvEJE44Eecu6Htr6hcqxGcefbm5PHuwu1MW7iNnYfyiG8cwbX9kxk3IJlWsTbjaJUUFTi1h/KShefPY+X8K4dGnEgQUXEQFun0VYQ2KPMzwmkKC4useN1JP8uW4ZYb4sepSMwpajrp3FnAjcA1wGLgNWCWetGmJCKXAM/htP9PUdUnReQxYLGqzhRnHOGfgJFAEfCkqk6vrExLBGeuwqJivlmfxVvztzJ3QxahIcKFXeOZMKgtQzo0P3M6luuCgjzI3V1xojiyz7nDXOFxZ7qO0p/5nFLjqC4JPX0SadDIqal4PiKbnLqsYRPrYD8Nn0w6596k5jLgRZwP7deAv9b2dNSWCILD1v1HeGfBNmYs3s7BowWkxEUxfmAyV/dNIrZR+OkLMP6h6nRsF+Y7V2YX5jsJo+R56c/8cpJIBYnllH2Pu9vmO/enOJbt1GDysk8eoltWeNSJpHDSz9MkkQZRQdEk5ovho71wagWXAF8CbwNDgYmqmua7UE/PEkFwySso4rOVu5g6fytLt2UTERbC6NQEJgxqS2pSk0CHZ2pTcTHkH3aSQskjL9vjdfbJ60pfHzgxnUh5QsIrThwlj/CGICFOLSYk1Hle8rN0WajTKX/KslAn0ZyyLMSjHI91nmWfsiy02h3/NW0aWgJkA5OB91U132PdB6p6ZbWiqiZLBMFr9c5DTJ2/jX8v28HR40X0bBPLxEFtGZWaQMMG1t5sKqDqXDVeYQIp+8g+kUSOV2fkvB+d/Wu48A/V2rWm9yx+UFX/t1pH9gNLBOZwXgEf/biDt37Yyk97c2kcGcZVfROZMKgtHYJ1CKrxj6ICJyEU5jkjt7TY/Vl04rUWObWVU5YVeawru6zISVCnLCuu/BhJ/aF9erXeit2z2JyRVJWFWw4wdcE2vli1i4IiZUiH5kwY1JYLu8UTHkxDUI05DbtnsTnjZeXkM2Pxdt5ZsI0d2cdoGRPBtQOSGTcgidaxDQMdnjEBZ/csNkGjqFiZvX4vb83fypwNWYSIcH6Xlkwc3JazO8TZEFQTtGp0ZbGqpvg+JGP8IzREOL9rPOd3jWfb/qO8s9AZgjprzR7aNW/EhEFtGds3kSaN7HaVxpTwdvhoD6AbUHqZp6q+6ce4KmQ1AlNV+YVFfL5yN1Pnb2Xx1oNEhIVwWa8EJg5uS2pibN29P4IxPlTTpqFHgHScRPAZcDHwnaqO9XGcXrFEYGpi7a7DTJ2/lY9+3MGR40X0aNOYCQPbMjotgUYNbCZPc+aqaSJYCaQCP6pqqojEA1NV9ULfh3p6lgiML+S4Q1Cnzt/G+j05xESGMW5AMjednWLzG5kzUk1nHz2mqsUiUigijYG9nDyrqDH1TkxkOBMHt2PCoLYs3nqQN+Zl8Oq3m3nt+y1c0bsNt53bnrNangH3STDGC94kgsUi0gR4BViCc7vKH/wZlDG1RUTo364Z/ds1Y/uBo7zy7WZmLN7OjMWZXNA1njvT29O3bbNAh2mMX1XpxjTuzesbq+oKv0V0GtY0ZPxtf24+b/ywlTd/yCD7aAH92jbljmEdOK9LSxt+auotX0w61wZoi0cNQlXn+izCKrBEYGrL0eOFzFi0nVe+3cKO7GN0bBnNbee2Z0xaGxqE2VXLpn6paWfxMzj3IliDM/00OBeUjfZplF6yRGBqW0FRMZ+t3MVLczazdtdhWjWO5Kah7Rg3IJmYSJsS29QPNU0E64FenrOOBpIlAhMoqsrcn/bx0uxN/LB5PzGRYUwY1JYbz25HyxgbaWTqtpqOGtoMhAN1IhEYEygiwrBOLRjWqQXLt2fz8tzN/HPOJiZ/u4Wr+rbh1nPa095mPzX1kDc1gvdxriP4Co9koKp3+ze08lmNwNQlGfuO8Mq3m/nXkkwKioq5qFsr7kjvQJrdNMfUMTVtGrqhvOWq+oYPYqsySwSmLsrKyeeNeRm8+UMGh/MKGZjSjDvSO5DeqYVNYWHqBJ/cs7iusERg6rLc/EKmL9zG5O+2sOtQHl1axXDbue0ZlZpg90cwAVWtRCAiM1T1Z+4UE6dspKq9fBumdywRmPqgoKiYmct28s+5m9iwJ5eE2EhuPqc91/ZPIirC5jQyta+6iaC1qu4SkbblrVfVrT6M0WuWCEx9oqp8s34vL83ZzMItB4htGM71g9tyw5B2xEVHBDo8E0R81jQkInHAfvVyJxEZCfwVCAVeVdWnK9juKuA9oL+qVvopb4nA1FdLtx3kn3M2MWvNHhqEhnB1v0RuPac9bZtHBTo0EwSqWyMYBDwNHAAeB94C4oAQ4HpV/eI0Bw0FNgAXApnAImCcqq4ps10M8CnQAPilJQJzptuUlcur327m/SU7KCwu5uKerbnj3A70TIwNdGjmDFZZIqis9+rvwP8C04CvgVtUtRVwLvCUF8cdAGxU1c2qehyYDowpZ7vHgWeAPC/KNKbe69Aimqeu7MV3Dwzn9mEdmLs+i1F//47xr85nzoYsiorr1wAOU/9V1msVpqqzAETkMVWdD6Cq67wcDtcG2O7xOhMY6LmBiPQBklT1UxG5r0qRG1PPtWwcyQMju/Dz9A5Mc0ca3TBlIXHRDbigazwjusczpEMckeGhgQ7VnOEqSwTFHs+PlVlX468sIhIC/BmY5MW2twG3ASQnJ9f00MbUKTGR4dx2bgcmDUlh1prdfLl6D5+s2MX0RduJahBKeueWjOgeT3rnlsQ2tLmNjO9V1kdQBBwBBGgIHC1ZBUSqaqV/kSIyGHhUVS9yXz8EoKpPua9jgU049zcAaIXTHzG6sn4C6yMwwSC/sIj5mw8wa/Vu/rNmD3tz8gkLEQZ3aM6IbvFc0C2e1rENAx2mqUcCckGZiIThdBafD+zA6Sy+TlVXV7D9bOBe6yw25mTFxcryzGxmrdnDl6t3sznrCACpibGM6N6KEd3iOatltF3BbCoVsCuLReQS4Dmc4aNTVPVJEXkMWKyqM8tsOxtLBMac1sa9ucxas5tZq/ewbHs2AClxUYzo5vQr9E5qajfQMaewKSaMOUPtOZzHf9bsYdaaPfywaR8FRUpcdAQXdmvJiO6tGNKhORFh1tlsLBEYExQO5xUwe30Ws1bvZvb6LHLzC53O5i4tGdEtnuFdWtLYbqQTtCwRGBNk8guL+GHTfmat2cN/1uwhKyef8FBhUPvmjOjeigu7xtMq1m6mE0wsERgTxIqLlR+3Z5f2K2zZ53Y2JzVhRLd4LuoeT4cW1tl8prNEYIwBnEnwNmXl8uVqp19hudvZ3D4uigu7xzOiWyt6JzWxzuYzkCUCY0y5dh/K4z9r9zBr9W5+2LSfwmKlRUwEF3aLZ1SvBAamNLOkcIawRGCMOa1DxwqYvX4vs9bs4Zt1ezl6vIj4xhGM6pXA6LQEeraJteajeswSgTGmSo4eL+SrtXuZuXwns9fvpaBISYmLYlRqAqNTEzirZXSgQzRVZInAGFNth44W8MXqXcxcvpN5m/ajCt0TGjM6NYFRqQkkNLGpLuoDSwTGGJ/YeziPT1Y4SaHkquYB7ZoxKi2BS3u2pllUg8AGaCpkicAY43Nb9x/h4+U7+feynfy0N5fQEOGcjnGMTk1gRPdWRNu9mesUSwTGGL9RVdbtzmHm8p3MXLaTHdnHiAgL4YKu8YxKTSC9cwu7p0IdYInAGFMrVJWl2w4yc9lOPlmxi/1HjhMTGcbI7q0YnZbA4PbNCQut7MaIxl8sERhjal1hUTHzNu1n5vKdfLlqNzn5hcRFR3BZr9aMSk2gT3ITG45aiywRGGMCKq+giNnrneGo/127l+OFxSQ1a8ioXgmMSWtD51YxgQ7xjGeJwBhTZ+TkFTBr9R7+vXwn32/cR1Gx0jk+htFpzjUKSc0aBTrEM5IlAmNMnbQvN5/PV+7i38t2snjrQQB6JzdhdGoCl/ZqTcsYmyHVVywRGGPqvMyDR/lkhZMU1u46TIhA7+SmdG0dQ+f4GDrFx9C5VQxNGtm1CtVhicAYU69s3JvDzGXOlczr9+SQk1dYui6+cYSTFOJj6NQqhi6tYjirZTSNGth1C5WxRGCMqbdUlT2H81m3+zAb9uSwfncuG/bksGFPDvmFxQCIQHKzRqUJonMr55ESF0W4DVcFKk8ElkKNMXWaiNAqNpJWsZGkd25ZuryoWNl24Cjrd+e4CSKH9Xty+HrdXoqKnS+44aFC+7jo0sRQkigSmza06bU9WCIwxtRLoSFCSlwUKXFRjOzRqnR5fmERm7OOlCaGDbtznIvclu8s3aZRg1A6xsfQOT66tO+hc3wMLWIigvLaBksExpgzSkRYKF1bN6Zr68YnLc/JK+Cnvbls2J3DOrcW8fW6vcxYnFm6TdNG4aWJwfNnbMPw2n4btcoSgTEmKMREhtMnuSl9kpuetHxfbj4bSmoPbhPTB0t3kJt/ooM6sWlDUpOakJbYhNSkJvRo0/iM6pz26zsRkZHAX4FQ4FVVfbrM+v8H3AIUAlnATaq61Z8xGWOMp7joCOLOimDIWXGly1SVHdnH2LDHqT2s3nGYZduy+XTFLgBCBDrFx5CW5CSG1MQmdIqPrrfzKPlt1JCIhAIbgAuBTGARME5V13hsMxxYoKpHReROIF1Vr6msXBs1ZIwJlKycfFZkZrN8ezbLMg+xfHs2h44VABAZHkLPNrGkurWGtKQmJDZtWGf6HAI1amgAsFFVN7tBTAfGAKWJQFW/8dh+PjDBj/EYY0yNtIiJ4Pyu8ZzfNR5wag5b9x9leWY2y7Y7CeLN+Vs5/t0WAJpFNSA1MdapNbg1h7p48x5/JoI2wHaP15nAwEq2vxn4vLwVInIbcBtAcnKyr+IzxpgaERHaxUXRLi6KMWltACgoKmb97pzSxLA8M5vZG7IoaXxJataQ1MQmpc1KPRJiadggsPdrqBO9HSIyAegHDCtvvaq+DLwMTtNQLYZmjDFVEh4aQo82sfRoE8uEQW0ByM0vZGXmIZa7zUpLtx7kE7e/ITRE3P6GE81KHVvWbn+DPxPBDiDJ43Wiu+wkInIB8D/AMFXN92M8xhgTENERYQzu0JzBHZqXLtt7OI/lbj/D8kynI3raQqcRpWF4qNPfkBRb2qTkz/4Gf3YWh+F0Fp+PkwAWAdep6mqPbXoD7wEjVfUnb8q1zmJjzJmouFjZeuCo0xHtJofVOw9z3J1Go3lUA+4Y1oFbz21frfID0lmsqoUi8kvgS5zho1NUdbWIPAYsVtWZwB+BaOBfbqbbpqqj/RWTMcbUVSEeV0pf3tvpbzhe6PY3uE1K8bH+mZbbJp0zxpggUFmNoH5e/WCMMcZnLBEYY0yQs0RgjDFBzhKBMcYEOUsExhgT5CwRGGNMkLNEYIwxQc4SgTHGBLl6d0GZiGQB1b15TRywz4fh1Hd2Pk5m5+MEOxcnOxPOR1tVbVHeinqXCGpCRBZXdGVdMLLzcTI7HyfYuTjZmX4+rGnIGGOCnCUCY4wJcsGWCF4OdAB1jJ2Pk9n5OMHOxcnO6PMRVH0ExhhjThVsNQJjjDFlWCIwxpggFzSJQERGish6EdkoIg8GOp5AEpEkEflGRNaIyGoR+VWgYwo0EQkVkR9F5JNAxxJoItJERN4TkXUislZEBgc6pkARkXvc/5FVIjJNRPxzi7AAC4pEICKhwD+Ai4FuwDgR6RbYqAKqEPiNqnYDBgG/CPLzAfArYG2gg6gj/gp8oapdgFSC9LyISBvgbqCfqvbAueXutYGNyj+CIhEAA4CNqrpZVY8D04ExAY4pYFR1l6oudZ/n4PyjtwlsVIEjIonApcCrgY4l0EQkFjgXmAygqsdVNTugQQVWGNBQRMKARsDOAMfjF8GSCNoA2z1eZxLEH3yeRKQd0BtYEOBQAuk54H6gOMBx1AUpQBbwmttU9qqIRAU6qEBQ1R3As8A2YBdwSFVnBTYq/wiWRGDKISLRwPvAr1X1cKDjCQQRuQzYq6pLAh1LHREG9AFeVNXewBEgKPvURKQpTstBCpAARInIhMBG5R/Bkgh2AEkerxPdZUFLRMJxksDbqvpBoOMJoLOB0SKSgdNkeJ6ITA1sSAGVCWSqakkN8T2cxBCMLgC2qGqWqhYAHwBDAhyTXwRLIlgEdBSRFBFpgNPhMzPAMQWMiAhOG/BaVf1zoOMJJFV9SFUTVbUdzt/F16p6Rn7r84aq7ga2i0hnd9H5wJoAhhRI24BBItLI/Z85nzO04zws0AHUBlUtFJFfAl/i9PxPUdXVAQ4rkM4GJgIrRWSZu+y3qvpZ4EIydchdwNvul6bNwI0BjicgVHWBiLwHLMUZafcjZ+hUEzbFhDHGBLlgaRoyxhhTAUsExhgT5CwRGGNMkLNEYIwxQc4SgTHGBDlLBKbOEhEVkT95vL5XRB71Udmvi8hYX5R1muNc7c7g+U2Z5e1E5JiILPN4XO/D46bbTKrGW0FxHYGpt/KBK0XkKVXdF+hgSohImKoWern5zcCtqvpdOes2qWqa7yIzpnqsRmDqskKcC3juKbui7Dd6Ecl1f6aLyBwR+beIbBaRp0VkvIgsFJGVItLBo5gLRGSxiGxw5xwquS/BH0VkkYisEJHbPcr9VkRmUs6VtiIyzi1/lYg84y77PTAUmCwif/T2TYtIroj8xZ0H/ysRaeEuTxOR+W5cH7pz4SAiZ4nIf0VkuYgs9XiP0R73FXjbvToW95yscct51tu4zBlMVe1hjzr5AHKBxkAGEAvcCzzqrnsdGOu5rfszHcgGWgMROHNK/cFd9yvgOY/9v8D5MtQRZ46dSOA24GF3mwhgMc6kY+k4E7CllBNnAs50BC1watlfA5e762bjzGdfdp92wDFgmcfjHHedAuPd578H/u4+XwEMc58/5vFeFgBXuM8jcaZLTgcO4cyrFQL8gJOUmgPrOXExaZNA/57tEfiH1QhMnabOrKhv4twgxFuL1LnnQj6wCSiZOnglzgdwiRmqWqyqP+FMpdAFGAFc7069sQDng7Oju/1CVd1SzvH6A7PVmZysEHgbZ07/09mkqmkej2/d5cXAu+7zqcBQ9z4BTVR1jrv8DeBcEYkB2qjqhwCqmqeqRz3izVTVYpxE0w4nOeTh1FKuBEq2NUHMEoGpD57DaWv3nBe/EPfvV0RCgAYe6/I9nhd7vC7m5H6xsvOrKCDAXR4fzil6Yg76IzV5EzVQ3XlgPM9DEVDStzEAZ1bRy3BqRSbIWSIwdZ6qHgBm4CSDEhlAX/f5aCC8GkVfLSIhbpt6e5wmky+BO91puhGRTl7cmGUhMExE4tzboo4D5pxmn8qEACX9H9cB36nqIeCgiJzjLp8IzFHnDnOZInK5G2+EiDSqqGD3HhSx6kwweA/OrShNkLNRQ6a++BPwS4/XrwD/FpHlON9qq/NtfRvOh3hj4A5VzRORV3GaUJa6natZwOWVFaKqu0TkQeAbnBrFp6r6by+O38Fj9ldwZsV9Hue9DBCRh4G9wDXu+huAl9wPes9ZQScC/xSRx4AC4OpKjhmDc94i3Vj/nxdxmjOczT5qTB0jIrmqGh3oOEzwsKYhY4wJclYjMMaYIGc1AmOMCXKWCIwxJshZIjDGmCBnicAYY4KcJQJjjAly/x8fIzjFH07abgAAAABJRU5ErkJggg==",
536 | "text/plain": [
537 | ""
538 | ]
539 | },
540 | "metadata": {
541 | "needs_background": "light"
542 | },
543 | "output_type": "display_data"
544 | }
545 | ],
546 | "source": [
547 | "plt.title(\"Text Classification (CNN): Learning Curves\")\n",
548 | "plt.xlabel(\"Number of Epochs\")\n",
549 | "plt.ylabel(\"Cross Entropy Loss\")\n",
550 | "plt.plot(train_losses, label = \"Training Loss\")\n",
551 | "plt.plot(valid_losses, label= \"Validation Loss\")\n",
552 | "plt.legend()\n",
553 | "plt.show()"
554 | ]
555 | },
556 | {
557 | "cell_type": "markdown",
558 | "metadata": {},
559 | "source": [
560 | "# Testing"
561 | ]
562 | },
563 | {
564 | "cell_type": "code",
565 | "execution_count": 23,
566 | "metadata": {
567 | "id": "YLpKGbjyu2OK"
568 | },
569 | "outputs": [
570 | {
571 | "name": "stdout",
572 | "output_type": "stream",
573 | "text": [
574 | "Test Loss: 0.3947\n",
575 | "Test Accuracy: 84.90%\n"
576 | ]
577 | }
578 | ],
579 | "source": [
580 | "model.load_state_dict(torch.load('CNN.pt'))\n",
581 | "\n",
582 | "test_loss, test_acc = Evaluate(model, test_iterator, criterion)\n",
583 | "\n",
584 | "print(f'Test Loss: {test_loss:.4f}')\n",
585 | "print(f'Test Accuracy: {test_acc*100:.2f}%')"
586 | ]
587 | },
588 | {
589 | "cell_type": "markdown",
590 | "metadata": {
591 | "id": "o3vZ2NNn8sM6"
592 | },
593 | "source": [
594 | "# Sampling"
595 | ]
596 | },
597 | {
598 | "cell_type": "code",
599 | "execution_count": 24,
600 | "metadata": {
601 | "id": "e0hc5XtDsqIH"
602 | },
603 | "outputs": [],
604 | "source": [
605 | "import spacy\n",
606 | "nlp = spacy.load('en_core_web_sm')\n",
607 | "\n",
608 | "def predict_class(model, text, min_len = 4):\n",
609 | " model.eval()\n",
610 | " tokenized = [tok.text for tok in nlp.tokenizer(text)]\n",
611 | " if len(tokenized) < min_len:\n",
612 | " tokenized += [''] * (min_len - len(tokenized))\n",
613 | " indexed = [FIELD.vocab.stoi[t] for t in tokenized]\n",
614 | " tensor = torch.LongTensor(indexed).to(device)\n",
615 | " tensor = tensor.unsqueeze(1)\n",
616 | " preds = model(tensor)\n",
617 | " max_preds = preds.argmax(dim = 1)\n",
618 | " return max_preds.item()"
619 | ]
620 | },
621 | {
622 | "cell_type": "code",
623 | "execution_count": 42,
624 | "metadata": {},
625 | "outputs": [
626 | {
627 | "name": "stdout",
628 | "output_type": "stream",
629 | "text": [
630 | "['ENTY', 'HUM', 'DESC', 'NUM', 'LOC', 'ABBR']\n"
631 | ]
632 | }
633 | ],
634 | "source": [
635 | "print(LABEL.vocab.itos)"
636 | ]
637 | },
638 | {
639 | "cell_type": "code",
640 | "execution_count": 52,
641 | "metadata": {
642 | "id": "vc0zO3gTws-d"
643 | },
644 | "outputs": [
645 | {
646 | "name": "stdout",
647 | "output_type": "stream",
648 | "text": [
649 | "Predicted class is: ABBR\n"
650 | ]
651 | }
652 | ],
653 | "source": [
654 | "pred_class = predict_class(model, \"What does CNN stand for?\")\n",
655 | "print(f'Predicted class is: {LABEL.vocab.itos[pred_class]}')"
656 | ]
657 | },
658 | {
659 | "cell_type": "code",
660 | "execution_count": 53,
661 | "metadata": {
662 | "id": "Umdey8pIw8yp"
663 | },
664 | "outputs": [
665 | {
666 | "name": "stdout",
667 | "output_type": "stream",
668 | "text": [
669 | "Predicted class is: NUM\n"
670 | ]
671 | }
672 | ],
673 | "source": [
674 | "pred_class = predict_class(model, \"How long did it take to train an epoch?\")\n",
675 | "print(f'Predicted class is: {LABEL.vocab.itos[pred_class]}')"
676 | ]
677 | },
678 | {
679 | "cell_type": "code",
680 | "execution_count": 57,
681 | "metadata": {
682 | "id": "8rbQ8FYRw-TM"
683 | },
684 | "outputs": [
685 | {
686 | "name": "stdout",
687 | "output_type": "stream",
688 | "text": [
689 | "Predicted class is: DESC\n"
690 | ]
691 | }
692 | ],
693 | "source": [
694 | "pred_class = predict_class(model, \"What is the model of GPU used?\")\n",
695 | "print(f'Predicted class is: {LABEL.vocab.itos[pred_class]}')"
696 | ]
697 | }
698 | ],
699 | "metadata": {
700 | "accelerator": "GPU",
701 | "colab": {
702 | "collapsed_sections": [],
703 | "name": "CNN.ipynb",
704 | "provenance": []
705 | },
706 | "kernelspec": {
707 | "display_name": "Python 3",
708 | "name": "python3"
709 | },
710 | "language_info": {
711 | "codemirror_mode": {
712 | "name": "ipython",
713 | "version": 3
714 | },
715 | "file_extension": ".py",
716 | "mimetype": "text/x-python",
717 | "name": "python",
718 | "nbconvert_exporter": "python",
719 | "pygments_lexer": "ipython3",
720 | "version": "3.9.7"
721 | }
722 | },
723 | "nbformat": 4,
724 | "nbformat_minor": 0
725 | }
726 |
--------------------------------------------------------------------------------
/text_classification/notebooks/FastText.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Installing Packages"
8 | ]
9 | },
10 | {
11 | "cell_type": "code",
12 | "execution_count": null,
13 | "metadata": {},
14 | "outputs": [],
15 | "source": [
16 | "!pip install -U torchtext==0.6.0"
17 | ]
18 | },
19 | {
20 | "cell_type": "markdown",
21 | "metadata": {
22 | "id": "bbMNpP192mb2"
23 | },
24 | "source": [
25 | "# Importing Required Libraries"
26 | ]
27 | },
28 | {
29 | "cell_type": "code",
30 | "execution_count": 2,
31 | "metadata": {
32 | "id": "slw2Y5t3taWQ"
33 | },
34 | "outputs": [],
35 | "source": [
36 | "import torch\n",
37 | "from torchtext import data, datasets\n",
38 | "import torch.nn as nn\n",
39 | "import torch.optim as optim\n",
40 | "import torch.nn.functional as F\n",
41 | "from torchtext.data import Field, LabelField, BucketIterator\n",
42 | "\n",
43 | "import random\n",
44 | "\n",
45 | "import matplotlib.pyplot as plt"
46 | ]
47 | },
48 | {
49 | "cell_type": "code",
50 | "execution_count": 3,
51 | "metadata": {
52 | "colab": {
53 | "base_uri": "https://localhost:8080/"
54 | },
55 | "id": "7kAMYpKr2tJV",
56 | "outputId": "5d7e7127-e114-496d-c1f0-08a57498694c"
57 | },
58 | "outputs": [
59 | {
60 | "name": "stdout",
61 | "output_type": "stream",
62 | "text": [
63 | "Notebook is running on cuda\n"
64 | ]
65 | }
66 | ],
67 | "source": [
68 | "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
69 | "print(\"Notebook is running on\", device)"
70 | ]
71 | },
72 | {
73 | "cell_type": "markdown",
74 | "metadata": {
75 | "id": "8ZOcRV2w2vpH"
76 | },
77 | "source": [
78 | "Fixing SEED for reproducibility of results"
79 | ]
80 | },
81 | {
82 | "cell_type": "code",
83 | "execution_count": 4,
84 | "metadata": {
85 | "id": "N99mW-8XtnWr"
86 | },
87 | "outputs": [],
88 | "source": [
89 | "SEED = 4444\n",
90 | "\n",
91 | "random.seed(SEED)\n",
92 | "torch.manual_seed(SEED)\n",
93 | "torch.cuda.manual_seed(SEED)\n",
94 | "torch.backends.cudnn.deterministic = True"
95 | ]
96 | },
97 | {
98 | "cell_type": "markdown",
99 | "metadata": {
100 | "id": "sMxBvhY8y0wq"
101 | },
102 | "source": [
103 | "Generate NGRAMS for FastText"
104 | ]
105 | },
106 | {
107 | "cell_type": "code",
108 | "execution_count": 5,
109 | "metadata": {
110 | "id": "jk054NHay0fN"
111 | },
112 | "outputs": [],
113 | "source": [
114 | "def generate_n_grams(x, n=2):\n",
115 | " n_grams = set(zip(*[x[i:] for i in range(n)]))\n",
116 | " for n_gram in n_grams:\n",
117 | " x.append(' '.join(n_gram))\n",
118 | " return list(set(x))"
119 | ]
120 | },
121 | {
122 | "cell_type": "code",
123 | "execution_count": 6,
124 | "metadata": {
125 | "id": "O5NaQke7touV"
126 | },
127 | "outputs": [],
128 | "source": [
129 | "FIELD = Field(tokenize = 'spacy',tokenizer_language = 'en_core_web_sm', preprocessing = generate_n_grams)\n",
130 | "\n",
131 | "LABEL = LabelField(dtype = torch.float)"
132 | ]
133 | },
134 | {
135 | "cell_type": "markdown",
136 | "metadata": {
137 | "id": "RWXCd0V73YK7"
138 | },
139 | "source": [
140 | "# Splitting the data"
141 | ]
142 | },
143 | {
144 | "cell_type": "code",
145 | "execution_count": 7,
146 | "metadata": {
147 | "colab": {
148 | "base_uri": "https://localhost:8080/"
149 | },
150 | "id": "79ih2w2sttNq",
151 | "outputId": "b33d07a7-3983-4b49-a81d-db6f2ca0b908"
152 | },
153 | "outputs": [],
154 | "source": [
155 | "train_data, test_data = datasets.IMDB.splits(FIELD, LABEL)"
156 | ]
157 | },
158 | {
159 | "cell_type": "code",
160 | "execution_count": 8,
161 | "metadata": {
162 | "id": "nxpdowpRtwNQ"
163 | },
164 | "outputs": [],
165 | "source": [
166 | "train_data, valid_data = train_data.split(random_state = random.seed(SEED))"
167 | ]
168 | },
169 | {
170 | "cell_type": "code",
171 | "execution_count": 9,
172 | "metadata": {
173 | "colab": {
174 | "base_uri": "https://localhost:8080/"
175 | },
176 | "id": "BSWt_wDwtzLt",
177 | "outputId": "59e4e52f-b3d7-4300-ef8f-876519cd0092"
178 | },
179 | "outputs": [
180 | {
181 | "name": "stdout",
182 | "output_type": "stream",
183 | "text": [
184 | "Number of training examples: 17500\n",
185 | "Number of validation examples: 7500\n",
186 | "Number of testing examples: 25000\n"
187 | ]
188 | }
189 | ],
190 | "source": [
191 | "print(f'Number of training examples: {len(train_data)}')\n",
192 | "print(f'Number of validation examples: {len(valid_data)}')\n",
193 | "print(f'Number of testing examples: {len(test_data)}')"
194 | ]
195 | },
196 | {
197 | "cell_type": "code",
198 | "execution_count": 10,
199 | "metadata": {
200 | "colab": {
201 | "base_uri": "https://localhost:8080/"
202 | },
203 | "id": "trFz-SmNt0fO",
204 | "outputId": "6535fa21-210f-49a5-c8ae-01e80e80b092"
205 | },
206 | "outputs": [],
207 | "source": [
208 | "MAX_VOCAB_SIZE = 25000 # excluding and token\n",
209 | "\n",
210 | "FIELD.build_vocab(train_data, max_size = MAX_VOCAB_SIZE, vectors=\"glove.6B.100d\", unk_init = torch.Tensor.normal_)\n",
211 | "LABEL.build_vocab(train_data)"
212 | ]
213 | },
214 | {
215 | "cell_type": "code",
216 | "execution_count": 11,
217 | "metadata": {
218 | "colab": {
219 | "base_uri": "https://localhost:8080/"
220 | },
221 | "id": "PQUUTTgat5hg",
222 | "outputId": "965f4522-3b08-4866-bb7d-f0a32c2c5649"
223 | },
224 | "outputs": [
225 | {
226 | "name": "stdout",
227 | "output_type": "stream",
228 | "text": [
229 | "Unique tokens in FIELD vocabulary: 25002\n",
230 | "Unique tokens in LABEL vocabulary: 2\n"
231 | ]
232 | }
233 | ],
234 | "source": [
235 | "print(f\"Unique tokens in FIELD vocabulary: {len(FIELD.vocab)}\")\n",
236 | "print(f\"Unique tokens in LABEL vocabulary: {len(LABEL.vocab)}\")"
237 | ]
238 | },
239 | {
240 | "cell_type": "markdown",
241 | "metadata": {
242 | "id": "BIZMVoIA4S2K"
243 | },
244 | "source": [
245 | "# Model Definition"
246 | ]
247 | },
248 | {
249 | "cell_type": "code",
250 | "execution_count": 12,
251 | "metadata": {
252 | "id": "egyx2GtzuEdP"
253 | },
254 | "outputs": [],
255 | "source": [
256 | "class FastText(nn.Module):\n",
257 | " def __init__(self, vocab_size, emb_dim, output_dim, pad_idx):\n",
258 | " super().__init__()\n",
259 | " self.embedding = nn.Embedding(vocab_size, emb_dim, padding_idx=pad_idx)\n",
260 | " self.fc = nn.Linear(emb_dim, output_dim)\n",
261 | " \n",
262 | " def forward(self, input): # [input] = [input_length, batch_size]\n",
263 | " embedded = self.embedding(input) # [embedded] = [seq_len, batch_size, emb_dim]\n",
264 | " embedded = embedded.permute(1, 0, 2) # [embedded] = [batch_size, seq_len, emb_dim]\n",
265 | " pooled = F.avg_pool2d(embedded, (embedded.shape[1], 1)) # [pooled] = [batch_size, 1, emb_dim] \n",
266 | " pooled = pooled.squeeze(1) # [pooled] = [batch size, embedding_dim]\n",
267 | " return self.fc(pooled)"
268 | ]
269 | },
270 | {
271 | "cell_type": "code",
272 | "execution_count": 13,
273 | "metadata": {
274 | "id": "2PDdgoXiufHr"
275 | },
276 | "outputs": [],
277 | "source": [
278 | "def batch_accuracy(preds, y):\n",
279 | " #round predictions to the closest integer\n",
280 | " rounded_preds = torch.round(torch.sigmoid(preds))\n",
281 | " correct = (rounded_preds == y).float()\n",
282 | " acc = correct.sum() / len(correct)\n",
283 | " return acc"
284 | ]
285 | },
286 | {
287 | "cell_type": "markdown",
288 | "metadata": {
289 | "id": "OpnqXXA1uvuD"
290 | },
291 | "source": [
292 | "# Training and Evaluation Functions"
293 | ]
294 | },
295 | {
296 | "cell_type": "code",
297 | "execution_count": 14,
298 | "metadata": {
299 | "id": "gKSFO3hRutOc"
300 | },
301 | "outputs": [],
302 | "source": [
303 | "def Train(model, iterator, optimizer, criterion):\n",
304 | " epoch_loss = 0\n",
305 | " epoch_acc = 0\n",
306 | " model.train()\n",
307 | " for batch in iterator:\n",
308 | " optimizer.zero_grad()\n",
309 | " inp = batch.text\n",
310 | " label = batch.label \n",
311 | " predictions = model(inp).squeeze(1)\n",
312 | " loss = criterion(predictions, label)\n",
313 | " acc = batch_accuracy(predictions, label)\n",
314 | " loss.backward()\n",
315 | " optimizer.step()\n",
316 | " epoch_loss += loss.item()\n",
317 | " epoch_acc += acc.item() \n",
318 | " return epoch_loss / len(iterator), epoch_acc / len(iterator)"
319 | ]
320 | },
321 | {
322 | "cell_type": "code",
323 | "execution_count": 15,
324 | "metadata": {
325 | "id": "YqTp93Aruwoe"
326 | },
327 | "outputs": [],
328 | "source": [
329 | "def Evaluate(model, iterator, criterion):\n",
330 | " epoch_loss = 0\n",
331 | " epoch_acc = 0\n",
332 | " model.eval()\n",
333 | " with torch.no_grad():\n",
334 | " for batch in iterator: \n",
335 | " inp = batch.text\n",
336 | " label = batch.label \n",
337 | " predictions = model(inp).squeeze(1)\n",
338 | " loss = criterion(predictions, label)\n",
339 | " acc = batch_accuracy(predictions, label)\n",
340 | " epoch_loss += loss.item()\n",
341 | " epoch_acc += acc.item() \n",
342 | " return epoch_loss / len(iterator), epoch_acc / len(iterator)"
343 | ]
344 | },
345 | {
346 | "cell_type": "markdown",
347 | "metadata": {
348 | "id": "v1_3yX3-42vO"
349 | },
350 | "source": [
351 | "# Data Iterators, Hyperparameters and Model Initialization"
352 | ]
353 | },
354 | {
355 | "cell_type": "code",
356 | "execution_count": 16,
357 | "metadata": {
358 | "id": "yIHX-jbs5ClS"
359 | },
360 | "outputs": [],
361 | "source": [
362 | "BATCH_SIZE = 64\n",
363 | "\n",
364 | "train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits((train_data, valid_data, test_data), batch_size = BATCH_SIZE, device = device)"
365 | ]
366 | },
367 | {
368 | "cell_type": "code",
369 | "execution_count": 17,
370 | "metadata": {
371 | "id": "o6c7ld9wuYh4"
372 | },
373 | "outputs": [],
374 | "source": [
375 | "VOCAB_SIZE = len(FIELD.vocab) # dimension of one-hot vector / vocabulary\n",
376 | "EMB_DIM = 100 # dimension of word embeddings\n",
377 | "OUTPUT_DIM = 1 # dimension of output layer\n",
378 | "\n",
379 | "NUM_EPOCHS = 10\n",
380 | "LR = 0.001"
381 | ]
382 | },
383 | {
384 | "cell_type": "code",
385 | "execution_count": 18,
386 | "metadata": {
387 | "id": "j5U3vbZcwEgr"
388 | },
389 | "outputs": [],
390 | "source": [
391 | "model = FastText(VOCAB_SIZE, EMB_DIM, OUTPUT_DIM, FIELD.vocab.stoi[FIELD.pad_token])"
392 | ]
393 | },
394 | {
395 | "cell_type": "code",
396 | "execution_count": 19,
397 | "metadata": {
398 | "id": "ziglTF1D5voa"
399 | },
400 | "outputs": [],
401 | "source": [
402 | "optimizer = optim.Adam(model.parameters(), lr=LR)\n",
403 | "\n",
404 | "criterion = nn.BCEWithLogitsLoss()\n",
405 | "\n",
406 | "model = model.to(device)\n",
407 | "criterion = criterion.to(device)"
408 | ]
409 | },
410 | {
411 | "cell_type": "code",
412 | "execution_count": 20,
413 | "metadata": {
414 | "colab": {
415 | "base_uri": "https://localhost:8080/"
416 | },
417 | "id": "X7Mm3BnuuZ2F",
418 | "outputId": "1c65ec04-2ac8-4f1d-fb03-08ef3b8f4114"
419 | },
420 | "outputs": [
421 | {
422 | "name": "stdout",
423 | "output_type": "stream",
424 | "text": [
425 | "The model has 2,500,301 trainable parameters\n"
426 | ]
427 | }
428 | ],
429 | "source": [
430 | "def count_parameters(model):\n",
431 | " return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
432 | "\n",
433 | "print(f'The model has {count_parameters(model):,} trainable parameters')"
434 | ]
435 | },
436 | {
437 | "cell_type": "markdown",
438 | "metadata": {
439 | "id": "99qvK1qw5aAV"
440 | },
441 | "source": [
442 | "# Training"
443 | ]
444 | },
445 | {
446 | "cell_type": "code",
447 | "execution_count": 21,
448 | "metadata": {
449 | "id": "WT1J71IjuySn"
450 | },
451 | "outputs": [],
452 | "source": [
453 | "import time\n",
454 | "\n",
455 | "def Epoch_time(start_time, end_time):\n",
456 | " elapsed_time = end_time - start_time\n",
457 | " elapsed_mins = int(elapsed_time / 60)\n",
458 | " elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
459 | " return elapsed_mins, elapsed_secs"
460 | ]
461 | },
462 | {
463 | "cell_type": "code",
464 | "execution_count": 22,
465 | "metadata": {
466 | "colab": {
467 | "base_uri": "https://localhost:8080/"
468 | },
469 | "id": "TKDOAVH5uz2s",
470 | "outputId": "aafc2306-93a3-4ee0-8678-4e8a276ea1e1"
471 | },
472 | "outputs": [
473 | {
474 | "name": "stdout",
475 | "output_type": "stream",
476 | "text": [
477 | "Learning Rate: 0.001\n",
478 | "Time taken for epoch 1: 0m 9s\n",
479 | "Training Loss: 0.6900 | Validation Loss: 0.6643\n",
480 | "Training Accuracy: 53.88 %| Validation Accuracy: 63.70 %\n",
481 | "Time taken for epoch 2: 0m 8s\n",
482 | "Training Loss: 0.6689 | Validation Loss: 0.5680\n",
483 | "Training Accuracy: 70.73 %| Validation Accuracy: 75.88 %\n",
484 | "Time taken for epoch 3: 0m 8s\n",
485 | "Training Loss: 0.6161 | Validation Loss: 0.4792\n",
486 | "Training Accuracy: 78.73 %| Validation Accuracy: 78.10 %\n",
487 | "Time taken for epoch 4: 0m 8s\n",
488 | "Training Loss: 0.5431 | Validation Loss: 0.3829\n",
489 | "Training Accuracy: 83.51 %| Validation Accuracy: 83.00 %\n",
490 | "Time taken for epoch 5: 0m 8s\n",
491 | "Training Loss: 0.4712 | Validation Loss: 0.3528\n",
492 | "Training Accuracy: 86.64 %| Validation Accuracy: 85.13 %\n",
493 | "Time taken for epoch 6: 0m 8s\n",
494 | "Training Loss: 0.4132 | Validation Loss: 0.3434\n",
495 | "Training Accuracy: 88.35 %| Validation Accuracy: 86.67 %\n",
496 | "Time taken for epoch 7: 0m 8s\n",
497 | "Training Loss: 0.3627 | Validation Loss: 0.3479\n",
498 | "Training Accuracy: 89.37 %| Validation Accuracy: 87.30 %\n",
499 | "Time taken for epoch 8: 0m 8s\n",
500 | "Training Loss: 0.3263 | Validation Loss: 0.3536\n",
501 | "Training Accuracy: 90.39 %| Validation Accuracy: 87.87 %\n",
502 | "Time taken for epoch 9: 0m 8s\n",
503 | "Training Loss: 0.2976 | Validation Loss: 0.3649\n",
504 | "Training Accuracy: 91.05 %| Validation Accuracy: 88.28 %\n",
505 | "Time taken for epoch 10: 0m 8s\n",
506 | "Training Loss: 0.2725 | Validation Loss: 0.3771\n",
507 | "Training Accuracy: 91.55 %| Validation Accuracy: 88.56 %\n",
508 | "Model with Train Loss 0.4132, Validation Loss: 0.3434 was saved.\n"
509 | ]
510 | }
511 | ],
512 | "source": [
513 | "print(f\"Learning Rate: {LR}\")\n",
514 | "train_losses = []\n",
515 | "valid_losses = []\n",
516 | "min_losses = [float('inf'), float('inf')]\n",
517 | "\n",
518 | "start_time = time.time()\n",
519 | "for epoch in range(1, NUM_EPOCHS+1):\n",
520 | " \n",
521 | " train_loss, train_acc = Train(model, train_iterator, optimizer, criterion)\n",
522 | " train_losses.append(train_loss)\n",
523 | " valid_loss, valid_acc = Evaluate(model, valid_iterator, criterion)\n",
524 | " valid_losses.append(valid_loss)\n",
525 | "\n",
526 | " if valid_loss < min_losses[0]:\n",
527 | " min_losses[0] = valid_loss\n",
528 | " min_losses[1] = train_loss\n",
529 | " torch.save(model.state_dict(), 'FastText.pt')\n",
530 | "\n",
531 | " elapsed_time = Epoch_time(start_time, time.time())\n",
532 | " print(f\"Time taken for epoch {epoch}: {elapsed_time[0]}m {elapsed_time[1]}s\")\n",
533 | " start_time = time.time()\n",
534 | " print(f\"Training Loss: {train_loss:.4f} | Validation Loss: {valid_loss:.4f}\")\n",
535 | " print(f\"Training Accuracy: {train_acc*100:.2f} %| Validation Accuracy: {valid_acc*100:.2f} %\")\n",
536 | "\n",
537 | "print(f\"Model with Train Loss {min_losses[1]:.4f}, Validation Loss: {min_losses[0]:.4f} was saved.\")"
538 | ]
539 | },
540 | {
541 | "cell_type": "code",
542 | "execution_count": 23,
543 | "metadata": {
544 | "colab": {
545 | "base_uri": "https://localhost:8080/",
546 | "height": 295
547 | },
548 | "id": "OkjBpvPt8BmJ",
549 | "outputId": "04320b26-1bc8-44d2-c904-17650335ee40"
550 | },
551 | "outputs": [
552 | {
553 | "data": {
554 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABKdElEQVR4nO3dd3wUdfrA8c+TTkmooZfQQpESOgoIiAUBQSkKioqcWE7Fcra731lPT+/snBUVu4CCAiqKoiKI0oKA0pHeCb2T8vz+mAksIWUJ2cwm+7xfr3lld3Z25tnZzTzz/X5nvl9RVYwxxoSuMK8DMMYY4y1LBMYYE+IsERhjTIizRGCMMSHOEoExxoQ4SwTGGBPiLBEECRF5XUQe8jqOwiYij4rIh2e5jnzvOxEZIyKXn832g4mITBCRS72O40yF6u8/WFgiyIWIdBKRX0Rkn4jsFpFZItK2ANY7VER+9p2nqreo6r/Odt35iMXvA7GITBeRPSISHei4zkR+952INAdaAJPc50NFJF1EDvpML+c3LhFZJyIXuo87+6zzkIholu3Uysf6s/vu/gM8cQbr6Coim8502wUtkL9/EYly99Uqd9+vE5HRIpIQiO0VRZYIciAiccCXwP+A8kB14DHgmJdxecX9p+kMKNDH22gKzM3AR3rqXZW/qmppn+n2gtiQqs7MXCdwjju7rM92NhTQduYCcSLSpiDWVxBEJMLjEMbj/GavBsrgJP9koPuZrigIPktgqKpN2UxAG2BvHssMA5YBe4CpQG2f1xS4BVgF7AVeAQRoDBwF0oGDmdsA3gWecB93BTYB9wM7gK3A5UBPYCWwG/iHz7bCgAeBP4FdwCdAefe1BDeW64ENQArwf+5rPYDjQKoby6JcPuvDwCzgeeDLLK+9636+r4ADwBygns/rLwEbgf04/4CdfV57FPjQffwVcEeWdS8GrnD33Qvu/tgP/A40zWbfVcRJ4Hvd/TQTCMvhM60BOvk8Hwr8nM1yN7jf8wH3PTf7vJbt9oAPgAzgiLtv7/d5T+Z3EuE+LwO87X7Pm3HO6MOBKGBh5j5x581yv4scvzvgTeARP3/nXYFNObxWDZgA7ATWAiN8XmsH/Op+7q3Ay0BUlt//bTi//7Wc/E3/jZO/6Ruy/Iay/v5zWrYC8IX7O5jn7q/Tvjd32Qvd76BmLvtgHXBhDr/JzO/qLzj/PzOAr4Hbs6xjEdDPfdwI+M79PawArvRZriew1P0tbQbu9eoYd0r8XgcQrBMQh3NQfQ+4FCiX5fW+wGqcA3sE8E/gF5/XFecAURao5f4z9XBfG5r1h5vNP0Ka+w8fCQx33/8xEItzRnkEqOMufycwG6gBRANvAGOy/JDfBErgnA0dAxpn/dHnsT9WA38FWuMcfCpniX0XzsEhAvgIGOvz+hD3nzfC/efeBsRk3T5wJTDH530t3PVGAZfgJJGynEyoVbPZd08Br7v7LRKnFCPZfJ5S7n6J95l32vfizu8F1HO32wU4DLTKa3tkOcD4rC/zO8lMBJ+731kpoBIwFzfZAE1xTjQaA//nfs/huX13wD3AZz7PFwNX5/C9diWbRICTzJJxfoNRQF2cJHiJ+3proIP7nSbgJMq7svz+v8MpTZfg5G/6cXc/9XT3Y7lcfv85LTvWnUoCTXBOMnJKBE8DP+Xx2z7leyL7RPC++/2UAK4DZvks3wQnIUa7y2zEOXmIAFrinHw1cZfdinsiBJTD/R15PVnVUA5UdT/QiZMH0Z0iMllEKruL3AI8parLVDUN+DeQJCK1fVbztKruVafY/yOQdAYhpAJPqmoqzo++IvCSqh5Q1SU4ZxUtfGL5P1XdpKrHcH7IA7IUYx9T1SOqugjn7KUFfhKRTkBt4BNVTcYpeVydZbHPVXWuuy8+8v2sqvqhqu5S1TRVfQ7nH6ZhNpuaDCSKSAP3+bXAOFXNPPONxTnbEne/b81mHalAVZzSWao6VTLZdahV1v17IMv8DiKy12fqoKpfqeqf6vgJ+BbngH8m28uW+3vqiXMQPaSqO3BKPoMAVPUPnDPeicC9wLWqmp7Hag/4fD5UtbmqfuxvTK62OEnycVU9rqprcP4PMuNKVtXZ7ne6DieRdcmyjqdUdbeqHnGfpwKPu/tpCk5JJrvfQY7Likg40B+nxHNYVZfinKzlpALOwfdsPep+P0dwErfv//o1OIn3GNAbWKeq77j75jecUtVAn8/VRETiVHWPqi4ogNjOmiWCXLgHm6GqWgPnzKwa8KL7cm3gpcwDBk4xUHDaEjJt83l8GCh9Bpvf5fMPn/mPtN3n9SM+66sNfO4TyzKcqqfKPsufTSzXA9+qaor7/GN3nq8c1y8i94rIMrfRfS9OVUjFrBtR1aPAOGCIiIQBg3GqWFDVH3CqH14BdojIKLcdJ6tncEov34rIGhF5MIfPtNf9G5tl/mxVLeszzRaRS0VktnvBwF6cA3dm/P5uLye1cc56t/p8f2/glAwyvecuN0VVV/mxzlifz5dftYFqvkkR+Afub0pEEkXkSxHZJiL7cU6Esn6nG7M83+WeKGTK7XeY07LxOGfavuvOup1T1oOTqM/WiW2o6gGcasxB7qzBOCc/4Oy39ln22zVAFff1/ji/n/Ui8pOInFsAsZ01SwR+UtXlOMXXpu6sjTjFd9+DRglV/cWf1RVweBuBS7PEEqOqm882FhEpgVNl08X9p98G3A20EJE8SxUi0hmnreNKnKJ9WWAfTtLMzns4/zjdgcOq+uuJQFVHqmprnKJ4InDfaR/GKTH9TVXr4jQQ3iMipzUKquohnJJNYh7xR+Oc0T2LUx1WFpiSGX8e2/Pne96IU1VX0ee7i1PVc3yWeRWnmvESt3R24mPksM7GOKW+s7ERWJvlNxWrqj3d118DlgMNVDUOJ0lk/U4L+ncOThVpGk41aKaauSw/DWgnIjVyWeYQTjVTpirZLJP1s4wBBrsH8hicEj84++2nLPuttKreCqCq81S1L06in4jTnuc5SwQ5EJFGIvK3zB+QiNTEyfyz3UVeB/4uIue4r5cRkYHZr+0024EaIhJVQOG+DjyZWVQVkXgR6XsGsSS4Z+DZuRyndNEEp7onCedAMxOnrjQvsTj/uDuBCBF5GKf9JVvugT8DeA63NAAgIm1FpL2IROL84x51lzuFiPQWkfoiIjgJJz275VxTOL06I6sonKqsnUCaONfoX+zn9rbj1K3nyK3e+hZ4TkTiRCRMROqJSBd3/dfi1McPBUYA74lI5ll0Tt9dF5wGTb+JSIzvhNNOcUBEHhCREiISLiJN5eTl07E4jbUHRaQRcOuZbC+/3FLyZ8CjIlLS3XaOv0NVnYbTVvG5iLQWkQgRiRWRW0RkmLvYQmCQiES6V1sN8COUKThn/4/jVF9mfudf4lRvXuuuL9L97TYW5zLWa0SkjDpVvvvJ+bdZqCwR5OwA0B6YIyKHcBLAHziNnajq5zjXbI91i8Z/4DQq++MHYAmwTURS8lrYDy/h1K9/KyIH3Fjb+/neT92/u0Qku/rK64F3VHWDqm7LnHCqaa7x43K6qcA3OFc7rcc5gOdWlAenYa4Z4HuNfBxOHfUedz27cKplsmqAcxZ4EOeqlldV9cdslgMY5X6GnEonmdUAI3DO3PbgtI1M9nN7TwH/dKsI7s3x0zoHsiicdp89OJc7VhXn3oIXgetU9aBbzz8fpw0Bsvnu3AP1QXUuI8Wdt0RErsll+9Vxqhp9pzo49d1JOFf9pABv4VTrgdNecTXO/8mbOFV6heV2N45tOCcLY8j9su4BOAfucTjJ+g+cqwKnua8/hHMxwB6cS8TzbE9x2wM+w7kq6WOf+QdwThQGAVvcGP+DczIBTrvXOveYcQtO6ddzmVc3GBM0ROQ64CZV7ZTnwme/rY9xGsEnBnpbhUFEJgBvuw2sIUFE/gNUUdWs7VbGT5YITFARkZI4JaZXVfV9r+MxwcetDorCuZekLc7Z/o3FJZl7waqGTNAQkUtw6uK340fx3ISsWJxqmUM41T3P4XYTYvLHSgTGGBPirERgjDEhrsh1oFSxYkVNSEjwOgxjjClSkpOTU1Q1PrvXilwiSEhIYP78+V6HYYwxRYqIrM/ptYBWDYlIDxFZISKrs7v1XkReEJGF7rTSvR3bGGNMIQpYiUCczqFeAS7C6VJ2nohMdjuJAkBV7/ZZ/g6cnvqMMcYUokCWCNoBq1V1jTq9R47F6bo5J4Nx7hA0xhhTiALZRlCdU7sS2EQO3R64feTUwbmRKLvXbwJuAqhV64xH9DPG5ENqaiqbNm3i6NGjXodizkBMTAw1atQgMjLS7/cES2PxIGB8Tv2sq+oonH5haNOmjd34YEwh2LRpE7GxsSQkJJBLd0wmiKgqu3btYtOmTdSpU8fv9wWyamgzp3YPW8Odl51BWLWQMUHl6NGjVKhQwZJAESIiVKhQ4YxLcYFMBPOABiJSx+1ueRCn9toInOg3pBxOz43GmCBiSaDoyc93FrBE4I4udDtON8TLcHp4XCIij4tIH59FB+GMbxvQKp+FG/fy8g+rWLplP9athjHGnBTQ+whUdYqqJqpqPVV90p33sKpO9lnmUVU90+H9zticNbt49tuV9Bw5k/Oe/oG/f/Y705Zu5/DxtLzfbIwpdLt27SIpKYmkpCSqVKlC9erVTzw/fvx4ru+dP38+I0aMyHMb5513XoHEOn36dHr37l0g6/JCsDQWB9zNXepxRavqTF+xkx+X72Dyws2MmbuBqIgwzq1bge6NK9GtYSVqli+Z98qMMQFXoUIFFi5cCMCjjz5K6dKluffek+P7pKWlERGR/SGsTZs2tGnTJs9t/PKLPyPLFn8h1elcpdgYrmxTk9eGtOa3hy/moxvbM6R9bTbsPszDk5bQ+b8/ctHzP/HU18uYs2YXaelBMYqcMcY1dOhQbrnlFtq3b8/999/P3LlzOffcc2nZsiXnnXceK1asAE49Q3/00UcZNmwYXbt2pW7duowcOfLE+kqXLn1i+a5duzJgwAAaNWrENddcc6IKecqUKTRq1IjWrVszYsSIMzrzHzNmDM2aNaNp06Y88MADAKSnpzN06FCaNm1Ks2bNeOEFZ8C5kSNH0qRJE5o3b86gQYPOfmedgZApEWQVFRFGx/oV6Vi/Ig9f1oQ1Ow/yw/Id/LhiB6N/XssbP60hLiaCLg0rcUGjeLokVqJ8qYIaYtiYouWxL5awdMv+Al1nk2pxPHLZOWf8vk2bNvHLL78QHh7O/v37mTlzJhEREUybNo1//OMfTJgw4bT3LF++nB9//JEDBw7QsGFDbr311tOus//tt99YsmQJ1apVo2PHjsyaNYs2bdpw8803M2PGDOrUqcPgwYP9jnPLli088MADJCcnU65cOS6++GImTpxIzZo12bx5M3/88QcAe/fuBeDpp59m7dq1REdHn5hXWEI2EWRVN740deNLc2Pnuhw4msrPq1JOJIYvFm0hTKBlrXJc0KgSFzSqRKMqsXZFhTEeGDhwIOHh4QDs27eP66+/nlWrViEipKamZvueXr16ER0dTXR0NJUqVWL79u3UqFHjlGXatWt3Yl5SUhLr1q2jdOnS1K1b98Q1+YMHD2bUqFF+xTlv3jy6du1KfLzT4ec111zDjBkzeOihh1izZg133HEHvXr14uKLLwagefPmXHPNNVx++eVcfvnlZ7xfzoYlgmzExkRyabOqXNqsKhkZyu+b9/HD8h38sHwHz0xdwTNTV1CtTAzd3KRwXr2KlIgK9zpsYwImP2fugVKqVKkTjx966CG6devG559/zrp16+jatWu274mOjj7xODw8nLS00y8S8WeZglCuXDkWLVrE1KlTef311/nkk08YPXo0X331FTNmzOCLL77gySef5Pfff8+xDaSgWSLIQ1iY0KJmWVrULMvdFyWyY/9Rpq/YyffLtzPxt818NGcD0RFhnFuvAt0bVaJbo0rUKGcNzsYUhn379lG9enUA3n333QJff8OGDVmzZg3r1q0jISGBcePG+f3edu3aMWLECFJSUihXrhxjxozhjjvuICUlhaioKPr370/Dhg0ZMmQIGRkZbNy4kW7dutGpUyfGjh3LwYMHKVu2bIF/puxYIjhDleJiuLJtTa5sW5NjaenMXbv7RGnhoUlLYNISGlaOPVFaaFWrLBHhIdUmb0yhuf/++7n++ut54okn6NWrV4Gvv0SJErz66qv06NGDUqVK0bZt2xyX/f7770+pbvr00095+umn6datG6pKr1696Nu3L4sWLeKGG24gI8O5GOWpp54iPT2dIUOGsG/fPlSVESNGFFoSgCI4ZnGbNm00GAemUVXWpBzix+U7+H7ZDuat201ahlKmRCRdEuO5oFEluiTGU84anE0RsWzZMho3bux1GJ47ePAgpUuXRlW57bbbaNCgAXfffXfeb/RQdt+diCSrarbX1FqJoICICPXiS1PPbXDe7zY4f79sB9NX7GCy2+DcqlY5+rWqwaC2NQkLs8ZmY4Ldm2++yXvvvcfx48dp2bIlN998s9chFTgrERSCjAxl0aa9/Lh8B98t28Gyrfu5oFElnh3Ywi5JNUHLSgRF15mWCEKn8vrwbvjtI082HRYmtKxVjnsubsiUEZ14rM85/LwqhZ4vzWTOml2exGSMMZlCJxHMfg0m/RWWT/E0DBHh+vMS+Oyv5xETGcbgN2cz8vtVpGcUrZKZMab4CJ1EcP69ULUFTLwV9m7Me/kAa1q9DF+O6EyfFtV4/ruVXPv2HHbst5GgjDGFL3QSQUQ0DHgHMtJhwl8gPfs7EAtT6egIXrgqif8OaM6CDXu49KWZ/LRyp9dhGWNCTOgkAoAK9eCyF2HjHPjx315HAzhVRVe2qckXt3eiYulorh89l/98s5xU6/DOhLhu3boxderUU+a9+OKL3HrrrTm+p2vXrmReTNKzZ89s++x59NFHefbZZ3Pd9sSJE1m6dOmJ5w8//DDTpk07g+izF6zdVYdWIgBoNgBaXQ8/Pw+rz/6LLSgNKscy8baODG5Xi9em/8lVb/zKpj2HvQ7LGM8MHjyYsWPHnjJv7Nixfnf8NmXKlHzflJU1ETz++ONceOGF+VpXURB6iQCgx9MQ3xg+uxkObPM6mhNKRIXzVL9m/G9wS1ZuP0jPl2YydUnwxGdMYRowYABfffXViUFo1q1bx5YtW+jcuTO33norbdq04ZxzzuGRRx7J9v0JCQmkpKQA8OSTT5KYmEinTp1OdFUNzj0Cbdu2pUWLFvTv35/Dhw/zyy+/MHnyZO677z6SkpL4888/GTp0KOPHjwecO4hbtmxJs2bNGDZsGMeOHTuxvUceeYRWrVrRrFkzli9f7vdn9bq76tC8oSyqJAx8F0Z1hc+Gw7UTISx4Oo27rEU1mtcow+0f/8bNHyRz/bm1+XvPxsREBk+MJsR8/SBs+71g11mlGVz6dI4vly9fnnbt2vH111/Tt29fxo4dy5VXXomI8OSTT1K+fHnS09Pp3r07ixcvpnnz5tmuJzk5mbFjx7Jw4ULS0tJo1aoVrVu3BqBfv34MHz4cgH/+85+8/fbb3HHHHfTp04fevXszYMCAU9Z19OhRhg4dyvfff09iYiLXXXcdr732GnfddRcAFStWZMGCBbz66qs8++yzvPXWW3nuhmDorjo0SwQAlRpBr2dh7QyY+ZzX0ZymdoVSjL/1XIZ1rMN7v66n/2u/sDblkNdhGVOofKuHfKuFPvnkE1q1akXLli1ZsmTJKdU4Wc2cOZMrrriCkiVLEhcXR58+J4dM/+OPP+jcuTPNmjXjo48+YsmSJbnGs2LFCurUqUNiYiIA119/PTNmzDjxer9+/QBo3bo169at8+sz+nZXHRERcaK76rp1657orvqbb74hLi4OONld9YcfflhgvZOGZokgU9I1TiKY/hTUPg8SOnkd0SmiI8J5+LImnFevAveOX0TvkTP5d79m9E2q7nVoJtTkcuYeSH379uXuu+9mwYIFHD58mNatW7N27VqeffZZ5s2bR7ly5Rg6dChHj+bv0uuhQ4cyceJEWrRowbvvvsv06dPPKt7MrqwLohvrwuyuOnRLBAAi0Os5KF8XJtwIh1K8jihbFzapzJQRnWlcNY47xy7k/vGLOHw8MH2lGxNMSpcuTbdu3Rg2bNiJ0sD+/fspVaoUZcqUYfv27Xz99de5ruP8889n4sSJHDlyhAMHDvDFF1+ceO3AgQNUrVqV1NRUPvroZM8DsbGxHDhw4LR1NWzYkHXr1rF69WoAPvjgA7p06XJWn7Fdu3b89NNPpKSkkJ6ezpgxY+jSpQspKSlkZGTQv39/nnjiCRYsWHBKd9X/+c9/2LdvHwcPHjyr7UOolwgAomOd+wveuhA+vwWu/gTCgi8/VitbgrE3deDFaat4Zfpqftuwl5evbkXDKrFeh2ZMQA0ePJgrrrjiRBVRixYtaNmyJY0aNaJmzZp07Ngx1/e3atWKq666ihYtWlCpUqVTupL+17/+Rfv27YmPj6d9+/YnDv6DBg1i+PDhjBw58kQjMUBMTAzvvPMOAwcOJC0tjbZt23LLLbec0ecJxu6qrdO5THPfhCn3wkX/go4jCn79BejnVSncNW4hB46m8mifcxjUtqYNm2kKnHU6V3RZp3P51fZGaNwHvn8MNs7zOppcdWpQkSl3dqJtQnn+/tnv3DHmNw4c9f5OaWNM0WSJIJMI9PkfxFWD8cPgyB6vI8pVpdgY3h/WjvsuacjXf2yj18ifWbxpr9dhGWOKIEsEvkqUhQHvwoEtMOl2CPJqs7Aw4bZu9Rl7UwdS0zPo/9ovvP3zWopadZ8JXvZbKnry851ZIsiqRmu48DFY/qXTblAEtE0oz5QRnemSWIl/fbmU4e8ns+fQca/DMkVcTEwMu3btsmRQhKgqu3btIiYm5ozeZ43F2VGFMYPgzx/gL99BtaTAbq+AqCrvzFrHU18vo2LpaEYObknbhPJeh2WKqNTUVDZt2pTva/SNN2JiYqhRowaRkZGnzM+tsdgSQU4O7YLXO0FkDNw8w7nMtIhYvGkvd4z5jU17jnDPRYnc2qWejY9sTIizq4byo1QFGPA27FkHX9wV9O0FvprXKMuXd3SiZ7OqPDN1BdeNnsuOA3ZWZ4zJniWC3NQ+D7r9A/4YD7994HU0ZyQ2JpKRg5J4ul8z5q3bTc+XfubnVcF557QxxluWCPLS6R6o2xWm3A/bc+7YKhiJCIPa1WLy7Z0oVzKSa0fP4dmpK0izQW+MMT4sEeQlLByuGOW0EYy/AY4XvcFiGlaJZdLtHRnYugYv/7iaQaNms2XvEa/DMsYECUsE/oitDP1Gwc4V8PX9XkeTLyWjIvjvgBa8NCiJZVv303PkTKav2OF1WMaYIGCJwF/1ukHnvzltBYs/8TqafOubVJ0vR3SmSlwMw9+fz4/LLRkYE+osEZyJrn+HWufCl3dDymqvo8m3OhVLMe6mc0msHMvNHyYzc9VOr0Myxngoz0QgIgNFJNZ9/E8R+UxEWgU+tCAUHgH934bwKBg/FFKL7iWZZUpG8uFf2lO3YilufG8+v/xpVxQZE6r8KRE8pKoHRKQTcCHwNvBaYMMKYmWqwxWvO+O3fvtPr6M5K+VKRfHRje2pVb4kf3l3PnPX7vY6JGOMB/xJBOnu317AKFX9CogKXEhFQOIlcO7tMO9NWDrZ62jOSoXS0Xw0vD1Vy8ZwwztzSV4f3L2uGmMKnj+JYLOIvAFcBUwRkWg/34eI9BCRFSKyWkQezGGZK0VkqYgsEZGP/Q/dY90fgeqtnV5K96zzOpqzUik2hjHDOxAfG83Q0XNZtHGv1yEZYwqRPwf0K4GpwCWquhcoD9yX15tEJBx4BbgUaAIMFpEmWZZpAPwd6Kiq5wB3nUnwnoqIggGjncfjh0Fa0e7ts3JcDB8P70DZUpFc+/Yc/ti8z+uQjDGFxJ9EUBX4SlVXiUhXYCAw14/3tQNWq+oaVT0OjAX6ZllmOPCKqu4BUNWidS1juQToMxI2J8MPj3sdzVmrVrYEH9/YgdiYSIa8PYdlW/d7HZIxphD4kwgmAOkiUh8YBdQE/KnCqQ5s9Hm+yZ3nKxFIFJFZIjJbRHpktyIRuUlE5ovI/J07g+xSx3Mud4a5/OV/sHKq19GctZrlS/Lx8PbERIQz5K05rNp+wOuQjDEB5k8iyFDVNKAf8D9VvQ+nlFAQIoAGQFdgMPCmiJTNupCqjlLVNqraJj4+voA2XYAufhIqN4PPb4F9m72O5qzVrlCKj4e3JyxMGPzmHP7cedDrkIwxAeRPIkgVkcHAdcCX7rzIXJbPtBmn9JCphjvP1yZgsqqmqupaYCVOYihaImNg4LuQdgwm3AjpaV5HdNbqxpdmzPD2gHL1m7NZl3LI65CMMQHiTyK4ATgXeFJV14pIHcCfPpnnAQ1EpI6IRAGDgKzXWk7EKQ0gIhVxqorW+Bd6kKlYH3q/ABt+gZ/+43U0BaJ+pVg+urEDx9MyuPrN2WzcXfQ63DPG5C3PRKCqS4F7gd9FpCmwSVXzPNK51Um341xxtAz4RFWXiMjjItLHXWwqsEtElgI/Avep6q58fhbvtbgKkobAjGdgzXSvoykQDavE8uGN7Tl0PJ3Bb85ms/Vaakyxk+dQle6VQu8B6wDBqe65XlVnBDi2bBXaUJX5dfwQjOoGR/bArbOgdCWvIyoQv2/ax9VvzaZcySg+uflcqpQ5s8GxjTHeOtuhKp8DLlbVLqp6PnAJ8EJBBlisRJVy2guO7YfPboKM4jEITLMaZXh/WDt2HzrO1W/OZsf+otvPkjHmVP4kgkhVXZH5RFVX4l9jceiq3AQu/S+s+RFmFZ+c2bJWOd69oS3b9h/l6rfmkHLwmNchGWMKgD+JYL6IvCUiXd3pTSCI62aCRKvroOkA+OFJWP+r19EUmDYJ5Rk9tC2b9hxmyFtz2H2oaN9RbYzxLxHcCiwFRrjTUuCWQAZVLIg4VxGVrQUT/gKHi0/Pnh3qVuDt69uyNuUQQ96aw97DlgyMKcr8uWromKo+r6r93OkFnCt8TF5i4pz2gkM7YeKtkEfDfFHSsX5FRl3XhtU7DnLt23PZdyTV65CMMfmU3xHKahVoFMVZtSS4+AlY+Q3MftXraApUl8R4XhvSiuXb9jP0nbkcOGrJwJiiKL+JoPic2haGdjdBo97w3SNOB3XFSPfGlXn56lb8vmkfw96dx6FjRf+uamNCTY73EYhIv5zeA7yuqp50+hP09xHk5MgeeP18p+3glpkQU8briArUV4u3cseYBbSrU553hrajRFS41yEZY3zk9z6Cy3KYenOyzyHjrxLlnPEL9m92BrMpRu0FAL2aV+WFq5KYu3Y3w9+fz9HU9LzfZIwJChE5vaCqNxRmICGhZlu48FFnrONZL0Knu72OqED1TapOarpy3/hF3PxBMqOua010hJUMjAl2+W0jMPl17u3QtD9MewxWTfM6mgI3oHUNnrqiGT+t3MltHy3geFrxuLPamOLMEkFhE4E+L0OVpjBhGOz60+uICtygdrX41+VNmbZsB3eMWUBquiUDY4KZJQIvRJWEqz4CCYexV8Ox4jcK2LUdavPIZU2YumQ7d41bSJolA2OCVp6JQESSReQ2ESlXGAGFjHK1YeA7kLLKGdmsmHRO5+uGjnX4v56N+WrxVu79dBHpGcWrgdyY4sKfEsFVQDVgnoiMFZFLREQCHFdoqNvVudls+Zcw8zmvowmI4efX5b5LGjJx4RYemLCYDEsGxgSdHK8ayqSqq4H/E5GHcC4dHY0zmP07wEuqWnw60fFCh1th6yL48Umn3aDhpV5HVOBu61af1PQMXpy2ishw4cnLmxEWZucSxgQLv9oIRKQ5zrgEzwATgIHAfuCHwIUWIkTgshehagtn/IKdK72OKCDu7N6A27rVY8zcjTz6xRLyGhDJGFN48iwRiEgysBd4G3hQVTM7oZ8jIh0DGFvoiCwBgz6CN7o4jcfDvy92dx6LCPde3JDUdGXUjDVEhIXxUO/GWC2jMd7LMxEAA1U12wHlVTWnbijMmSpTA658H97v45QMBo2BsOJ1UZeI8PdLG5GansHoWWuJDBcevLSRJQNjPObPkWafiIwUkQXuFUQviUiFgEcWihI6Qo+nnZ5Kpz/ldTQBISI83LsJQzrU4o0Za3j+u+JZFWZMUeJPiWAsMAPo7z6/BhgHXBiooEJa2xth60KY8V+o2hwaX+Z1RAVORHi8T1PS0pX//bCa8DDhrgsTvQ7LmJDlTyKoqqr/8nn+hIhcFaiAQp4I9HwOdix37i+oUB8qNfY6qgIXFib8+4pmpGUoL05bRZgII7o38DosY0KSP1VD34rIIBEJc6crgamBDiykRcbAVR9AVCmn8fjIHq8jCoiwMOE//ZvTv1UNnv9uJf/7fpXXIRkTkvxJBMOBj4Hj7jQWuFlEDojI/kAGF9LiqsGVH8DejTDhRsgont06h4cJ/x3QnH4tq/Pcdyt5+QdLBsYUNn/GLI5V1TBVjXCnMHderKrGFUaQIatWe+j5DKyeBj/8K+/li6jwMOGZgS24omV1nv12Ja/8uNrrkIwJKf60ESAifYDz3afTVdUGpiksbW5w7jz++QWo0hyaFs8rdsPDhGcHtkBVeWbqCkTgr13rex2WMSHBnxvKngbaAh+5s+4UkY6q+veARmZOuvS/sGMZTLoNKjaAKs28jiggwsOE565MQoH/frOCMBFu6VLP67CMKfb8aSPoCVykqqNVdTTQA+gV2LDMKSKinJvNYsrA2GvgcPHt3ik8THhuYAsua1GNp79ezhs/Fb/xGowJNv7eulrW53Hx6vugqIit7IxhcGArjL8B0tO8jihgIsLDeOHKFvRuXpWnvl7OmzOyvbHdGFNA/EkE/wZ+E5F3ReQ9IBl4MrBhmWzVaA29X4A102HaI15HE1AR4WG8eFUSvZpX5ckpy3hrpiUDYwIl1zYCEQkDMoAOOO0EAA+o6rZAB2Zy0HKI03j868tOj6XNr/Q6ooCJCA/jpauSQOGJr5YBcGPnut4GZUwxlGsiUNUMEblfVT8BJhdSTCYvl/wbti+ByXdAxUSoluR1RAETER7Gi4OSyFDlia+WESbCsE51vA7LmGLFn6qhaSJyr4jUFJHymVPAIzM5C4+Ege9ByYpO4/HBnV5HFFCR4WGMHNySHudU4fEvl/LOrLVeh2RMseLvUJW34XQ8l+xO8wMZlPFD6XgY9CEcToFPh0J6qtcRBVRkeBj/u7oll5xTmce+WMp7v6zzOiRjig1/EkFjVa3jOwFNAh2Y8UO1lnDZSFj/M3z7T6+jCbjI8DD+N7gVFzepzCOTl/D+r+u8DsmYYsGfRPCLn/OMF1pcBR1ugzmvw28f5b18ERcVEcbLV7fioiaVeXjSEj6wZGDMWcuxsVhEqgDVgRIi0hLIHEYqDihZCLEZf130OGz/Hb68G+IbOZeZFmNREWG8cnUr/vpRMg9NWoKIMKRDba/DMqbIyq1EcAnwLFADeB5n8PrngHuAfwQ+NOO38AgY8K5z09m4IXBgu9cRBVxURBivXNOK7o0q8c+Jf/DRnPVeh2RMkZVjIlDV91S1GzBUVbv5TH1U9TN/Vi4iPURkhYisFpEHs3l9qIjsFJGF7nTjWXyW0FaqAgz62Bm74JPrIO241xEFXHREOK8OacUFjSrxf5//wcdzNngdkjFFkj+9j34pIlcDCb7Lq+rjub1JRMKBV4CLgE3APBGZrKpLsyw6TlVvP6OoTfaqNIPLX4Hxw+CbB5y7kIu56IhwXhvSils+SOYfn/9OmMCgdrW8DsuYIsWfxuJJQF8gDTjkM+WlHbBaVdeoauaANn3zG6jxU9P+0PEumD8akt/1OppC4SSD1nRtGM+Dn/3OuHlWMjDmTPhTIqihqj3yse7qwEaf55uA9tks119EzgdWAner6sasC4jITcBNALVq2dlenro/DNt+h6/uhfjGzgA3xVxMZDivD2nNzR8k8+BnvyMIV7at6XVYxhQJfl0+KiKB6gD/CyBBVZsD3wHvZbeQqo5S1Taq2iY+Pj5AoRQjYeHQ/y0oUwM+uRb2b/U6okIRExnOG9e2pnODeB74bDGfzD/tnMIYkw1/EkEnINlt9F0sIr+LyGI/3rcZ8D0lq+HOO0FVd6nqMffpW0Dxvu6xMJUs7zQeHzvoXEmUdizv9xQDMZHhjLq2NZ3qV+SBCYsZn7zJ65CMCXr+JIJLgQbAxcBlQG/3b17mAQ1EpI6IRAGDyNJxnYhU9XnaB1jmT9DGT5WbwBWvw+b58NU9oOp1RIUiJjKcN69rQ6f6Fblv/CImWDIwJlc5JgIRuQBAVdcDYaq6PnPCjzN3VU0Dbgem4hzgP1HVJSLyuDsGMsAIEVkiIouAEcDQs/s45jRN+sD598FvH8K8t7yOptBkJoOO9Spy7/hFfLbAkoExORHN4SxRRBaoaqusj7N7XpjatGmj8+dbn3dnJCMDxg6G1dPgusmQ0NHriArNkePp/OW9efy6ZhfPX9mCK1rW8DokYzwhIsmq2ia713KrGpIcHmf33ASzsDDoNwrKJcCn18O+0Dk7LhEVztvXt6VDnQr87ZNFTFq4Oe83GRNicksEmsPj7J6bYBdTxmk8Tj3qjGGQesTriApNiahw3h7ahnZ1ynP3uIWWDIzJIrdEUFdEJovIFz6PM5/bEFFFUXxDp2SwdaHTQV2INB4DlIyKYPTQtrRNcJLB5EVbvA7JmKCR2w1lvncBP5vltazPTVHRqCd0/QdM/7cz5nGHW72OqNCUjIrgnRvaMvSdedw19jcEuKxFNa/DMsZzOSYCVf2pMAMxhej8+5xSwbcPQY12xb7bal8loyJ4Z2hbbnhnHneNW0iYCL2aV837jcYUY/7cR2CKm7Aw6PsKxFaB8TfA0X1eR1SoSkU7JYNWtcoyYuxvTPk9NO68NiYnlghCVcnyMGC0cwXR5DtCqr0AMpNBO1rWLMsdY37ja0sGJoSdUSIQkTARiQtUMKaQ1WzndFC3dJLTW2mIKR0dwbvD2pHkJoNP5lnfRCY05ZkIRORjEYkTkVLAH8BSEbkv8KGZQnHeCKh/IXzzd6fH0hBTOjqCd29oS4e6Fbh/wmIenbyE1PQMr8MyplD5UyJooqr7gcuBr3EuHb02kEGZQhQWBle84VQVfTrU6aQuxMTGRPLuDW35S6c6vPvLOq4fPZc9h4r/CG/GZPInEUSKSCROIpisqqnYDWXFS6mKTrfVu9eEVOd0viLCw3iodxOeHdiC+ev30OeVn1m+bb/XYRlTKPxJBG8A64BSwAwRqQ3Yf0hxk9AJujwIi8fBwo+9jsYzA1rXYNxNHTiWmkG/V3+xRmQTEvJMBKo6UlWrq2pPdawHuhVCbKawnX8vJHSGKffCjuVeR+OZlrXK8cUdnUisHMutHy3g+e9WkpEReqUkEzr8aSy+020sFhF5W0QWABcUQmymsGWObBZZ0rm/4PhhryPyTOW4GMbe1IH+rWow8vtV3PJhMgePpXkdljEB4U/V0DC3sfhioBxOQ/HTAY3KeCe2CvR7A3YshW8e9DoaT8VEhvPswOY83LsJ3y/fQb9XZ7F+1yGvwzKmwPmTCDK7nO4JfKCqS7BuqIu3+hdCp7thwXvw+3ivo/GUiDCsUx3eu6Ed2/cfo8/Ls/h5VYrXYRlToPxJBMki8i1OIpgqIrGAXWhd3HX7P6jZHr64E3b96XU0nuvUoCKTb+9I5bhorhs9h7d/XktOgzoZU9T4kwj+AjwItFXVw0AUcENAozLeC4+E/m9DWITTXpB2zOuIPFe7Qik++2tHLmxcmX99uZT7xi/maGq612EZc9b8uWooA6gB/FNEngXOU9XFAY/MeK9sTbj8Ndi6CL572OtogkLp6AheH9KaO7s3YHzyJgaNms32/Ue9DsuYs+LPVUNPA3cCS91phIj8O9CBmSDRqCd0+CvMeR2Wfel1NEEhLEy4+6JEXh/SipXbD3DZ/37mtw17vA7LmHzzp2qoJ3CRqo5W1dFAD6B3YMMyQeXCx6BaS5j0V9i7wetogkaPplX57K/nER0ZxlVvzGZ8cuiMBW2KF397Hy3r87hMAOIwwSwiyumyWhXGD4P0VK8jChqNqsQx+bZOtEkox72fLuLxL5aSZp3WmSLGn0Twb+A3EXlXRN4DkoEnAxuWCTrl68JlL8GmefDDv7yOJqiUKxXF+8PaMfS8BEbPWsvQd+ax97B1WmeKjlwTgYiE4Vwq2gH4DJgAnKuq4wohNhNsmvaDNsNg1kuw6juvowkqEeFhPNrnHP7bvzlz1+6mz8uzWLn9gNdhGeOXXBOBe8XQ/aq6VVUnu9O2QorNBKNL/g2Vm8LnN8P+LV5HE3SubFuTMTd14EhqOle8MoupS+zfxQQ/f6qGponIvSJSU0TKZ04Bj8wEp8gSMOAdSD0CE26EdOt/J6vWtcvxxe2dqF+pNDd/kMxL01ZZp3UmqPmTCK4CbgNm4LQPJAPzAxmUCXLxidDreVg/C2b81+toglKVMjGMu/lc+rWszgvTVnLbxws4ZJ3WmSAVkdcCqlqnMAIxRUzSYFg7A376L9TuCHW7eB1R0ImJDOe5K1vQpFoc/56yjLUph3jzujbULF/S69CMOUWOJQIRGSIipw1JKSLXisjVgQ3LFAk9n4GKDeCz4XBwp9fRBCUR4cbOdXn3hnZs2XuEPi//zC+rrdM6E1xyqxq6A/g8m/mfAX8LTDimSIku7bQXHN0Hn98EGXb9fE7OT4xn8u2dqFA6mmtHz+W9X9ZZp3UmaOSWCCJV9bSRzFX1EBAZuJBMkVKlKfR4Gv78AWa96HU0QS2hYik+/+t5dGtYiUcmL+HBCb9zLM06rTPeyy0RlBCRUllnut1QRwUuJFPktB4K5/SDH56ADbO9jiaoxcZEMura1oy4oD7j5m9k8KjZ7DhgndYZb+WWCN4GxruD1QMgIgnAWPc1Yxwizl3HZWvB+L/A4d1eRxTUwsKEey5uyCtXt2LZ1gP0+d8sFm3c63VYJoTlmAhU9VlgEjBDRHaJyC7gJ+BLVX2msAI0RURMHAx8Bw5uh4l/dfolMrnq1bwqE249j/AwYeAbv/L5b9ZpnfFGXncWv66qtYEEIEFVa6vqa4USmSl6qrWEi5+AlV/DbPuZ+KNJtTgm396RVrXKcve4RTz51VJSrdM6U8j86n1UVQ+oqnWcYvLW/mZo2MsZyGZzstfRFAkVSkfzwV/ac925tXlz5loGvP4r61IOeR2WCSH+dkNtjH9EoO/LULoyfHqDc2mpyVNkeBiP923KK1e3Yl3KIXqOnMnYuRvsElNTKCwRmIJXsrwzfsG+TfDFndZecAZ6Na/KN3d1JqlmWR787Hdu/iCZ3YesS2sTWP4MVZksIreJSLkzXbmI9BCRFSKyWkQezGW5/iKiItLmTLdhglSt9tD9IVjyOSS/43U0RUrVMiX48C/t+b+ejZm+Yic9XpzBjJV257YJHH87nasGzBORsSJyiYhIXm8SkXDgFeBSoAkwWESaZLNcLM6YyHPOKHIT/M67E+p1h68fhG1/eB1NkRIWJgw/vy4Tb+tImRKRXDd6Lo99sYSjqXYDmil4eSYCVV2tqv8HJAIfA6OB9SLyWB7dUbcDVqvqGlU9jnP/Qd9slvsX8B/A7qopbsLC4Io3oEQ5+HQoHDvtRnWThybV4vjijk4MPS+Bd2ato+/Ls1i+bb/XYZlixq82AhFpDjwHPIMzStlAYD/wQy5vqw5s9Hm+yZ3nu95WQE1V/eoMYjZFSel46P8W7P4TptzrdTRFUkxkOI/2OYd3bmjLrkPH6fO/Wbw1c42NcWAKjF9tBMALwDyguaqOUNU5qvocsCa/G3aHwXwePzqwE5GbRGS+iMzfudPqSoucOp2hywOwaAws/NjraIqsbg0rMfWuzpyfGM8TXy3jutFz2b7fCtLm7PkzZvEEVe2uqh+r6jHf11W1Xy5v3wzU9Hlew52XKRZoCkwXkXU44yJPzq7BWFVHqWobVW0THx+f6wcyQer8+yChM3z1N9i5wutoiqwKpaN587rW/PuKZiSv38MlL87gmz+2eh2WKeL8GbM4t4N9buYBDUSkjohEAYOAyT7r3qeqFVU1QVUTgNlAH1W10c+Ko7Bw6PcmRJZ02gtSj3gdUZElIlzdvhZfjehErfIlueXDBdw/fpGNgGbyLWBjFqtqGnA7MBVYBnyiqktE5HER6XOWcZuiKK4q9HsDdiyFb3K8mtj4qW58aSbceh63davHp8mb6DlyJgs27PE6LFMESV53LorI2mxmq6rWDUxIuWvTpo3On2+FhiLtu0ecsQv6vw3NBngdTbEwd+1u7h63kG37j3LHBfW5vVt9IsLtflFzkogkq2q292rlmQiCjSWCYiA9Fd7pCTuWwc0/QYV6XkdULOw/msojk5bw+W+baVWrLC9e1ZJaFWx8ZOPILRH4e/loUxG5UkSuy5wKNkQTUsIjYcDbTrvB+GGQdizv95g8xcVE8sJVSYwc3JJVOw5y6Usz+HT+RuuvyOTJn8tHHwH+507dgP8CVsdvzk7ZWnD5q7B1IbzWEeaMgqN2o1RB6NOiGt/cdT5Nq5fhvvGLue3jBew9bP0VmZz5UyIYAHQHtqnqDUALoExAozKhoVEvuPJ9iCkDX98HzzeGr+61y0sLQPWyJfh4eAce6NGIb5dsp8eLM5m1OsXrsEyQ8icRHHEvI00TkThgB6feH2BM/jXpC8O/h+E/QOM+sOB9eKUdvHcZLPsC0u2SyPwKDxNu7VqPibd1pGR0ONe8NYcnv1rKsTTrr6hISU+DlFWwdDLszu7anbPnz1VDrwL/wLkP4G/AQWChWzoodNZYXMwdSnGSwfzRsG8jxNWAtsOg1fVQqqLX0RVZR46n8+SUpXw4ewONq8bx0qAkEivHeh2W8ZWRAXvXOxdR7Fzm/N2xHFJWQrrbjtbjP9DhlnytvsCuGnIHr49T1cX5iqQAWCIIEelpsPIbmPcmrJkO4VHQtD+0HQ41WnsdXZH1/bLt3D9+MQePpfH3Sxtx/XkJ+NGZsClIqrB/s3ugX3bywL9zBaQePrlcXA2o1AgqNYb4xu7fRhCVvyvBzjoRiEh1oDYQcfKz6Ix8RXOWLBGEoJ0rYN5bTj9Fxw9CtVbQ7iY45wqIjPE6uiJn54Fj3D9+ET+u2EmXxHieGdCcSnG2HwucKhzc4dxAuXO583fHcufxMZ8LI0pXPvVgX6kxxDd02s4K0FklAhH5D86YBEuBzMpFVVVPrhyyRBDCju6HxeNg7iinuFyyglNl1GYYlLVmqzOhqnw4ez1PfLWMUtERPN2vGRefU8XrsIquQ7t8qnOWnTzwH/G507tEeajU5PSz/JJ5dtRQIM42EazA6XU0KC72tkRgUIW1P8HcN2HFFGdew57QbjjU6eKMm2z8snrHAe4cu5AlW/YzuF1NHurdhJJREXm/MVQd3eec1Wc9yz+04+Qy0WVOP9hXagyl4j39bZ5tIvgaGKiqQTGqiCUCc4q9G52G5QXvweFdULGhkxBaDIJoawz1x/G0DJ7/biVvzPiThAqlePGqJFrULOt1WN46vNupkkxZ6UyZZ/n7fTpQjizlHPDjG5964I+rFpQnI2ebCCbg3DvwPXCiVKCqIwoySH9ZIjDZSj3qjI889w3Y8htExULSYKdxOT7R6+iKhF//3MU9nyxk54Fj3H5BfW46v27xLh1kZMD+Tc6BfudKSFnh/l0Jh33uuYiIgYqJPvX37t8yNZ1R+IqIs00E12c3X1XfK4DYzpglApOnTclOO8KSzyD9uFNd1O4mSOwB4cX4wFYA9h1O5aFJfzB50RYqlo7ili71GNKhNjGR4V6Hln9px50R8nwP+CkrnWvzfa/SKVHOKVHGJzoH/szHZWo63aEUcdbpnAlNB3fCb+/DvNHOmV+Zmk7Dcqvr7J6EPCSv380L363i59UpxMdGc1vXegxqVyu4E8LR/c7BPWXFqdU6u9eC+txEV6amc6CPbwgVG7gH/IbOxQdBWKVTUPKVCETkE1W9UkR+B05bSFWbF2yY/rFEYM5Yehqs/NopJaydAeHRzj0J7YZD9VZeRxfUZq/ZxfPfrWTu2t1ULRPDbd3qc2WbmkRFeFQlogoHtp08yPse8A/4jNQWFun0anvigJ95lt8Aokp5E7vH8psIqqrqVhGpnd3rqrq+AGP0myUCc1Z2LHduUls4BlIPQfXWJ+9JiIj2OrqgpKr8+ucunvtuJcnr91C9bAlGdK9Pv1Y1iAzEmAdpx+HIbucu870bTq27T1kFx/adXDYq1q3KyVKlUy7BqgGzKMg7iysCu9TD+iRLBKZAHN0Hi8Y6l6DuWuVc412lmXMAyTqVKFesqwz8parMWJXC89+tZNHGvdQqX5IR3RtweVK1nAfBUYVjB5zG18Puwf1winOF1yF3Xtbnvgf6TLFVT63GyXwcW8W+Gz/lt0TQAXga2A38C/gAqIjTUd11qvpNYMLNnSUCU6BUnS4sFo11EsKe9adeMQLOdeHlartTgs9Ux6lvjogq/Li9kp6KHt7F3CWrmPjLYvalbCMx9hg96kSQGHucsMO7nIO675SeQxfY4dFOW03J8lCyolNHX6qi+7i88ziuunPQL+C7bENRfhPBfJzO5soAo4BLVXW2iDQCxqhqy0AFnBtLBCbgjh1wEsKedadOe9c789N9760UKFPDTQy1oWzCqcmiVMXgPGNNO+50c3Bkr1M6Opr5152O7MlyFr/LeX40m7N11wFKIaUrUqpsFaRUBefAfuLgXsE5wJ+YX9Gpqw/GfVNM5ZYIcqtEi1DVb90VPK6qswFUdbl1UmWKtehYqNLUmbLKyICD205PEnvWw6ppzmu+IktlKUX4lCrK1oLIEvmLMT3NPZDvOfUAftq0N/v5vpdNZics0ucAXgGqtjj9bN09oGeUKM83a47zwg9rWbXjII0iYrmrQyKXnFPZOrQrInJLBBk+j49kea1oXXNqTEEJC3PuHI2rBrXPO/3144edBs7TShJrYc2Ppx+AY6uemihKV4bjh/I+kB/P40Z/CXeqU3ynipV9npeFEmVPXyZziizp99l6GNAzCS5pXpMvF2/hpWmruOXDZM6pFsc9FyVyQaNKlhCCXG5VQ+nAIUCAEkDmL1iAGFWNLJQIs7CqIVNkqcKhnaeXJDIf79/MyXMsyeEgXTb3A3jmFFXas2qXtPQMJi3cwkvfr2LD7sO0qFGGuy9KpEtivCUED9kNZcYUBWnHnPr46FjnQF6Eui/ITmp6Bp8t2MTI71ezee8RWtcuxz0XJXJevQqWEDxgicAY45njaRl8mryRl39YzdZ9R2lXpzz3XJRIh7oVvA4tpFgiMMZ47mhqOuPmbeSVH1ez48AxOtavwD0XJdK6duH0xx/qLBEYY4LG0dR0Ppy9ntd/+pOUg8c5PzGeey5KJCnUu74OMEsExpigc/h4Gh/86iSEPYdT6d6oEndflEjT6nbzWCBYIjDGBK2Dx9J475d1jJqxhn1HUrm4SWXuviiRxlXjvA6tWLFEYIwJevuPpvLOz+t4a+YaDhxLo1ezqtx+QX1LCAXEEoExpsjYdziVt35ew+if13LoeDqta5djcLta9G5eNbjHQwhylgiMMUXO3sPHGZ+8iY/nbGBNyiHiYiLo16oG17SvRYPKNh71mbJEYIwpslSV2Wt2M2buBr75YxvH0zNom+CUEno2s1KCvywRGGOKhV0HjzFhwSbGzN3I2pRDlCkRSf9WNbi6fU3qV7JSQm4sERhjipXMUdM+nruBqUu2kZqutKtTnqvb1aJH0ypWSsiGJQJjTLGVcvAY45M3MWbuBtbvOky5kk4pYXD7WtSLL+11eEHDEoExptjLyFB++XMXY9xSQlqG0qFueQa7pYToiNAuJVgiMMaElJ0HjvFp8kbGzN3Axt1HKF8qigGtazC4XS3qVCzldXiesERgjAlJGRnKz6tTGDN3A98t3U5ahnJevQoMbleLS86pQlRE0e7q+0xYIjDGhLwd+4/yqduWsGnPESqUimJAmxoMbluLhBAoJXiWCESkB/ASEA68papPZ3n9FuA2IB04CNykqktzW6clAmPM2cjIUGas2smYuRuYtmwH6RlKp/oVGdyuFhc1qVxsSwmeJAIRCQdWAhcBm4B5wGDfA72IxKnqfvdxH+Cvqtojt/VaIjDGFJTt+4/yybyNjJ23kc17j1CxdDQD3VJCrQolvQ6vQOWWCHIbvP5stQNWq+oaN4ixQF/gRCLITAKuUpwcsNUYYwKuclwMd3RvwF+71WfGyp18PHcDb/z0J69N/5PODSpyTftadG9cmcjw4llKyBTIRFAd2OjzfBPQPutCInIbcA8QBVyQ3YpE5CbgJoBatWoVeKDGmNAWHiZ0a1SJbo0qsW3fUcbN28i4eRu45cMFxMdG069ldfokVaNJ1bhiOd5yIKuGBgA9VPVG9/m1QHtVvT2H5a8GLlHV63Nbr1UNGWMKQ3qGMn3FDsbM3cD0FTtJy1DqVypN3xbV6JNUjdoVilYDs1dVQ5uBmj7Pa7jzcjIWeC2A8RhjjN/Cw4TujSvTvXFl9hw6zpQ/tjJp4Rae+24lz323kqSaZembVI3ezasRHxvtdbhnJZAlggicxuLuOAlgHnC1qi7xWaaBqq5yH18GPJJTxspkJQJjjJe27D3CF4u2MGnhFpZu3U+YQMf6FembVJ1LzqlMbEyk1yFmy8vLR3sCL+JcPjpaVZ8UkceB+ao6WUReAi4EUoE9wO2+iSI7lgiMMcFi1fYDTHaTwobdh4mKCOPCxpXo06I6XRvGB1Xnd3ZDmTHGBJCqsnDjXiYt3MKXi7eQcvA4sTER9DinCn2TqnNuvQqEh3nbyGyJwBhjCklaega/rtnFpIVb+OaPbRw8lkZ8bDS9m1elb1J1WtQo48mVR5YIjDHGA0dT0/lx+Q4mLdzCD8t3cDw9g9oVSrpXHlWnfqXC6ybbEoExxnhs35FUpi7ZxuSFW/jlzxQyFM6pFsflSdXp3aIqVcuUCOj2LREYY0wQ2bH/KF8u3sqkRVtYtHEvItC+Tnn6JlXn0qZVKFsyqsC3aYnAGGOC1LqUQ0xetIWJCzezZuchIsOFLonx9EmqzoWNK1EyqmBu97JEYIwxQU5VWbJlP5MXbWHywi1s23+UklHhXNykMn2TqtOpQcWz6vPIEoExxhQhGRnK3HW7mbRwC1N+38q+I6mUKxnJo33OoW9S9Xyt06suJowxxuRDWJjQoW4FOtStwGN9zmHGyp1MWrSFamUD06BsicAYY4JYVEQYFzapzIVNKgdsG8W7k21jjDF5skRgjDEhzhKBMcaEOEsExhgT4iwRGGNMiLNEYIwxIc4SgTHGhDhLBMYYE+KKXBcTIrITWJ/Pt1cEUgownKLO9sepbH+cZPviVMVhf9RW1fjsXihyieBsiMj8nPraCEW2P05l++Mk2xenKu77w6qGjDEmxFkiMMaYEBdqiWCU1wEEGdsfp7L9cZLti1MV6/0RUm0ExhhjThdqJQJjjDFZWCIwxpgQFzKJQER6iMgKEVktIg96HY9XRKSmiPwoIktFZImI3Ol1TMFARMJF5DcR+dLrWLwmImVFZLyILBeRZSJyrtcxeUVE7nb/T/4QkTEiEuN1TIEQEolARMKBV4BLgSbAYBFp4m1UnkkD/qaqTYAOwG0hvC983Qks8zqIIPES8I2qNgJaEKL7RUSqAyOANqraFAgHBnkbVWCERCIA2gGrVXWNqh4HxgJ9PY7JE6q6VVUXuI8P4PyT52807GJCRGoAvYC3vI7FayJSBjgfeBtAVY+r6l5Pg/JWBFBCRCKAksAWj+MJiFBJBNWBjT7PNxHiBz8AEUkAWgJzPA7Fay8C9wMZHscRDOoAO4F33Kqyt0SklNdBeUFVNwPPAhuArcA+Vf3W26gCI1QSgclCREoDE4C7VHW/1/F4RUR6AztUNdnrWIJEBNAKeE1VWwKHgJBsUxORcjg1B3WAakApERnibVSBESqJYDNQ0+d5DXdeSBKRSJwk8JGqfuZ1PB7rCPQRkXU4VYYXiMiH3obkqU3AJlXNLCWOx0kMoehCYK2q7lTVVOAz4DyPYwqIUEkE84AGIlJHRKJwGnwmexyTJ0REcOp/l6nq817H4zVV/buq1lDVBJzfxQ+qWizP+vyhqtuAjSLS0J3VHVjqYUhe2gB0EJGS7v9Nd4ppw3mE1wEUBlVNE5Hbgak4Lf+jVXWJx2F5pSNwLfC7iCx05/1DVad4F5IJMncAH7knTWuAGzyOxxOqOkdExgMLcK62+41i2tWEdTFhjDEhLlSqhowxxuTAEoExxoQ4SwTGGBPiLBEYY0yIs0RgjDEhzhKBCVoioiLynM/ze0Xk0QJa97siMqAg1pXHdga6PXj+mGV+gogcEZGFPtN1BbjdrtaTqvFXSNxHYIqsY0A/EXlKVVO8DiaTiESoapqfi/8FGK6qP2fz2p+qmlRwkRmTP1YiMMEsDecGnruzvpD1jF5EDrp/u4rITyIySUTWiMjTInKNiMwVkd9FpJ7Pai4UkfkistLtcyhzXIJnRGSeiCwWkZt91jtTRCaTzZ22IjLYXf8fIvIfd97DQCfgbRF5xt8PLSIHReQFtx/870Uk3p2fJCKz3bg+d/vCQUTqi8g0EVkkIgt8PmNpn3EFPnLvjsXdJ0vd9Tzrb1ymGFNVm2wKygk4CMQB64AywL3Ao+5r7wIDfJd1/3YF9gJVgWicPqUec1+7E3jR5/3f4JwMNcDpYycGuAn4p7tMNDAfp9OxrjgdsNXJJs5qON0RxOOUsn8ALndfm47Tn33W9yQAR4CFPlNn9zUFrnEfPwy87D5eDHRxHz/u81nmAFe4j2NwukvuCuzD6VcrDPgVJylVAFZw8mbSsl5/zzZ5P1mJwAQ1dXpGfR9ngBB/zVNn3IVjwJ9AZtfBv+McgDN9oqoZqroKpyuFRsDFwHVu9xtzcA6cDdzl56rq2my21xaYrk7nZGnARzh9+uflT1VN8plmuvMzgHHu4w+BTu44AWVV9Sd3/nvA+SISC1RX1c8BVPWoqh72iXeTqmbgJJoEnORwFKeU0g/IXNaEMEsEpih4Eaeu3bdf/DTc36+IhAFRPq8d83mc4fM8g1PbxbL2r6KAAHf4HJzr6Mk+6A+dzYc4C/ntB8Z3P6QDmW0b7XB6Fe2NUyoyIc4SgQl6qrob+AQnGWRaB7R2H/cBIvOx6oEiEubWqdfFqTKZCtzqdtWNiCT6MTDLXKCLiFR0h0UdDPyUx3tyEwZktn9cDfysqvuAPSLS2Z1/LfCTOqPMbRKRy914o0WkZE4rdsehKKNOJ4N34wxFaUKcXTVkiorngNt9nr8JTBKRRThntfk5W9+AcxCPA25R1aMi8hZOFcoCt3F1J3B5bitR1a0i8iDwI06J4itVneTH9uv59AALTq+4I3E+SzsR+SewA7jKff164HX3QO/bK+i1wBsi8jiQCgzMZZuxOPstxo31Hj/iNMWc9T5qTJARkYOqWtrrOEzosKohY4wJcVYiMMaYEGclAmOMCXGWCIwxJsRZIjDGmBBnicAYY0KcJQJjjAlx/w/huHB936ummwAAAABJRU5ErkJggg==",
555 | "text/plain": [
556 | ""
557 | ]
558 | },
559 | "metadata": {
560 | "needs_background": "light"
561 | },
562 | "output_type": "display_data"
563 | }
564 | ],
565 | "source": [
566 | "plt.title(\"Sentiment Analysis (FastText): Learning Curves\")\n",
567 | "plt.xlabel(\"Number of Epochs\")\n",
568 | "plt.ylabel(\"Binary Cross Entropy Loss\")\n",
569 | "plt.plot(train_losses, label = \"Training Loss\")\n",
570 | "plt.plot(valid_losses, label= \"Validation Loss\")\n",
571 | "plt.legend()\n",
572 | "plt.show()"
573 | ]
574 | },
575 | {
576 | "cell_type": "markdown",
577 | "metadata": {},
578 | "source": [
579 | "# Testing"
580 | ]
581 | },
582 | {
583 | "cell_type": "code",
584 | "execution_count": 24,
585 | "metadata": {
586 | "colab": {
587 | "base_uri": "https://localhost:8080/"
588 | },
589 | "id": "YLpKGbjyu2OK",
590 | "outputId": "fbad5f78-802f-441e-8f09-a461c781732e"
591 | },
592 | "outputs": [
593 | {
594 | "name": "stdout",
595 | "output_type": "stream",
596 | "text": [
597 | "Test Loss: 0.3499\n",
598 | "Test Accuracy: 86.28%\n"
599 | ]
600 | }
601 | ],
602 | "source": [
603 | "model.load_state_dict(torch.load('FastText.pt'))\n",
604 | "\n",
605 | "test_loss, test_acc = Evaluate(model, test_iterator, criterion)\n",
606 | "\n",
607 | "print(f'Test Loss: {test_loss:.4f}')\n",
608 | "print(f'Test Accuracy: {test_acc*100:.2f}%')"
609 | ]
610 | },
611 | {
612 | "cell_type": "markdown",
613 | "metadata": {
614 | "id": "o3vZ2NNn8sM6"
615 | },
616 | "source": [
617 | "# Sampling"
618 | ]
619 | },
620 | {
621 | "cell_type": "code",
622 | "execution_count": 25,
623 | "metadata": {
624 | "id": "e0hc5XtDsqIH"
625 | },
626 | "outputs": [],
627 | "source": [
628 | "import spacy\n",
629 | "nlp = spacy.load('en_core_web_sm')\n",
630 | "\n",
631 | "def predict_sentiment(model, text):\n",
632 | " model.eval()\n",
633 | " tokenized = generate_n_grams([tok.text for tok in nlp.tokenizer(text)], n=2)\n",
634 | " indexed = [FIELD.vocab.stoi[t] for t in tokenized]\n",
635 | " tensor = torch.LongTensor(indexed).to(device)\n",
636 | " tensor = tensor.unsqueeze(1)\n",
637 | " prediction = torch.sigmoid(model(tensor))\n",
638 | " return prediction.item()"
639 | ]
640 | },
641 | {
642 | "cell_type": "code",
643 | "execution_count": 26,
644 | "metadata": {
645 | "colab": {
646 | "base_uri": "https://localhost:8080/"
647 | },
648 | "id": "vc0zO3gTws-d",
649 | "outputId": "1ab04c54-b9e4-4323-943d-31cf2600127a"
650 | },
651 | "outputs": [
652 | {
653 | "data": {
654 | "text/plain": [
655 | "4.2760206042657956e-07"
656 | ]
657 | },
658 | "execution_count": 26,
659 | "metadata": {},
660 | "output_type": "execute_result"
661 | }
662 | ],
663 | "source": [
664 | "predict_sentiment(model, \"This film is not bad\")"
665 | ]
666 | },
667 | {
668 | "cell_type": "code",
669 | "execution_count": 27,
670 | "metadata": {
671 | "colab": {
672 | "base_uri": "https://localhost:8080/"
673 | },
674 | "id": "Umdey8pIw8yp",
675 | "outputId": "de24d02b-3c28-4dfa-f0b8-c4f0a4650340"
676 | },
677 | "outputs": [
678 | {
679 | "data": {
680 | "text/plain": [
681 | "1.0"
682 | ]
683 | },
684 | "execution_count": 27,
685 | "metadata": {},
686 | "output_type": "execute_result"
687 | }
688 | ],
689 | "source": [
690 | "predict_sentiment(model, \"This film is excellent\")"
691 | ]
692 | },
693 | {
694 | "cell_type": "code",
695 | "execution_count": 28,
696 | "metadata": {
697 | "colab": {
698 | "base_uri": "https://localhost:8080/"
699 | },
700 | "id": "8rbQ8FYRw-TM",
701 | "outputId": "37bb1381-ac04-42ac-de39-e48de16ae9db"
702 | },
703 | "outputs": [
704 | {
705 | "data": {
706 | "text/plain": [
707 | "2.538240104210665e-11"
708 | ]
709 | },
710 | "execution_count": 28,
711 | "metadata": {},
712 | "output_type": "execute_result"
713 | }
714 | ],
715 | "source": [
716 | "predict_sentiment(model, \"This film is bad\")"
717 | ]
718 | }
719 | ],
720 | "metadata": {
721 | "accelerator": "GPU",
722 | "colab": {
723 | "name": "FastText.ipynb",
724 | "provenance": []
725 | },
726 | "kernelspec": {
727 | "display_name": "Python 3",
728 | "name": "python3"
729 | },
730 | "language_info": {
731 | "codemirror_mode": {
732 | "name": "ipython",
733 | "version": 3
734 | },
735 | "file_extension": ".py",
736 | "mimetype": "text/x-python",
737 | "name": "python",
738 | "nbconvert_exporter": "python",
739 | "pygments_lexer": "ipython3",
740 | "version": "3.9.7"
741 | }
742 | },
743 | "nbformat": 4,
744 | "nbformat_minor": 0
745 | }
746 |
--------------------------------------------------------------------------------
/text_classification/notebooks/LSTM.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Installing Packages"
8 | ]
9 | },
10 | {
11 | "cell_type": "markdown",
12 | "metadata": {
13 | "id": "bbMNpP192mb2"
14 | },
15 | "source": [
16 | "# Importing Required Libraries"
17 | ]
18 | },
19 | {
20 | "cell_type": "code",
21 | "execution_count": 1,
22 | "metadata": {
23 | "id": "slw2Y5t3taWQ"
24 | },
25 | "outputs": [],
26 | "source": [
27 | "import torch\n",
28 | "from torchtext import data, datasets\n",
29 | "import torch.nn as nn\n",
30 | "import torch.optim as optim\n",
31 | "from torch.nn.utils.rnn import pack_padded_sequence\n",
32 | "from torchtext.data import Field, LabelField, BucketIterator\n",
33 | "\n",
34 | "import random\n",
35 | "\n",
36 | "import matplotlib.pyplot as plt"
37 | ]
38 | },
39 | {
40 | "cell_type": "code",
41 | "execution_count": 2,
42 | "metadata": {
43 | "colab": {
44 | "base_uri": "https://localhost:8080/"
45 | },
46 | "id": "7kAMYpKr2tJV",
47 | "outputId": "1b33d632-6261-4881-8703-5a572fa8c210"
48 | },
49 | "outputs": [
50 | {
51 | "name": "stdout",
52 | "output_type": "stream",
53 | "text": [
54 | "Notebook is running on cuda\n"
55 | ]
56 | }
57 | ],
58 | "source": [
59 | "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
60 | "print(\"Notebook is running on\", device)"
61 | ]
62 | },
63 | {
64 | "cell_type": "markdown",
65 | "metadata": {
66 | "id": "8ZOcRV2w2vpH"
67 | },
68 | "source": [
69 | "Fixing SEED for reproducibility of results"
70 | ]
71 | },
72 | {
73 | "cell_type": "code",
74 | "execution_count": 3,
75 | "metadata": {
76 | "id": "N99mW-8XtnWr"
77 | },
78 | "outputs": [],
79 | "source": [
80 | "SEED = 4444\n",
81 | "\n",
82 | "random.seed(SEED)\n",
83 | "torch.manual_seed(SEED)\n",
84 | "torch.cuda.manual_seed(SEED)\n",
85 | "torch.backends.cudnn.deterministic = True"
86 | ]
87 | },
88 | {
89 | "cell_type": "code",
90 | "execution_count": 4,
91 | "metadata": {
92 | "id": "O5NaQke7touV"
93 | },
94 | "outputs": [],
95 | "source": [
96 | "FIELD = Field(tokenize = 'spacy', tokenizer_language = 'en_core_web_sm', include_lengths=True)\n",
97 | "\n",
98 | "LABEL = LabelField(dtype = torch.float)"
99 | ]
100 | },
101 | {
102 | "cell_type": "markdown",
103 | "metadata": {
104 | "id": "RWXCd0V73YK7"
105 | },
106 | "source": [
107 | "# Splitting the data"
108 | ]
109 | },
110 | {
111 | "cell_type": "code",
112 | "execution_count": 5,
113 | "metadata": {
114 | "id": "79ih2w2sttNq"
115 | },
116 | "outputs": [],
117 | "source": [
118 | "train_data, test_data = datasets.IMDB.splits(FIELD, LABEL)"
119 | ]
120 | },
121 | {
122 | "cell_type": "code",
123 | "execution_count": 6,
124 | "metadata": {
125 | "id": "nxpdowpRtwNQ"
126 | },
127 | "outputs": [],
128 | "source": [
129 | "train_data, valid_data = train_data.split(random_state = random.seed(SEED))"
130 | ]
131 | },
132 | {
133 | "cell_type": "code",
134 | "execution_count": 7,
135 | "metadata": {
136 | "colab": {
137 | "base_uri": "https://localhost:8080/"
138 | },
139 | "id": "BSWt_wDwtzLt",
140 | "outputId": "a5c136ea-d381-4ee1-8e91-a908e5f8ece1"
141 | },
142 | "outputs": [
143 | {
144 | "name": "stdout",
145 | "output_type": "stream",
146 | "text": [
147 | "Number of training examples: 17500\n",
148 | "Number of validation examples: 7500\n",
149 | "Number of testing examples: 25000\n"
150 | ]
151 | }
152 | ],
153 | "source": [
154 | "print(f'Number of training examples: {len(train_data)}')\n",
155 | "print(f'Number of validation examples: {len(valid_data)}')\n",
156 | "print(f'Number of testing examples: {len(test_data)}')"
157 | ]
158 | },
159 | {
160 | "cell_type": "code",
161 | "execution_count": 8,
162 | "metadata": {
163 | "id": "trFz-SmNt0fO"
164 | },
165 | "outputs": [],
166 | "source": [
167 | "MAX_VOCAB_SIZE = 25000 # excluding and token\n",
168 | "\n",
169 | "FIELD.build_vocab(train_data, max_size = MAX_VOCAB_SIZE, vectors=\"glove.6B.100d\", unk_init = torch.Tensor.normal_)\n",
170 | "LABEL.build_vocab(train_data)"
171 | ]
172 | },
173 | {
174 | "cell_type": "code",
175 | "execution_count": 9,
176 | "metadata": {
177 | "colab": {
178 | "base_uri": "https://localhost:8080/"
179 | },
180 | "id": "PQUUTTgat5hg",
181 | "outputId": "6fde1f1d-4150-4e21-aba1-f66137439cee"
182 | },
183 | "outputs": [
184 | {
185 | "name": "stdout",
186 | "output_type": "stream",
187 | "text": [
188 | "Unique tokens in FIELD vocabulary: 25002\n",
189 | "Unique tokens in LABEL vocabulary: 2\n"
190 | ]
191 | }
192 | ],
193 | "source": [
194 | "print(f\"Unique tokens in FIELD vocabulary: {len(FIELD.vocab)}\")\n",
195 | "print(f\"Unique tokens in LABEL vocabulary: {len(LABEL.vocab)}\")"
196 | ]
197 | },
198 | {
199 | "cell_type": "markdown",
200 | "metadata": {
201 | "id": "BIZMVoIA4S2K"
202 | },
203 | "source": [
204 | "# Model Definition"
205 | ]
206 | },
207 | {
208 | "cell_type": "code",
209 | "execution_count": 10,
210 | "metadata": {
211 | "id": "egyx2GtzuEdP"
212 | },
213 | "outputs": [],
214 | "source": [
215 | "class LSTM(nn.Module):\n",
216 | " def __init__(self, vocab_size, emb_size, hidden_size, output_size, num_layers, pad_idx):\n",
217 | " super().__init__()\n",
218 | " self.embedding = nn.Embedding(vocab_size, emb_size, padding_idx = pad_idx)\n",
219 | " self.lstm = nn.LSTM(emb_size, hidden_size, num_layers = num_layers, bidirectional=True, dropout=0.5)\n",
220 | " self.fc = nn.Linear(hidden_size*2, output_size)\n",
221 | " self.dropout = nn.Dropout(0.5)\n",
222 | " \n",
223 | " def forward(self, input, input_lengths): # [input] = [seq_len, batch_size]\n",
224 | " embedded = self.dropout(self.embedding(input)) # [embedded] = [sent_len, batch_size, embedding_size]\n",
225 | " packed_embedded = pack_padded_sequence(embedded, input_lengths.to('cpu')) # inputs lengths need to be on CPU\n",
226 | " packed_output, (hidden, cell) = self.lstm(packed_embedded)\n",
227 | " output, output_lengths = nn.utils.rnn.pad_packed_sequence(packed_output)\n",
228 | " # [output] = [seq_len, batch_size, hidden_size*2]\n",
229 | " # [hidden] = [num_layers * 2, batch_size, hidden_size]\n",
230 | " # [cell] = [num_layers * 2, batch_size, hidden_size]\n",
231 | " hidden = self.dropout(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1)) # [hidden] = [batch_size, hidden_size * 2]\n",
232 | " return self.fc(hidden)"
233 | ]
234 | },
235 | {
236 | "cell_type": "code",
237 | "execution_count": 11,
238 | "metadata": {
239 | "id": "2PDdgoXiufHr"
240 | },
241 | "outputs": [],
242 | "source": [
243 | "def batch_accuracy(preds, y):\n",
244 | "\n",
245 | " #round predictions to the closest integer\n",
246 | " rounded_preds = torch.round(torch.sigmoid(preds))\n",
247 | " correct = (rounded_preds == y).float()\n",
248 | " acc = correct.sum() / len(correct)\n",
249 | " return acc"
250 | ]
251 | },
252 | {
253 | "cell_type": "markdown",
254 | "metadata": {
255 | "id": "OpnqXXA1uvuD"
256 | },
257 | "source": [
258 | "# Training and Evaluation Functions"
259 | ]
260 | },
261 | {
262 | "cell_type": "code",
263 | "execution_count": 12,
264 | "metadata": {
265 | "id": "gKSFO3hRutOc"
266 | },
267 | "outputs": [],
268 | "source": [
269 | "def Train(model, iterator, optimizer, criterion):\n",
270 | " epoch_loss = 0\n",
271 | " epoch_acc = 0\n",
272 | " model.train()\n",
273 | " for batch in iterator:\n",
274 | " optimizer.zero_grad()\n",
275 | " inp, inp_lengths = batch.text\n",
276 | " label = batch.label \n",
277 | " predictions = model(inp, inp_lengths).squeeze(1)\n",
278 | " loss = criterion(predictions, label)\n",
279 | " acc = batch_accuracy(predictions, label)\n",
280 | " loss.backward()\n",
281 | " optimizer.step()\n",
282 | " epoch_loss += loss.item()\n",
283 | " epoch_acc += acc.item()\n",
284 | " return epoch_loss / len(iterator), epoch_acc / len(iterator)"
285 | ]
286 | },
287 | {
288 | "cell_type": "code",
289 | "execution_count": 13,
290 | "metadata": {
291 | "id": "YqTp93Aruwoe"
292 | },
293 | "outputs": [],
294 | "source": [
295 | "def Evaluate(model, iterator, criterion):\n",
296 | " epoch_loss = 0\n",
297 | " epoch_acc = 0\n",
298 | " model.eval()\n",
299 | " with torch.no_grad():\n",
300 | " for batch in iterator:\n",
301 | " inp, inp_lengths = batch.text\n",
302 | " label = batch.label \n",
303 | " predictions = model(inp, inp_lengths).squeeze(1)\n",
304 | " loss = criterion(predictions, label)\n",
305 | " acc = batch_accuracy(predictions, label)\n",
306 | " epoch_loss += loss.item()\n",
307 | " epoch_acc += acc.item()\n",
308 | " return epoch_loss / len(iterator), epoch_acc / len(iterator)"
309 | ]
310 | },
311 | {
312 | "cell_type": "markdown",
313 | "metadata": {
314 | "id": "v1_3yX3-42vO"
315 | },
316 | "source": [
317 | "# Data Iterators, Hyperparameters and Model Initialization"
318 | ]
319 | },
320 | {
321 | "cell_type": "code",
322 | "execution_count": 14,
323 | "metadata": {
324 | "id": "yIHX-jbs5ClS"
325 | },
326 | "outputs": [],
327 | "source": [
328 | "BATCH_SIZE = 64\n",
329 | "\n",
330 | "train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits((train_data, valid_data, test_data), batch_size = BATCH_SIZE, sort_within_batch = True, device = device)"
331 | ]
332 | },
333 | {
334 | "cell_type": "code",
335 | "execution_count": 15,
336 | "metadata": {
337 | "id": "o6c7ld9wuYh4"
338 | },
339 | "outputs": [],
340 | "source": [
341 | "VOCAB_SIZE = len(FIELD.vocab) # dimension of one-hot vector / vocabulary\n",
342 | "EMB_DIM = 100 # dimension of word embeddings\n",
343 | "HIDDEN_DIM = 256 # dimension of hidden layer\n",
344 | "OUTPUT_DIM = 1 # dimension of output layer\n",
345 | "NUM_LAYERS = 2\n",
346 | "\n",
347 | "NUM_EPOCHS = 10\n",
348 | "LR = 0.001"
349 | ]
350 | },
351 | {
352 | "cell_type": "code",
353 | "execution_count": 16,
354 | "metadata": {
355 | "id": "j5U3vbZcwEgr"
356 | },
357 | "outputs": [],
358 | "source": [
359 | "model = LSTM(VOCAB_SIZE, EMB_DIM, HIDDEN_DIM, OUTPUT_DIM, NUM_LAYERS, FIELD.vocab.stoi[FIELD.pad_token])"
360 | ]
361 | },
362 | {
363 | "cell_type": "code",
364 | "execution_count": 17,
365 | "metadata": {
366 | "id": "ziglTF1D5voa"
367 | },
368 | "outputs": [],
369 | "source": [
370 | "optimizer = optim.Adam(model.parameters(), lr=LR)\n",
371 | "\n",
372 | "criterion = nn.BCEWithLogitsLoss()\n",
373 | "\n",
374 | "model = model.to(device)\n",
375 | "criterion = criterion.to(device)"
376 | ]
377 | },
378 | {
379 | "cell_type": "code",
380 | "execution_count": 18,
381 | "metadata": {
382 | "colab": {
383 | "base_uri": "https://localhost:8080/"
384 | },
385 | "id": "X7Mm3BnuuZ2F",
386 | "outputId": "6645fdd0-8293-41df-a4ff-cd29eea63919"
387 | },
388 | "outputs": [
389 | {
390 | "name": "stdout",
391 | "output_type": "stream",
392 | "text": [
393 | "The model has 4,810,857 trainable parameters\n"
394 | ]
395 | }
396 | ],
397 | "source": [
398 | "def count_parameters(model):\n",
399 | " return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
400 | "\n",
401 | "print(f'The model has {count_parameters(model):,} trainable parameters')"
402 | ]
403 | },
404 | {
405 | "cell_type": "markdown",
406 | "metadata": {
407 | "id": "99qvK1qw5aAV"
408 | },
409 | "source": [
410 | "# Training"
411 | ]
412 | },
413 | {
414 | "cell_type": "code",
415 | "execution_count": 19,
416 | "metadata": {
417 | "id": "WT1J71IjuySn"
418 | },
419 | "outputs": [],
420 | "source": [
421 | "import time\n",
422 | "\n",
423 | "def Epoch_time(start_time, end_time):\n",
424 | " elapsed_time = end_time - start_time\n",
425 | " elapsed_mins = int(elapsed_time / 60)\n",
426 | " elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
427 | " return elapsed_mins, elapsed_secs"
428 | ]
429 | },
430 | {
431 | "cell_type": "code",
432 | "execution_count": 20,
433 | "metadata": {
434 | "colab": {
435 | "base_uri": "https://localhost:8080/"
436 | },
437 | "id": "TKDOAVH5uz2s",
438 | "outputId": "57357c85-df7f-4e9a-a609-dd367866adea"
439 | },
440 | "outputs": [
441 | {
442 | "name": "stdout",
443 | "output_type": "stream",
444 | "text": [
445 | "Learning Rate: 0.001, Hidden Dimensions: 256\n",
446 | "Time taken for epoch 1: 0m 35s\n",
447 | "Training Loss: 0.6791 | Validation Loss: 0.7054\n",
448 | "Training Accuracy: 56.48 %| Validation Accuracy: 52.33 %\n",
449 | "Time taken for epoch 2: 0m 34s\n",
450 | "Training Loss: 0.6756 | Validation Loss: 0.6665\n",
451 | "Training Accuracy: 56.92 %| Validation Accuracy: 57.87 %\n",
452 | "Time taken for epoch 3: 0m 35s\n",
453 | "Training Loss: 0.6262 | Validation Loss: 0.5267\n",
454 | "Training Accuracy: 64.63 %| Validation Accuracy: 74.09 %\n",
455 | "Time taken for epoch 4: 0m 35s\n",
456 | "Training Loss: 0.5106 | Validation Loss: 0.4333\n",
457 | "Training Accuracy: 75.00 %| Validation Accuracy: 79.46 %\n",
458 | "Time taken for epoch 5: 0m 35s\n",
459 | "Training Loss: 0.4344 | Validation Loss: 0.3877\n",
460 | "Training Accuracy: 80.39 %| Validation Accuracy: 82.77 %\n",
461 | "Time taken for epoch 6: 0m 35s\n",
462 | "Training Loss: 0.3672 | Validation Loss: 0.3312\n",
463 | "Training Accuracy: 84.22 %| Validation Accuracy: 86.01 %\n",
464 | "Time taken for epoch 7: 0m 35s\n",
465 | "Training Loss: 0.3312 | Validation Loss: 0.3588\n",
466 | "Training Accuracy: 86.08 %| Validation Accuracy: 84.35 %\n",
467 | "Time taken for epoch 8: 0m 35s\n",
468 | "Training Loss: 0.2889 | Validation Loss: 0.3066\n",
469 | "Training Accuracy: 87.88 %| Validation Accuracy: 87.55 %\n",
470 | "Time taken for epoch 9: 0m 35s\n",
471 | "Training Loss: 0.2686 | Validation Loss: 0.2816\n",
472 | "Training Accuracy: 89.10 %| Validation Accuracy: 88.37 %\n",
473 | "Time taken for epoch 10: 0m 35s\n",
474 | "Training Loss: 0.2380 | Validation Loss: 0.3224\n",
475 | "Training Accuracy: 90.45 %| Validation Accuracy: 87.09 %\n",
476 | "Model with Train Loss 0.2686, Validation Loss: 0.2816 was saved.\n"
477 | ]
478 | }
479 | ],
480 | "source": [
481 | "print(f\"Learning Rate: {LR}, Hidden Dimensions: {HIDDEN_DIM}\")\n",
482 | "train_losses = []\n",
483 | "valid_losses = []\n",
484 | "min_losses = [float('inf'), float('inf')]\n",
485 | "\n",
486 | "start_time = time.time()\n",
487 | "for epoch in range(1, NUM_EPOCHS+1):\n",
488 | " \n",
489 | " train_loss, train_acc = Train(model, train_iterator, optimizer, criterion)\n",
490 | " train_losses.append(train_loss)\n",
491 | " valid_loss, valid_acc = Evaluate(model, valid_iterator, criterion)\n",
492 | " valid_losses.append(valid_loss)\n",
493 | "\n",
494 | " if valid_loss < min_losses[0]:\n",
495 | " min_losses[0] = valid_loss\n",
496 | " min_losses[1] = train_loss\n",
497 | " torch.save(model.state_dict(), 'LSTM.pt')\n",
498 | "\n",
499 | " elapsed_time = Epoch_time(start_time, time.time())\n",
500 | " print(f\"Time taken for epoch {epoch}: {elapsed_time[0]}m {elapsed_time[1]}s\")\n",
501 | " start_time = time.time()\n",
502 | " print(f\"Training Loss: {train_loss:.4f} | Validation Loss: {valid_loss:.4f}\")\n",
503 | " print(f\"Training Accuracy: {train_acc*100:.2f} %| Validation Accuracy: {valid_acc*100:.2f} %\")\n",
504 | "\n",
505 | "print(f\"Model with Train Loss {min_losses[1]:.4f}, Validation Loss: {min_losses[0]:.4f} was saved.\")"
506 | ]
507 | },
508 | {
509 | "cell_type": "code",
510 | "execution_count": 21,
511 | "metadata": {
512 | "colab": {
513 | "base_uri": "https://localhost:8080/",
514 | "height": 295
515 | },
516 | "id": "OkjBpvPt8BmJ",
517 | "outputId": "e7d60c8f-a210-43a1-e465-afc3ca5153f8"
518 | },
519 | "outputs": [
520 | {
521 | "data": {
522 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABKOElEQVR4nO3dd3hU1dbA4d9KIQFCJ/Teew0g0sWCimABBLFgxYpir1eviuUT671ib9eGiIqI2FAQVFrovbfQa+iQsr4/9gmEEJJJMpNJWe/zzJOZOWfOXjOZmTXn7H3WFlXFGGNM4RUS7ACMMcYElyUCY4wp5CwRGGNMIWeJwBhjCjlLBMYYU8hZIjDGmELOEkE+ISJvi8gTwY4jt4nIUyLyWQ63ke3XTkS+FJFLc9K+v4jIyyJyW7DjyCoReVRE3g92HObMLBHkgIh0FpF/RCReRPaIyN8i0s4P2x0iIn+lvk9Vb1XVZ3K67WzE4vMXsYhMEZG9IhIR6LiyIruvnYi0AFoC33u3T/u/pFq3qYj86r0P9onIHBG5SEQGi8hB73JERJJT3T7oPXa9iBwXkfJptjlPRFREanl3jQQeFZEiPsZfy3t8WFafuz+p6nOqelMgti3OMBFZLCKHRCRORL4WkeaBaK+gskSQTSJSEpgA/AcoC1QF/g0cC2ZcweJ9WXUBFOgT3Gj8Zijwufp21uUPwG9AJaACMAzYr6qfq2qUqkYBFwJbUm5796VYBwxKueF9kRVL3YCqbgWWk4de32AnGeB14G7c610WaACMAy7O6obywHMJHlW1SzYuQAywL5N1bgCWAXuBX4CaqZYpcCuwCtgHvAkI0Bg4CiQBB1PaAD4GnvWudwfigAeBHcBW4FLgImAlsAd4NFVbIcDDwBpgNzAGKOstq+XFch2wEdgFPOYt6wUcBxK8WBZk8Fz/BfwNvAJMSLPsY+/5/QgcAGYCdVMtfx3YBOwH5gBdUi17CvjMu/4jcFeabS8ELvNeu1e912M/sAhols5rVx6XwPd5r9M0IOQMz2kt0DnV7SHAX+msV957DUtn8n7oDsSlc/964HFgdqr7RgKPedutler+x4CPfHyPpvxvw9JZVgr4wHvvbAaeBUK9ZXWBP7z3yi7g89TPzYv3Ie+1PwbUO9N7KJ3/4Rnfb97yosAnuM/MMtx7/LTXzFu3Pu5z0j6D12AKcNOZ/odeLHfgPofrgLeAkWm28T1wr3e9CvANsNNbf1iq9doDsd77bzvwSqC/h/x1CXoA+fUClPQ+KJ/gfumVSbO8L7Aa98Ue5n3Q/0m1XHFfSKWBGt4bq5e37JQ3q3ffx5yaCBJxX77hwM3e478ASgBNgSNAbW/9u4EZQDUgAngH+NJblvLBfM/7ELb0PtyNveUnPsSZvB6rgduBtrjEUTFN7Lu9D0oY7otldKrlVwPlvGX3AduAyLTtAwOAmake19LbbhHgAlwSKc3JhFo5ndfueeBt73ULx+3FSDrPp7j3ukSnuu+0/4t3v+C+SCbgEnLFM7xG3TlzIjgXWOHFHYpL9DU5PRFcDsxNdXsC8PAZ2kv536aXCL7z3gfFcXsws4Ch3rJ6wHneeyUamAq8libe+UB17z3j83vIh3VfAP4EyuDerwvTe828dW8FNmTyvpxC5ongN9zeRFGgK+5HiXjLy+A+S1VwP6jm4D53RYA6uB8LF3jrTgeu8a5HAWcF+3vK14sdGsomVd0PdObkm3qniIwXkYreKrcCz6vqMlVNBJ4DWolIzVSbeUFV96nqRmAy0CoLISQAI1Q1ARiN+1X6uqoeUNUlwFLchywllsdUNU5Vj+E+mP3S7Ar/W1WPqOoCYEGqx2ZKRDrjvrTGqOoc3J7HVWlW+05VZ3mvxeepn6uqfqaqu1U1UVVfxn0BNUynqfFAAxGp792+BvhKVVP2WkoAjXAf4mXqDqWklQBUxu2dJajqNPU+uWmU9v4eyOz5e4/vgfuCfBnYKiJTU8Xpq0+Ba3Ffwstwv9TTOpAqNlS1t6q+kJVGvPfoRcA9qnpIVXfg9qYGettcraq/qeoxVd2J28vrlmYzb6jqJlU9kuq+rLyHzrTuAOA5Vd2rqnHAGxlsoxxujyannlfVPd5zmYb7THfxlvUDpqvqFqAd7ofB06p6XFXX4j77A711E4B6IlJeVQ+q6gw/xJYrLBHkgPdlM0RVqwHNcL8aXvMW1wRe9zoO9+EOQwiuLyHFtlTXD+N+Rfhqt6omeddTPozbUy0/kmp7NYHvUsWyDLdLXTHV+jmJ5TrgV1Xd5d3+wrsvtTNuX0TuF5FlXqf7Ptxhi/JpHo+qHgW+Aq4WkRDcMfVPvWV/AP/FHYLaISLvev04ab2E23v5VUTWisjDZ3hO+7y/Jc6wPG1scap6p6rWxb3eh4D/+fLYVD7FJdAhGTy2RKrYsqsmbm9oa6r3xDu4PQNEpKKIjBaRzSKyH/iM0/8fm9LZblbeQ2dat0qabafXTorduKSeUyfa8JL6aE7211yF++EC7nWrkvKaea/bo5z8HN2I66NYLiKzRaS3H2LLFZYI/ERVl+MOQTTz7tqE29UunepSVFX/8WVzfg5vE3BhmlgiVTW9X5xZikVEiuJ+xXUTkW0isg0YDrQUkUz3KkSkC+448ADc4bXSQDwuaabnE2Aw0BM4rKrTTwSq+oaqtgWa4D6QD5z2ZNwe032qWgfX6XqviPRMZ71DuD2bBpk9h3QeuwmXkJpltm6ax23AHXe+CPj2DKs1xv2CzolNuMMx5VO9H0qqalNv+XO4/3tzVS2JO3SX9v/h7/doiq24Q0Ipqmew7u9ANRGJyWCdQ5za6V4pnXXSPpcvcXvMNYEOuD4BcK/bujSfoxKqehGAqq5S1UG4hPoiMFZEimcQW55hiSCbRKSRiNwnItW829VxvyJSdgffBh4Rkabe8lIi0t/HzW/HvcF9Gibog7eBESmHpUQkWkT6ZiGWWt4v8PRcitu7aII73NMK92U1DXeYIzMlcP0dO4EwEfkXrv8lXd4XfzLuEMynKfeLSDsR6SAi4bgP/1FvvVOISG8RqScigks4Semt55nI6YdEREQi01zKiMi/ve2GeMNAb+DkeyErbgTO8RJReroBP2VxmxGp48X9T38FXhaRkl7MdUUk5bmWwA0OiBeRqqSTUANoDO5zU8Zr+84zraiqq4BRwJci0l1EinjPcWCqPb35wOUiUkxE6uFe3wyp6jxcJ/b7wC+qus9bNAs4ICIPiUhREQkVkWbiDRkXkatFJFpVkzm513am91aeYokg+w7gfi3MFJFDuA/9YlxnJ6r6He5XwWhv93oxrlPZF38AS4BtIrIrs5V98Dru+PqvInLAi7WDj4/92vu7W0TmprP8Otwolo2qui3lgjtMM9iHIXm/AD/jRjttwH2BZ3Q4ANxhk+a4QxYpSuKO1+71trMbdxgorfrAJNwX3XRglKpOPkM773rPIfWv4bNxh91SX5JxnaCTcCNGFuN+cQ/J5HmcRlXXqGpsestEpDIu4Y5Ldd9PIvJoJps9mCbec3BJugiuL2kvMJaTh1n+DbTBJcofOfPeSSA8jesoX4d7PceS8ZDsYZw8JLgPtxd3GW44L7i+j+O45PcJJw/zZOYLXAf+Fyl3eIdie+N+7KzjZLIo5a3SC1gi7vyQ14GBafpQ8qyUnnFj8g0RuRa4RVU750JbX+A6wccFui0fYnkZWKOqo4IdS24Rdyb1QFVNu2dm/MgSgclXRKQYbo9plKpmtTPW5HHeXk8d3N5afdweyX9V9bVgxlXQ2aEhk2+IyAW4voTtpNplNwVKEdwIpgO4hP89rh/ABJDtERhjTCFnewTGGFPI5bsiS+XLl9datWoFOwxjjMlX5syZs0tVo9Nblu8SQa1atYiNTXd0nTHGmDMQkQ1nWmaHhowxppCzRGCMMYWcJQJjjCnk8l0fgTEmdyQkJBAXF8fRo0eDHYrJgsjISKpVq0Z4eLjPj7FEYIxJV1xcHCVKlKBWrVqcWm7J5FWqyu7du4mLi6N27do+P84ODRlj0nX06FHKlStnSSAfERHKlSuX5b04SwTGmDOyJJD/ZOd/FtBEICK9RGSFiKxObyYoEXlVROZ7l5XejD+BsW0RTHkRjgSuCWOMyY8ClghEJBRXI/xCXA31QSLSJPU6qjpcVVupaivgPwSy7vnq32HKc/Bac/j9GTi0O2BNGWNybvfu3bRq1YpWrVpRqVIlqlateuL28ePHM3xsbGwsw4YNy7SNs88+2y+xTpkyhd69883MlKcJZGdxe2C1N8EzIjIa6IubCCM9g4AnAxZN53ug7jkw7WV3mTEKYm6As4dBiYqZPtwYk7vKlSvH/PnzAXjqqaeIiori/vvvP7E8MTGRsLD0v8JiYmKIicloBkvnn398mTm24AvkoaGqnDrTVBynTtx+gjeFYm1c2dnAqdwCBnwCt8+Axpe4ZPB6C5j4IMTHBbRpY0zODRkyhFtvvZUOHTrw4IMPMmvWLDp27Ejr1q05++yzWbFiBXDqL/SnnnqKG264ge7du1OnTh3eeOONE9uLioo6sX737t3p168fjRo1YvDgwaRUZp44cSKNGjWibdu2DBs2LEu//L/88kuaN29Os2bNeOihhwBISkpiyJAhNGvWjObNm/Pqq68C8MYbb9CkSRNatGjBwIEDc/5iZUFeGT46EBjrTQV3GhG5BbgFoEaNGjlvrUIjuPxd6PYQ/PUKxH4AsR9Cq6ug83Ao6/uwK2MKg3//sISlW/b7dZtNqpTkyUuaZvlxcXFx/PPPP4SGhrJ//36mTZtGWFgYkyZN4tFHH+Wbb7457THLly9n8uTJHDhwgIYNG3LbbbedNs5+3rx5LFmyhCpVqtCpUyf+/vtvYmJiGDp0KFOnTqV27doMGjTI5zi3bNnCQw89xJw5cyhTpgznn38+48aNo3r16mzevJnFixcDsG/fPgBeeOEF1q1bR0RExIn7cksg9wg2A9VT3a7m3ZeegcCXZ9qQqr6rqjGqGhMdnW7xvOwpVxf6vgnD5kHb62DBl/CftvDdbbBrlf/aMcb4Tf/+/QkNDQUgPj6e/v3706xZM4YPH86SJUvSfczFF19MREQE5cuXp0KFCmzfvv20ddq3b0+1atUICQmhVatWrF+/nuXLl1OnTp0TY/Kzkghmz55N9+7diY6OJiwsjMGDBzN16lTq1KnD2rVrueuuu/j5558pWbIkAC1atGDw4MF89tlnZzzkFSiBbG02UF9EauMSwEDgqrQriUgjoAxuarrgKF0DLn4ZutwP//zH7R0s+BKaXgZd74eKWf/VYkxBkp1f7oFSvHjxE9efeOIJevTowXfffcf69evp3r17uo+JiIg4cT00NJTExMRsreMPZcqUYcGCBfzyyy+8/fbbjBkzhg8//JAff/yRqVOn8sMPPzBixAgWLVqUawkhYHsEqpoI3An8AizDTQC+RESeFpE+qVYdCIzWvDBVWsnK0Os5uGcRdLobVv0Kb50NowfDlnnBjs4Yk0Z8fDxVq7qux48//tjv22/YsCFr165l/fr1AHz11Vc+P7Z9+/b8+eef7Nq1i6SkJL788ku6devGrl27SE5O5oorruDZZ59l7ty5JCcns2nTJnr06MGLL75IfHw8Bw8e9PvzOZOAphtVnQhMTHPfv9LcfiqQMWRLVDSc92+XDGa+AzPfguUToP750PUBqN4+2BEaY4AHH3yQ6667jmeffZaLL77Y79svWrQoo0aNolevXhQvXpx27dqdcd3ff/+datWqnbj99ddf88ILL9CjRw9UlYsvvpi+ffuyYMECrr/+epKTkwF4/vnnSUpK4uqrryY+Ph5VZdiwYZQuXdrvz+dM8t2cxTExMZrrE9McjYfZ78P0N+HwbqjdFbo+CLU6g515aQqoZcuW0bhx42CHEXQHDx4kKioKVeWOO+6gfv36DB8+PNhhZSi9/52IzFHVdMfUWokJX0SWgi73uUNG5z8LO5bDJ73howth9STIZ8nUGOO79957j1atWtG0aVPi4+MZOnRosEPyu0KzR7B8234Wb95PsSKhFC0SSrHwUIoVCXPXvUvRIqEUCQ3JvFZHwhGY+yn8/Rrs3wxV2rhDRg0vtD0EU2DYHkH+ldU9grxyHkHATV6+kxd/Xp7peqEhQrHwUCJTkkN4SqJIkzTCOxPV+Gxa751IzKaPKTF6EPElG7Kh6e0cqH0hRSOLuHXDTz6uaHgoISGWKIwxeUuhSQTXdKzJRc0rcfh4EoePJ3HkeBKHjydyJCHlehJHEtx9J5en3J/IoeOJ7Dp4LNXjEzmckIRqU0J5gb4hf3PHvu9pMf1uVv/9Iv9NvJQfkjuSROgpcUSGh9CoUkme6tOUVtVLB+fFMMaYVApNIoiKCCMqwr9PV1U5lpjskkbCeRw5+ggbVvxAlXlv8Nq+UYwoPoGldW9kRcWLOZgY4pLIsUR+WLiFy0b9zeAONXjggkaUKur7TELGGONvhSYRBIKIEBkeSmR4KGUAKAqVroEug2HFRIpPfYl2C5+kXan33VDUs6+B8EjuPrc+r/y2kk/+Wc/Pi7fzRO/G9GlZxWq/G2OCwkYNBUJICDTuDbdMgcFjoURlmHg/vN4Spr9JiZDjPHlJU8bf2ZkqpSO5e/R8rv1wFut2HQp25MbkGT169OCXX3455b7XXnuN22677YyP6d69OymDSS666KJ0a/Y89dRTjBw5MsO2x40bx9KlJwsl/+tf/2LSpElZiD59ebVctSWCQBKB+ufBjb/CteOhfH345VF4/1xIPE6zqqX47vZOPN23KfM37uOC16by+qRVHEtMt/aeMYXKoEGDGD169Cn3jR492ud6PxMnTsz2SVlpE8HTTz/Nueeem61t5QeWCHKDCNTpBkMmwBUfwI6l7gQ13CilazvWYtJ93Ti/SUVenbSSC1+bxj+rdwU5aGOCq1+/fvz4448nJqFZv349W7ZsoUuXLtx2223ExMTQtGlTnnwy/WlMatWqxa5d7nM0YsQIGjRoQOfOnU+UqgZ3jkC7du1o2bIlV1xxBYcPH+aff/5h/PjxPPDAA7Rq1Yo1a9YwZMgQxo4dC7gziFu3bk3z5s254YYbOHbs2In2nnzySdq0aUPz5s1ZvjzzUYopgl2u2voIcluzK2D+5/DnC9DiSiheDoCKJSP571Vt6B+zkyfGLeaq92dyWeuqPHpRY6JLRGSyUWMC7KeH3XSv/lSpOVz4whkXly1blvbt2/PTTz/Rt29fRo8ezYABAxARRowYQdmyZUlKSqJnz54sXLiQFi1apLudOXPmMHr0aObPn09iYiJt2rShbdu2AFx++eXcfPPNADz++ON88MEH3HXXXfTp04fevXvTr1+/U7Z19OhRhgwZwu+//06DBg249tpreeutt7jnnnsAKF++PHPnzmXUqFGMHDmS999/P9OXIS+Uq7Y9gtwmAhc8B8cOuqkz0+jWIJpfh3flrnPqMWHhFnq+PIUvZm4kOTl/nfhnjD+kPjyU+rDQmDFjaNOmDa1bt2bJkiWnHMZJa9q0aVx22WUUK1aMkiVL0qfPyZqXixcvpkuXLjRv3pzPP//8jGWsU6xYsYLatWvToEEDAK677jqmTp16Yvnll18OQNu2bU8UqstMXihXbXsEwVChsZsmM/YDiLkRKp4ylTOR4aHcd35D+raqyuPjFvHod4sYO2cTIy5rTuPKJYMUtCnUMvjlHkh9+/Zl+PDhzJ07l8OHD9O2bVvWrVvHyJEjmT17NmXKlGHIkCEcPXo0W9sfMmQI48aNo2XLlnz88cdMmTIlR/GmlLL2Rxnr3CxXbXsEwdL9EYgo4TqPz1Dmo16FKL68+Sxe7t+S9bsP0/s/f/HcxGUcOhaYOunG5DVRUVH06NGDG2644cTewP79+ylevDilSpVi+/bt/PTTTxluo2vXrowbN44jR45w4MABfvjhhxPLDhw4QOXKlUlISODzzz8/cX+JEiU4cODAadtq2LAh69evZ/Xq1QB8+umndOvWLUfPMS+Uq7Y9gmApXs4lg58fhpW/QMNe6a4mIlzRtho9G1fgxZ+X8+7UtUxYsIWn+jTl/KaVcjloY3LfoEGDuOyyy04cImrZsiWtW7emUaNGVK9enU6dOmX4+DZt2nDllVfSsmVLKlSocEop6WeeeYYOHToQHR1Nhw4dTnz5Dxw4kJtvvpk33njjRCcxQGRkJB999BH9+/cnMTGRdu3aceutt2bp+eTFctWFpuhcnpSUAKM6gibD7TMgrEimD4ldv4fHvlvMiu0HOK9JRZ7q05SqpYvmQrCmsLGic/mXlaHOT0LDXcfxnjUw+z2fHhJTqywThnXmkQsb8deqXZz78p+8O3UNCUnJAQ7WGFNQWSIItvrnQd2eMOVFOLTbp4eEh4YwtFtdfru3K53qleO5icu55D9/MWfD3gAHa4wpiCwRBFvKcNLjB2HyiCw9tFqZYrx3bQzvXNOW+CMJ9Hv7Hx79bhHxhxMCFKwpbPLboWOTvf+ZJYK8oEIjaHcjzPkItp95PHR6RIQLmlZi0r3duLFTbb6avYmer0zhu3lx9iE2ORIZGcnu3bvtfZSPqCq7d+8mMjIyS4+zzuK84vAeeKM1VGkF14zL9kxnS7bE89h3i5m/aR9n1y3HM5c2o250lF9DNYVDQkICcXFx2R6jb4IjMjKSatWqER5+ann7jDqLLRHkJTPehp8fgkGj3bSX2ZScrHwxayMv/rycYwnJ3Nq9Lrd3r0tkeGjmDzbGFEg2aii/aHcjlG8AvzwGicezvZmQEOHqs2ryx33dubB5Jd74fRW9XpvKtFU7/RisMaagsESQl6QeTjrr3RxvLrpEBK8PbM1nN3ZARLjmg1kM+3IeOw7Yrr4x5iRLBHlN/fOg3rnw5//BIf+Uou5cvzw/3d2Fu3vW5+fF2+j58p98OmODFbIzxgCWCPKm80dkazhpRiLDQxl+XgN+vqcLLaqV4olxixk2ep6NCDHGWCLIkyo0gnY3wZyPYXvGZXGzqk50FJ/d2IF7zq3PhIVb+XbuZr9u3xiT/1giyKu6PwwRJeHnR85YnTS7RIS7zqlP+1pleWr8EjbvO+LX7Rtj8hdLBHlVsbLQ41FY9yesyLjMbnaEhggvD2hJsir3j1lg/QXGFGKZJgIR6S8iJbzrj4vItyLSJvChGWJucMNJf30MEo/5ffPVyxbjyUuaMn3tbj78e53ft2+MyR982SN4QlUPiEhn4FzgA+CtwIZlgFTDSdf6ZThpevrHVOPcxhX5v19WsHL76RNxGGMKPl8SQZL392LgXVX9Eci8cL7xj/rnQb3z/DqcNDUR4YUrmlMiIozhX83neKKVszamsPElEWwWkXeAK4GJIhLh4+OMv1wwAo4fgj+eDcjmy0dF8PzlzVmyZT9v/L4qIG0YY/IuX77QBwC/ABeo6j6gLPBAIIMyaUQ3hPY3w9xPYNvigDRxftNK9G9bjVFTVjNnw56AtGGMyZt8SQSVgR9VdZWIdAf6A7N82biI9BKRFSKyWkQePsM6A0RkqYgsEZEvfA280On2EESWgl/8P5w0xb8uaUKV0kW5d8wCDh1LDEgbxpi8x5dE8A2QJCL1gHeB6kCmX9giEgq8CVwINAEGiUiTNOvUBx4BOqlqU+CeLEVfmBQrC90fhXVTYcXEgDRRIjKcl/u3ZOOew4yYuCwgbRhj8h5fEkGyqiYClwP/UdUHcHsJmWkPrFbVtap6HBgN9E2zzs3Am6q6F0BVd/geeiEUcz2Ub+hVJ/X/cFKADnXKcXOXOnwxcyOTl9u/w5jCwJdEkCAig4BrgQnefeEZrJ+iKrAp1e04777UGgANRORvEZkhIr3S25CI3CIisSISu3NnIS6lnDKcdO86mPlOwJq597wGNKxYgge/WcieQ9kvh22MyR98SQTXAx2BEaq6TkRqA5/6qf0woD7QHRgEvCcipdOupKrvqmqMqsZER0f7qel8qv65UP98mPoSHAxMUowMD+XVK1ux7/BxHh+3yArTGVPAZZoIVHUpcD+wSESaAXGq+qIP296M609IUc27L7U4YLyqJqjqOmAlLjGYjJw/AhIOw+TADCcFaFKlJMPPa8DERdsYN98K0xlTkPlSYqI7sArX8TsKWCkiXX3Y9mygvojUFpEiwEBgfJp1xuH2BhCR8rhDRWt9C70Qi24A7W6Guf+DbYsC1szQrnWJqVmGf32/hC1WmM6YAsuXQ0MvA+erajdV7QpcALya2YO8DuY7cecgLAPGqOoSEXlaRPp4q/0C7BaRpcBk4AFV3Z2dJ1LodHvQDScNQHXSFCmF6ZKSlfu/tsJ0xhRUviSCcFVdkXJDVVfiW2cxqjpRVRuoal1VHeHd9y9VHe9dV1W9V1WbqGpzVR2dnSdRKBUrCz0eg/XTYPmPAWumZrniPNG7Cf+s2c3H/6wPWDvGmODxJRHEisj7ItLdu7wHxAY6MOODttdDdCP49fGADScFGNiuOj0bVeDFn5ezeocVpjOmoPElEdwGLAWGeZelwK2BDMr4KDTM1SHauw5mvh2wZkSE569oTrEioQz/agEJSVaYzpiCxJdRQ8dU9RVVvdy7vIo7nm/ygnrnQv0L4M+X4GDgTgCrUCKS5y9vzqLN8fzHCtMZU6Bkt4poDb9GYXLmghGQeCRg1UlT9GpWmSvaVOPNKWuYt3FvQNsyxuSe7CYCGz6Sl5SvD+1vccNJty4MaFNP9mlCpZKR3DtmAYePW2E6YwqCMyYCEbn8DJcrgKK5GKPxRbcHoWhp+OXRgA0nBSgZGc7I/i1Zv/sQz09cHrB2jDG5JyyDZZdksGxCBstMMBQt44aTTrwflk+Axhn9+3KmY91y3NipNu//tY5zm1SkW4NCXvbDmHxO8lsdmZiYGI2NtdGr6UpKhLc7QeJRuGMWhEUErKmjCUn0+e9f7DucwK/Du1K6mM1eakxeJiJzVDUmvWU25WRBEhrmVSddDzPeCmhTkeGhvDKgFXsPH+fxcYGZNc0YkzssERQ09XpCg14wdWRAh5MCNKtainvObcCEhVv53grTGZNvWSIoiM5/1htO+kzAmxratQ5tapTmiXGL2RpvhemMyY98qT46R0TuEJEyuRGQ8YPy9aH9UJj7acCHk4aFhvDKgFYkJisPjl1ohemMyYd82SO4EqgCzBaR0SJygYhIgOMyOdXtATeSKIDVSVPUKl+cxy5uzLRVu/h0xoaAtmWM8T9fSkysVtXHcHMFfAF8CGwQkX+LSNlAB2iyqWgZ6PEobPgLlv0Q8Oaual+DHg2jef6nZazecTDg7Rlj/MenPgIRaYGbl+Al4BugP7Af+CNwoZkca3s9RDd21UkTjga0KRHhxStaUDQ8lHvHzLfCdMbkIz71EeAmopkNtFDVYao6U1VfxmYTy9tCw6DXc7BvA8wM7HBSgAolIxlxWXMWxsXz3z9WB7w9Y4x/+LJH0F9Ve6rqF6p6StF7Vb08QHEZf6l7DjS4EKa+DAe2B7y5i5pX5rLWVfnv5NXM37Qv4O0ZY3LOl0QQLyJviMhcbwTR6yJSLuCRGf85/1l3tnEuDCcFeKpPUyqUiODer+Zz5HhSrrRpjMk+XxLBaGAncAXQz7v+VSCDMn5Wvh50GArzPoOtCwLeXKmirjDd2l2HeOGnZQFvzxiTM74kgsqq+oyqrvMuzwIVAx2Y8bOuuTecFKBTvfJc36kWn0zfwLRVOwPenjEm+3xJBL+KyEARCfEuA4BfAh2Y8bOipeGcx2DD37BsfK40+VCvRtSrEMUDXy8k/nBCrrRpjMk6XxLBzbjzB457l9HAUBE5ICL7Axmc8bM2Q6BCk1wZTgquMN2rA1qx6+AxnvjeCtMZk1f5ckJZCVUNUdUw7xLi3VdCVUvmRpDGT1Kqk+7bCDNG5UqTzauVYljP+oxfsIUfFmzJlTaNMVnj6wllfURkpHfpHeigTADV7QENL4JpuTOcFOD27nVpVb00j49bzLb4wO+JGGOyxpcTyl4A7gaWepe7ReT5QAdmAuj8ZyHxGPz0QK50HLvCdC05lpjEg98sJL9NhmRMQefLHsFFwHmq+qGqfgj0Ai4ObFgmoMrVdXWIln4PU3Inp9eJjuKxixozdeVOPrPCdMbkKb7OR1A61fVSAYjD5LbOw6HVYPjzRVgwOleavPqsmnRtEM2IictYu9MK0xmTV/iSCJ4D5onIxyLyCTAHGBHYsEzAiUDv16BWF/j+Tlj/dy40KbzUrwWR4aEMH7OARCtMZ0yekGEiEJEQIBk4C/gWV3m0o6ramcUFQVgRuPJTKFMLvhoMu9cEvMmKJSN59tJmLNi0j1FTAt+eMSZzGSYCVU0GHlTVrao63rtsy6XYTG4oWgYGjwEJgc/7weE9AW+yd4sq9G1VhTd+X8WiuPiAt2eMyZgvh4Ymicj9IlJdRMqmXAIemck9ZevAwC8hfjOMvsqNKAqwp/s0o3xUBPd8NY+jCVaYzphg8nWqyjuAqbj+gTlAbCCDMkFQowNcOgo2TofxdwV8WGmpYq4w3Zqdhxj+1XyOJ1p/gTHB4ksiaKyqtVNfgCaBDswEQfN+0ONxWPiVG00UYJ3rl+eJ3k34afE2hn4aa3sGxgSJL4ngHx/vO42I9BKRFSKyWkQeTmf5EBHZKSLzvctNvmzXBFDX+6HlVe78goVjAt7cjZ1r8/zlzZmycifXfzSbg8cSA96mMeZUYWdaICKVgKpAURFpDYi3qCRQLLMNi0go8CZwHhAHzBaR8aq6NM2qX6nqndkJ3gSACFzyOsRvgu/vgFLVoObZAW1yUPsaFA0P5b6vF3DNBzP5eEh7ShULD2ibxpiTMtojuAAYCVQDXsFNXv8ycC/wqA/bbg+sVtW1qppStbRvzsI1uSKsCAz4H5Su4TqPc2FY6aWtq/LmVW1Ysnk/g96bwe6Dge+wNsY4Z0wEqvqJqvYAhqhqj1SXPqr6rQ/brgpsSnU7zrsvrStEZKGIjBWR6ultSERuEZFYEYndudMmOckVxcrC4K8Bgc/758qw0l7NKvHedTGs3XWQK9+dYQXqjMklvvQRTBCRq0TkURH5V8rFT+3/ANRS1RbAb8An6a2kqu+qaoyqxkRHR/upaZOpsnVg4BfuMNFXV+fKsNJuDaL55Pr2bIs/yoB3prNpz+GAt2lMYedLIvged0gnETiU6pKZzUDqX/jVvPtOUNXdqpry7fI+0NaH7ZrcVLMj9B3lZjYbPyxXqpV2qFOOz27qQPyRBAa8M501VpfImIDyJRFUU9UrVfX/VPXllIsPj5sN1BeR2iJSBBgInDJHoohUTnWzD2AznedFLfpDj8dg4WiY+lKuNNmqemlG33IWCUnJXPnOdJZttcnwjAkUn4aPikjzrG5YVROBO3HzGy8DxqjqEhF5WkT6eKsNE5ElIrIAGAYMyWo7Jpd0fQBaDoLJI2Dh17nSZOPKJflqaEfCQ0MY+O4M5m/alyvtGlPYSGaThIjIUqAesA44hhtGqt5x/VwXExOjsbF2YnNQJB6DTy+DuNlw7Xh32CgXbNpzmMHvz2T3wWN8OKQdHeqUy5V2jSlIRGSOqsakt8yXPYILgfrA+cAlQG/vrylswiLgys+gVPVcG1YKUL1sMcYM7UilUpFc99Es/lxpI8eM8aczJgIROQdAVTcAIaq6IeWCdeoWXieGlSp8MSBXhpUCVCoVyZihHalTPoqbP4nllyVWBNcYf8loj2BkquvfpFn2eABiMflFubpuWOm+jTDmWkg8njvNRkXw5S1n0bRqSW7/fC7j5m3O/EHGmExllAjkDNfTu20Km5pnQ983Yf00+OHuXBlWClCqaDif3diB9rXKMnzMfL6ctTFX2jWmIMsoEegZrqd32xRGLQZA90dgwRcwbWTm6/tJ8YgwPrq+Hd0bRPPIt4v44K91uda2MQXRGYvOAXVEZDzu13/KdbzbtQMemckfuj0Ee9bCH89CmdqulHUuiAwP5Z1rYrjnq3k8M2Eph48lcuc59RCxnVVjsiqjRJC6QFzan3u59/PP5G0i0Oc/sG8TjLvdjSiq0SFXmi4SFsIbA1sTGb6Ql39bycHjiTzcq5ElA2Oy6IyJQFX/zM1ATD4WFgEDP4f3e8LoQXDTJFenKDeaDg1hZL+WFCsSyjt/ruXI8SSeuqQpISGWDIzxlS/nERiTuWJlYfBY0GT4fAAc2ZtrTYeECM/0bcbQrnX43/QNPPjNQhKTbOpLY3xlicD4T7m6cOXnsHc9fHVNrg0rBRARHr6wEfee14Cxc+K4e7TNg2yMr7KUCEQkRERKBioYUwDU6nRyWOmE4bk2rBRcMhjWsz6PX9yYHxdt5dbP5tg8yMb4INNEICJfiEhJESkOLAaWisgDgQ/N5Fstr3SjieZ/BtN8KVTrXzd1qcOIy5oxecUObvh4NodsHmRjMuTLHkETVd0PXAr8hBs6ek0ggzIFQPdHoHl/+OMZWJz2xPTAG9yhJq8MaMnMdXu45oOZxB9JyPUYjMkvfEkE4SISjksE41U1ATuhzGRGBPr8F2p0hO9ug02zcj2Ey1pX482rWrNoczxX2TzIxpyRL4ngHWA9UByYKiI1AZslxGQuPNJ1HpesAl8Ogj25fwZwr2aVee/aGFbvOMjAd2ewfb/Ng2xMWpkmAlV9Q1WrqupF6mwAeuRCbKYgKF7ODStNTnTVSnNxWGmK7g0r8MkN7dmy7wgD3plO3F6bB9mY1HzpLL7b6ywWEflAROYC5+RCbKagKF/PnXC2Z12uVitN7SxvHuS9h47T/+3prLV5kI05wZdDQzd4ncXnA2VwHcUvBDQqU/DU6uxKUaybCj/m7rDSFK1rlGH0LR05npjMgHdmsHybHeE0BnxLBCnn6l8EfKqqS7Ay1CY7Wg2Crg/CvM/gr1eDEkKTKm4e5LAQ4cp3ZrDA5kE2xqdEMEdEfsUlgl9EpARgp2ya7OnxKDTrB7//G5Z8F5QQ6lWI4utbO1KyaBiD35/JrHW5M8uaMXmVL4ngRuBhoJ2qHgaKANcHNCpTcIm4M4+rnwXf3QqbZgcljOpli/H10LOpWDKCaz+cyVSbB9kUYr6MGkoGqgGPi8hI4GxVXRjwyEzBFR7pOo9LVHLVSneuCEoYlUpF8tXQjtQuH8VNn8QyfsGWoMRhTLD5MmroBeBuYKl3GSYizwU6MFPAFS9/cljpW53gl8fgyL5cD6N8VASjbz6LVtVLM+zLebz620o0CB3ZxgSTZPamF5GFQCtvzwARCQXmqWqLXIjvNDExMRobGxuMpk0gHNjuylDM+8yVsu7xGLS5DkIzmjPJ/44lJvHYd4sZOyeO3i0qM7J/SyLDQ3M1BmMCSUTmqGpMest8rT5aOtX1UjmOyJgUJSpC3//C0D8hujH8eC+80wXWTsnVMCLCQnmpXwsevrARPy7aypXvzmDHATsL2RQOviSC54B5IvKxiHwCzAFGBDYsU+hUbglDJsCAT+H4IfhfX1eWYveaXAtBRLi1W13evrotK7cd4NL//s3SLXaugSn4MkwEIhKCGyp6FvAt8A3QUVW/yoXYTGEjAk36wB2z4NynYN00eLNDrvcfXNC0El/f2hEF+r39D78t3Z5rbRsTDL70EcSe6bhSMFgfQSFyYDtMfhbmfhqU/oMd+49y8/9iWbg5nod7NeKWrnUQsXMpTf6U0z6CSSJyv4hUF5GyKRc/x2jM6UpUdGUphk49tf9gzeRcab5CSTe89KLmlXn+p+U8OHahTX9pCiRf9gjSqx2sqlonMCFlzPYICilVWPYD/Po47NsADS6EC0a4eZIDLDlZee33Vbzx+yra1y7L21e3pWzxIgFv1xh/ymiPINNEkNdYIijkEo7CzLdh6khIPAodhkLXB6Bo6YA3/f38zTwwdiGVS0XywXXtqFchKuBtGuMv2To0JCJXi8hpU1KKyDUicpU/AzTGZ+GR0PkeGDbXFbGb/ib8pw3Mfh+SAjs3cd9WVRl9y1kcOpbIZaP+trIUpsDIqI/gLiC9qmDfAvcFJhxjfBRVIU3/wX1e/8EfAW22TY0yjLujE1VLF+X6j2fz6fT1AW3PmNyQUSIIV9XTZu9Q1UNAuC8bF5FeIrJCRFaLyMMZrHeFiKiI5JnRSSafqNzCnX9w5WeQcBg+vQy+GAi7VgesyWplijH2trPp3iCaJ75fwpPfLyYxyTqRTf6VUSIoKiLF097plaHOtKfMK0XxJnAh0AQYJCJNzrC9u4GZvgZtzClEoPEl3vkH/4b1f8GoDvDzowE7/yAqIox3r43h5i61+WT6Bm74JJb9RxMC0pYxgZZRIvgAGOtNVg+AiNQCRnvLMtMeWK2qa1X1uPe4vums9wzwImDn85ucCYtI1X9wFcwYBW+0Dlj/QWiI8NjFTXjxiub8s3oXl4/6h427bT5kk/+cMRGo6kjge2CqiOwWkd3An8AEVX3Jh21XBTaluh3n3XeCiLQBqqvqjxltSERuEZFYEYndudM66EwmUvcfVGzq+g/e7hyw/oMr29Xg0xs7sOvgMfq++ZdNdGPynQxPKFPVt1W1JlALqKWqNVX1LX807JWveAUfOp5V9V1VjVHVmOjoaH80bwqDyi3guh9c/0HikYD2H3SsW47vbu9EmWJFGPz+DMbOifN7G8YEik/VR1X1gKoeyOK2NwPVU92u5t2XogTQDJgiIutx9YzGW4ex8avU/QfnPZ2m/2CvX5uqXb44393eiQ61y3H/1wt44aflJCfnr/N0TOHkaxnq7JgN1BeR2iJSBBgIjE9ZqKrxqlpeVWupai1gBtBHVe1sMeN/YRHQ6W6v/2Cw13/QBma959f+g1LFwvno+nYM7lCDt/9cw62fzeHw8cCe32BMTgUsEahqInAn8AuwDBijqktE5GkR6ROodo3JUFQF6PPGyf6Diff7vf8gPDSEZy9txpOXNGHSsu30e2s6W+OP+G37xvibL7WG5gAfAl+oqn/3pbPBSkwYv1GF5RNc/aK966HlILjw/yCypN+amLxiB3d9MY9iRUJ579oYWlYv7bdtG5MVOa0+eiVQBZgtIqNF5AKxWrymIEjdf9D1QVj4FbzdCTbO8FsTPRpW4Nvbz6ZIWAgD3pnOjwu3+m3bxvhLpolAVVer6mNAA+AL3N7BBhH5t5WjNgVCWASc8xjc8AtICHx0IfzxLCT55wSxBhVL8P0dnWhetRR3fDGXN35fRX4r9mgKNp/6CESkBfAy8BJulrL+wH4gsIVdjMlN1dvDrX+5Q0RTX4IPzvfbVJnloiL4/OYOXN6mKq/8tpJ7vprP0YQkv2zbmJzKNBF4fQSv4kYBtVDVYao6U1VfBtYGOkBjclVECbh0FPT/BPasdR3Jcz52/Qk53XRYKC/3b8kDFzTk+/lbGPTeDHYeOJbzmI3JIV/mLP5GVXuq6heqesq7VlUvD2h0xgRL00vh9ulQrR38cDeMHgyHduV4syLCHT3q8fbVbVi2dT+Xvvk3y7ftz3m8xuRAZmcWJwP2ZW8Kp5JV4JpxcMFzsPo3eOtsWDXJL5vu1awyY289m8TkZK4Y9Q+/L9vul+0akx02Z7ExGQkJgY53wM2ToVg5+PwKmPgAJOT8vIBmVUsx/s7O1ImO4qb/xfL+tLXWiWyCwuYsNsZXCUdh0lMw8y2IbgSXv+fqGeXQkeNJ3Pf1fCYu2saAmGo81KsR5aIich6vManYnMXG+NPq32Hc7XB4N/T8F3S80+055EBysvLKbyv57+TVRISFcEXbatzYuTZ1o21eZOMfOU4EItIMN7lMZMp9qvo/v0WYBZYITJ5waDf8MMydmVyrC1z2NpSqluPNrt5xgA/+Wsc3czdzPDGZcxtX5OYutWlfuyx2HqfJiRwlAhF5EuiOSwQTcTOO/aWq/fwcp08sEZg8QxXmfQY/PQShYdD7VWh2hV82vfPAMT6dsYHPZmxgz6HjtKhWipu61OGiZpUICw1krUhTUOU0ESwCWgLzVLWliFQEPlPV8/wfauYsEZg8Z/ca+PYW2BwLLQbCRS/5rV7R0YQkvpkbxwfT1rF21yGqli7K9Z1qcWW76pSI9GnqcN8d2QfLf4TF38Dxg3DeM1Cjg3/bMEGT00QwS1XbeyeW9QAOAMtUtZH/Q82cJQKTJyUlwNSRMPX/3CGiy96Fmh39tvnkZOX35Tt4b9paZq3bQ4mIMAZ1qMGQs2tRpXTR7G/4+CFY+TMs+sYNkU06DqVrQnIS7N8MHYa6fpAip01fbvKZnCaCUcCjuPkE7gMOAvNV9Xp/B+oLSwQmT9s0C769GfZthM73QveHIdS/v9wXbNrHe9PW8tPibQjQu0VlbupSh2ZVS/m2gcRjrsN78VhY8RMkHIaoStDscndoq2pbt0cw6d8w+z2XGPq8AXW6+/V5mNzlt1FD3uT1JVV1oZ9iyzJLBCbPO3YAfnoY5n8GVdq4Yabl6/m9mbi9h/no7/WMnrWRQ8eT6FinHDd3rU33BhUICUnTsZyUCOunul/+y36AY/FQtCw06eu+/GueDSGhpzey4R/4/k7YswbaXAvnPwuRPiYck6f4Y9RQVaAmEJZyn6pO9VuEWWCJwOQbS7935SkSj7mzk9sOcaWv/Sz+SAKjZ23ko7/Xs23/UepViOKmzrW5tFVlIrfOdsf8l4yDw7ugSAlo3Bua9YM63XzbW0k4AlOeh3/+A1EVXad4wwv9/jxMYOX00NCLuDkJlgIp5RJVVYMyy5glApOv7N8C426DtVOg4UXQ5z9QvHxAmkpISubHBVuYPOVXmu6ZRJ+wGVRiNxoWiTToBc37Qb3zIDwy842lZ/Nct3ewY4lLJBe+GLDnYvwvp4lgBa7qaJ4ok2iJwOQ7ycnubORJT0FkaVfdtL6fB93tWOZ++S/+BvasJTkknAURbfkovi1/h8bQq019buxcmzo5PUEt8Tj89YrrGI8s6WZ0a3ZFQPZ0jH/lNBH8BPRX1YOBCC6rLBGYfGv7EvjmJtixFNrfAuc9DeE5GPGzZy0s/tZddixxk+rU7uq+mBv1hmJlWbXdnaD27dzNJCQn07ORn05Q274Uvr8Dtsx1ezoXvwIlK2d/eybgcpoIvsGdR/A7cGKvQFWH+TNIX1kiMPlawlH4/WmY8SaUbwhXvJ+1ekX7t8CS79wv/81z3H3Vz3Jf/k36QomK6T5s54FjfDp9PZ/O2MDewwm09E5QuzAnJ6glJcKMUTB5BIRGwAUjoPXVtneQR+U0EVyX3v2q+okfYssySwSmQFjzB3x3m1ev6AnoeNeZ6xUd2uU6nhd/Cxv+BhQqt3Rf/k0vg9I1fG72yHHvBLW/1rHOO0Hths61ubJddaIiwjLfQHp2r4Hxd7nY6vSAS16HMjWzty0TMFZ0zpi86PAeV69o2Q+n1ys6Gu/O8l001nU0axKUb+A6aZtdkePhqMnJyqRl23l/2jpmrd9DicgwrmpfgyGdalG5VDYOVyUnw5wP4bcnXemNc5+EdjfnuBif8Z9sJQIRGaOqA7wSE6etpKo5r7+bDZYITIGiCvM/d/WKQkKh0z3ukM+qX72zfGu4L/5m/aBi04AcdpmfcoLaoq2EiNC7RWVu6VqXJlWyUSZj3yaYcA+snuQOWfX9L5Sv7/eYTdZlNxFUVtWtIpLuPp6qbvBjjD6zRGAKpD1rXb2iuNmnn+WbS8fcN+1xJ6h9NdudoHZxi8rcd16DrI80UoUFo+Hnh905CN0fhrOHucJ8Jmj8eWZxeWC3BvF4kiUCU2AlJbqEUK5u+mf55pL4Iwl8MG0t7/+1jmOJyQyIqcawnvWzfsjowHaYeD8sG+/6NPq+CZWaByZok6ns7hGcBbwA7AGeAT4FyuOmt7xWVX8OTLgZs0RgTO7YeeAYb05ezeczNxAiwnVn1+K2bnUpU7xI1ja0ZJxLCEf2Qufh0PUBCLMZ2HJbdhNBLK7YXCngXeBCVZ0hIo2AL1W1daACzoglAmNy16Y9h3l10kq+m7eZqCJh3NK1Djd0rk3xrIwyOrwHfn4EFo5203z2fROqpfudZNJzZC9MHwWtBkHZ7M0SnN1EMF9VW3nXl6lq41TL5lkiMKZwWbHtACN/XcFvS7dTPqoId/aox6AONYgIy8JhrJW/us7kA1vhrNuhx2NQpFjAYs73ju6HGW/B9DddocCLRkL7m7O1qewmgrmq2ibt9fRu5yZLBMYE19yNe3nxp+XMXLeHamWKcu95DejbqiqhaSuensnR/TDpSYj9EMrUdvWXancJbND5zbGDMOsdV+jvyF53pnj3h3PUx5LdRJAEHAIEKAocTlkERKqqn6dH8o0lAmOCT1WZumoXL/2ynMWb99OgYhT3n9+Q85pU9L10xbpp7kS0veug7fWu5IafZnbLt44fhtgP4K/XXLXY+hdAj0egSs4PwNgJZcaYgEhOViYu3srLv65k3a5DtK5Rmod6NeKsOuV828Dxw65ExYxRUKIy9H4NGpwf0JjzpISjMOdjV9Dv4Haoew50fxSqt/NbE5YIjDEBlZCUzNg5cbw+aRXb9h+la4NoHrygoe+zpsXFuiJ2O5dDiyuh1wtQrGxgg84LEo/DvP/B1JfhwBZ3hnmPR91EQX5micAYkyuOJiTxv+nrGTVlDfsOJ9C7RWXuO78htcv7MOdx4jFX3vqvV6BoGdcx2vTSgMccFEkJMP8LmPoSxG+C6h1cx3mdbgFr0hKBMSZX7T+awHtT1/L+tHUcT0pmQEx17u5Zn0qlfJgUZ9siNwHO1vlQs7NLBg0vglJVAx124CUlwqIx8OeLsHe9O3O8x6NQt2fAzyAPWiIQkV7A60Ao8L6qvpBm+a3AHbiZzw4Ct6jq0oy2aYnAmPwj7UlpQ86uxa2+nJSWlOgm85n7P9i10t1XpQ00utiNoIlumL/KXScnueqxf74Au1dDpRZuD6DBBbn2PIKSCEQkFFgJnAfEAbOBQam/6EWkpKru9673AW5X1V4ZbdcSgTH5T9qT0oZ2q8P1nXw8KW3nSlg+wVVj3ex99svVO5kUqsbk3SqnycmuxMaU513/R4Umbg+gUe9cT2TBSgQdgadU9QLv9iMAqvr8GdYfhCtdkeGs2JYIjMm/Tj0pLYK7zqnHoPY1KBLm4xf5/q2w4keXFNZNheREiKoIDS+ERpe48xHyQvkKVVgxESY/D9sXuRLi3R+BJpcGLWkFKxH0A3qp6k3e7WuADqp6Z5r17gDuBYoA56jqqnS2dQtwC0CNGjXabtgQlMKnxhg/mbNhL//3szsprXrZogw/N4snpQEc2efKXS+fAKt+g+MHoUgJNx90o4vd30gfRy35i6qLZfII18dRtg50exia9wtqIUHI44kg1fpXAReoarozoqWwPQJjCoaUk9L+7+flLNmyn4YVS3D/BQ05t3GFrM+nnHDU7SEsn+B+iR/aCSHhbhROo4tdZ3OJSoF5IuASwNrJMPk5V0q8dA3o9hC0GJhnym/nl0NDIcBeVc0whVsiMKZgSU5Wfly0lVd+cyeltalRmgezclLaaRtMcl/GyyfAsgnuzGUEqrU72a+QwxneTrH+L/hjBGz8B0pWg673Q6vBEJbFKq0BFqxEEIbrLO4JbMZ1Fl+lqktSrVM/5VCQiFwCPHmmQFNYIjCmYEo5Ke21SSvZvv8Y3RpEM7RbHdrWLJO1wnapqcKOZa5PYfkEd7gGoHxDaNzbJYbKrbN33H7jTJj8rNsTiarkEkCba/NGH0U6gjl89CLgNdzw0Q9VdYSIPA3Equp4EXkdOBdIAPYCd6ZOFOmxRGBMwXY0IYlP/nEnpcUfSSAiLIS2NcvQsU45OtYtR4tqpX3vXE5r3yZ36Gj5BFj/t5sLukQVaHSRSwq1ukBoJmXU4ua4PoA1v0PxaOh8L8RcD+HZmOs5F9kJZcaYfOfgsUT+Wb2L6Wt3M33NbpZvOwBA0fBQYmqVoWPdcnSsU47mVUsRFpqNxHB4j5sbetkPsPp3SDwCEaXc2P5GF0O9cyEi1TSdWxe4PoCVP0PRstDpblcSuogPZ03nAZYIjDH53p5Dx5m5djcz1u5m+trdrNx+EIDiRUJpV7vsiT2GplVKZW30Ebjid2unuENIKybCkT0QGgF1e7jRR2smu72IyFJw9l3Q4VaIKOH/JxlAlgiMMQXOroPHXFJY4xLD2p2HACgRGUaH2mU5q045zqpTjiaVSxKSlcSQlAibZriksGwCxG+EiJJuIp2Ot+f+kFQ/sURgjCnwtu8/yoyUPYY1u1m/202hUqpoOB1ql3WHkuqWo0GFEr4nBlXYtQqiol0hvHzMEoExptDZsu/IKXsMcXuPAFC2eBHOqnPyUFLd6Kisn7eQD1kiMMYUepv2HGa6t8cwY81utsQfBaB8VIRLDF7nc+3yxQtkYsgoEeSNU96MMSbAqpctRvWyxRgQUx1VZeOewyf2Fqav2c2EhVsBqFgygo5e/0LHuuWoUbZYgUwMqVkiMMYUOiJCzXLFqVmuOAPb10BVWbfr0Imk8NfqXYybvwWAKqUiOb9pJfrHVKNplfzZUZwZOzRkjDFpqCqrdxxk+trd/L16F5OX7+R4UjJNq5RkQEx1+raqQulieauERGasj8AYY3Jg3+HjjF+whTGxm1i8eT9FQkM4v2lFBsRUp1O98lk/byEILBEYY4yfLNkSz9excYybv5l9hxOoUiqSK9pWo3/b6tQoVyzY4Z2RJQJjjPGzY4lJTFq6gzGxm5i6aieqcFadsgyIqc6FzSpTtEhw5x9IyxKBMcYE0JZ9R/h2bhxjYuPYuOcwJSLC6N2yCgNiqtGqeuk8MerIEoExxuSC5GRl1vo9jIndxMRFWzmakEz9ClEMiKnOpa2rEl0ieCWqLREYY0wuO3A0gQkLtzImdhPzNu4jLEQ4p1EFBsRUp3vD6OxVTM0BSwTGGBNEq3cc4OvYOL6Zu5ldB48RXSKCy1tXpX9MdepViMp8A35gicAYY/KAhKRkpqzYyZjYTfyxfAdJyUqbGqUZEFOdi1tUpkRkJpPi5IAlAmOMyWN2HDjKuHmbGRMbx+odBykaHspFzSszIKYa7WuX9XsHsyUCY4zJo1SVeZv28XXsJn5YsJWDxxKpVa4Y/WOqc3mbqlQu5Z8pMC0RGGNMPnD4eCI/L97GmNhNzFi7hxCBrg2iGRBTnZ6NKxARlv1zEywRGGNMPrNh9yHGzolj7Jw4tsYfpXSxcP7dpyl9W1XN1vasDLUxxuQzNcsV577zG3LPuQ34a/UuxsRuompp/xwmSssSgTHG5GGhIUK3BtF0axAdsDZy94wGY4wxeY4lAmOMKeQsERhjTCFnicAYYwo5SwTGGFPIWSIwxphCzhKBMcYUcpYIjDGmkMt3JSZEZCewIZsPLw/s8mM4+Z29Hqey1+Mkey1OVRBej5qqmu5ZafkuEeSEiMSeqdZGYWSvx6ns9TjJXotTFfTXww4NGWNMIWeJwBhjCrnClgjeDXYAeYy9Hqey1+Mkey1OVaBfj0LVR2CMMeZ0hW2PwBhjTBqWCIwxppArNIlARHqJyAoRWS0iDwc7nmARkeoiMllElorIEhG5O9gx5QUiEioi80RkQrBjCTYRKS0iY0VkuYgsE5GOwY4pWERkuPc5WSwiX4pIZLBjCoRCkQhEJBR4E7gQaAIMEpEmwY0qaBKB+1S1CXAWcEchfi1SuxtYFuwg8ojXgZ9VtRHQkkL6uohIVWAYEKOqzYBQYGBwowqMQpEIgPbAalVdq6rHgdFA3yDHFBSqulVV53rXD+A+5NmbDbuAEJFqwMXA+8GOJdhEpBTQFfgAQFWPq+q+oAYVXGFAUREJA4oBW4IcT0AUlkRQFdiU6nYchfzLD0BEagGtgZlBDiXYXgMeBJKDHEdeUBvYCXzkHSp7X0SKBzuoYFDVzcBIYCOwFYhX1V+DG1VgFJZEYNIQkSjgG+AeVd0f7HiCRUR6AztUdU6wY8kjwoA2wFuq2ho4BBTKPjURKYM7clAbqAIUF5GrgxtVYBSWRLAZqJ7qdjXvvkJJRMJxSeBzVf022PEEWSegj4isxx0yPEdEPgtuSEEVB8Spaspe4lhcYiiMzgXWqepOVU0AvgXODnJMAVFYEsFsoL6I1BaRIrgOn/FBjikoRERwx3+XqeorwY4n2FT1EVWtpqq1cO+LP1S1QP7q84WqbgM2iUhD766ewNIghhRMG4GzRKSY97npSQHtOA8LdgC5QVUTReRO4Bdcz/+HqrokyGEFSyfgGmCRiMz37ntUVScGLySTx9wFfO79aFoLXB/keIJCVWeKyFhgLm603TwKaKkJKzFhjDGFXGE5NGSMMeYMLBEYY0whZ4nAGGMKOUsExhhTyFkiMMaYQs4SgcmzRERF5OVUt+8Xkaf8tO2PRaSfP7aVSTv9vQqek9PcX0tEjojI/FSXa/3YbnerpGp8VSjOIzD51jHgchF5XlV3BTuYFCISpqqJPq5+I3Czqv6VzrI1qtrKf5EZkz22R2DyskTcCTzD0y5I+4teRA56f7uLyJ8i8r2IrBWRF0RksIjMEpFFIlI31WbOFZFYEVnp1RxKmZfgJRGZLSILRWRoqu1OE5HxpHOmrYgM8ra/WERe9O77F9AZ+EBEXvL1SYvIQRF51auD/7uIRHv3txKRGV5c33m1cBCReiIySUQWiMjcVM8xKtW8Ap97Z8fivSZLve2M9DUuU4Cpql3skicvwEGgJLAeKAXcDzzlLfsY6Jd6Xe9vd2AfUBmIwNWU+re37G7gtVSP/xn3Y6g+rsZOJHAL8Li3TgQQiys61h1XgK12OnFWwZUjiMbtZf8BXOotm4KrZ5/2MbWAI8D8VJcu3jIFBnvX/wX817u+EOjmXX861XOZCVzmXY/ElUvuDsTj6mqFANNxSakcsIKTJ5OWDvb/2S7Bv9gegcnT1FVG/R9ughBfzVY378IxYA2QUjp4Ee4LOMUYVU1W1VW4UgqNgPOBa73yGzNxX5z1vfVnqeq6dNprB0xRV5wsEfgcV9M/M2tUtVWqyzTv/mTgK+/6Z0Bnb56A0qr6p3f/J0BXESkBVFXV7wBU9aiqHk4Vb5yqJuMSTS1ccjiK20u5HEhZ1xRilghMfvAa7lh76rr4iXjvXxEJAYqkWnYs1fXkVLeTObVfLG19FQUEuCvVl3NtPVmD/lBOnkQOZLcOTOrXIQlI6dtoj6sq2hu3V2QKOUsEJs9T1T3AGFwySLEeaOtd7wOEZ2PT/UUkxDumXgd3yOQX4DavVDci0sCHiVlmAd1EpLw3Leog4M9MHpORECCl/+Mq4C9VjQf2ikgX7/5rgD/VzTIXJyKXevFGiEixM23Ym4eilLoig8NxU1GaQs5GDZn84mXgzlS33wO+F5EFuF+12fm1vhH3JV4SuFVVj4rI+7hDKHO9ztWdwKUZbURVt4rIw8Bk3B7Fj6r6vQ/t101VARZcVdw3cM+lvYg8DuwArvSWXwe87X3Rp64Keg3wjog8DSQA/TNoswTudYv0Yr3XhzhNAWfVR43JY0TkoKpGBTsOU3jYoSFjjCnkbI/AGGMKOdsjMMaYQs4SgTHGFHKWCIwxppCzRGCMMYWcJQJjjCnk/h+nlpudrykNIwAAAABJRU5ErkJggg==",
523 | "text/plain": [
524 | ""
525 | ]
526 | },
527 | "metadata": {
528 | "needs_background": "light"
529 | },
530 | "output_type": "display_data"
531 | }
532 | ],
533 | "source": [
534 | "plt.title(\"Sentiment Analysis (LSTM): Learning Curves\")\n",
535 | "plt.xlabel(\"Number of Epochs\")\n",
536 | "plt.ylabel(\"Binary Cross Entropy Loss\")\n",
537 | "plt.plot(train_losses, label = \"Training Loss\")\n",
538 | "plt.plot(valid_losses, label= \"Validation Loss\")\n",
539 | "plt.legend()\n",
540 | "plt.show()"
541 | ]
542 | },
543 | {
544 | "cell_type": "markdown",
545 | "metadata": {},
546 | "source": [
547 | "# Testing"
548 | ]
549 | },
550 | {
551 | "cell_type": "code",
552 | "execution_count": 22,
553 | "metadata": {
554 | "colab": {
555 | "base_uri": "https://localhost:8080/"
556 | },
557 | "id": "YLpKGbjyu2OK",
558 | "outputId": "2ba60055-1f76-4b99-a5b9-4b4315e8e5df"
559 | },
560 | "outputs": [
561 | {
562 | "name": "stdout",
563 | "output_type": "stream",
564 | "text": [
565 | "Test Loss: 0.2986\n",
566 | "Test Accuracy: 87.62%\n"
567 | ]
568 | }
569 | ],
570 | "source": [
571 | "model.load_state_dict(torch.load('LSTM.pt'))\n",
572 | "\n",
573 | "test_loss, test_acc = Evaluate(model, test_iterator, criterion)\n",
574 | "\n",
575 | "print(f'Test Loss: {test_loss:.4f}')\n",
576 | "print(f'Test Accuracy: {test_acc*100:.2f}%')"
577 | ]
578 | },
579 | {
580 | "cell_type": "markdown",
581 | "metadata": {
582 | "id": "o3vZ2NNn8sM6"
583 | },
584 | "source": [
585 | "# Sampling"
586 | ]
587 | },
588 | {
589 | "cell_type": "code",
590 | "execution_count": 23,
591 | "metadata": {
592 | "id": "e0hc5XtDsqIH"
593 | },
594 | "outputs": [],
595 | "source": [
596 | "import spacy\n",
597 | "nlp = spacy.load('en_core_web_sm')\n",
598 | "\n",
599 | "def predict_sentiment(model, text):\n",
600 | " model.eval()\n",
601 | " tokenized = [tok.text for tok in nlp.tokenizer(text)]\n",
602 | " indexed = [FIELD.vocab.stoi[t] for t in tokenized]\n",
603 | " length = [len(indexed)]\n",
604 | " tensor = torch.LongTensor(indexed).to(device)\n",
605 | " tensor = tensor.unsqueeze(1)\n",
606 | " length_tensor = torch.LongTensor(length)\n",
607 | " prediction = torch.sigmoid(model(tensor, length_tensor))\n",
608 | " return prediction.item()"
609 | ]
610 | },
611 | {
612 | "cell_type": "code",
613 | "execution_count": 24,
614 | "metadata": {
615 | "colab": {
616 | "base_uri": "https://localhost:8080/",
617 | "height": 311
618 | },
619 | "id": "vc0zO3gTws-d",
620 | "outputId": "fde91c09-6b91-4f2b-e49d-124d528c52aa"
621 | },
622 | "outputs": [
623 | {
624 | "data": {
625 | "text/plain": [
626 | "0.011234746314585209"
627 | ]
628 | },
629 | "execution_count": 24,
630 | "metadata": {},
631 | "output_type": "execute_result"
632 | }
633 | ],
634 | "source": [
635 | "predict_sentiment(model, \"This film is not bad\")"
636 | ]
637 | },
638 | {
639 | "cell_type": "code",
640 | "execution_count": 25,
641 | "metadata": {
642 | "id": "Umdey8pIw8yp"
643 | },
644 | "outputs": [
645 | {
646 | "data": {
647 | "text/plain": [
648 | "0.939202070236206"
649 | ]
650 | },
651 | "execution_count": 25,
652 | "metadata": {},
653 | "output_type": "execute_result"
654 | }
655 | ],
656 | "source": [
657 | "predict_sentiment(model, \"This film is excellent\")"
658 | ]
659 | },
660 | {
661 | "cell_type": "code",
662 | "execution_count": 26,
663 | "metadata": {
664 | "id": "8rbQ8FYRw-TM"
665 | },
666 | "outputs": [
667 | {
668 | "data": {
669 | "text/plain": [
670 | "0.0072547681629657745"
671 | ]
672 | },
673 | "execution_count": 26,
674 | "metadata": {},
675 | "output_type": "execute_result"
676 | }
677 | ],
678 | "source": [
679 | "predict_sentiment(model, \"This film is bad\")"
680 | ]
681 | }
682 | ],
683 | "metadata": {
684 | "accelerator": "GPU",
685 | "colab": {
686 | "name": "LSTM.ipynb",
687 | "provenance": []
688 | },
689 | "kernelspec": {
690 | "display_name": "Python 3",
691 | "name": "python3"
692 | },
693 | "language_info": {
694 | "codemirror_mode": {
695 | "name": "ipython",
696 | "version": 3
697 | },
698 | "file_extension": ".py",
699 | "mimetype": "text/x-python",
700 | "name": "python",
701 | "nbconvert_exporter": "python",
702 | "pygments_lexer": "ipython3",
703 | "version": "3.9.7"
704 | }
705 | },
706 | "nbformat": 4,
707 | "nbformat_minor": 0
708 | }
709 |
--------------------------------------------------------------------------------
/text_classification/plots/BERT.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IvLabs/Natural-Language-Processing/dcfd38d00a8d5137b122c778a67445d7e95cf53e/text_classification/plots/BERT.png
--------------------------------------------------------------------------------
/text_classification/plots/CNN.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IvLabs/Natural-Language-Processing/dcfd38d00a8d5137b122c778a67445d7e95cf53e/text_classification/plots/CNN.png
--------------------------------------------------------------------------------
/text_classification/plots/FastText.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IvLabs/Natural-Language-Processing/dcfd38d00a8d5137b122c778a67445d7e95cf53e/text_classification/plots/FastText.png
--------------------------------------------------------------------------------
/text_classification/plots/LSTM.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IvLabs/Natural-Language-Processing/dcfd38d00a8d5137b122c778a67445d7e95cf53e/text_classification/plots/LSTM.png
--------------------------------------------------------------------------------
/word_rnn/README.md:
--------------------------------------------------------------------------------
1 | # Paragraph generation (Language modeling)
2 |
3 | Model was trained over Book 1 - Philosopher's stone.txt which is 1st edition of Harry Potter franchise.
4 |
5 | Text consists of
6 | 5978 distinct words (vocabilary size)
7 | 3275 usable paragraphs
8 |
9 |
10 | > It is observed that model trained with nn.RNN is unable to form long term dependencies. Paragraphs formed were very short.
11 | Meaning of sentence is lost just in 3-4 words . Although it is able to recognise end of paragraph. Model is wise enough to choose proper punctuation marks. For example- double quotes are closed in almost every trial.
12 |
13 | > Whereas , model trained with nn.LSTM is able to form long term dependencies. Even next sentence formed after one shows some context shared. LSTM shows better results that RNN and GRU. Sentences and paragraphs are ended at proper position and proper meaning than other two.
14 |
15 | > GRU Gives somewhat similar results like LSTM. Sentences show some what meaning and shared context. But next sentence formed looses previous context. Sentence ending and paragraph ending is done properly. Punctuations are also maintained. Also you can observe that , length of sequence formed decreases.
16 |
17 |
18 | GRU Loss function
19 |
20 |
21 | 
22 |
23 | #embedding = 128
24 | #hidden_size = 256
25 | #num_layers = 4
26 | #learning_rate = 0.0006
27 | #epochs = 25
28 | #3,782,234 trainable parameters
29 |
30 | LSTM Loss function
31 |
32 |
33 | 
34 |
35 | #embedding = 128
36 | #hidden_size = 256
37 | #num_layers = 4
38 | #learning_rate = 0.0006
39 | #epochs = 25
40 | #4,275,802 trainable parameters
41 |
42 | RNN Loss function
43 |
44 | 
45 |
46 | #embedding = 150
47 | #hidden_size = 400
48 | #num_layers = 3
49 | #learning_rate = 0.0006
50 | #epochs = 25
51 | #4,156,278 trainable parameters
52 | > Since RNN has no cell states or gates those help in remembering details from previous inputs, we have increased size of hidden layer and embedding layer. Increment in number of parameters show better results.
53 |
54 |
55 |
56 |
57 |
58 |
59 |
60 | ## __ Results __ -
61 |
62 | 
63 |
64 | ## LSTM
65 |
66 | 1. testing2('“ i tell you , we’re going to win that')
67 | - “ i tell you , we’re going to win that in the summer - year waffle , something that he is to die — dumbledore says nothing about it , i’ll therefore nicolas ‘ rude with all with a fifty - points - points points that year , you were steamrollered ! ”
68 |
69 | 2. testing2('there was a very good chance they were going to get caught')
70 | - 'there was a very good chance they were going to get caught some point \\ ”'
71 |
72 |
73 | 3. testing2('they pulled on their bathrobes , picked up their wands ,')
74 | - 'they pulled on their bathrobes , picked up their wands , insisting themselves , and snape into the slytherin table . “ aren’t you ? ” screeched dumbledore , but she passed , but mr . dursley was , mysterious a long bite with broad soon .'
75 |
76 | 4. testing2('“ i am scared ”')
77 | - '“ i am scared ” . . . for the soul . “ i can knew it , ” said dumbledore , grinning , grinning down the mom , “ mom , mom . ” '
78 |
79 | 5. testing2('hermione took out a list of subjects and titles she had')
80 | - 'hermione took out a list of subjects and titles she had found talking to the ceiling . harry found a quiver again at the barrier , but the whisperers bear the platform as dressed there , too .'
81 |
82 | ## GRU
83 |
84 | 1. testing2('“ i tell you , we’re going to win that')
85 | - '“ i tell you , we’re going to win that , ” said harry .'
86 |
87 | 2. testing2('there was a very good chance they were going to get caught')
88 | - 'there was a very good chance they were going to get caught the fact . some and its common families and fifty trolls in the air ; it turned on the note .'
89 |
90 | 3. testing2('they pulled on their bathrobes , picked up their wands ,')
91 | - 'they pulled on their bathrobes , picked up their wands , stone - green - muggle words . the same thing was still around at quirrell’s mouth , a purple hand , and a still screamed of books into the ceiling , and a same same through the same one ; the last last time in harry’s hand , reached into the same wall . the rest of the first thing they were talking and down , and waving into the table and winked , it was a terrible nostrils in'
92 |
93 | 4. testing2('“ i am scared ”')
94 | - '“ i am scared ” . . . weasley face at harry . “ i said it , however , i thought . . . ”'
95 |
96 | 5. testing2('hermione took out a list of subjects and titles she had')
97 | - 'hermione took out a list of subjects and titles she had already passed at the table , and she looked forward to herself to be allowed to be looking at them . she looked very high , she looked at them .'
98 |
99 |
100 | ## RNN
101 |
102 | 1. testing2('“ i tell you , we’re going to win that')
103 | - “ i tell you , we’re going to win that hole . ”'
104 |
105 | 2. testing2('there was a very good chance they were going to get caught')
106 | - 'there was a very good chance they were going to get caught the whole day .'
107 |
108 | 3. testing2('they pulled on their bathrobes , picked up their wands ,')
109 | 'they pulled on their bathrobes , picked up their wands , and fell into blackness .'
110 |
111 | 4. testing2('“ i am scared ”')
112 | - '“ i am scared ” gasped . . . . ”'
113 |
114 | 5. testing2('hermione took out a list of subjects and titles she had')
115 | - 'hermione took out a list of subjects and titles she had to say .'
116 |
117 |
118 | ## Resources :
119 | https://youtu.be/iWea12EAu6U CS224N course over NLP.
120 |
--------------------------------------------------------------------------------
/word_rnn/ezgif.com-gif-maker.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IvLabs/Natural-Language-Processing/dcfd38d00a8d5137b122c778a67445d7e95cf53e/word_rnn/ezgif.com-gif-maker.gif
--------------------------------------------------------------------------------
/word_rnn/paragraph generation loss gru.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IvLabs/Natural-Language-Processing/dcfd38d00a8d5137b122c778a67445d7e95cf53e/word_rnn/paragraph generation loss gru.png
--------------------------------------------------------------------------------
/word_rnn/paragraph generation loss lstm.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IvLabs/Natural-Language-Processing/dcfd38d00a8d5137b122c778a67445d7e95cf53e/word_rnn/paragraph generation loss lstm.png
--------------------------------------------------------------------------------
/word_rnn/paragraph generation loss rnn.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IvLabs/Natural-Language-Processing/dcfd38d00a8d5137b122c778a67445d7e95cf53e/word_rnn/paragraph generation loss rnn.png
--------------------------------------------------------------------------------
/word_rnn/wordRNN_paragraph_gru2.pth:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IvLabs/Natural-Language-Processing/dcfd38d00a8d5137b122c778a67445d7e95cf53e/word_rnn/wordRNN_paragraph_gru2.pth
--------------------------------------------------------------------------------
/word_rnn/wordRNN_paragraph_lstm2.pth:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IvLabs/Natural-Language-Processing/dcfd38d00a8d5137b122c778a67445d7e95cf53e/word_rnn/wordRNN_paragraph_lstm2.pth
--------------------------------------------------------------------------------
/word_rnn/wordRNN_paragraph_rnn2.pth:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IvLabs/Natural-Language-Processing/dcfd38d00a8d5137b122c778a67445d7e95cf53e/word_rnn/wordRNN_paragraph_rnn2.pth
--------------------------------------------------------------------------------
61 | 
62 | 
63 | 
64 | 
52 | 
53 | 
54 | 
55 |