├── README.md
├── information_retrieval.ipynb
├── output_kmeans.csv
├── pipeline.png
└── siamese.png


/README.md:
--------------------------------------------------------------------------------
 1 | # Information Retrieval System
 2 | 
 3 | <p align="center">
 4 |   <b><h2><ins>Motivation</ins></h2></b><br>
 5 | </p>  
 6 | 
 7 | In the following steps, I will discuss how I approached the problem of quora question pairs and the motivation for coming up with an architecture like Siamese network.
 8 | 
 9 | <b><ins>How and why I chose the Siamese based network?</ins></b>
10 | <ul>
11 |   <li>Having worked on computer vision in domains like medical imaging, biometrics, and document image super-resolution, I approached the problem of deciding the architecture for the quora question pairs dataset based on specifically biometrics.</li>
12 |    <li>While I was working on medical imaging at IIT Mandi, I also worked on a bit of biometrics domain wherein I had palm & knuckle print images of the left & right hand. Similar to that, in this problem I had question 1 & question 2 as an input. Hence, I went with the best possible approach of using a <b>Siamese based network.</b></li>
13 |    <li>Even though the task was to find from a pool of questions which top-3 questions are closer to the user inputted question, to solve this problem, classifying given two questions are they similar or dissimilar was essential. The dataset had same questions repeated multiple times against different set of questions, so the network learned meaningful and discriminative features which eventually helped in solving the prime task of finding top-3 suggestions.</li>
14 |    <li>A siamese network had two inputs and it works on a phenomenon of sharing weights. The network’s input was question 1 and question 2 as tokenized vectors each of length 103, label being 0 or 1.
15 | </li>
16 |    <li>I tried a variety of architectures for the siamese network. Note that there are three functions in my network namely <b>Embedding()</b>, <b>Middle()</b>, <b>Predict()</b>. Following are some of architectural changes I made in the <b>Middle()</b> function:
17 |      <ul>
18 |        <li>Fully Conv1D siamese network which gave a validation accuracy of around 75%. This had multiple combinations of conv1D layers.</li>
19 |        <li>Conv1D + LSTM + Dense (In sequence) along with MaxPool1D layer which gave me a 2% boost in validation accuracy (~78%).</li>
20 |        <li>Then, finally I added a MaxPool1D layer just after the embedding layer output to capture important features instead of directly passing the embedding layer output to conv or lstm layers. This further improved my performance and I finally achieved a training accuracy of around 98-99% and validation accuracy of around 80-81% approximately. The final architecture is simple and smaller in size consisting of just one maxpooling, lstm and dense layer in the Siamese function.
21 | </li>
22 |        </li>
23 |   </ul>
24 | <p align="center">
25 |   <img src="https://github.com/aditya-AI/Top-3-Question-Suggestions-For-A-Given-Query/blob/master/pipeline.png">
26 | </p>
27 | 
28 | <b><ins>Finding the top-3 closest match for the given query question</ins></b>
29 | 
30 | <ul>
31 |   <li>Since the goal of this task was to find from a pool of questions (training data) which top-3 questions are closer to the user inputted question (validation data).
32 | </li>
33 |    <li>In order to accomplish this, I broke my siamese network and created a new model which had just two functions namely Embedding and Siamese functions.
34 | </li>
35 |    <li>At test time the input was fed to the embedding layer and further extracted features from the Siamese function that had a Dense layer outputting 128 features maps. (Input being training and validation questions fed one at a time).
36 | </li>
37 |    <li>Finally, saved these features representations to the disk especially the training feature maps so that I did not have to compute the predictions again and again.
38 | </li>
39 |    <li>This process of extracting features was computationally efficient as I got features for all the training data (~650K) by processing them only and only once.
40 | </li>
41 |    <li>The feature maps were further reshaped to get a two-dimensional array in which the first dimension was all the training questions (1 & 2) and the second dimension was the size of the feature map (128).
42 | </li>
43 |   </ul>
44 |   
45 |   <b><ins>Reducing the search space of finding the top-3 closest match to the query question</ins></b>
46 |   
47 |   <ul>
48 |   <li>Finding the top-3 closest match for a given query from a pool of 650K or even more questions is a time-consuming task and is also not computationally efficient if done through the brute-force method.
49 | </li>
50 |   <li>To reduce the search space I used K-Means clustering to reduce the time complexity by a big margin.
51 | </li>
52 |   <li>For example: Finding the top-3 closest match for 100 questions from a pool of 650K questions took 313 seconds when done using the brute-force method. Whereas, using the K-Means clustering technique it took 200 seconds for finding the top-3 closest match for <b>1000</b> questions from a pool of 650K questions.
53 | </li>
54 |   <li>Not only that clustering the data points did not affect the final results and the clustering was robust when compared with the brute-force method since both the methods gave a same top-3 closest match for each given query question.
55 | </li>
56 |  </ul>
57 |  
58 |  <p align="center">
59 |   <img src="https://github.com/aditya-AI/Top-3-Question-Suggestions-For-A-Given-Query/blob/master/siamese.png">
60 | </p>
61 | 


--------------------------------------------------------------------------------
/information_retrieval.ipynb:
--------------------------------------------------------------------------------
   1 | {
   2 |  "cells": [
   3 |   {
   4 |    "cell_type": "markdown",
   5 |    "metadata": {},
   6 |    "source": [
   7 |     "## Quora Question Pairs"
   8 |    ]
   9 |   },
  10 |   {
  11 |    "cell_type": "markdown",
  12 |    "metadata": {},
  13 |    "source": [
  14 |     "#### In this task, for a given user query I had to find which is the most similar question from the given set of questions. A system is to be designed which will be able to give top-3 question suggestions for the given query."
  15 |    ]
  16 |   },
  17 |   {
  18 |    "cell_type": "markdown",
  19 |    "metadata": {},
  20 |    "source": [
  21 |     "<b><a href=\"http://qim.fs.quoracdn.net/quora_duplicate_questions.tsv\">DataSet Link</a></b>"
  22 |    ]
  23 |   },
  24 |   {
  25 |    "cell_type": "markdown",
  26 |    "metadata": {},
  27 |    "source": [
  28 |     "<img src=\"pipeline.png\" alt=\"Italian Trulli\">"
  29 |    ]
  30 |   },
  31 |   {
  32 |    "cell_type": "markdown",
  33 |    "metadata": {},
  34 |    "source": [
  35 |     "I started by importing the necessary libraries. `re` was used for preprocessing the text of the data, removing unwanted characters or noise from the data. `Pandas` is used for reading the data in a row and column format binding it in a dataframe. `Matplotlib` was used for plotting the statistics of the data. To visualise the plots within jupyter notebook `%matplotlib inline` was added. `Pickle` was used to save the modified dataframes, vocabulary and embeddings of the data, so that it can be later reused by just loading the specific pickle file. `BeautifulSoup` was imported for removing any kind of *HTML* tags present in the data. `Numpy` was imported to handle multi-dimensional arrays and since the input, output, predictions from the model were an array, numpy was quite useful. Then, I imported `NLTK` a natural language processing toolkit to remove stopwords from the data like *if, or, him* etc. which does not give any important information. `Gensim` library was imported to load Google's `Word2Vec` which converted each of the word in to a 300 dimensional vector."
  36 |    ]
  37 |   },
  38 |   {
  39 |    "cell_type": "markdown",
  40 |    "metadata": {},
  41 |    "source": [
  42 |     "**Note**: I did not use stemming, since stemming usually takes each word to its root and sometimes that root word is not there in the pre-trained word2vec models. "
  43 |    ]
  44 |   },
  45 |   {
  46 |    "cell_type": "markdown",
  47 |    "metadata": {},
  48 |    "source": [
  49 |     "The below libraries were imported as and when they were required I collated them to one cell. There are many other libraries that were used which I have not put in this cell and you will find them in later sections as the implementation proceeds."
  50 |    ]
  51 |   },
  52 |   {
  53 |    "cell_type": "code",
  54 |    "execution_count": 2,
  55 |    "metadata": {},
  56 |    "outputs": [
  57 |     {
  58 |      "name": "stderr",
  59 |      "output_type": "stream",
  60 |      "text": [
  61 |       "[nltk_data] Downloading package stopwords to\n",
  62 |       "[nltk_data]     /users/aditya.sharma/nltk_data...\n",
  63 |       "[nltk_data]   Package stopwords is already up-to-date!\n"
  64 |      ]
  65 |     }
  66 |    ],
  67 |    "source": [
  68 |     "import re\n",
  69 |     "import pandas as pd\n",
  70 |     "import matplotlib.pyplot as plt\n",
  71 |     "%matplotlib inline\n",
  72 |     "import pickle\n",
  73 |     "from bs4 import BeautifulSoup\n",
  74 |     "import numpy as np                                                                \n",
  75 |     "import nltk                                      \n",
  76 |     "nltk.download('stopwords')\n",
  77 |     "from nltk.corpus import stopwords                                \n",
  78 |     "from gensim.models import KeyedVectors"
  79 |    ]
  80 |   },
  81 |   {
  82 |    "cell_type": "markdown",
  83 |    "metadata": {},
  84 |    "source": [
  85 |     "### Loading the data"
  86 |    ]
  87 |   },
  88 |   {
  89 |    "cell_type": "code",
  90 |    "execution_count": 3,
  91 |    "metadata": {},
  92 |    "outputs": [],
  93 |    "source": [
  94 |     "url=\"http://qim.fs.quoracdn.net/quora_duplicate_questions.tsv\"\n",
  95 |     "original_data=pd.read_csv(url,error_bad_lines=False,sep='\\t')"
  96 |    ]
  97 |   },
  98 |   {
  99 |    "cell_type": "markdown",
 100 |    "metadata": {},
 101 |    "source": [
 102 |     "#### Output first few rows of the data to see how it looks like. As per my observation, we need three columns for this task namely *question1*, *question2* and *is_duplicate*."
 103 |    ]
 104 |   },
 105 |   {
 106 |    "cell_type": "code",
 107 |    "execution_count": 4,
 108 |    "metadata": {
 109 |     "scrolled": true
 110 |    },
 111 |    "outputs": [
 112 |     {
 113 |      "data": {
 114 |       "text/html": [
 115 |        "<div>\n",
 116 |        "<style scoped>\n",
 117 |        "    .dataframe tbody tr th:only-of-type {\n",
 118 |        "        vertical-align: middle;\n",
 119 |        "    }\n",
 120 |        "\n",
 121 |        "    .dataframe tbody tr th {\n",
 122 |        "        vertical-align: top;\n",
 123 |        "    }\n",
 124 |        "\n",
 125 |        "    .dataframe thead th {\n",
 126 |        "        text-align: right;\n",
 127 |        "    }\n",
 128 |        "</style>\n",
 129 |        "<table border=\"1\" class=\"dataframe\">\n",
 130 |        "  <thead>\n",
 131 |        "    <tr style=\"text-align: right;\">\n",
 132 |        "      <th></th>\n",
 133 |        "      <th>id</th>\n",
 134 |        "      <th>qid1</th>\n",
 135 |        "      <th>qid2</th>\n",
 136 |        "      <th>question1</th>\n",
 137 |        "      <th>question2</th>\n",
 138 |        "      <th>is_duplicate</th>\n",
 139 |        "    </tr>\n",
 140 |        "  </thead>\n",
 141 |        "  <tbody>\n",
 142 |        "    <tr>\n",
 143 |        "      <th>0</th>\n",
 144 |        "      <td>0</td>\n",
 145 |        "      <td>1</td>\n",
 146 |        "      <td>2</td>\n",
 147 |        "      <td>What is the step by step guide to invest in sh...</td>\n",
 148 |        "      <td>What is the step by step guide to invest in sh...</td>\n",
 149 |        "      <td>0</td>\n",
 150 |        "    </tr>\n",
 151 |        "    <tr>\n",
 152 |        "      <th>1</th>\n",
 153 |        "      <td>1</td>\n",
 154 |        "      <td>3</td>\n",
 155 |        "      <td>4</td>\n",
 156 |        "      <td>What is the story of Kohinoor (Koh-i-Noor) Dia...</td>\n",
 157 |        "      <td>What would happen if the Indian government sto...</td>\n",
 158 |        "      <td>0</td>\n",
 159 |        "    </tr>\n",
 160 |        "    <tr>\n",
 161 |        "      <th>2</th>\n",
 162 |        "      <td>2</td>\n",
 163 |        "      <td>5</td>\n",
 164 |        "      <td>6</td>\n",
 165 |        "      <td>How can I increase the speed of my internet co...</td>\n",
 166 |        "      <td>How can Internet speed be increased by hacking...</td>\n",
 167 |        "      <td>0</td>\n",
 168 |        "    </tr>\n",
 169 |        "    <tr>\n",
 170 |        "      <th>3</th>\n",
 171 |        "      <td>3</td>\n",
 172 |        "      <td>7</td>\n",
 173 |        "      <td>8</td>\n",
 174 |        "      <td>Why am I mentally very lonely? How can I solve...</td>\n",
 175 |        "      <td>Find the remainder when [math]23^{24}[/math] i...</td>\n",
 176 |        "      <td>0</td>\n",
 177 |        "    </tr>\n",
 178 |        "    <tr>\n",
 179 |        "      <th>4</th>\n",
 180 |        "      <td>4</td>\n",
 181 |        "      <td>9</td>\n",
 182 |        "      <td>10</td>\n",
 183 |        "      <td>Which one dissolve in water quikly sugar, salt...</td>\n",
 184 |        "      <td>Which fish would survive in salt water?</td>\n",
 185 |        "      <td>0</td>\n",
 186 |        "    </tr>\n",
 187 |        "  </tbody>\n",
 188 |        "</table>\n",
 189 |        "</div>"
 190 |       ],
 191 |       "text/plain": [
 192 |        "   id  qid1  qid2                                          question1  \\\n",
 193 |        "0   0     1     2  What is the step by step guide to invest in sh...   \n",
 194 |        "1   1     3     4  What is the story of Kohinoor (Koh-i-Noor) Dia...   \n",
 195 |        "2   2     5     6  How can I increase the speed of my internet co...   \n",
 196 |        "3   3     7     8  Why am I mentally very lonely? How can I solve...   \n",
 197 |        "4   4     9    10  Which one dissolve in water quikly sugar, salt...   \n",
 198 |        "\n",
 199 |        "                                           question2  is_duplicate  \n",
 200 |        "0  What is the step by step guide to invest in sh...             0  \n",
 201 |        "1  What would happen if the Indian government sto...             0  \n",
 202 |        "2  How can Internet speed be increased by hacking...             0  \n",
 203 |        "3  Find the remainder when [math]23^{24}[/math] i...             0  \n",
 204 |        "4            Which fish would survive in salt water?             0  "
 205 |       ]
 206 |      },
 207 |      "execution_count": 4,
 208 |      "metadata": {},
 209 |      "output_type": "execute_result"
 210 |     }
 211 |    ],
 212 |    "source": [
 213 |     "original_data.head()"
 214 |    ]
 215 |   },
 216 |   {
 217 |    "cell_type": "markdown",
 218 |    "metadata": {},
 219 |    "source": [
 220 |     "Check if there are any `null` values in the dataset!"
 221 |    ]
 222 |   },
 223 |   {
 224 |    "cell_type": "code",
 225 |    "execution_count": 5,
 226 |    "metadata": {},
 227 |    "outputs": [
 228 |     {
 229 |      "data": {
 230 |       "text/plain": [
 231 |        "3"
 232 |       ]
 233 |      },
 234 |      "execution_count": 5,
 235 |      "metadata": {},
 236 |      "output_type": "execute_result"
 237 |     }
 238 |    ],
 239 |    "source": [
 240 |     "original_data.isnull().values.sum()"
 241 |    ]
 242 |   },
 243 |   {
 244 |    "cell_type": "markdown",
 245 |    "metadata": {},
 246 |    "source": [
 247 |     "Drop the three null values!"
 248 |    ]
 249 |   },
 250 |   {
 251 |    "cell_type": "code",
 252 |    "execution_count": 6,
 253 |    "metadata": {},
 254 |    "outputs": [],
 255 |    "source": [
 256 |     "original_data = original_data.dropna()"
 257 |    ]
 258 |   },
 259 |   {
 260 |    "cell_type": "code",
 261 |    "execution_count": 7,
 262 |    "metadata": {},
 263 |    "outputs": [
 264 |     {
 265 |      "data": {
 266 |       "text/plain": [
 267 |        "(404287, 6)"
 268 |       ]
 269 |      },
 270 |      "execution_count": 7,
 271 |      "metadata": {},
 272 |      "output_type": "execute_result"
 273 |     }
 274 |    ],
 275 |    "source": [
 276 |     "original_data.shape"
 277 |    ]
 278 |   },
 279 |   {
 280 |    "cell_type": "markdown",
 281 |    "metadata": {},
 282 |    "source": [
 283 |     "Since I will need the `original_data` later on, I copy it into a new variable called `data`."
 284 |    ]
 285 |   },
 286 |   {
 287 |    "cell_type": "code",
 288 |    "execution_count": 8,
 289 |    "metadata": {},
 290 |    "outputs": [],
 291 |    "source": [
 292 |     "data = original_data.copy()"
 293 |    ]
 294 |   },
 295 |   {
 296 |    "cell_type": "code",
 297 |    "execution_count": 9,
 298 |    "metadata": {},
 299 |    "outputs": [
 300 |     {
 301 |      "data": {
 302 |       "text/html": [
 303 |        "<div>\n",
 304 |        "<style scoped>\n",
 305 |        "    .dataframe tbody tr th:only-of-type {\n",
 306 |        "        vertical-align: middle;\n",
 307 |        "    }\n",
 308 |        "\n",
 309 |        "    .dataframe tbody tr th {\n",
 310 |        "        vertical-align: top;\n",
 311 |        "    }\n",
 312 |        "\n",
 313 |        "    .dataframe thead th {\n",
 314 |        "        text-align: right;\n",
 315 |        "    }\n",
 316 |        "</style>\n",
 317 |        "<table border=\"1\" class=\"dataframe\">\n",
 318 |        "  <thead>\n",
 319 |        "    <tr style=\"text-align: right;\">\n",
 320 |        "      <th></th>\n",
 321 |        "      <th>id</th>\n",
 322 |        "      <th>qid1</th>\n",
 323 |        "      <th>qid2</th>\n",
 324 |        "      <th>question1</th>\n",
 325 |        "      <th>question2</th>\n",
 326 |        "      <th>is_duplicate</th>\n",
 327 |        "    </tr>\n",
 328 |        "  </thead>\n",
 329 |        "  <tbody>\n",
 330 |        "    <tr>\n",
 331 |        "      <th>0</th>\n",
 332 |        "      <td>0</td>\n",
 333 |        "      <td>1</td>\n",
 334 |        "      <td>2</td>\n",
 335 |        "      <td>What is the step by step guide to invest in sh...</td>\n",
 336 |        "      <td>What is the step by step guide to invest in sh...</td>\n",
 337 |        "      <td>0</td>\n",
 338 |        "    </tr>\n",
 339 |        "    <tr>\n",
 340 |        "      <th>1</th>\n",
 341 |        "      <td>1</td>\n",
 342 |        "      <td>3</td>\n",
 343 |        "      <td>4</td>\n",
 344 |        "      <td>What is the story of Kohinoor (Koh-i-Noor) Dia...</td>\n",
 345 |        "      <td>What would happen if the Indian government sto...</td>\n",
 346 |        "      <td>0</td>\n",
 347 |        "    </tr>\n",
 348 |        "    <tr>\n",
 349 |        "      <th>2</th>\n",
 350 |        "      <td>2</td>\n",
 351 |        "      <td>5</td>\n",
 352 |        "      <td>6</td>\n",
 353 |        "      <td>How can I increase the speed of my internet co...</td>\n",
 354 |        "      <td>How can Internet speed be increased by hacking...</td>\n",
 355 |        "      <td>0</td>\n",
 356 |        "    </tr>\n",
 357 |        "    <tr>\n",
 358 |        "      <th>3</th>\n",
 359 |        "      <td>3</td>\n",
 360 |        "      <td>7</td>\n",
 361 |        "      <td>8</td>\n",
 362 |        "      <td>Why am I mentally very lonely? How can I solve...</td>\n",
 363 |        "      <td>Find the remainder when [math]23^{24}[/math] i...</td>\n",
 364 |        "      <td>0</td>\n",
 365 |        "    </tr>\n",
 366 |        "    <tr>\n",
 367 |        "      <th>4</th>\n",
 368 |        "      <td>4</td>\n",
 369 |        "      <td>9</td>\n",
 370 |        "      <td>10</td>\n",
 371 |        "      <td>Which one dissolve in water quikly sugar, salt...</td>\n",
 372 |        "      <td>Which fish would survive in salt water?</td>\n",
 373 |        "      <td>0</td>\n",
 374 |        "    </tr>\n",
 375 |        "  </tbody>\n",
 376 |        "</table>\n",
 377 |        "</div>"
 378 |       ],
 379 |       "text/plain": [
 380 |        "   id  qid1  qid2                                          question1  \\\n",
 381 |        "0   0     1     2  What is the step by step guide to invest in sh...   \n",
 382 |        "1   1     3     4  What is the story of Kohinoor (Koh-i-Noor) Dia...   \n",
 383 |        "2   2     5     6  How can I increase the speed of my internet co...   \n",
 384 |        "3   3     7     8  Why am I mentally very lonely? How can I solve...   \n",
 385 |        "4   4     9    10  Which one dissolve in water quikly sugar, salt...   \n",
 386 |        "\n",
 387 |        "                                           question2  is_duplicate  \n",
 388 |        "0  What is the step by step guide to invest in sh...             0  \n",
 389 |        "1  What would happen if the Indian government sto...             0  \n",
 390 |        "2  How can Internet speed be increased by hacking...             0  \n",
 391 |        "3  Find the remainder when [math]23^{24}[/math] i...             0  \n",
 392 |        "4            Which fish would survive in salt water?             0  "
 393 |       ]
 394 |      },
 395 |      "execution_count": 9,
 396 |      "metadata": {},
 397 |      "output_type": "execute_result"
 398 |     }
 399 |    ],
 400 |    "source": [
 401 |     "data.head()"
 402 |    ]
 403 |   },
 404 |   {
 405 |    "cell_type": "markdown",
 406 |    "metadata": {},
 407 |    "source": [
 408 |     "#### Check for data imbalancing!"
 409 |    ]
 410 |   },
 411 |   {
 412 |    "cell_type": "code",
 413 |    "execution_count": 168,
 414 |    "metadata": {},
 415 |    "outputs": [
 416 |     {
 417 |      "name": "stdout",
 418 |      "output_type": "stream",
 419 |      "text": [
 420 |       "id              149263\n",
 421 |       "qid1            149263\n",
 422 |       "qid2            149263\n",
 423 |       "question1       149263\n",
 424 |       "question2       149263\n",
 425 |       "is_duplicate    149263\n",
 426 |       "dtype: int64\n"
 427 |      ]
 428 |     }
 429 |    ],
 430 |    "source": [
 431 |     "print (data[data.is_duplicate == 1].count())"
 432 |    ]
 433 |   },
 434 |   {
 435 |    "cell_type": "markdown",
 436 |    "metadata": {},
 437 |    "source": [
 438 |     "As per my observation this dataset doesn't have data imbalancing issue, since both the classes have if not equal but fair amount of records in each class. The class label `0` has around 250K records whereas class label `1` has around 150K records."
 439 |    ]
 440 |   },
 441 |   {
 442 |    "cell_type": "code",
 443 |    "execution_count": 169,
 444 |    "metadata": {},
 445 |    "outputs": [
 446 |     {
 447 |      "data": {
 448 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjsAAAIICAYAAACIOC9YAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzs3Xm4JFV9//H3l0HZRDZxRFEHCKBGXAeXUWCAqKhEFEEwRFFU4sKijCYGUZHgElkVjAgKQ8QECKgEQYlxvGyjbC7gj8g+KooCggMDDAp8f3+c09A0fZe+3Ze+t3i/nqefmq46p+r06TtzP1N16lRkJpIkSU21wrAbIEmSNJUMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO9IMFBELIyIjYs4UHuPAeoz5U3UMPXoiYn79Pg8cdlseDfWzjgy7HZoeDDvSFKn/2Dpr5wwQxU4RcUZE/C4i/hwRf4yI8yJi34hYedhtHE9EzKk/cwuH3ZapEBFLWn+n6uuBiPhTRCyOiA9ExIpTcMzHVEBssoH/cEjSTBIRawKnAq8ClgJnA0uAtYHXAEcCH4iI12fmNcNq5wBcDDwbuHXYDenTF4A/AbOADYA3Ay8HtgV2HGK7NI0ZdiQ9ZkXECsB/AX8DnAPslpl/bNu+InAQ8M/A9yPihZl5+1Aa26fMvBv45bDbMQBHZuaS1puI+CxwCfCmiNgqM88dWss0bXkZS5oGIuKNEXFSRFwdEXfV12URsU/9hTyaFSJiv4j4ZUQsj4gbI+KIiHjiKMdZPyKOjojrI+LeeqnmvyNi8x7aukVEnFmPdW9E/D4ifhwRn5xA3V3rZYEjRtm+UkTcHhE3tS5LRMTjaz/8pG67u17SOCMi/mai7R7F31GCznXAju1BByAz78vM/YFTgGcCD/uMETEy2qXKiHhH/azv6LJtwt9DRKweER+PiF9ExB0RcWdEXBcRp0TEi2uZA4EbapXdOy73vKOWGfWSTERsHBH/HhG/rZfwflffb9yl7INjueqlv4vrd3JbRJwcEU/rUmfDiDg2Iq6NiHtq2Ssi4piIWKdb/01UZv4/YKS+fcl45SNijYj4bERcVf/O3B4R53T+LNXLgT+sbz/Z0afz+2mzHn2e2ZGmh88BDwAXAb8F1gC2oZyy3xx42yj1jgC2pFyGOYNy2eWDwBYR8crMXN4qGBEvAv6HcnnmHOCbwJOANwIXRMSbMvPssRoZEdsBZwF3AP9d27o25fLI+4FPjfM5v025VPR3EfGRzLyvY/sOwJrAYW3bFgJvBX4B/DtwD/BU4JXAdsD/jnPMsbynLg+rZz5GcxCwC/Cu2u6/TPaAvXwPERHA94B5wI+ArwL3AesDWwPnA5dRftmvCewL/JzSzy0/G6c9m1P6cHXKd3ol8Czg74EdIuJvMvOSLlXfD7yh1jkXeCmlj54fES/IzHvr/tejnHl5IuUS4enAypRLUG8Djgb++Ii99ybqcswxcvWS5YXAc2qbjqT0/VuA/4mI92XmV2rxVh/uXj/fSNuulvTZXj3aMtOXL19T8KL8w5sTLLtRl3UrACfW/by0Y9vCuv5W4JkddU6v2z7etn5F4FpgObBVx76eSgktNwErta0/sO5nftu61r6f36W9T5rgZ/1K3cf2XbadVbdtVt+vQQmBlwKzupRfp4/vZ0Xg3nq8jSdQ/re17Eva1o2M9h0D76jl3zHZ7wHYrO7jW6P8fKzV9n5OLbtwlPbMr9sPbFsXwP/V9bt1lN+lrv8lsEKXn4s7Wt9T27b/qNve0rZu77pu3y5tWg1YZYLf15K6nzkd6/8auLtu26JtfQIjo/zsfQWItvUbU0L4ve3779Znvmbmy8tY0jSQmdd1WfcA5cwOlDM23XwhM3/VUecjlICwR1u51wMbAUdlx5iGzPwd8HngKZRBnhNxT5f2TnTg64l1uXv7yoh4CuVz/jQzr2jtlvIL+V7KZ+o8Zj9nBNYGHl///JsJlG+VWb+PY072e+jW3w9k/+OH5lHO4vwoM7/Rsf9TgAuATSln0Tp9se17ajmuLrtdTur2Ge7KzEesH8cH66W0f4mIkyhnaFahBMLzR6sUEY+nnK1aBvxzZj54FijLwPMvUn4e3t5jezQDeBlLmgbquIWPAK8DNqT8j7fdI8ZBVI8YjJmZ10fEb4A5EbFmZv6JcrcKwDO7jdmg/M8WyuWosS5lfYNyx8tFEXEKZUzDhZl54xh1Otu3OCKuBv42ItZq+4W9G+UOm4VtZe+IiDOBvwV+FhGnUy7dXJRjX3aaSv3cht7r93Al5TLUWyPimZRLlRcAl2bmn/toR8uL6nLRKNsXUYLOC4HzOrZd2qV8KxCu1bbuv4HPAF+KiNdQLt1dCFzZHjh6sG9dJiW4XA6cBBwzTr1NgVUpP6+3ddm+CDiA8lnVMIYdacjqOIJLKGMYLqaMS7mNMjajNQ5jpVGq/2GU9b+nDKhdg3KbbmsQ6M7jNOcJY23MzG9GxPbAAsqZo3+on+Eyyv+Wvz/O/ltOBD4N7Ap8ua7bHfgL5VJIu12Af6IMJm6NCVoeEacBH87M0fpgPLcBf6b8b/7pwHi3lT+9Lm+Z5PGgx+8hM++PiG2ATwA7Af9at98ZESdS+nxZH+1Zoy5vGmV7a/2aXbb9qcu61jirWa0VmfmriHgJ5fLXdjx0e/hvIuLQzPxiTy2GDbLtbqwe9PNZNcN5GUsavndTgs6nMvOlmfn+zDwgMw+k3AU0ltmjrH9KXS7tWO6QmTHGa7wBxmTmWZm5DeV/79tSBkn/NfCdiHjOePWrr1MuS+0OEBEvpIxPObvzclhm3pOZB2bmJsAzKJciLqjL0yZ4vG6f4z7KgHAod2SNKiKeTRlT8wDw07ZND9Tt3f7j2O2XZs/fQ2benpkfysynU878vJsyjmYvHgqKk9Vqz1NG2b5eR7lJycz/y8xdKGFvLvBRyu+fL0TEu/rZdw8elc+q6cmwIw3fX9Xl6V22bTVO3Udsj4gNKWchltRLWAA/rsstJtXCLup4i0WZuR/lMsXjgddOsO5vKJcNXhoRm/LQ+J0TR69V6tWxJa+hDPR9ZZ+3Lh9bl/tFxCpjlDugLr/fEcZal+CeziPN7bKur+8hM6/NzK9RvvdllLvXWu6vy1mPqDi6VnCbP8r2revyJz3sc1RZbuW/LDP/lXKHHZS70B4NV1EGMj+/nk3t1O2zTqZPNQ0ZdqThW1KX89tX1rMd/zxO3X3rWI5WnRWAQyh/t09oK3cGZS6ZD0TE67rtKCJeHhGrjnWwiNhylLMYrTNMvYyjWViX76L84rsV+E7H8daNiM261F2NcqnnPsqlqFb5Z0TEs8b7HG3+E/gBJXCeFhHtY02IiFkRcRDlEtrdlMtp7S6uy/d01NuWh36Zt+vpe4iIDWp47bQW5dJm++De2ynjWJ7Rbb+juJASAl4ZETt1tGMnSii7mnImbVIi4sURsUaXTZP5mZm0OsbpG5Rb7P+lfVtEbATsQ7mM+vW2Ta0B8L30qaYhx+xIUyzGflbR+yljdD4CHBkRW1PGjmwMbE+Zg2WXMepfSBm4ewrl9PtrgOdT5l75fKtQZv4lInakDA49KyIWUwa+3k05K7E5ZWD0eoz9y+eLwNMi4kJKSPsz8GLKnEC/Ak4eo26nb1FuX/4g8DjKHUqd89c8DfhpRFxBGYj6G8p8LdtTLkd8MTPvbCv/75SzHlvz8HlRuqpjYnaizKL8OuD6iDirfpbW4yI2oNwNtltm/rxjFydQvrt/jojnUwYUb0I5w/UtyqMM2o/X6/fwfOCbEXEJ5Rbx3wHrUs7oPI6HxvCQmcsi4iLKHEvfoISU+4H/zszLR/n8GRG7A98HTomIMyiXyDalnHG5E3h7vctvst4G/ENEXEAJerdT7kj7W0q/HtnHvnv1UUqA26vOL/RDHppnZ3Vgr8y8oa38VZTpAHaNiL9Qfi4S+Hr7XZCaAR7te919+XqsvKjz7IzzWrOWfQ7lrpWbgbsoYeXdjDJ3Cg/Ns7MhZbDwLylzt/yW8svjiaO06cmUCQx/QflluowSrk6jjIFZsa3sgTxynp23UM6GXFPr3lH39Wlg3Un00Vfb+uLFXbavSRmcu6h+tnspA0lHKGdOoqP8SGebJ9iOoAwaPpMyuPu+tnYtZox5eCjjlc6mBINltQ1b0WWenV6/B8pt7p+hhNrf189/I/Bd4LVd9vtX9TP8kTKe6MHjM8acMZRw8/Xat3+py5OATbuUfcTPRdu2R/y8UiYb/DJlssPbKGejrqUExef28B0tocs8O+P8/RsZ5WfqX2t/30sZaP194NWj7Gdzytm/pW192tPPl6/hv6J+mZKkNhHxXErIuAvYMjOvHXKTJE2SY3YkqYvM/AXlTNa6wKL2sVGSZhbP7EjSGCJiB8pEc9dl5tfHKy9p+jHsSJKkRvMyliRJajRvPW+IJz3pSTlnzpyB7vOuu+5itdU6H9GkXtiH/bMP+2cf9s8+7N9U9OFll112a2auO145w05DzJkzh0sv7fZcvskbGRlh/vz5A93nY4192D/7sH/2Yf/sw/5NRR9GxITmO/IyliRJajTDjiRJajTDjiRJajTDjiRJajTDjiRJajTDjiRJajTDjiRJajTDjiRJajTDjiRJajTDjiRJajTDjiRJajTDjiRJajTDjiRJajTDjiRJajTDjiRJajTDjiRJajTDjiRJajTDjiRJajTDjiRJarQVh90ATV9X/HYp7/joWcNuxqiWfO71w26CJGkG8MyOJElqtKGHnYhYJyLeHRHfiohrI+KeiFgaERdExLsiYoWO8nMiIsd4nTzGsXaPiIsjYlk9xkhEbD9G+VkR8aGIuLy267aIODsi5o1RZ5WI+FREXBURyyPi5og4NSKePUadtSPiyIhYEhH3RsTvIuL4iFh/vP6TJEljmw6XsXYGvgzcBPwQ+DUwG9gR+Crw2ojYOTOzo97PgW932d8vuh0kIg4FFgA3AscBjwd2Bc6MiL0z8+iO8gGcDOwEXAUcDawN7AKcFxFvzswzOuqsBHwfeAVwKfAF4On1M74+IrbJzIs66qwDLAY2ARbVYz4LeGet8/LMvL7bZ5IkSeObDmHnauANwFmZ+UBrZUTsD1wMvJkSfE7vqPezzDxwIgeoZ2IWANcBm2fm7XX9IcBlwKER8Z3MXNJWbVdK0FkMbJuZy2udY4ALgOMiYlFm3tlWZz9K0DkN2KX1eSLiFEowOz4iNmv/nMBnKEHn8Mxc0NbmfShh6d+A7SbyOSVJ0iMN/TJWZi7KzDM7AgCZ+XvgmPp2fp+HeW9dfroVdOoxlgBfAlainElp9766PKAVdGqdS4BTgHUpYQh48ExQ6zj/2P556hmg84HnAFu11XkC8DbgLuDAjuMfDfwKeE1EbDjxjypJktoNPeyM4y91eV+XbU+NiH+IiP3r8nlj7Gebuvxel23f7ShDRKwMzAPupoSUcesAGwHPAK7OzBsmWOdlwCrAhR1niKhh6Zz6dusu+5MkSRMwHS5jdRURKwJvr2+7hZRX1Vd7nRFg98z8ddu61YCnAcsy86Yu+7mmLjdpW7cRMAu4PjO7Ba1udTaty6u7lB9kHUmS1INpG3aAzwHPBc7OzHPa1t8N/AtlDExr4O7zKJeBtgZ+EBEvyMy76rY16nLpKMdprV+zbd10rvOgiNgT2BNg9uzZjIyMjLKbyZm9CizYrFvWmx4G/XmnwrJly2ZEO6cz+7B/9mH/7MP+DbMPp2XYqYNzFwC/pIxpeVBm3gx8oqPKeRHxasrA4ZcC76YM7m20zDwWOBZg7ty5OX/+/IHu/6hvnMFhV0zLHxEAluw2f9hNGNfIyAiD/l4ea+zD/tmH/bMP+zfMPpx2Y3YiYi9KULkS2Dozb5tIvXq56av17ZZtm1pnR9agu9b6P82QOpIkqQfTKuxExAeBoyhz5Wxd78jqxS11uVprRb2c9VvgCRGxXpc6G9dl+7iZ64D7gQ3r2KGJ1LmqLkcbXzOoOpIkqQfTJuxExD8BRwA/owSdmyexm5fVZeckfIvqstt8Na/tKEO91XwxsCqwxUTqUALSr4FNImKDCdb5MXAP8IqIWL29cJ05+tX17Q+77E+SJE3AtAg7EfFxyoDkyygT+N06RtkXdT5Coq7fFvhQfXtSx+bWfD0fi4i12urMAT4A3Auc0FHny3V5cL0VvVVnc8osyrfQNtFhneG5dZzPt7cxInaghKYrgXPb6iwDvk45E3Vgx/H3AuYA5ziDsiRJkzf00acRsTtwEOWy0fnAPmV+vodZkpkL658PBzaOiMWURz9AuRurNX/NxzNzcXvlzFwcEYdTZji+PCJOozwuYhfKIyD27pg9GcpjG3akTBz404g4E1in1pkFvCcz7+iocziwfa1zUUT8gDL3zs6Uu8j26Jw8EdifMmnifhHxAsqs0c8GdgBupoQxSZI0SUMPO0Drks8s4IOjlDkXWFj//HXgTcDmlEtDjwP+AJwKHJ2Z3SYBJDMXRMQVlPCwJ/AA8BPgkMz8TpfyGRFvpVzO2gPYG1gOnAcc3Bmoap17I+JVwEeBt1LONN1BuU3+k5l5ZZc6f4yIlwOfBN5IOQP0R8qZpk9k5o2ddSRJ0sQNPezU51sd2EP5rwFfm+SxFvJQaJpI+fso44iO6KHO3ZRb4ztvjx+rzm3AvvUlSZIGaFqM2ZEkSZoqhh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRoQw87EbFORLw7Ir4VEddGxD0RsTQiLoiId0VE1zZGxLyIODsibqt1Lo+ID0bErDGOtX1EjNT9L4uIiyJi93Hat3tEXFzLL631tx+j/KyI+FBtzz21fWdHxLwx6qwSEZ+KiKsiYnlE3BwRp0bEs8dqmyRJGt/Qww6wM3Ac8FLgIuBI4HTgucBXgVMjItorRMQOwHnAlsC3gKOBxwNHACd3O0hE7AWcWfd7Uj3mU4GFEXHoKHUOBRYC69XyJwGbAWfW/XWWj3r8w2t7jq7t2xI4r7a7s85KwPeBTwB3AF8A/hd4E3BpRLy0W9skSdLErDjsBgBXA28AzsrMB1orI2J/4GLgzcCOlABERDyREjzuB+Zn5qV1/ceBRcBOEbFrZp7ctq85wKHAbcDczFxS1x8EXAIsiIjTM/NHbXXmAQuA64DNM/P2uv4Q4DLg0Ij4Tmtf1a7ATsBiYNvMXF7rHANcABwXEYsy8862OvsBrwBOA3Zp9UFEnAJ8Gzg+IjZr7xtJkjRxQz+zk5mLMvPMzl/mmfl74Jj6dn7bpp2AdYGTW0Gnll8OHFDfvq/jMHsAKwFHt4eTGmA+U9++t6NO6/2nW0Gn1lkCfKnu750ddVrHPaAVdGqdS4BTart3aq2vZ4Jax/nH9j7IzDOA84HnAFshSZImZehhZxx/qcv72tZtU5ff61L+POBuYF69PDSROt/tKDOpOhGxMjCvHv/8CR5nI+AZwNWZeUMPbZMkSRM0bcNORKwIvL2+bQ8cm9bl1Z11MvM+4AbK5bkNJ1jnJuAuYP2IWLUeezXgacCyur3TNXW5Sdu6jYBZwPW1HROpM2q7xqgjSZJ6MB3G7Izmc5TBxGdn5jlt69eoy6Wj1GutX7PHOqvVcndP4TEGUedBEbEnsCfA7NmzGRkZGWU3kzN7FViwWbfcNj0M+vNOhWXLls2Idk5n9mH/7MP+2Yf9G2YfTsuwExH7UAYH/xJ425CbM21l5rHAsQBz587N+fPnD3T/R33jDA67Ylr+iACwZLf5w27CuEZGRhj09/JYYx/2zz7sn33Yv2H24bS7jFVv6f4CcCWwdWbe1lGkdbZjDbprrf/TJOos7VhOxTH6rSNJknowrcJORHwQOAr4BSXo/L5Lsavq8hHjWOo4nw0oA5qvn2Cd9SiXsG7MzLsBMvMu4LfAE+r2ThvXZftYm+sot8NvWNsxkTqjtmuMOpIkqQfTJuxExD9RJgX8GSXo3DxK0UV1uV2XbVsCqwKLM/PeCdZ5bUeZSdWpt5ovrsffYoLHuQ74NbBJRGzQQ9skSdIETYuwUycE/Bxlsr5tM/PWMYqfBtwK7BoRc9v2sTJwcH375Y46JwD3AnvVCQZbddYC9q9vj+mo03r/sVquVWcO8IG6vxM66rSOe3BtT6vO5sAuwC3UyREBMjPbjvP59kdj1NmWt6BczjsXSZI0KUMffVqfTXUQ5RLQ+cA+HU+HAFiSmQsBMvOOiHgPJfSMRMTJlJmR30C5lfs0ygR+D8rMGyLiI8AXKY9gOAX4M2WCv/WBw9pnT651FkfE4ZQZji+PiNMoj4DYBVgb2Ltj9mQoj4rYse73pxFxJrBOrTMLeE9m3tFR53Bg+1rnooj4AWXunZ0pd4bt4ezJkiRN3tDDDmWMDZQw8MFRypxLeUYVAJn57YjYCvgY5XESKwPXUoLJF+sZk4fJzKMiYgnwYcr8PStQzpockJkndjtoZi6IiCsoZ3L2BB4AfgIckpnf6VI+I+KtlMtZewB7A8spkx0enJmLu9S5NyJeBXwUeCvwIcozsr4NfDIzrxylTyRJ0gQMPexk5oHAgZOodyHwuh7rnEl5GGgvdRbSFrQmUP4+ytijI3qoczflQaCf6KVtkiRpfNNizI4kSdJUMexIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRGM+xIkqRG6ynsRMQLI+L9EbFG27rVIuLEiPhTRPwuIvYdfDMlSZImp9czO/8EfCwzl7at+yzwtrqvdYDDI+LVA2qfJElSX3oNO3OBH7beRMTjgN2Bi4EnAxsAtwL7DKqBkiRJ/eg17DwZuLHt/VxgdeArmbk8M38HnAE8b0DtkyRJ6kuvYSeBFdvev7KuO7dt3S3Aun22S5IkaSB6DTu/Bl7W9n4H4MbMvL5t3VOB2/ttmCRJ0iD0GnZOBeZFxGkRcRLwcuC0jjLPBq4bROMkSZL6teL4RR7mCGA7YMf6/mfAQa2NEbEBsDnlDi1JkqSh6ynsZOYy4BUR8dy66srMfKC9CCUIXTqg9kmSJPWl1zM7AGTmL0ZZvwRY0kd7JEmSBsrHRUiSpEbr+cxORGwM7Au8BFgLmNWlWGbmRn22TZIkqW+9Phvr5ZRBye8HXgCsDESXV6/73SkijoqI8yPijojIerdXt7Jz6vbRXiePcZzdI+LiiFgWEUsjYiQith+j/KyI+FBEXB4R90TEbRFxdkTMG6POKhHxqYi4KiKWR8TNEXFqRDx7jDprR8SREbEkIu6tzxg7PiLWH62OJEmamF7P7HwWWAl4L3B8Zt43oHYcADwfWEaZoflZE6jzc+DbXdZ3HU8UEYcCC+r+jwMeD+wKnBkRe2fm0R3lAzgZ2Am4CjgaWBvYBTgvIt6cmWd01FkJ+D7wCsog7S8ATwd2Bl4fEdtk5kUdddYBFgObAIvqMZ8FvLPWeXnHPEaSJKkHvYadzYHTMvPYAbfjQ5QQci2wFW3P3xrDzzLzwInsvJ6JWUCZ/2fzzLy9rj8EuAw4NCK+UwdYt+xKCTqLgW0zc3mtcwxwAXBcRCzKzDvb6uxHCTqnAbu07lSLiFMowez4iNis4w62z1CCzuGZuaCtzftQwtK/UW73lyRJk9DrAOU/U2ZRHqjM/GFmXpOZOeh9V++ty0+3gk497hLgS5SzVe/sqPO+ujygFXRqnUuAUyiPxNiptb6eCWod5x/bA009A3Q+8BxKmGvVeQLlifF3AQd2HP9o4FfAayJiw4l/VEmS1K7XsLMYeOFUNGQSnhoR/xAR+9flWA8f3aYuv9dl23c7yhARKwPzgLspIWXcOsBGwDOAqzPzhgnWeRmwCnBhxxkialg6p77dusv+JEnSBPR6GWt/YHFEvC0zvz4VDerBq+rrQRExAuyemb9uW7ca8DRgWWbe1GU/19TlJm3rNqLcZXb9KOOSutXZtC6vHqW9g6ojSZJ60GvY2YEyiHZhRLybMt7lT13KZWb+S7+NG8XdwL9QxsC0Bu4+j3IZaGvgBxHxgsy8q25boy6XjrK/1vo129ZN5zoPiog9gT0BZs+ezcjIyCi7mZzZq8CCzQY1Bn3wBv15p8KyZctmRDunM/uwf/Zh/+zD/g2zD3sNOwe2/XmL+uomKYFk4DLzZuATHavPi4hXUwYOvxR4N2Vwb6PVgeLHAsydOzfnz58/0P0f9Y0zOOyKSU2y/ahYstv8YTdhXCMjIwz6e3mssQ/7Zx/2byb04ZyPnjXsJoxp4XZPGFof9vqcKtXrAAAgAElEQVSbbNqOHcnM+yLiq5SwsyUPhZ3W2ZE1ulZ8aH37GarpXEeSJPWg1weBnjtVDRmQW+pytdaKzLwrIn4LPC0i1usybmfjumwfN3MdcD+wYUSs2GXcTrc6V9XlaONrBlVHkiT1oGnPxnpZXXZOwreoLrvNV/PajjLUW80XA6vS/VLdI+pQAtKvgU0iYoMJ1vkxcA/lSfKrtxeOiBWAV9e3E5l3SJIkdTGpsBMRz4iIAyLi9Ij4QUR8s75/5qAb2OXYL6pBoHP9tpTJCQE6HzVxTF1+LCLWaqszB/gAcC9wQkedL9flwfVW9FadzSmzKN8CnN5aX+cIah3n8+1tjIgdKKHpSuDctjrLgK9TzkQd2HH8vYA5wDnOoCxJ0uRN5kGg7wG+SHncQrRteiNwQETsm5lf6XGfb6z1AZ5Sly+PiIX1z7dm5ofrnw8HNo6IxZRZl6HcjdWav+bjmbm4ff+ZuTgiDqfMcHx5RJxW278L5REQe3fMngzlsQ07UiYO/GlEnAmsU+vMAt6TmXd01Dkc2L7WuSgifkCZe2dnyl1ke3TMngzldv75wH4R8QLgYuDZlDvfbqaEMUmSNEk9hZ169uQY4E7gEMolmZuA9ShhYx/gSxFxbWb+oIddvwDYvWPdhvUFZSbhVtj5OvAmyqMrXgs8DvgDcCpwdGZ2mwSQzFwQEVdQwsOewAPAT4BDMvM7XcpnRLyVcjlrD2BvYDlwHnBwZ6Cqde6NiFcBHwXeSjnTdAflNvlPZuaVXer8sT5g9ZOUwLcF8EfKmaZPZOaNnXUkSdLE9Xpm5yOUoPPizLyubf1VwEhEnEiZe+cjwITDTn3G1YETLPs14GsT3XdH3YXAwh7K3wccUV8TrXM35db4ztvjx6pzG7BvfUmSpAHqdczOS4BTO4LOg+r6/6rlJEmShq7XsLMKcOs4ZW6p5SRJkoau17DzKx7+IMtutmYKnowuSZI0Gb2GnW8Bm0fEv0XEw57XFBFPjIgvUC5hfXNQDZQkSepHrwOUPwu8AXgvsFtE/JxyN9ZTgOcDTwR+WctJkiQNXU9nduq8MvOA4yhzzbySMofMFpTgdBzwii7zz0iSJA1Fz5MKZuZS4B8iYi9gU8rDKpcCV2XmXwbcPkmSpL70HHZaarD5xQDbIkmSNHBNexCoJEnSw4x5ZiciFgEJ7J6ZN9b3E5GZuW3frZMkSerTeJex5lPCzqpt7yciJ9keSZKkgRoz7GTmCmO9lyRJmu4ML5IkqdF6CjsRcXxEvGGcMttHxPH9NUuSJGkwej2z8w7gBeOUeT6w+6RaI0mSNGBTcRlrJeD+KdivJElSzyYTdka90yoiVgK2BH4/6RZJkiQN0LgzKEfE9R2rPhQR7+xSdBawLuXMzjEDaJskSVLfJvK4iBV46GxOAlFfnf4CXAH8ADh4IK2TJEnq07hhJzPntP4cEQ8AR2TmQVPZKEmSpEHp9UGgWwNLpqAdkiRJU6KnsJOZ53ZbHxGPA54L3J2ZVw2iYZIkSYPQ66SCb4mIUyNi7bZ1GwH/D7gUuDIivhkRvZ4xkiRJmhK93nq+B/CszLytbd1hwF8BPwQuB3YAut2tJUmS9KjrNew8B7ik9SYingi8Djg1M/8GeAnwSww7kiRpmug17KwL3NT2/uWUcT8nA2TmX4DvAxsNpHWSJEl96jXs3Ams0fZ+K8rcOxe0rVsOrN5nuyRJkgai14HE1wCvrY+FSOAtwOWZeWtbmWcCNw+ofZIkSX3p9czOscCGlNDzf8AGwAkdZV5MuTtLkiRp6HoKO5l5IvA5YFXK5ayjgaNa2yNiHg/dmSVJkjR0Pc+Hk5n7A/uPsvlSYC3grn4aJUmSNCgDnfwvM/8M/HmQ+5QkSerHpMJORDwP+Dvg2cBqdY4dImIOZa6d72fm7QNqoyRJ0qT1HHYi4iDKZazWeJ9s27wC8J/AB2kbyyNJkjQsvT4ba1fgAMrEgS8APtu+PTOvp4zbecOgGihJktSPXm893we4FtghMy+n+/ic/wM27rdhkiRJg9Br2NkMOKcORB7N74DZk2+SJEnS4PQadgJ4YJwysymPjJAkSRq6XsPONcC80TZGxArAK3EGZUmSNE30GnZOBV4UEQtG2b4/ZQbl/+irVZIkSQPS663nRwI7A5+PiLdQbzuPiEOBLYC5wI8pz9CSJEkaup7CTmbeExFbA18AdgNm1U37UcbynATslZn3DbSVkiRJkzSZZ2MtBd4REfsBmwPrAEuBizPzlgG3T5IkqS+TfjZWZt4GnDPAtkiSJA1crwOUJUmSZpSezuxExPETLJqZ+a5JtEeSJGmger2M9Y5xtidl4sEEDDuSJGnoeg07G4yyfk3KYOWPA4uBj/bTKEmSpEHp9dbzX42y6VfAzyPiHOBy4H+Br/XZNkmSpL4NdIByZv4GOBPYd5D7lSRJmqypuBvrD8DGU7BfSZKkng007ETELGAbyiSDkiRJQ9frredbjrGfpwPvBF4AfLXPdkmSJA1Er3djjVAf/jmKAM4DPjLZBkmSJA1Sr2HnILqHnQeA2ynPx7q471ZJkiQNSK+3nh84Re2QJEmaEj4bS5IkNVqvA5Svn+RxMjM3mmRdSZKkSet1zM4KwOOA9er7+4FbgScBs+q6m4A/d9SLyTZQkiSpH71exnoe8Fvgx8DWwMqZuR6wMmV+nYuAG4HnZeYG7a9BNlqSJGmieg07n6Y89HN+Zp6bmfcDZOb9mTlCCUBr13KSJElD12vYeRNwRmZ2XqYCIDOXA2cAO/bbMEmSpEHoNeysQxmzM5bH1XKSJElD12vYuQ7YKSLW6LYxItYCdgIme9eWJEnSQPUado4BngpcHBFvj4g5EbFKXe5OGaD8FOBLg26oJEnSZPQ6g/LREbExsDdwQpciARyVmf82iMZJkiT1q+cZlDNzX+AVwPHATymXrH4KfA14Zd0+YRGxU0QcFRHnR8QdEZERcdI4deZFxNkRcVtE3BMRl0fEByNi1hh1to+IkYhYGhHLIuKiejZqrOPsHhEX1/JLa/3txyg/KyI+VNtzT23f2RExb4w6q0TEpyLiqohYHhE3R8SpEfHssdomSZImptdJBQHIzB8BPxpQGw4Ang8so8zR86yxCkfEDsDpwHLgFOA24G+BIyghbOcudfYCjgL+CJxEmfRwJ2BhRGyWmR/uUudQYEFt03HA44FdgTMjYu/MPLqjfAAn1/1eBRxNuQ1/F+C8iHhzZp7RUWcl4Pu13ZcCXwCeXj/D6yNim8y8aKz+kCRJY5tU2BmwD1ECxbXAVsAPRysYEU+kBI/7KXP9XFrXfxxYRBk8vWtmntxWZw5wKCUUzc3MJXX9QcAlwIKIOL0GuFadeZSgcx2weWbeXtcfAlwGHBoR32ntq9qVEnQWA9vW2/CJiGOAC4DjImJRZt7ZVmc/StA5DdglMx+odU4Bvg0cX8PYAxPoR0mS1MXQHwSamT/MzGsyMydQfCdgXeDkVtCp+1hOOUME8L6OOnsAKwFHt4eTGmA+U9++t6NO6/2nW0Gn1llCGXy9EvDOjjqt4x7QCjq1ziWUM1Dr1vYDD54Jah3nH9sDTT0DdD7wHEoAlCRJkzT0sNOjberye122nQfcDcyrl4cmUue7HWUmVSciVgbm1eOfP8HjbAQ8A7g6M2/ooW2SJKkHMy3sbFqXV3duyMz7gBsol+Y2nGCdm4C7gPUjYlWAiFgNeBqwrG7vdE1dbtK2biPKg1Cvr+2YSJ1R2zVGHUmS1KPpMGanF63JDJeOsr21fs0e66xWy909hccYRJ2HiYg9gT0BZs+ezcjIyGhFJ2X2KrBgs27ZbXoY9OedCsuWLZsR7ZzO7MP+2Yf9mwl9OJ3/vYbh9uFMCztqk5nHAscCzJ07N+fPnz/Q/R/1jTM47Irp+yOyZLf5w27CuEZGRhj09/JYYx/2zz7s30zow3d89KxhN2FMC7dbbWh9ONMuY7XOdnR9XEXb+j9Nos7SjuVUHKPfOpIkqUczLexcVZePGMcSESsCGwD38fBnc41VZz3KJawbM/NugMy8C/gt8IS6vdPGddk+1uY6yu3wG9Z2TKTOqO0ao44kSerRTAs7i+pyuy7btgRWBRZn5r0TrPPajjKTqlNvNV9cj7/FBI9zHfBrYJOI2KCHtkmSpB7MtLBzGnArsGtEzG2trLd+H1zffrmjzgnAvcBedYLBVp21gP3r22M66rTef6yWa9WZA3yg7q/z2WCt4x5c29OqszllFuVbKDM/A1DnFWod5/MRsUJbnR0ooelK4FwkSdKkDX30aUS8EXhjffuUunx5RCysf7619TiHzLwjIt5DCT0jEXEyZWbkN1Bu5T6NMoHfgzLzhoj4CPBF4NI6O3HrcRHrA4e1z55c6yyOiMMpMxxfHhGnUR4XsQvlERB7d8yeDOVRETvW/f40Is4E1ql1ZgHvycw7OuocDmxf61wUET+gzL2zM+XOsD2cPVmSpP4MPewALwA6H8i5IQ/NlfMr4MFnV2XmtyNiK+BjwJuBlSmPmtgP+GK3mZgz86iIWFL383bKGa0rKbMdn9itUZm5ICKuoJzJ2RN4APgJcEhmfqdL+YyIt1IuZ+1BeTL8cspkhwdn5uIude6NiFcBHwXeSnl0xh2UR0V8MjOv7NY2SZI0cUMPO5l5IHBgj3UuBF7XY50zgTN7rLMQWNhD+fsoDyQ9ooc6dwOfqC9JkjRgM23MjiRJUk8MO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdEMO5IkqdFmbNiJiCURkaO8fj9KnXkRcXZE3BYR90TE5RHxwYiYNcZxto+IkYhYGhHLIuKiiNh9nLbtHhEX1/JLa/3txyg/KyI+VNtzT23f2RExb+I9IkmSullx2A3o01LgyC7rl3WuiIgdgNOB5cApwG3A3wJHAK8Adu5SZy/gKOCPwEnAn4GdgIURsVlmfrhLnUOBBcCNwHHA44FdgTMjYu/MPLqjfAAn1/1eBRwNrA3sApwXEW/OzDPG7QlJktTVTA87f8rMA8crFBFPpASP+4H5mXlpXf9xYBGwU0Tsmpknt9WZAxxKCUVzM3NJXX8QcAmwICJOz8wftdWZRwk61wGbZ+btdf0hwGXAoRHxnda+ql0pQWcxsG1mLq91jgEuAI6LiEWZeWdvXSNJkmAGX8bq0U7AusDJraADUIPFAfXt+zrq7AGsBBzdHk5qgPlMffvejjqt959uBZ1aZwnwpbq/d3bUaR33gFbQqXUuoZyBWre2X5IkTcJMDzsrRcTfR8T+EbFvRGw9yvibberye122nQfcDcyLiJUmWOe7HWUmVSciVgbm1eOf38NxJEnSBM30y1hPAb7ese6GiHhnZp7btm7Tury6cweZeV9E3AD8NbAh8H8TqHNTRNwFrB8Rq2bm3RGxGvA0YFlm3tSlrdfU5SZt6zYCZgHXZ+Z9E6wjSZJ6MJPP7JwAbEsJPKsBmwFfAeYA342I57eVXaMul46yr9b6NSdRZ42O5VQcY81RtkuSpHHM2DM7mfmpjlW/AN4bEcsog4QPBN70aLfr0RQRewJ7AsyePZuRkZGB7n/2KrBgs24nnKaHQX/eqbBs2bIZ0c7pzD7sn33Yv5nQh9P532sYbh/O2LAzhmMoYWfLtnWdZ2E6tdb/qaPOk+q2P45RZ2nHstdj9FrnQZl5LHAswNy5c3P+/Pmj7GZyjvrGGRx2xfT9EVmy2/xhN2FcIyMjDPp7eayxD/tnH/ZvJvThOz561rCbMKaF2602tD6cyZexRnNLXa7Wtu6qunzE2JeIWBHYALgPuH6Cddar+78xM+8GyMy7gN8CT6jbO21cl+1jgK6j3A6/YW3HROpIkqQeNDHsvKwu24PLorrcrkv5LYFVgcWZee8E67y2o8yk6tRbzRfX42/Rw3EkSdIEzciwExHPrnc/da6fQ5mBGMqMxy2nAbcCu0bE3LbyKwMH17df7tjdCcC9wF51v606awH717fHdNRpvf9YLdferg/U/Z3QUad13INre1p1NqfMonwLZeZnSZI0CdN3QMbYdqHMYHwe8CvgTspt3K8HVgbOpsx+DEBm3hER76GEnpGIOJkyM/IbKLeYn0aZwI+2OjdExEeALwKXRsQpPPS4iPWBw9pnT651FkfE4cB+wOURcRrlcRG7UB4BsXfH7MlQHhWxY93vTyPiTGCdWmcW8J7MvGOyHSVJ0mPdTA07P6SElBdSnmu1GmUQ7wWUeXe+npnZXiEzvx0RWwEfA95MCUXXUoLJFzvL1zpHRcQS4MPA2ylnwq6kzHZ8YreGZeaCiLiCciZnT+AB4CfAIZn5nS7lMyLeSrmctQewN+X5XecBB2fm4h76RZIkdZiRYadOGHjuuAUfWe9C4HU91jkTOLPHOguBhT2Uv4/yQNIjejmOJEka34wcsyNJkjRRhh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohh1JktRohp1pIiLWj4jjI+J3EXFvRCyJiCMjYq1ht02SpJlsxWE3QBARGwGLgScDZwC/BF4C7AtsFxGvyMw/DrGJkiTNWJ7ZmR7+jRJ09snMN2bmRzNzG+AIYFPg00NtnSRJM5hhZ8jqWZ1XA0uAL3Vs/iRwF/C2iFjtUW6aJEmNYNgZvq3r8n8y84H2DZl5J3AhsCrwske7YZIkNYFhZ/g2rcurR9l+TV1u8ii0RZKkxnGA8vCtUZdLR9neWr9m54aI2BPYs75dFhFXDbhtTwJuHfA+Byb+ddgtmJBp3YczhH3YP/uwf/Zhn7b+1ynpw2dOpJBhZwbLzGOBY6dq/xFxaWbOnar9PxbYh/2zD/tnH/bPPuzfMPvQy1jD1zpzs8Yo21vr//QotEWSpMYx7Axf69LTaGNyNq7L0cb0SJKkMRh2hu+HdfnqiHjY9xERqwOvAO4GfvxoN4wpvET2GGIf9s8+7J992D/7sH9D68PIzGEdW1VEnEOZa2efzDyqbf3hwIeAr2Tme4fVPkmSZjLDzjTQ5XER/we8lDIHz9XAPB8XIUnS5Bh2pomIeDpwELAdsA5wE/At4FOZefsw2yZJ0kzmmJ1pIjN/k5nvzMz1MvPxmfnMzPzgoILOoJ6qHhFr13pL6n5+V/e7/iDaOZ3124cRsVpE7BYR/xERv4yIuyLizoi4NCIWRMTjp/ozDNugfg479rllRNwfERkRBw+yvdPVIPsxIl5UfyZvrPv6Q0ScGxFvn4q2TxcD/DfxlRFxRq2/PCJ+HRFnR8R2U9X26SAidoqIoyLi/Ii4o/79O2mS+xr4vwuPOIZndppvjKeqb025G2xCT1WPiHXqfjYBFgGXAM8CdgBuBl6emddPxWcYtkH0Yf3H77vAbZSB6dcCawFvAJ5S979tZi6foo8xVIP6OezY5+rA5ZQJ354AfDozDxhku6ebQfZjROwFfAG4HTgL+C2wNvBc4MbM3HXgH2AaGOC/ie+jPMj5LsqZ+BuB9YEdKY/5OSAzG/kg54j4GfB8YBnlcz8L+EZm/n2P+xn4vwtdZaavhr+Ac4AE9u5Yf3hdf8wE9/OVWv6wjvX71PXfG/Znnc59CLwA2A14fMf61YHL6n4WDPuzTuc+7LLP4ynhcf+6j4OH/TlnSj9Sbop4oO5v9S7bHzfszzqd+xB4HGX+s3uATTu2PRtYTrmTdqVhf94p6sOtKVOjBDC/9ttJw/guJnScYXeYr6l9ARvVH5gbgBU6tq1OSeV3AauNs58n1L+4yzr/YaRcDl1Sj7PhsD/zdO3DcY7xd/UYZw77886UPqScUUzg74F3PBbCziD7Efh5LbvOsD/XTOxDYHbdz89H2X553d74/p1s2Hk0/m1tvRyz03yDeqr6y4BVgAtrvfb9tP532H68Jnk0nkz/l7q8r499TGcD7cOIeDJwHPDtzJzUOIEZaiD9GBHPBZ4H/A9wW0RsHREfrmPHtu2c86thBvWzeDNwC7BJRGzcviEiNqGc9fhZeiftWB6Nf1sBByg/FgzqqeqP5aezPxqffY+6/F4f+5jOBt2Hx1H+/XqszT81qH7cvC5vBkYoY/AOAQ4F/hf4WUT81eSbOa0NpA+znH74AOXn8LKIODEiPhsR/065LP3/gJ0H0N4me9R+r/gg0Oab9FPVp2g/M9GUfvY6SHQ74GeUMShNNLA+jIg9KIO6d8nMPwygbTPJoPrxyXX5Lsqg5NcDF1AuzXyCcmnwrIjYLDP/PPnmTksD+1nMzP+KiN8B/wm03732B+AEoJE3bAzQo/Z7xTM70hBFxI7AkcDvgTdn5l/GqfKYFhFzKP31X5l56nBbM6O1/u2fBeyamWdn5h2ZeQ3ll/allP9Nv3lYDZwJIuLvKWfCzqcMSl61Ln8AHA2cPLzWqZ1hp/kG9VT1x/LT2afks0fEGyn/GN4MzM+G3rZfDaoPj6fc/fL+QTRqBhpUP7a2/z4zf9S+oV6eOaO+fUnPLZz+BtKHdVzO8ZTLVW/LzF9m5j2Z+UvgbZRLWTtHxPz+m9xYj9rvFcNO8w3qqeqP5aezD/yzR8TOwH9RTndvlZlXjVNlphtUH76IcgnmljqJWUZEUi4ZAHysrvt2f82dtgb993m0XyKtyUxXmWC7ZpJB9eGrKbefn9tlcO0DwHn17Ysn08jHiEft94pjdprvYU9Vb/9LGb09Vf3HlP9RvyIiVm+/I6veufHqjuM1yaD6sFVnN+BEyliJrRt+RqdlUH3475RLBZ02BrakjHu6DPhp3y2engb59/n/t3f/sVbXdRzHny9LJCwlrKEFcYVqFFrkwByKoGasLYV+Zz/GpVZuaSHMUlSQgmZpk0u2/vAPoz9yM5lUrJq6aZroZGa0YWkm3DthpCEgRpLkfffH53PidPieyznHe+/3cHg9trMv9/vj8+N7Lue87+fz+X4++4AuScdHxL6a46fl7dZBKHO7Gax7eFzevrXO8cr+ThvzNJgG9bN1QGU/n+/X0L9octIm0kyYkwvS8aSCr/0ezgdeJQ1cnFB2vY7Ee1gn7W6Ognl2BvM+kmZODmAVeTb9vP900h82B4BJZde3Xe8hqYsvSF/G76s5NjXfw35gStn1HYb7OZsB5tkhtYBNLvp9ava9aPXl5SKOAgXTcQ+4qnruFiAiVJNO7XIRG0mD8SrLRcyIiGeGuj5lGIx7KOk80mDGY0h9/c8WZLUnInqGqBqlGqzfwzppd5O6so7G5SJa/f98AvAA6Yv5UdKcJmNJSx28AbgiIlYPdX3KMIj38DZgAan1Zh3QB3QB84ARQE9ELBri6pQijzmcl388GZhD+iPu93nfzoi4Mp/bRWol7IuIrpp0mnovWlZ2ROjX8LyA8aQvgx2k/5h9pKda3lxwbpDHKRYcG0P6i7Avp7OD9MU9ruw6tvs95GDrw0Cv3rLr2c73cIB0K/e241t2BvM+ktcTI32p/Js0huce4MNl1/FIuIekpRK6SXMV7SZNCrqL9DTWZ8uu4xDfv+WNfpaRAsC6n2/NvBetvtyyY2ZmZh3NT2OZmZlZR3OwY2ZmZh3NwY6ZmZl1NAc7ZmZm1tEc7JiZmVlHc7BjZmZmHc3BjpmZmXU0BztmNmQkdeWFOdeUWIZeSb3tkrek7nxPussok9nRyMGOmVmHk7Q8B1izyy6LWRm86rmZDaXtpPXTXiy7IG1kHWkV5x1lF8TsaOFgx8yGTEQcAJ4suxztJCJexMGf2bByN5aZDZmiMTuSxkr6gaSnJO2TtCf/e42kiS3mI0mXS3pC0n5J2yX9SNKJdc6v261Tb5xRLl9ImihpsaQnc17bJK3Kq4g3Uta6Y3YkjZP0Q0lPS3pZ0i5JGyUtrTnvPEm3SvqzpL353M2Srpc0subcXuD6/OP9Oe+orORddd4oSUskbcrvyz8lPSLpkkbqZdbO3LJjZsNG0ihgAzAJuBdYT1o5egIwF1gLbGkh6R7gG6SuoVuBAzm9DwIjSCspD5ZVwLnAz4FfAnOAK4CZks6JiP2tJCppGnA3MPQrDecAAATbSURBVAZ4ELgLGAW8l7TC9Iqq068CJgMPA78GRgJn5/NmS/pQRLyaz+0B5gGzgJ8CvQV5jwbuAz4APA7cRvpjeA5wu6QpEXFdK/UyawcOdsxsOF1ACnR6ImJR9QFJI4Djmk1Q0gxSoPMMcGZE7Mr7rwXuB04B+l5juaudDUyNiL6czxLgTuDjwDf5/6CkIbnud5ICnc9HxO01x8fVXPI1YGtE1LbOrACuAz4J3AEQET05mJkFrImI3xUUoYcU6FwVETdWpTcS+AVwjaS1EbGp2bqZtQN3Y5lZGV6u3RERr0TESy2ktSBvv1sJdHJ6+4ElLZZvIKsrgU7Op58U5PQDX2oxzYuALuBXtYFOzmNbzc9bagOdbFXezmk0Y0knAV8AHqsOdHI++0mtSAI+12iaZu3GLTtmNpweID2hdbWkM4DfkLq1NlV1uzTrjKq0az0EtJpuPYfkExFbJD0LdEkaHRF7mkzzrLz9bSMnSzoeWAh8DHg38CZSQFLx9ibyng68DghJywuOH5u372kiTbO24mDHzIZNROyVdBbwbeBiDrZA7JT0Y2BlfoKrGZVByM8V5PcfSTtbLnCxQ/LJ/k4ae3Qi0GywMzpvtx/uREnHksbXnAlsJnVX/YM0TgnSYORmugNPytvp+VXPG5tI06ytONgxs2GVu2S+LEmkwbfnA5cBy0hd60sHuLxI5THusdQMbpb0euAtwLaaa/rztugzcHTBvmpjgacK9p9cU55mVIKjRlpk5pICnTURsaD6gKRTOPjkVaMq5V0VEYubvNbsiOAxO2ZWikieiIhbgAvz7nktJPV43s4qOHYOqYum1u68HV9wbNph8jskn/zI/Higt4UuLEiTDAJ8pIFz35m3dzVStqzSlVd0LzaSgr+ZDeRtdkRysGNmw0bSFEljCw5V9v2rhWTX5O21ksZU5TUSuKHONRvzdkFu/alcM57UwjSQhZImVF1zDHAT6fP0J80V/X/Wkx4Jv7hoXpuap7F683Z2zTkTge/XSf+FvH1H7YGIeB74GTBN0lJJhwREkiZJOnXgKpi1L3djmdlwuhC4SdIjwF+B54FxpK6ZflLQ0JSI2CDpFuDrwGZJazk4z85uCpZliIhHJT1Imi9no6T7SAHXRaS5bopafCo2AJsk3UHqApoDvB/4A3DjANcNVIdXJH0KuIc0r82lpNaekaSBwRdw8PN6PfA3YLGk04E/koKYj5Lm3DkkoCE9gt8P3CDpNHLLVkSszMcvB94FfAf4oqSHSGOT3pbznw5cAmxtpX5mZXOwY2bD6W7Sl/G5pGDkBFIwci9wc0Q83GK6C0nB02XApaSWjHXANcCf6lwzlxRczSUFSk8D3yIFHJ8eIK9FpKegvkJ6XPwFYDWwrNUJBQEi4jFJU4GrSd1ZM4CXSIHNsqrz9kk6H/geqXVnJmms0grgZuAzBWn/RdJ84ErSHD2VWZZX5uN7Jc0Cvkp6xPwT+ZznSPdlEek9MjsiqXiqBjMzq5aXj5gPnBoRveWWxsya4TE7ZmZm1tEc7JiZmVlH85gdM2srkrqA7gZP72nxUW8zO4p4zI6ZtRVJs0lPDzXC42fM7LAc7JiZmVlH85gdMzMz62gOdszMzKyjOdgxMzOzjuZgx8zMzDqagx0zMzPraA52zMzMrKP9FxkuuqbbgrbvAAAAAElFTkSuQmCC\n",
 449 |       "text/plain": [
 450 |        "<Figure size 576x576 with 1 Axes>"
 451 |       ]
 452 |      },
 453 |      "metadata": {
 454 |       "needs_background": "light"
 455 |      },
 456 |      "output_type": "display_data"
 457 |     }
 458 |    ],
 459 |    "source": [
 460 |     "fig = plt.figure(figsize = (8,8))\n",
 461 |     "ax = fig.gca()\n",
 462 |     "data['is_duplicate'].hist(ax=ax)\n",
 463 |     "plt.xlabel('is_duplicate',fontsize=20)\n",
 464 |     "plt.ylabel('questions',fontsize=20)\n",
 465 |     "plt.xticks(fontsize=20)\n",
 466 |     "plt.yticks(fontsize=20)\n",
 467 |     "plt.title('Labels vs. Questions Plot',fontsize=20)\n",
 468 |     "plt.show()"
 469 |    ]
 470 |   },
 471 |   {
 472 |    "cell_type": "markdown",
 473 |    "metadata": {},
 474 |    "source": [
 475 |     "Below are the stopwords that are there in the `nltk` library which will be removed from the dataset."
 476 |    ]
 477 |   },
 478 |   {
 479 |    "cell_type": "code",
 480 |    "execution_count": 170,
 481 |    "metadata": {
 482 |     "scrolled": true
 483 |    },
 484 |    "outputs": [
 485 |     {
 486 |      "name": "stdout",
 487 |      "output_type": "stream",
 488 |      "text": [
 489 |       "{'she', 'when', 'has', 'once', \"couldn't\", 'my', \"doesn't\", 'as', 'can', 'hers', 'own', 'that', 'shouldn', 'not', 'shan', \"you'll\", 'are', 'doing', 'ours', 'under', \"haven't\", 'hasn', \"hasn't\", 'who', 'they', 'weren', 'he', 'll', 'them', 'how', 'am', 'didn', 'or', \"mustn't\", 'him', 'itself', 'now', \"it's\", \"mightn't\", 'a', 'until', 'up', 'myself', 'aren', 'we', 'into', 'below', 'been', 'should', 'during', \"wouldn't\", \"weren't\", 'there', 'mustn', 'of', 'with', 'himself', 'off', 'any', 'against', 'further', 'herself', 'where', \"you're\", 'these', 'such', 'what', 'ma', 'the', \"you've\", 'than', 'to', 'again', \"aren't\", 'being', 'but', 've', 'only', 'which', 'while', \"won't\", 'ain', 'their', \"shouldn't\", 'those', 'you', 'won', 'theirs', \"shan't\", 'out', 're', 'through', 't', \"wasn't\", 'this', 'was', 'in', 'if', 'it', 'at', 'other', 'about', 'd', \"she's\", 'needn', 'why', 'over', 'more', 'don', 'm', 'o', 'mightn', 'so', 'down', 'haven', 'whom', 'too', 'then', 'i', 'just', 'ourselves', 'were', \"that'll\", 'very', 'will', 'y', 'our', 'her', 'here', 'having', 'and', 'yourselves', 'doesn', 'its', 'each', 'do', 's', 'had', 'few', 'wouldn', 'for', \"should've\", 'by', 'is', \"don't\", 'an', 'your', 'because', 'yourself', 'have', 'yours', 'before', 'from', 'same', 'nor', 'between', 'does', 'above', \"hadn't\", \"needn't\", 'all', 'no', \"didn't\", 'on', 'couldn', 'hadn', \"isn't\", 'after', 'both', 'did', 'most', 'me', 'wasn', 'his', 'be', 'isn', \"you'd\", 'some', 'themselves'}\n"
 490 |      ]
 491 |     }
 492 |    ],
 493 |    "source": [
 494 |     "stop = set(stopwords.words('english')) \n",
 495 |     "print(stop)"
 496 |    ]
 497 |   },
 498 |   {
 499 |    "cell_type": "markdown",
 500 |    "metadata": {},
 501 |    "source": [
 502 |     "Next, I defined `sentence_to_wordlist` which takes each sentence from one of the two columns `question1` and `question2`. On each sentence it applies beautifulsoup for removing html tags if any. Then on that string I applied `regular expression` for some text preprocessing like include upper and lower case alphabets, 0-9 numbers, correct short forms and so on.\n",
 503 |     "Then applied `.lower()` to bring everything to lowercase to maintain regularity in every word and `.split()` method to convert the sentence into list of words. Finally, on these list of words I iterated one by one to check if that word is not a `stopword` if not then it is returned. "
 504 |    ]
 505 |   },
 506 |   {
 507 |    "cell_type": "code",
 508 |    "execution_count": 9,
 509 |    "metadata": {},
 510 |    "outputs": [],
 511 |    "source": [
 512 |     "def sentence_to_wordlist(sentence):\n",
 513 |     "    \n",
 514 |     "    \n",
 515 |     "    sentence = BeautifulSoup(sentence)  \n",
 516 |     "    sentence = sentence.get_text()\n",
 517 |     "\n",
 518 |     "    sentence = re.sub(r\"[^A-Za-z%@#$&*]\", \" \", sentence)\n",
 519 |     "    sentence = re.sub(r\"what's\", \"what is \", sentence)\n",
 520 |     "    sentence = re.sub(r\"\\'s\", \" \", sentence)\n",
 521 |     "    sentence = re.sub(r\"\\'ve\", \" have \", sentence)\n",
 522 |     "    sentence = re.sub(r\"can't\", \"cannot \", sentence)\n",
 523 |     "    sentence = re.sub(r\"n't\", \" not \", sentence)\n",
 524 |     "    sentence = re.sub(r\"i'm\", \"i am \", sentence)\n",
 525 |     "    sentence = re.sub(r\"\\'re\", \" are \", sentence)\n",
 526 |     "    sentence = re.sub(r\"\\'d\", \" would \", sentence)\n",
 527 |     "    sentence = re.sub(r\"\\'ll\", \" will \", sentence)\n",
 528 |     "    sentence = re.sub(r\",\", \" \", sentence)\n",
 529 |     "    sentence = re.sub(r\"\\.\", \" \", sentence)\n",
 530 |     "    sentence = re.sub(r\"\\/\", \" \", sentence)\n",
 531 |     "    sentence = re.sub(r\"\\^\", \" ^ \", sentence)\n",
 532 |     "    sentence = re.sub(r\"\\+\", \" + \", sentence)\n",
 533 |     "    sentence = re.sub(r\"\\=\", \" = \", sentence)\n",
 534 |     "    sentence = re.sub(r\"'\", \" \", sentence)\n",
 535 |     "    sentence = re.sub(r\"(\\d+)(k)\", r\"\\g<1>000\", sentence)\n",
 536 |     "    sentence = re.sub(r\":\", \" : \", sentence)\n",
 537 |     "    sentence = re.sub(r\" e g \", \" eg \", sentence)\n",
 538 |     "    sentence = re.sub(r\" b g \", \" bg \", sentence)\n",
 539 |     "    sentence = re.sub(r\" u s \", \" american \", sentence)\n",
 540 |     "    sentence = re.sub(r\"\\0s\", \"0\", sentence)\n",
 541 |     "    sentence = re.sub(r\"e - mail\", \"email\", sentence)\n",
 542 |     "    sentence = re.sub(r\"j k\", \"jk\", sentence)\n",
 543 |     "    sentence = re.sub(r\"\\s{2,}\", \" \", sentence)\n",
 544 |     "    \n",
 545 |     "    sentence = sentence.lower().split()\n",
 546 |     "\n",
 547 |     "\n",
 548 |     "    stops = set(stopwords.words(\"english\"))\n",
 549 |     "    sentence = [w for w in sentence if not w in stops]\n",
 550 |     "\n",
 551 |     "    return(sentence)"
 552 |    ]
 553 |   },
 554 |   {
 555 |    "cell_type": "markdown",
 556 |    "metadata": {},
 557 |    "source": [
 558 |     "Here I defined the two columns on which I applied the preprocessing."
 559 |    ]
 560 |   },
 561 |   {
 562 |    "cell_type": "code",
 563 |    "execution_count": 18,
 564 |    "metadata": {},
 565 |    "outputs": [],
 566 |    "source": [
 567 |     "columns = ['question1', 'question2']"
 568 |    ]
 569 |   },
 570 |   {
 571 |    "cell_type": "markdown",
 572 |    "metadata": {},
 573 |    "source": [
 574 |     "Next, I iterated over each row by using dataframes function called `iterrows()` which iterates over each record or row of the dataframe. For each row I iterated over the two columns namely `question1`, `question2` and then for each record I called the `sentence_to_wordlist()` function passing in the row and one of the two columns. Using pandas `at()` function I updated for each row both columns `question1` and `question2`."
 575 |    ]
 576 |   },
 577 |   {
 578 |    "cell_type": "code",
 579 |    "execution_count": 12,
 580 |    "metadata": {
 581 |     "scrolled": true
 582 |    },
 583 |    "outputs": [
 584 |     {
 585 |      "name": "stderr",
 586 |      "output_type": "stream",
 587 |      "text": [
 588 |       "/users/aditya.sharma/super-resolution-master_new/SR_new/lib/python3.5/site-packages/bs4/__init__.py:273: UserWarning: \"b'.'\" looks like a filename, not markup. You should probably open this file and pass the filehandle into Beautiful Soup.\n",
 589 |       "  ' Beautiful Soup.' % markup)\n"
 590 |      ]
 591 |     }
 592 |    ],
 593 |    "source": [
 594 |     "for indices, record in data.iterrows():\n",
 595 |     "        # Iterate through the text of both questions of the row\n",
 596 |     "        for column in columns:\n",
 597 |     "            data.at[indices, column] =  sentence_to_wordlist(record[column])"
 598 |    ]
 599 |   },
 600 |   {
 601 |    "cell_type": "markdown",
 602 |    "metadata": {},
 603 |    "source": [
 604 |     "Next, I print the first few rows of the modified dataframe and we can see that both the `question1` and `question2` columns have been converted into list of words."
 605 |    ]
 606 |   },
 607 |   {
 608 |    "cell_type": "code",
 609 |    "execution_count": 55,
 610 |    "metadata": {},
 611 |    "outputs": [
 612 |     {
 613 |      "data": {
 614 |       "text/html": [
 615 |        "<div>\n",
 616 |        "<style scoped>\n",
 617 |        "    .dataframe tbody tr th:only-of-type {\n",
 618 |        "        vertical-align: middle;\n",
 619 |        "    }\n",
 620 |        "\n",
 621 |        "    .dataframe tbody tr th {\n",
 622 |        "        vertical-align: top;\n",
 623 |        "    }\n",
 624 |        "\n",
 625 |        "    .dataframe thead th {\n",
 626 |        "        text-align: right;\n",
 627 |        "    }\n",
 628 |        "</style>\n",
 629 |        "<table border=\"1\" class=\"dataframe\">\n",
 630 |        "  <thead>\n",
 631 |        "    <tr style=\"text-align: right;\">\n",
 632 |        "      <th></th>\n",
 633 |        "      <th>id</th>\n",
 634 |        "      <th>qid1</th>\n",
 635 |        "      <th>qid2</th>\n",
 636 |        "      <th>question1</th>\n",
 637 |        "      <th>question2</th>\n",
 638 |        "      <th>is_duplicate</th>\n",
 639 |        "    </tr>\n",
 640 |        "  </thead>\n",
 641 |        "  <tbody>\n",
 642 |        "    <tr>\n",
 643 |        "      <th>0</th>\n",
 644 |        "      <td>0</td>\n",
 645 |        "      <td>1</td>\n",
 646 |        "      <td>2</td>\n",
 647 |        "      <td>[step, step, guide, invest, share, market, india]</td>\n",
 648 |        "      <td>[step, step, guide, invest, share, market]</td>\n",
 649 |        "      <td>0</td>\n",
 650 |        "    </tr>\n",
 651 |        "    <tr>\n",
 652 |        "      <th>1</th>\n",
 653 |        "      <td>1</td>\n",
 654 |        "      <td>3</td>\n",
 655 |        "      <td>4</td>\n",
 656 |        "      <td>[story, kohinoor, koh, noor, diamond]</td>\n",
 657 |        "      <td>[would, happen, indian, government, stole, koh...</td>\n",
 658 |        "      <td>0</td>\n",
 659 |        "    </tr>\n",
 660 |        "    <tr>\n",
 661 |        "      <th>2</th>\n",
 662 |        "      <td>2</td>\n",
 663 |        "      <td>5</td>\n",
 664 |        "      <td>6</td>\n",
 665 |        "      <td>[increase, speed, internet, connection, using,...</td>\n",
 666 |        "      <td>[internet, speed, increased, hacking, dns]</td>\n",
 667 |        "      <td>0</td>\n",
 668 |        "    </tr>\n",
 669 |        "    <tr>\n",
 670 |        "      <th>3</th>\n",
 671 |        "      <td>3</td>\n",
 672 |        "      <td>7</td>\n",
 673 |        "      <td>8</td>\n",
 674 |        "      <td>[mentally, lonely, solve]</td>\n",
 675 |        "      <td>[find, remainder, math, 23, 24, math, divided,...</td>\n",
 676 |        "      <td>0</td>\n",
 677 |        "    </tr>\n",
 678 |        "    <tr>\n",
 679 |        "      <th>4</th>\n",
 680 |        "      <td>4</td>\n",
 681 |        "      <td>9</td>\n",
 682 |        "      <td>10</td>\n",
 683 |        "      <td>[one, dissolve, water, quikly, sugar, salt, me...</td>\n",
 684 |        "      <td>[fish, would, survive, salt, water]</td>\n",
 685 |        "      <td>0</td>\n",
 686 |        "    </tr>\n",
 687 |        "  </tbody>\n",
 688 |        "</table>\n",
 689 |        "</div>"
 690 |       ],
 691 |       "text/plain": [
 692 |        "   id  qid1  qid2                                          question1  \\\n",
 693 |        "0   0     1     2  [step, step, guide, invest, share, market, india]   \n",
 694 |        "1   1     3     4              [story, kohinoor, koh, noor, diamond]   \n",
 695 |        "2   2     5     6  [increase, speed, internet, connection, using,...   \n",
 696 |        "3   3     7     8                          [mentally, lonely, solve]   \n",
 697 |        "4   4     9    10  [one, dissolve, water, quikly, sugar, salt, me...   \n",
 698 |        "\n",
 699 |        "                                           question2  is_duplicate  \n",
 700 |        "0         [step, step, guide, invest, share, market]             0  \n",
 701 |        "1  [would, happen, indian, government, stole, koh...             0  \n",
 702 |        "2         [internet, speed, increased, hacking, dns]             0  \n",
 703 |        "3  [find, remainder, math, 23, 24, math, divided,...             0  \n",
 704 |        "4                [fish, would, survive, salt, water]             0  "
 705 |       ]
 706 |      },
 707 |      "execution_count": 55,
 708 |      "metadata": {},
 709 |      "output_type": "execute_result"
 710 |     }
 711 |    ],
 712 |    "source": [
 713 |     "data.head()"
 714 |    ]
 715 |   },
 716 |   {
 717 |    "cell_type": "markdown",
 718 |    "metadata": {},
 719 |    "source": [
 720 |     "I save the `preprocessed_data` as pickle file so that I can just load it and reuse it later."
 721 |    ]
 722 |   },
 723 |   {
 724 |    "cell_type": "markdown",
 725 |    "metadata": {},
 726 |    "source": [
 727 |     "**Note**: Saving the dataframe as a pickle file acquires less space on the disk as well as keeps the format intact when reloaded."
 728 |    ]
 729 |   },
 730 |   {
 731 |    "cell_type": "code",
 732 |    "execution_count": 14,
 733 |    "metadata": {},
 734 |    "outputs": [],
 735 |    "source": [
 736 |     "with open('data_preprocessed.pickle', 'wb') as sub_data:\n",
 737 |     "    pickle.dump(data, sub_data, protocol=pickle.HIGHEST_PROTOCOL)"
 738 |    ]
 739 |   },
 740 |   {
 741 |    "cell_type": "code",
 742 |    "execution_count": 54,
 743 |    "metadata": {},
 744 |    "outputs": [],
 745 |    "source": [
 746 |     "with open('data_preprocessed.pickle', 'rb') as handle:\n",
 747 |     "    data = pickle.load(handle)"
 748 |    ]
 749 |   },
 750 |   {
 751 |    "cell_type": "markdown",
 752 |    "metadata": {},
 753 |    "source": [
 754 |     "After I preprocessed the sentences into list of words, next I assigned each unique word in the whole corpuse a number, so that I could pass this as an input to the model and also create a word2vec representation of these vocabularies (numbers)."
 755 |    ]
 756 |   },
 757 |   {
 758 |    "cell_type": "markdown",
 759 |    "metadata": {},
 760 |    "source": [
 761 |     "For doing this, I initialised a dictionary of variable `vocabulary()` which stored each word as a key and a number as a value respectively. Another variable called `inverse_vocabulary()` which is a list that holds the value or number for each unique word. It was initialised which an `<unk>` token since we want to zero pad the words with a number zero I did not want to assign any word a number 0. Hence, initialised with a `<unk>` token which holds the value zero."
 762 |    ]
 763 |   },
 764 |   {
 765 |    "cell_type": "markdown",
 766 |    "metadata": {},
 767 |    "source": [
 768 |     "Similar to above, I again iterate over the dataframe using `iterrows()` function, for each question in a row I iterate over all the words one by one. First I check whether the word is already in the dictionary `vocabulary()` if the word is not there then a value based on the length of the `inverse_vocabulary` is assigned to that new word (key), the inverse_vocabulary is updated with the new value along with it."
 769 |    ]
 770 |   },
 771 |   {
 772 |    "cell_type": "markdown",
 773 |    "metadata": {},
 774 |    "source": [
 775 |     "To update the dataframe `data` with numbers, I have a list named `sentence_to_numbers` which will append a value (number) for each word. Then using `at()` function the dataframe for each question of the particular row was updated with the list of word indices."
 776 |    ]
 777 |   },
 778 |   {
 779 |    "cell_type": "code",
 780 |    "execution_count": 15,
 781 |    "metadata": {},
 782 |    "outputs": [],
 783 |    "source": [
 784 |     "vocabulary = dict()\n",
 785 |     "inverse_vocabulary = ['<unk>']  \n",
 786 |     "\n",
 787 |     "for indices, record in data.iterrows():\n",
 788 |     "         for column in columns:\n",
 789 |     "\n",
 790 |     "            sentence_to_numbers = []  \n",
 791 |     "            for word in record[column]:\n",
 792 |     "\n",
 793 |     "               \n",
 794 |     "                if word not in vocabulary:\n",
 795 |     "                    vocabulary[word] = len(inverse_vocabulary)\n",
 796 |     "                    sentence_to_numbers.append(len(inverse_vocabulary))\n",
 797 |     "                    inverse_vocabulary.append(word)\n",
 798 |     "                else:\n",
 799 |     "                    sentence_to_numbers.append(vocabulary[word])\n",
 800 |     "\n",
 801 |     "            data.at[indices, column] =  sentence_to_numbers"
 802 |    ]
 803 |   },
 804 |   {
 805 |    "cell_type": "markdown",
 806 |    "metadata": {},
 807 |    "source": [
 808 |     "Now, I will save the `data_to_number` representation in a pickle file, so that I can reuse it and save time for later. "
 809 |    ]
 810 |   },
 811 |   {
 812 |    "cell_type": "code",
 813 |    "execution_count": 267,
 814 |    "metadata": {},
 815 |    "outputs": [],
 816 |    "source": [
 817 |     "with open('data_number_representation.pickle', 'wb') as sub_data:\n",
 818 |     "    pickle.dump(data, sub_data, protocol=pickle.HIGHEST_PROTOCOL)"
 819 |    ]
 820 |   },
 821 |   {
 822 |    "cell_type": "markdown",
 823 |    "metadata": {},
 824 |    "source": [
 825 |     "I will load it as `modified_data`."
 826 |    ]
 827 |   },
 828 |   {
 829 |    "cell_type": "code",
 830 |    "execution_count": 10,
 831 |    "metadata": {},
 832 |    "outputs": [],
 833 |    "source": [
 834 |     "with open('data_number_representation.pickle', 'rb') as handle:\n",
 835 |     "    data = pickle.load(handle)"
 836 |    ]
 837 |   },
 838 |   {
 839 |    "cell_type": "code",
 840 |    "execution_count": 11,
 841 |    "metadata": {},
 842 |    "outputs": [
 843 |     {
 844 |      "data": {
 845 |       "text/html": [
 846 |        "<div>\n",
 847 |        "<style scoped>\n",
 848 |        "    .dataframe tbody tr th:only-of-type {\n",
 849 |        "        vertical-align: middle;\n",
 850 |        "    }\n",
 851 |        "\n",
 852 |        "    .dataframe tbody tr th {\n",
 853 |        "        vertical-align: top;\n",
 854 |        "    }\n",
 855 |        "\n",
 856 |        "    .dataframe thead th {\n",
 857 |        "        text-align: right;\n",
 858 |        "    }\n",
 859 |        "</style>\n",
 860 |        "<table border=\"1\" class=\"dataframe\">\n",
 861 |        "  <thead>\n",
 862 |        "    <tr style=\"text-align: right;\">\n",
 863 |        "      <th></th>\n",
 864 |        "      <th>id</th>\n",
 865 |        "      <th>qid1</th>\n",
 866 |        "      <th>qid2</th>\n",
 867 |        "      <th>question1</th>\n",
 868 |        "      <th>question2</th>\n",
 869 |        "      <th>is_duplicate</th>\n",
 870 |        "    </tr>\n",
 871 |        "  </thead>\n",
 872 |        "  <tbody>\n",
 873 |        "    <tr>\n",
 874 |        "      <th>0</th>\n",
 875 |        "      <td>0</td>\n",
 876 |        "      <td>1</td>\n",
 877 |        "      <td>2</td>\n",
 878 |        "      <td>[1, 1, 2, 3, 4, 5, 6]</td>\n",
 879 |        "      <td>[1, 1, 2, 3, 4, 5]</td>\n",
 880 |        "      <td>0</td>\n",
 881 |        "    </tr>\n",
 882 |        "    <tr>\n",
 883 |        "      <th>1</th>\n",
 884 |        "      <td>1</td>\n",
 885 |        "      <td>3</td>\n",
 886 |        "      <td>4</td>\n",
 887 |        "      <td>[7, 8, 9, 10, 11]</td>\n",
 888 |        "      <td>[12, 13, 14, 15, 16, 8, 9, 10, 11, 17]</td>\n",
 889 |        "      <td>0</td>\n",
 890 |        "    </tr>\n",
 891 |        "    <tr>\n",
 892 |        "      <th>2</th>\n",
 893 |        "      <td>2</td>\n",
 894 |        "      <td>5</td>\n",
 895 |        "      <td>6</td>\n",
 896 |        "      <td>[18, 19, 20, 21, 22, 23]</td>\n",
 897 |        "      <td>[20, 19, 24, 25, 26]</td>\n",
 898 |        "      <td>0</td>\n",
 899 |        "    </tr>\n",
 900 |        "    <tr>\n",
 901 |        "      <th>3</th>\n",
 902 |        "      <td>3</td>\n",
 903 |        "      <td>7</td>\n",
 904 |        "      <td>8</td>\n",
 905 |        "      <td>[27, 28, 29]</td>\n",
 906 |        "      <td>[30, 31, 32, 33, 34, 32, 35, 34, 33]</td>\n",
 907 |        "      <td>0</td>\n",
 908 |        "    </tr>\n",
 909 |        "    <tr>\n",
 910 |        "      <th>4</th>\n",
 911 |        "      <td>4</td>\n",
 912 |        "      <td>9</td>\n",
 913 |        "      <td>10</td>\n",
 914 |        "      <td>[36, 37, 38, 39, 40, 41, 42, 43, 44, 45]</td>\n",
 915 |        "      <td>[46, 12, 47, 41, 38]</td>\n",
 916 |        "      <td>0</td>\n",
 917 |        "    </tr>\n",
 918 |        "  </tbody>\n",
 919 |        "</table>\n",
 920 |        "</div>"
 921 |       ],
 922 |       "text/plain": [
 923 |        "   id  qid1  qid2                                 question1  \\\n",
 924 |        "0   0     1     2                     [1, 1, 2, 3, 4, 5, 6]   \n",
 925 |        "1   1     3     4                         [7, 8, 9, 10, 11]   \n",
 926 |        "2   2     5     6                  [18, 19, 20, 21, 22, 23]   \n",
 927 |        "3   3     7     8                              [27, 28, 29]   \n",
 928 |        "4   4     9    10  [36, 37, 38, 39, 40, 41, 42, 43, 44, 45]   \n",
 929 |        "\n",
 930 |        "                                question2  is_duplicate  \n",
 931 |        "0                      [1, 1, 2, 3, 4, 5]             0  \n",
 932 |        "1  [12, 13, 14, 15, 16, 8, 9, 10, 11, 17]             0  \n",
 933 |        "2                    [20, 19, 24, 25, 26]             0  \n",
 934 |        "3    [30, 31, 32, 33, 34, 32, 35, 34, 33]             0  \n",
 935 |        "4                    [46, 12, 47, 41, 38]             0  "
 936 |       ]
 937 |      },
 938 |      "execution_count": 11,
 939 |      "metadata": {},
 940 |      "output_type": "execute_result"
 941 |     }
 942 |    ],
 943 |    "source": [
 944 |     "data.head()"
 945 |    ]
 946 |   },
 947 |   {
 948 |    "cell_type": "markdown",
 949 |    "metadata": {},
 950 |    "source": [
 951 |     "I also saved the `vocabulary` and `inverse_vocabulary` variable in a pickle file, since we would need them while creating the embedding matrix. "
 952 |    ]
 953 |   },
 954 |   {
 955 |    "cell_type": "code",
 956 |    "execution_count": 270,
 957 |    "metadata": {},
 958 |    "outputs": [],
 959 |    "source": [
 960 |     "with open('vocabulary.pickle', 'wb') as vocab:\n",
 961 |     "    pickle.dump(vocabulary, vocab, protocol=pickle.HIGHEST_PROTOCOL)"
 962 |    ]
 963 |   },
 964 |   {
 965 |    "cell_type": "code",
 966 |    "execution_count": 271,
 967 |    "metadata": {},
 968 |    "outputs": [],
 969 |    "source": [
 970 |     "with open('inverse_vocab.pickle', 'wb') as inverse_vocab:\n",
 971 |     "    pickle.dump(inverse_vocabulary, inverse_vocab, protocol=pickle.HIGHEST_PROTOCOL)"
 972 |    ]
 973 |   },
 974 |   {
 975 |    "cell_type": "code",
 976 |    "execution_count": 12,
 977 |    "metadata": {},
 978 |    "outputs": [],
 979 |    "source": [
 980 |     "with open('vocabulary.pickle', 'rb') as handle:\n",
 981 |     "    vocabulary = pickle.load(handle)"
 982 |    ]
 983 |   },
 984 |   {
 985 |    "cell_type": "markdown",
 986 |    "metadata": {},
 987 |    "source": [
 988 |     "Next, I created the embedding matrix for my vocabulary. For which I used `gensim` library and google's pre-trained word2vec model. Google's pre-trained word2vec model gives a 300 dimensional vector for each word which will be fed to the `Embedding layer` of my model. Since, I will pad my sentences with a zero, I initialise the embedding matrix's zeroth element as zero. The size of the embedding matrix will be `(Size of Vocabulary + 1 (for zero) X 300 (embedding dim))`."
 989 |    ]
 990 |   },
 991 |   {
 992 |    "cell_type": "markdown",
 993 |    "metadata": {},
 994 |    "source": [
 995 |     "I iterated over vocabulary and for each word corresponding to its index I store the 300 dimensional vector in the `embeddings` numpy array."
 996 |    ]
 997 |   },
 998 |   {
 999 |    "cell_type": "markdown",
1000 |    "metadata": {},
1001 |    "source": [
1002 |     "If the word is not there in the Google's pretrained model then that word will be randomly initialised, since the `embeddings` array is initialised randomly beforehand."
1003 |    ]
1004 |   },
1005 |   {
1006 |    "cell_type": "code",
1007 |    "execution_count": 13,
1008 |    "metadata": {},
1009 |    "outputs": [],
1010 |    "source": [
1011 |     "word2vec = KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin',binary=True)"
1012 |    ]
1013 |   },
1014 |   {
1015 |    "cell_type": "code",
1016 |    "execution_count": 14,
1017 |    "metadata": {},
1018 |    "outputs": [],
1019 |    "source": [
1020 |     "embedding_dim = 300\n",
1021 |     "embeddings = 1 * np.random.randn(len(vocabulary) + 1, embedding_dim)  # Initialising the embedding matrix randomly\n",
1022 |     "embeddings[0] = 0  \n",
1023 |     "# Build the embedding matrix\n",
1024 |     "for word, index in vocabulary.items():\n",
1025 |     "    if word in word2vec.vocab:\n",
1026 |     "        embeddings[index] = word2vec.word_vec(word)"
1027 |    ]
1028 |   },
1029 |   {
1030 |    "cell_type": "code",
1031 |    "execution_count": 15,
1032 |    "metadata": {},
1033 |    "outputs": [
1034 |     {
1035 |      "data": {
1036 |       "text/plain": [
1037 |        "(85159, 300)"
1038 |       ]
1039 |      },
1040 |      "execution_count": 15,
1041 |      "metadata": {},
1042 |      "output_type": "execute_result"
1043 |     }
1044 |    ],
1045 |    "source": [
1046 |     "embeddings.shape #since there are 85158 words in the dataset and 0 is <unk> token which will be a zero padding"
1047 |    ]
1048 |   },
1049 |   {
1050 |    "cell_type": "markdown",
1051 |    "metadata": {},
1052 |    "source": [
1053 |     "Next, I stored the embedding matrix as a pickle file too."
1054 |    ]
1055 |   },
1056 |   {
1057 |    "cell_type": "code",
1058 |    "execution_count": 281,
1059 |    "metadata": {},
1060 |    "outputs": [],
1061 |    "source": [
1062 |     "with open('embeddings.pickle', 'wb') as embed:\n",
1063 |     "    pickle.dump(embeddings, embed, protocol=pickle.HIGHEST_PROTOCOL)"
1064 |    ]
1065 |   },
1066 |   {
1067 |    "cell_type": "markdown",
1068 |    "metadata": {},
1069 |    "source": [
1070 |     "Then I imported all the Model related libraries that I used for splitting the data, padding the sentences to equal length, keras conv, merge, dropout, maxpooling etc. layers and the Model (Funtional API)."
1071 |    ]
1072 |   },
1073 |   {
1074 |    "cell_type": "code",
1075 |    "execution_count": 16,
1076 |    "metadata": {},
1077 |    "outputs": [
1078 |     {
1079 |      "name": "stderr",
1080 |      "output_type": "stream",
1081 |      "text": [
1082 |       "Using TensorFlow backend.\n"
1083 |      ]
1084 |     }
1085 |    ],
1086 |    "source": [
1087 |     "from keras.preprocessing.sequence import pad_sequences\n",
1088 |     "from keras.layers import Dense, Input, Flatten, Conv1D, MaxPooling1D, Embedding, Merge, Dropout, GlobalMaxPooling1D\n",
1089 |     "from keras.models import Model\n",
1090 |     "from sklearn.model_selection import train_test_split\n",
1091 |     "import itertools"
1092 |    ]
1093 |   },
1094 |   {
1095 |    "cell_type": "markdown",
1096 |    "metadata": {},
1097 |    "source": [
1098 |     "To prepare my data for feeding it into the model, I will first split the data into training and validation data. Validation data will help me to tune my hyperparameters, change the architecture, optimizer. It will also tell me whether my Model is `overfitting` on the training data."
1099 |    ]
1100 |   },
1101 |   {
1102 |    "cell_type": "markdown",
1103 |    "metadata": {},
1104 |    "source": [
1105 |     "For this I define a new dataframe `new_data` which has only two columns namely `question1` and `question2`."
1106 |    ]
1107 |   },
1108 |   {
1109 |    "cell_type": "code",
1110 |    "execution_count": 19,
1111 |    "metadata": {},
1112 |    "outputs": [],
1113 |    "source": [
1114 |     "new_data = data[columns]"
1115 |    ]
1116 |   },
1117 |   {
1118 |    "cell_type": "code",
1119 |    "execution_count": 20,
1120 |    "metadata": {},
1121 |    "outputs": [],
1122 |    "source": [
1123 |     "labels = data['is_duplicate']"
1124 |    ]
1125 |   },
1126 |   {
1127 |    "cell_type": "markdown",
1128 |    "metadata": {},
1129 |    "source": [
1130 |     "Since the input to my model will be a fixed size input it is important to keep all the sentences/sequences of same length. For that, I pad all the sentences with zeros based on the length of the sentence that has the maximum words."
1131 |    ]
1132 |   },
1133 |   {
1134 |    "cell_type": "code",
1135 |    "execution_count": 21,
1136 |    "metadata": {},
1137 |    "outputs": [],
1138 |    "source": [
1139 |     "max_seq_length = max(new_data.question1.map(lambda x: len(x)).max(),new_data.question2.map(lambda x: len(x)).max())"
1140 |    ]
1141 |   },
1142 |   {
1143 |    "cell_type": "code",
1144 |    "execution_count": 22,
1145 |    "metadata": {},
1146 |    "outputs": [
1147 |     {
1148 |      "data": {
1149 |       "text/plain": [
1150 |        "103"
1151 |       ]
1152 |      },
1153 |      "execution_count": 22,
1154 |      "metadata": {},
1155 |      "output_type": "execute_result"
1156 |     }
1157 |    ],
1158 |    "source": [
1159 |     "max_seq_length"
1160 |    ]
1161 |   },
1162 |   {
1163 |    "cell_type": "markdown",
1164 |    "metadata": {},
1165 |    "source": [
1166 |     "I take `random_state=13` which will divide the training and validation data in the same fashion no matter how many times I run it. If I change the `random_state` the manner in which data is divided will also change."
1167 |    ]
1168 |   },
1169 |   {
1170 |    "cell_type": "code",
1171 |    "execution_count": 23,
1172 |    "metadata": {},
1173 |    "outputs": [],
1174 |    "source": [
1175 |     "X_train, X_validation, Y_train, Y_validation = train_test_split(new_data,labels,random_state=13, test_size=0.2)"
1176 |    ]
1177 |   },
1178 |   {
1179 |    "cell_type": "markdown",
1180 |    "metadata": {},
1181 |    "source": [
1182 |     "Since, my network will have two inputs, the data was divided as `question1` and `question2` in a dictionary fashion."
1183 |    ]
1184 |   },
1185 |   {
1186 |    "cell_type": "code",
1187 |    "execution_count": 24,
1188 |    "metadata": {},
1189 |    "outputs": [],
1190 |    "source": [
1191 |     "train = {'question1': X_train.question1, 'question2': X_train.question2}\n",
1192 |     "val = {'question1': X_validation.question1, 'question2': X_validation.question2}"
1193 |    ]
1194 |   },
1195 |   {
1196 |    "cell_type": "markdown",
1197 |    "metadata": {},
1198 |    "source": [
1199 |     "Next, using `itertools()` function on both training and validation data, I padded each sentence with zeros to make each sequence of same size i.e. `103`. By default, Keras will pad zeros in a `pre-order` i.e. before the sequence."
1200 |    ]
1201 |   },
1202 |   {
1203 |    "cell_type": "code",
1204 |    "execution_count": 25,
1205 |    "metadata": {
1206 |     "scrolled": true
1207 |    },
1208 |    "outputs": [],
1209 |    "source": [
1210 |     "for dataset, side in itertools.product([train, val], ['question1', 'question2']):\n",
1211 |     "    dataset[side] = pad_sequences(dataset[side], maxlen=max_seq_length)"
1212 |    ]
1213 |   },
1214 |   {
1215 |    "cell_type": "markdown",
1216 |    "metadata": {},
1217 |    "source": [
1218 |     "Finally, I printed the shape of training and validation data for both `question1` and `question2` respectively."
1219 |    ]
1220 |   },
1221 |   {
1222 |    "cell_type": "code",
1223 |    "execution_count": 26,
1224 |    "metadata": {},
1225 |    "outputs": [
1226 |     {
1227 |      "data": {
1228 |       "text/plain": [
1229 |        "((323429, 103), (323429, 103))"
1230 |       ]
1231 |      },
1232 |      "execution_count": 26,
1233 |      "metadata": {},
1234 |      "output_type": "execute_result"
1235 |     }
1236 |    ],
1237 |    "source": [
1238 |     "train['question1'].shape,train['question2'].shape"
1239 |    ]
1240 |   },
1241 |   {
1242 |    "cell_type": "code",
1243 |    "execution_count": 27,
1244 |    "metadata": {},
1245 |    "outputs": [
1246 |     {
1247 |      "data": {
1248 |       "text/plain": [
1249 |        "((80858, 103), (80858, 103))"
1250 |       ]
1251 |      },
1252 |      "execution_count": 27,
1253 |      "metadata": {},
1254 |      "output_type": "execute_result"
1255 |     }
1256 |    ],
1257 |    "source": [
1258 |     "val['question1'].shape,val['question2'].shape"
1259 |    ]
1260 |   },
1261 |   {
1262 |    "cell_type": "markdown",
1263 |    "metadata": {},
1264 |    "source": [
1265 |     "# Siamese Network"
1266 |    ]
1267 |   },
1268 |   {
1269 |    "cell_type": "markdown",
1270 |    "metadata": {},
1271 |    "source": [
1272 |     "<img src=\"siamese.png\" alt=\"Italian Trulli\">"
1273 |    ]
1274 |   },
1275 |   {
1276 |    "cell_type": "markdown",
1277 |    "metadata": {},
1278 |    "source": [
1279 |     "I use a `Siamese` based architecture. Since the data is distributed in such a fashion wherein there are two questions and we have to find similarity between them, so using siamese is good way to go about this problem statement."
1280 |    ]
1281 |   },
1282 |   {
1283 |    "cell_type": "markdown",
1284 |    "metadata": {},
1285 |    "source": [
1286 |     "I import few more modules like `LSTM`, `BatchNormalization` etc. for various experiments that I did. I also imported `earlystopping`, `modelcheckpoint` and `reducelronplateau` which will stop the model if the `validation_loss` will stop decreasing after a certain point, saving best weights in the complete training again based on `validation_loss` and finally decay the `learning_rate`."
1287 |    ]
1288 |   },
1289 |   {
1290 |    "cell_type": "code",
1291 |    "execution_count": 28,
1292 |    "metadata": {},
1293 |    "outputs": [],
1294 |    "source": [
1295 |     "import keras\n",
1296 |     "from keras.models import Model\n",
1297 |     "from keras.layers import Input, Embedding, LSTM, Lambda,Dropout,merge,Lambda,Reshape\n",
1298 |     "from keras.layers import BatchNormalization, Bidirectional, GlobalMaxPool1D\n",
1299 |     "import keras.backend as K\n",
1300 |     "from keras.optimizers import Adam\n",
1301 |     "from keras.callbacks import ModelCheckpoint\n",
1302 |     "from keras import regularizers\n",
1303 |     "import numpy.random as rng\n",
1304 |     "from keras.callbacks import EarlyStopping,ModelCheckpoint,ReduceLROnPlateau\n",
1305 |     "from keras import models\n",
1306 |     "import tensorflow as tf"
1307 |    ]
1308 |   },
1309 |   {
1310 |    "cell_type": "markdown",
1311 |    "metadata": {},
1312 |    "source": [
1313 |     "Used batch_size of 64, number of training epochs were 25 and weights were initialised using Xavier uniform initialisation."
1314 |    ]
1315 |   },
1316 |   {
1317 |    "cell_type": "code",
1318 |    "execution_count": 31,
1319 |    "metadata": {},
1320 |    "outputs": [],
1321 |    "source": [
1322 |     "batch_size = 64 \n",
1323 |     "n_epoch = 15\n",
1324 |     "W_init = keras.initializers.glorot_uniform(seed=None)"
1325 |    ]
1326 |   },
1327 |   {
1328 |    "cell_type": "markdown",
1329 |    "metadata": {},
1330 |    "source": [
1331 |     "I tried many architectures out of which I finalised the below architecture. I achieved `~81%` accuracy on the validation data and on training data I achieved `~100%` accuracy. The model definitely overfits."
1332 |    ]
1333 |   },
1334 |   {
1335 |    "cell_type": "markdown",
1336 |    "metadata": {},
1337 |    "source": [
1338 |     "Since I use Keras Functional API, I defined three functions namely `embedding()`, `middle()` and `predict()`. The embedding function took embedding matrix as an input and was Trainable as True when the network was getting trained. The embedding output is feeded into the middle function module as input on which the maxpooling, lstm and dense layers are applied. The dense layer outputs a 128 feature maps which are then passed to the predict function which computes an L1 distance on these feature maps and then using a Dense layer with one neuron outputs a prediction of 0 or 1."
1339 |    ]
1340 |   },
1341 |   {
1342 |    "cell_type": "code",
1343 |    "execution_count": 33,
1344 |    "metadata": {
1345 |     "scrolled": true
1346 |    },
1347 |    "outputs": [
1348 |     {
1349 |      "name": "stderr",
1350 |      "output_type": "stream",
1351 |      "text": [
1352 |       "/users/aditya.sharma/super-resolution-master_new/SR_new/lib/python3.5/site-packages/ipykernel_launcher.py:61: UserWarning: The `merge` function is deprecated and will be removed after 08/2017. Use instead layers from `keras.layers.merge`, e.g. `add`, `concatenate`, etc.\n"
1353 |      ]
1354 |     },
1355 |     {
1356 |      "name": "stdout",
1357 |      "output_type": "stream",
1358 |      "text": [
1359 |       "____________________________________________________________________________________________________\n",
1360 |       "Layer (type)                     Output Shape          Param #     Connected to                     \n",
1361 |       "====================================================================================================\n",
1362 |       "input_5 (InputLayer)             (None, 103)           0                                            \n",
1363 |       "____________________________________________________________________________________________________\n",
1364 |       "input_6 (InputLayer)             (None, 103)           0                                            \n",
1365 |       "____________________________________________________________________________________________________\n",
1366 |       "embedding_3 (Embedding)          (None, 103, 300)      25547700    input_5[0][0]                    \n",
1367 |       "                                                                   input_6[0][0]                    \n",
1368 |       "____________________________________________________________________________________________________\n",
1369 |       "max_pooling1d_4 (MaxPooling1D)   (None, 11, 300)       0           embedding_3[0][0]                \n",
1370 |       "____________________________________________________________________________________________________\n",
1371 |       "max_pooling1d_5 (MaxPooling1D)   (None, 11, 300)       0           embedding_3[1][0]                \n",
1372 |       "____________________________________________________________________________________________________\n",
1373 |       "lstm_4 (LSTM)                    (None, 200)           400800      max_pooling1d_4[0][0]            \n",
1374 |       "____________________________________________________________________________________________________\n",
1375 |       "lstm_5 (LSTM)                    (None, 200)           400800      max_pooling1d_5[0][0]            \n",
1376 |       "____________________________________________________________________________________________________\n",
1377 |       "dense_4 (Dense)                  (None, 128)           25728       lstm_4[0][0]                     \n",
1378 |       "____________________________________________________________________________________________________\n",
1379 |       "dense_5 (Dense)                  (None, 128)           25728       lstm_5[0][0]                     \n",
1380 |       "____________________________________________________________________________________________________\n",
1381 |       "merge_2 (Merge)                  (None, 128)           0           dense_4[0][0]                    \n",
1382 |       "                                                                   dense_5[0][0]                    \n",
1383 |       "____________________________________________________________________________________________________\n",
1384 |       "dense_6 (Dense)                  (None, 1)             129         merge_2[0][0]                    \n",
1385 |       "====================================================================================================\n",
1386 |       "Total params: 26,400,885\n",
1387 |       "Trainable params: 26,400,885\n",
1388 |       "Non-trainable params: 0\n",
1389 |       "____________________________________________________________________________________________________\n",
1390 |       "Train on 323429 samples, validate on 80858 samples\n",
1391 |       "Epoch 1/15\n",
1392 |       "323429/323429 [==============================] - 199s - loss: 0.4820 - acc: 0.7644 - val_loss: 0.4428 - val_acc: 0.7887\n",
1393 |       "Epoch 2/15\n",
1394 |       "323429/323429 [==============================] - 199s - loss: 0.3664 - acc: 0.8333 - val_loss: 0.4352 - val_acc: 0.8020\n",
1395 |       "Epoch 3/15\n",
1396 |       "323429/323429 [==============================] - 199s - loss: 0.2748 - acc: 0.8805 - val_loss: 0.4456 - val_acc: 0.8083\n",
1397 |       "Epoch 4/15\n",
1398 |       "323429/323429 [==============================] - 198s - loss: 0.1987 - acc: 0.9172 - val_loss: 0.4871 - val_acc: 0.8081\n",
1399 |       "Epoch 5/15\n",
1400 |       "323429/323429 [==============================] - 198s - loss: 0.1433 - acc: 0.9421 - val_loss: 0.5768 - val_acc: 0.8004\n",
1401 |       "Epoch 6/15\n",
1402 |       "323429/323429 [==============================] - 198s - loss: 0.1062 - acc: 0.9579 - val_loss: 0.6742 - val_acc: 0.8023\n",
1403 |       "Epoch 7/15\n",
1404 |       "323429/323429 [==============================] - 198s - loss: 0.0837 - acc: 0.9672 - val_loss: 0.7558 - val_acc: 0.7988\n",
1405 |       "Epoch 8/15\n",
1406 |       "323429/323429 [==============================] - 198s - loss: 0.0682 - acc: 0.9737 - val_loss: 0.8367 - val_acc: 0.7968\n",
1407 |       "Epoch 9/15\n",
1408 |       "323429/323429 [==============================] - 198s - loss: 0.0570 - acc: 0.9785 - val_loss: 0.8474 - val_acc: 0.7906\n",
1409 |       "Epoch 10/15\n",
1410 |       "323392/323429 [============================>.] - ETA: 0s - loss: 0.0495 - acc: 0.9813\n",
1411 |       "Epoch 00009: reducing learning rate to 0.00010000000474974513.\n",
1412 |       "323429/323429 [==============================] - 198s - loss: 0.0495 - acc: 0.9813 - val_loss: 0.8731 - val_acc: 0.8026\n",
1413 |       "Epoch 11/15\n",
1414 |       "323429/323429 [==============================] - 198s - loss: 0.0255 - acc: 0.9907 - val_loss: 1.0015 - val_acc: 0.7984\n",
1415 |       "Epoch 12/15\n",
1416 |       "323429/323429 [==============================] - 198s - loss: 0.0178 - acc: 0.9934 - val_loss: 1.0751 - val_acc: 0.8001\n",
1417 |       "Epoch 13/15\n",
1418 |       "323429/323429 [==============================] - 198s - loss: 0.0147 - acc: 0.9943 - val_loss: 1.1448 - val_acc: 0.7970\n",
1419 |       "Epoch 14/15\n",
1420 |       "323429/323429 [==============================] - 198s - loss: 0.0130 - acc: 0.9946 - val_loss: 1.1901 - val_acc: 0.7981\n",
1421 |       "Epoch 15/15\n",
1422 |       "323429/323429 [==============================] - 198s - loss: 0.0119 - acc: 0.9950 - val_loss: 1.2397 - val_acc: 0.7955\n"
1423 |      ]
1424 |     }
1425 |    ],
1426 |    "source": [
1427 |     "question1_input = Input(shape=(max_seq_length,), dtype='int32')\n",
1428 |     "question2_input = Input(shape=(max_seq_length,), dtype='int32')\n",
1429 |     "\n",
1430 |     "\n",
1431 |     "def embedding():\n",
1432 |     "    \n",
1433 |     "    embedding_layer = Embedding(len(embeddings), 300, weights=[embeddings], input_length=max_seq_length, trainable=False)\n",
1434 |     "    encoded_question1 = embedding_layer(question1_input)\n",
1435 |     "    encoded_question2 = embedding_layer(question2_input)\n",
1436 |     "    return encoded_question1,encoded_question2\n",
1437 |     "\n",
1438 |     "def middle(q): \n",
1439 |     "    x = MaxPooling1D(10,padding='same')(q)\n",
1440 |     "    x = LSTM(200, return_sequences=False,kernel_initializer=W_init)(x)\n",
1441 |     "    x = Dense(128, activation=\"relu\",kernel_initializer=W_init)(x)\n",
1442 |     "    return x\n",
1443 |     "\n",
1444 |     "\n",
1445 |     "def predict(encoded_q1,encoded_q2):\n",
1446 |     "# Calculates the distance\n",
1447 |     "    \n",
1448 |     "    L1_distance = lambda x: K.abs(x[0]-x[1])\n",
1449 |     "    both = merge([encoded_q1,encoded_q2], mode = L1_distance, output_shape=lambda x: x[0])\n",
1450 |     "    prediction = Dense(1,activation='sigmoid',kernel_initializer=W_init)(both)\n",
1451 |     "    return prediction\n",
1452 |     "\n",
1453 |     "\n",
1454 |     "\n",
1455 |     "encoded_question1,encoded_question2 = embedding()\n",
1456 |     "encoded_q1 = middle(encoded_question1)\n",
1457 |     "encoded_q2 = middle(encoded_question2)\n",
1458 |     "prediction = predict(encoded_q1,encoded_q2)\n",
1459 |     "    \n",
1460 |     "quora = Model([question1_input, question2_input], [prediction])\n",
1461 |     "\n",
1462 |     "optimizer = Adam(lr=0.001,decay=0.0)\n",
1463 |     "\n",
1464 |     "quora.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
1465 |     "quora.summary()\n",
1466 |     "\n",
1467 |     "earlyStopping = EarlyStopping(monitor='val_loss', patience=20, verbose=0,mode='auto')\n",
1468 |     "ckpt = ModelCheckpoint(filepath='quora_lstm_max10.h5', save_best_only=True,monitor='val_acc', mode='auto')\n",
1469 |     "reduce_lr_loss = ReduceLROnPlateau(monitor='val_loss', factor=0.1,patience=7, verbose=1, epsilon=1e-4,mode='auto')\n",
1470 |     "\n",
1471 |     "quora_trained = quora.fit([train['question1'], train['question2']], Y_train, batch_size=batch_size, epochs=n_epoch,\n",
1472 |     "                            callbacks=[earlyStopping, ckpt, reduce_lr_loss],\n",
1473 |     "                            validation_data=([val['question1'], val['question2']], Y_validation))\n",
1474 |     "\n"
1475 |    ]
1476 |   },
1477 |   {
1478 |    "cell_type": "markdown",
1479 |    "metadata": {},
1480 |    "source": [
1481 |     "### Breaking the network to extract output of 128 feature maps from the middle( ) function by creating a new model"
1482 |    ]
1483 |   },
1484 |   {
1485 |    "cell_type": "markdown",
1486 |    "metadata": {},
1487 |    "source": [
1488 |     "First I load the model again which has both weights and model, then just save the weights."
1489 |    ]
1490 |   },
1491 |   {
1492 |    "cell_type": "code",
1493 |    "execution_count": 34,
1494 |    "metadata": {},
1495 |    "outputs": [
1496 |     {
1497 |      "name": "stderr",
1498 |      "output_type": "stream",
1499 |      "text": [
1500 |       "/users/aditya.sharma/super-resolution-master_new/SR_new/lib/python3.5/site-packages/keras/engine/topology.py:1231: UserWarning: The `Merge` layer is deprecated and will be removed after 08/2017. Use instead layers from `keras.layers.merge`, e.g. `add`, `concatenate`, etc.\n",
1501 |       "  return cls(**config)\n"
1502 |      ]
1503 |     }
1504 |    ],
1505 |    "source": [
1506 |     "quora = models.load_model('quora_lstm_max10.h5')"
1507 |    ]
1508 |   },
1509 |   {
1510 |    "cell_type": "code",
1511 |    "execution_count": 35,
1512 |    "metadata": {},
1513 |    "outputs": [],
1514 |    "source": [
1515 |     "quora.save_weights('quora_lstm_max10_weights.h5')"
1516 |    ]
1517 |   },
1518 |   {
1519 |    "cell_type": "markdown",
1520 |    "metadata": {},
1521 |    "source": [
1522 |     "Next, I created a new model which takes two inputs and outputs two outputs, for each of the two questions it will output a feature map."
1523 |    ]
1524 |   },
1525 |   {
1526 |    "cell_type": "code",
1527 |    "execution_count": 36,
1528 |    "metadata": {},
1529 |    "outputs": [],
1530 |    "source": [
1531 |     "quora1 = Model([question1_input, question1_input], [encoded_q1,encoded_q2])"
1532 |    ]
1533 |   },
1534 |   {
1535 |    "cell_type": "markdown",
1536 |    "metadata": {},
1537 |    "source": [
1538 |     "Then, I saved the combined model & weights and just weights file for this new model."
1539 |    ]
1540 |   },
1541 |   {
1542 |    "cell_type": "code",
1543 |    "execution_count": 37,
1544 |    "metadata": {},
1545 |    "outputs": [],
1546 |    "source": [
1547 |     "quora1.save('quora_lstm_max10_dense.h5')"
1548 |    ]
1549 |   },
1550 |   {
1551 |    "cell_type": "code",
1552 |    "execution_count": 38,
1553 |    "metadata": {},
1554 |    "outputs": [],
1555 |    "source": [
1556 |     "quora1.save_weights('quora_lstm_max10_dense_weights.h5')"
1557 |    ]
1558 |   },
1559 |   {
1560 |    "cell_type": "markdown",
1561 |    "metadata": {},
1562 |    "source": [
1563 |     "#### Training data predictions!"
1564 |    ]
1565 |   },
1566 |   {
1567 |    "cell_type": "code",
1568 |    "execution_count": 39,
1569 |    "metadata": {},
1570 |    "outputs": [],
1571 |    "source": [
1572 |     "train_prediction = quora1.predict([train['question1'],train['question2']]) #training predictions"
1573 |    ]
1574 |   },
1575 |   {
1576 |    "cell_type": "code",
1577 |    "execution_count": 40,
1578 |    "metadata": {},
1579 |    "outputs": [],
1580 |    "source": [
1581 |     "train_prediction = np.array(train_prediction) #converting the list of predictions into numpy array"
1582 |    ]
1583 |   },
1584 |   {
1585 |    "cell_type": "code",
1586 |    "execution_count": 41,
1587 |    "metadata": {},
1588 |    "outputs": [
1589 |     {
1590 |      "data": {
1591 |       "text/plain": [
1592 |        "(2, 323429, 128)"
1593 |       ]
1594 |      },
1595 |      "execution_count": 41,
1596 |      "metadata": {},
1597 |      "output_type": "execute_result"
1598 |     }
1599 |    ],
1600 |    "source": [
1601 |     "train_prediction.shape"
1602 |    ]
1603 |   },
1604 |   {
1605 |    "cell_type": "markdown",
1606 |    "metadata": {},
1607 |    "source": [
1608 |     "Saving the predictions as a numpy array."
1609 |    ]
1610 |   },
1611 |   {
1612 |    "cell_type": "code",
1613 |    "execution_count": 42,
1614 |    "metadata": {},
1615 |    "outputs": [],
1616 |    "source": [
1617 |     "np.save('train_predictions.npy',train_prediction)"
1618 |    ]
1619 |   },
1620 |   {
1621 |    "cell_type": "markdown",
1622 |    "metadata": {},
1623 |    "source": [
1624 |     "Since the output of `train_prediction` returns two predictions for `question1` and `question2`. I now reshape it to a one output which will have a shape `(2 X 323429,128)`"
1625 |    ]
1626 |   },
1627 |   {
1628 |    "cell_type": "code",
1629 |    "execution_count": 43,
1630 |    "metadata": {},
1631 |    "outputs": [],
1632 |    "source": [
1633 |     "train_pred = np.reshape(train_prediction,(-1,128)) "
1634 |    ]
1635 |   },
1636 |   {
1637 |    "cell_type": "code",
1638 |    "execution_count": 44,
1639 |    "metadata": {},
1640 |    "outputs": [
1641 |     {
1642 |      "data": {
1643 |       "text/plain": [
1644 |        "(646858, 128)"
1645 |       ]
1646 |      },
1647 |      "execution_count": 44,
1648 |      "metadata": {},
1649 |      "output_type": "execute_result"
1650 |     }
1651 |    ],
1652 |    "source": [
1653 |     "train_pred.shape"
1654 |    ]
1655 |   },
1656 |   {
1657 |    "cell_type": "markdown",
1658 |    "metadata": {},
1659 |    "source": [
1660 |     "#### Validation data predictions!"
1661 |    ]
1662 |   },
1663 |   {
1664 |    "cell_type": "code",
1665 |    "execution_count": 177,
1666 |    "metadata": {},
1667 |    "outputs": [],
1668 |    "source": [
1669 |     "val_prediction = quora1.predict([val['question1'],val['question2']]) #valid predictions"
1670 |    ]
1671 |   },
1672 |   {
1673 |    "cell_type": "code",
1674 |    "execution_count": 178,
1675 |    "metadata": {},
1676 |    "outputs": [],
1677 |    "source": [
1678 |     "val_prediction = np.array(val_prediction) #converting the list of predictions into numpy array"
1679 |    ]
1680 |   },
1681 |   {
1682 |    "cell_type": "code",
1683 |    "execution_count": 179,
1684 |    "metadata": {},
1685 |    "outputs": [
1686 |     {
1687 |      "data": {
1688 |       "text/plain": [
1689 |        "(2, 80858, 128)"
1690 |       ]
1691 |      },
1692 |      "execution_count": 179,
1693 |      "metadata": {},
1694 |      "output_type": "execute_result"
1695 |     }
1696 |    ],
1697 |    "source": [
1698 |     "val_prediction.shape"
1699 |    ]
1700 |   },
1701 |   {
1702 |    "cell_type": "code",
1703 |    "execution_count": 180,
1704 |    "metadata": {},
1705 |    "outputs": [],
1706 |    "source": [
1707 |     "np.save('val_predictions.npy',val_prediction)"
1708 |    ]
1709 |   },
1710 |   {
1711 |    "cell_type": "code",
1712 |    "execution_count": 181,
1713 |    "metadata": {},
1714 |    "outputs": [],
1715 |    "source": [
1716 |     "val_pred = np.reshape(val_prediction,(-1,128)) "
1717 |    ]
1718 |   },
1719 |   {
1720 |    "cell_type": "code",
1721 |    "execution_count": 182,
1722 |    "metadata": {},
1723 |    "outputs": [
1724 |     {
1725 |      "data": {
1726 |       "text/plain": [
1727 |        "(161716, 128)"
1728 |       ]
1729 |      },
1730 |      "execution_count": 182,
1731 |      "metadata": {},
1732 |      "output_type": "execute_result"
1733 |     }
1734 |    ],
1735 |    "source": [
1736 |     "val_pred.shape"
1737 |    ]
1738 |   },
1739 |   {
1740 |    "cell_type": "markdown",
1741 |    "metadata": {},
1742 |    "source": [
1743 |     "Now, I use the `original_data` that I had defined in the starting and will append the `question2` with `question1` to make it one single column of data."
1744 |    ]
1745 |   },
1746 |   {
1747 |    "cell_type": "code",
1748 |    "execution_count": 167,
1749 |    "metadata": {},
1750 |    "outputs": [],
1751 |    "source": [
1752 |     "X_train, X_validation, Y_train, Y_validation = train_test_split(original_data,original_data.is_duplicate, random_state=13,test_size=0.2)"
1753 |    ]
1754 |   },
1755 |   {
1756 |    "cell_type": "code",
1757 |    "execution_count": 168,
1758 |    "metadata": {},
1759 |    "outputs": [],
1760 |    "source": [
1761 |     "train_data = X_train.question1.append(X_train.question2)"
1762 |    ]
1763 |   },
1764 |   {
1765 |    "cell_type": "code",
1766 |    "execution_count": 169,
1767 |    "metadata": {},
1768 |    "outputs": [],
1769 |    "source": [
1770 |     "train_data = train_data.reset_index()"
1771 |    ]
1772 |   },
1773 |   {
1774 |    "cell_type": "code",
1775 |    "execution_count": 170,
1776 |    "metadata": {
1777 |     "scrolled": true
1778 |    },
1779 |    "outputs": [],
1780 |    "source": [
1781 |     "train_data = train_data.drop(['index'], axis=1)"
1782 |    ]
1783 |   },
1784 |   {
1785 |    "cell_type": "code",
1786 |    "execution_count": 171,
1787 |    "metadata": {},
1788 |    "outputs": [],
1789 |    "source": [
1790 |     "train_data.columns = ['questions']"
1791 |    ]
1792 |   },
1793 |   {
1794 |    "cell_type": "code",
1795 |    "execution_count": 172,
1796 |    "metadata": {},
1797 |    "outputs": [
1798 |     {
1799 |      "data": {
1800 |       "text/plain": [
1801 |        "(646858, 1)"
1802 |       ]
1803 |      },
1804 |      "execution_count": 172,
1805 |      "metadata": {},
1806 |      "output_type": "execute_result"
1807 |     }
1808 |    ],
1809 |    "source": [
1810 |     "train_data.shape"
1811 |    ]
1812 |   },
1813 |   {
1814 |    "cell_type": "markdown",
1815 |    "metadata": {},
1816 |    "source": [
1817 |     "#### Validation dataframe"
1818 |    ]
1819 |   },
1820 |   {
1821 |    "cell_type": "code",
1822 |    "execution_count": 183,
1823 |    "metadata": {},
1824 |    "outputs": [],
1825 |    "source": [
1826 |     "val_data = X_validation.question1.append(X_validation.question2)"
1827 |    ]
1828 |   },
1829 |   {
1830 |    "cell_type": "code",
1831 |    "execution_count": 184,
1832 |    "metadata": {},
1833 |    "outputs": [],
1834 |    "source": [
1835 |     "val_data = val_data.reset_index()"
1836 |    ]
1837 |   },
1838 |   {
1839 |    "cell_type": "code",
1840 |    "execution_count": 185,
1841 |    "metadata": {},
1842 |    "outputs": [],
1843 |    "source": [
1844 |     "val_data = val_data.drop(['index'], axis=1)"
1845 |    ]
1846 |   },
1847 |   {
1848 |    "cell_type": "code",
1849 |    "execution_count": 187,
1850 |    "metadata": {},
1851 |    "outputs": [],
1852 |    "source": [
1853 |     "val_data.columns = ['questions']"
1854 |    ]
1855 |   },
1856 |   {
1857 |    "cell_type": "code",
1858 |    "execution_count": 188,
1859 |    "metadata": {},
1860 |    "outputs": [
1861 |     {
1862 |      "data": {
1863 |       "text/plain": [
1864 |        "(161716, 1)"
1865 |       ]
1866 |      },
1867 |      "execution_count": 188,
1868 |      "metadata": {},
1869 |      "output_type": "execute_result"
1870 |     }
1871 |    ],
1872 |    "source": [
1873 |     "val_data.shape"
1874 |    ]
1875 |   },
1876 |   {
1877 |    "cell_type": "markdown",
1878 |    "metadata": {},
1879 |    "source": [
1880 |     "**Note:** I use 400K questions against which a query question will be compared."
1881 |    ]
1882 |   },
1883 |   {
1884 |    "cell_type": "markdown",
1885 |    "metadata": {},
1886 |    "source": [
1887 |     "Next, I divide the `train_pred` into two 400k and after 400K take 2K predictions as a query."
1888 |    ]
1889 |   },
1890 |   {
1891 |    "cell_type": "code",
1892 |    "execution_count": 173,
1893 |    "metadata": {},
1894 |    "outputs": [],
1895 |    "source": [
1896 |     "data_pred = train_pred"
1897 |    ]
1898 |   },
1899 |   {
1900 |    "cell_type": "code",
1901 |    "execution_count": 174,
1902 |    "metadata": {},
1903 |    "outputs": [
1904 |     {
1905 |      "data": {
1906 |       "text/plain": [
1907 |        "(646858, 128)"
1908 |       ]
1909 |      },
1910 |      "execution_count": 174,
1911 |      "metadata": {},
1912 |      "output_type": "execute_result"
1913 |     }
1914 |    ],
1915 |    "source": [
1916 |     "data_pred.shape"
1917 |    ]
1918 |   },
1919 |   {
1920 |    "cell_type": "code",
1921 |    "execution_count": 189,
1922 |    "metadata": {},
1923 |    "outputs": [],
1924 |    "source": [
1925 |     "query_pred = val_pred"
1926 |    ]
1927 |   },
1928 |   {
1929 |    "cell_type": "code",
1930 |    "execution_count": 190,
1931 |    "metadata": {},
1932 |    "outputs": [
1933 |     {
1934 |      "data": {
1935 |       "text/plain": [
1936 |        "(161716, 128)"
1937 |       ]
1938 |      },
1939 |      "execution_count": 190,
1940 |      "metadata": {},
1941 |      "output_type": "execute_result"
1942 |     }
1943 |    ],
1944 |    "source": [
1945 |     "query_pred.shape"
1946 |    ]
1947 |   },
1948 |   {
1949 |    "cell_type": "code",
1950 |    "execution_count": 191,
1951 |    "metadata": {},
1952 |    "outputs": [],
1953 |    "source": [
1954 |     "query_questions = val_data"
1955 |    ]
1956 |   },
1957 |   {
1958 |    "cell_type": "code",
1959 |    "execution_count": 192,
1960 |    "metadata": {},
1961 |    "outputs": [
1962 |     {
1963 |      "data": {
1964 |       "text/plain": [
1965 |        "(161716, 1)"
1966 |       ]
1967 |      },
1968 |      "execution_count": 192,
1969 |      "metadata": {},
1970 |      "output_type": "execute_result"
1971 |     }
1972 |    ],
1973 |    "source": [
1974 |     "query_questions.shape"
1975 |    ]
1976 |   },
1977 |   {
1978 |    "cell_type": "code",
1979 |    "execution_count": 175,
1980 |    "metadata": {},
1981 |    "outputs": [],
1982 |    "source": [
1983 |     "data_questions = train_data"
1984 |    ]
1985 |   },
1986 |   {
1987 |    "cell_type": "code",
1988 |    "execution_count": 176,
1989 |    "metadata": {},
1990 |    "outputs": [
1991 |     {
1992 |      "data": {
1993 |       "text/plain": [
1994 |        "(646858, 1)"
1995 |       ]
1996 |      },
1997 |      "execution_count": 176,
1998 |      "metadata": {},
1999 |      "output_type": "execute_result"
2000 |     }
2001 |    ],
2002 |    "source": [
2003 |     "data_questions.shape"
2004 |    ]
2005 |   },
2006 |   {
2007 |    "cell_type": "markdown",
2008 |    "metadata": {},
2009 |    "source": [
2010 |     "## Brute-Force Method for Finding the Top-3 Closest from the Training data for a given Input Query"
2011 |    ]
2012 |   },
2013 |   {
2014 |    "cell_type": "code",
2015 |    "execution_count": 212,
2016 |    "metadata": {},
2017 |    "outputs": [],
2018 |    "source": [
2019 |     "import heapq"
2020 |    ]
2021 |   },
2022 |   {
2023 |    "cell_type": "code",
2024 |    "execution_count": 223,
2025 |    "metadata": {},
2026 |    "outputs": [],
2027 |    "source": [
2028 |     "main_array = np.zeros((100,3))"
2029 |    ]
2030 |   },
2031 |   {
2032 |    "cell_type": "code",
2033 |    "execution_count": 224,
2034 |    "metadata": {
2035 |     "scrolled": true
2036 |    },
2037 |    "outputs": [],
2038 |    "source": [
2039 |     "def comparison(query):\n",
2040 |     "    arr = []\n",
2041 |     "    for i in range(data_pred.shape[0]):\n",
2042 |     "            predict = np.linalg.norm(query - data_pred[i])\n",
2043 |     "            arr.append(predict)\n",
2044 |     "    hp = np.array(heapq.nsmallest(3, range(len(arr)), arr.__getitem__))\n",
2045 |     "    return hp"
2046 |    ]
2047 |   },
2048 |   {
2049 |    "cell_type": "markdown",
2050 |    "metadata": {},
2051 |    "source": [
2052 |     "I took only 100 query questions since this method takes 313 seconds to output top-3 suggestions for each input query time!"
2053 |    ]
2054 |   },
2055 |   {
2056 |    "cell_type": "code",
2057 |    "execution_count": 225,
2058 |    "metadata": {},
2059 |    "outputs": [
2060 |     {
2061 |      "name": "stdout",
2062 |      "output_type": "stream",
2063 |      "text": [
2064 |       "313.1226750000005\n"
2065 |      ]
2066 |     }
2067 |    ],
2068 |    "source": [
2069 |     "import time\n",
2070 |     "start = time.clock()\n",
2071 |     "for i in range(100):\n",
2072 |     "    main_array[i,:] = comparison(query_pred[i])\n",
2073 |     "print (time.clock() - start)   "
2074 |    ]
2075 |   },
2076 |   {
2077 |    "cell_type": "code",
2078 |    "execution_count": 226,
2079 |    "metadata": {},
2080 |    "outputs": [
2081 |     {
2082 |      "data": {
2083 |       "text/plain": [
2084 |        "(100, 3)"
2085 |       ]
2086 |      },
2087 |      "execution_count": 226,
2088 |      "metadata": {},
2089 |      "output_type": "execute_result"
2090 |     }
2091 |    ],
2092 |    "source": [
2093 |     "main_array.shape"
2094 |    ]
2095 |   },
2096 |   {
2097 |    "cell_type": "code",
2098 |    "execution_count": 227,
2099 |    "metadata": {},
2100 |    "outputs": [],
2101 |    "source": [
2102 |     "np.set_printoptions(suppress=True)"
2103 |    ]
2104 |   },
2105 |   {
2106 |    "cell_type": "code",
2107 |    "execution_count": 228,
2108 |    "metadata": {},
2109 |    "outputs": [],
2110 |    "source": [
2111 |     "main_array = main_array.astype(np.int64)"
2112 |    ]
2113 |   },
2114 |   {
2115 |    "cell_type": "code",
2116 |    "execution_count": 229,
2117 |    "metadata": {},
2118 |    "outputs": [],
2119 |    "source": [
2120 |     "pd.set_option('display.max_colwidth', -1)"
2121 |    ]
2122 |   },
2123 |   {
2124 |    "cell_type": "code",
2125 |    "execution_count": 230,
2126 |    "metadata": {},
2127 |    "outputs": [],
2128 |    "source": [
2129 |     "import csv\n",
2130 |     "import os\n",
2131 |     "filename = 'output_brute_force.csv'\n",
2132 |     "a = open(filename, 'a')\n",
2133 |     "\n",
2134 |     "headers = ['Query', 'Closest-1','Closest-2','Closest-3']\n",
2135 |     "writer = csv.DictWriter(a, delimiter='\\t', lineterminator='\\n',fieldnames=headers)\n",
2136 |     "fileEmpty = os.stat(filename).st_size == 0\n",
2137 |     "writer.writeheader()\n",
2138 |     "for i in range(len(main_array)):\n",
2139 |     "    a.write((str(query_questions.iloc[i])).split('\\n')[0].split('    ')[1]+'\\t'+str(data_questions.iloc[main_array[i][0]]).split('\\n')[0].split('    ')[1] +'\\t'+ str(data_questions.iloc[main_array[i][1]]).split('\\n')[0].split('    ')[1] +'\\t'+ str(data_questions.iloc[main_array[i][2]]).split('\\n')[0].split('    ')[1] + \"\\n\")\n",
2140 |     "a.close()    "
2141 |    ]
2142 |   },
2143 |   {
2144 |    "cell_type": "code",
2145 |    "execution_count": 274,
2146 |    "metadata": {},
2147 |    "outputs": [],
2148 |    "source": [
2149 |     "data_brute = pd.read_csv('output_brute_force.csv',sep='\\t')"
2150 |    ]
2151 |   },
2152 |   {
2153 |    "cell_type": "code",
2154 |    "execution_count": 275,
2155 |    "metadata": {},
2156 |    "outputs": [
2157 |     {
2158 |      "data": {
2159 |       "text/html": [
2160 |        "<div>\n",
2161 |        "<style scoped>\n",
2162 |        "    .dataframe tbody tr th:only-of-type {\n",
2163 |        "        vertical-align: middle;\n",
2164 |        "    }\n",
2165 |        "\n",
2166 |        "    .dataframe tbody tr th {\n",
2167 |        "        vertical-align: top;\n",
2168 |        "    }\n",
2169 |        "\n",
2170 |        "    .dataframe thead th {\n",
2171 |        "        text-align: right;\n",
2172 |        "    }\n",
2173 |        "</style>\n",
2174 |        "<table border=\"1\" class=\"dataframe\">\n",
2175 |        "  <thead>\n",
2176 |        "    <tr style=\"text-align: right;\">\n",
2177 |        "      <th></th>\n",
2178 |        "      <th>Query</th>\n",
2179 |        "      <th>Closest-1</th>\n",
2180 |        "      <th>Closest-2</th>\n",
2181 |        "      <th>Closest-3</th>\n",
2182 |        "    </tr>\n",
2183 |        "  </thead>\n",
2184 |        "  <tbody>\n",
2185 |        "    <tr>\n",
2186 |        "      <th>0</th>\n",
2187 |        "      <td>What are the things that we can do to bring change in Indian education system?</td>\n",
2188 |        "      <td>What are the things that we can do to bring change in Indian education system?</td>\n",
2189 |        "      <td>What are the things that we can do to bring change in Indian education system?</td>\n",
2190 |        "      <td>What are the things that we can do to bring change in Indian education system?</td>\n",
2191 |        "    </tr>\n",
2192 |        "    <tr>\n",
2193 |        "      <th>1</th>\n",
2194 |        "      <td>Is Donald Trump in league with Putin?</td>\n",
2195 |        "      <td>Is Donald Trump in league with Putin?</td>\n",
2196 |        "      <td>Is Donald Trump in league with Putin?</td>\n",
2197 |        "      <td>What are some ways to contact Jesse Ventura?</td>\n",
2198 |        "    </tr>\n",
2199 |        "    <tr>\n",
2200 |        "      <th>2</th>\n",
2201 |        "      <td>What does Bootstrap do?</td>\n",
2202 |        "      <td>What is Kairos like?</td>\n",
2203 |        "      <td>What is frugal?</td>\n",
2204 |        "      <td>What's so special about Grana?</td>\n",
2205 |        "    </tr>\n",
2206 |        "    <tr>\n",
2207 |        "      <th>3</th>\n",
2208 |        "      <td>Why did Sanskrit fail to become a suitable language for computers?</td>\n",
2209 |        "      <td>Why did Sanskrit fail to become a suitable language for computers?</td>\n",
2210 |        "      <td>Why did Sanskrit fail to become a suitable language for computers?</td>\n",
2211 |        "      <td>Technology: Are we compromising on wisdom over information?</td>\n",
2212 |        "    </tr>\n",
2213 |        "    <tr>\n",
2214 |        "      <th>4</th>\n",
2215 |        "      <td>How can I increase my typing speed fast?</td>\n",
2216 |        "      <td>How can I increase my typing speed fast?</td>\n",
2217 |        "      <td>How can I increase my typing speed fast?</td>\n",
2218 |        "      <td>How can I increase my typing speed fast?</td>\n",
2219 |        "    </tr>\n",
2220 |        "  </tbody>\n",
2221 |        "</table>\n",
2222 |        "</div>"
2223 |       ],
2224 |       "text/plain": [
2225 |        "                                                                            Query  \\\n",
2226 |        "0  What are the things that we can do to bring change in Indian education system?   \n",
2227 |        "1  Is Donald Trump in league with Putin?                                            \n",
2228 |        "2  What does Bootstrap do?                                                          \n",
2229 |        "3  Why did Sanskrit fail to become a suitable language for computers?               \n",
2230 |        "4  How can I increase my typing speed fast?                                         \n",
2231 |        "\n",
2232 |        "                                                                        Closest-1  \\\n",
2233 |        "0  What are the things that we can do to bring change in Indian education system?   \n",
2234 |        "1  Is Donald Trump in league with Putin?                                            \n",
2235 |        "2  What is Kairos like?                                                             \n",
2236 |        "3  Why did Sanskrit fail to become a suitable language for computers?               \n",
2237 |        "4  How can I increase my typing speed fast?                                         \n",
2238 |        "\n",
2239 |        "                                                                        Closest-2  \\\n",
2240 |        "0  What are the things that we can do to bring change in Indian education system?   \n",
2241 |        "1  Is Donald Trump in league with Putin?                                            \n",
2242 |        "2  What is frugal?                                                                  \n",
2243 |        "3  Why did Sanskrit fail to become a suitable language for computers?               \n",
2244 |        "4  How can I increase my typing speed fast?                                         \n",
2245 |        "\n",
2246 |        "                                                                        Closest-3  \n",
2247 |        "0  What are the things that we can do to bring change in Indian education system?  \n",
2248 |        "1  What are some ways to contact Jesse Ventura?                                    \n",
2249 |        "2  What's so special about Grana?                                                  \n",
2250 |        "3  Technology: Are we compromising on wisdom over information?                     \n",
2251 |        "4  How can I increase my typing speed fast?                                        "
2252 |       ]
2253 |      },
2254 |      "execution_count": 275,
2255 |      "metadata": {},
2256 |      "output_type": "execute_result"
2257 |     }
2258 |    ],
2259 |    "source": [
2260 |     "data_brute.head()"
2261 |    ]
2262 |   },
2263 |   {
2264 |    "cell_type": "markdown",
2265 |    "metadata": {},
2266 |    "source": [
2267 |     "## Finding the Top-3 Closest from the Training data for a given Input Query using KMeans Clustering"
2268 |    ]
2269 |   },
2270 |   {
2271 |    "cell_type": "markdown",
2272 |    "metadata": {},
2273 |    "source": [
2274 |     "To make the search space faster, I used KMeans clustering. Firstly, I trained Kmeans on the training data to get the cluster centers then predicted a cluster center for every new query. This improved the performance massively. Comparing to the brute-force method, Kmeans gave me top-3 results within `200secs` that too when I had not 100 but 1000 query questions against 650K questions and without any performance loss."
2275 |    ]
2276 |   },
2277 |   {
2278 |    "cell_type": "code",
2279 |    "execution_count": 84,
2280 |    "metadata": {},
2281 |    "outputs": [],
2282 |    "source": [
2283 |     "from sklearn.cluster import KMeans\n",
2284 |     "import os"
2285 |    ]
2286 |   },
2287 |   {
2288 |    "cell_type": "code",
2289 |    "execution_count": 234,
2290 |    "metadata": {},
2291 |    "outputs": [],
2292 |    "source": [
2293 |     "kmeans=KMeans(n_clusters=20, n_jobs=-1)"
2294 |    ]
2295 |   },
2296 |   {
2297 |    "cell_type": "code",
2298 |    "execution_count": 235,
2299 |    "metadata": {},
2300 |    "outputs": [],
2301 |    "source": [
2302 |     "kmeans_fit = kmeans.fit(data_pred)"
2303 |    ]
2304 |   },
2305 |   {
2306 |    "cell_type": "code",
2307 |    "execution_count": 236,
2308 |    "metadata": {},
2309 |    "outputs": [
2310 |     {
2311 |      "data": {
2312 |       "text/plain": [
2313 |        "array([19,  7, 16, ..., 17, 15, 14], dtype=int32)"
2314 |       ]
2315 |      },
2316 |      "execution_count": 236,
2317 |      "metadata": {},
2318 |      "output_type": "execute_result"
2319 |     }
2320 |    ],
2321 |    "source": [
2322 |     "kmeans_fit.labels_"
2323 |    ]
2324 |   },
2325 |   {
2326 |    "cell_type": "code",
2327 |    "execution_count": 237,
2328 |    "metadata": {},
2329 |    "outputs": [],
2330 |    "source": [
2331 |     "cluster_centers = kmeans.cluster_centers_"
2332 |    ]
2333 |   },
2334 |   {
2335 |    "cell_type": "code",
2336 |    "execution_count": 238,
2337 |    "metadata": {},
2338 |    "outputs": [
2339 |     {
2340 |      "data": {
2341 |       "text/plain": [
2342 |        "(20, 128)"
2343 |       ]
2344 |      },
2345 |      "execution_count": 238,
2346 |      "metadata": {},
2347 |      "output_type": "execute_result"
2348 |     }
2349 |    ],
2350 |    "source": [
2351 |     "cluster_centers.shape"
2352 |    ]
2353 |   },
2354 |   {
2355 |    "cell_type": "code",
2356 |    "execution_count": 239,
2357 |    "metadata": {},
2358 |    "outputs": [],
2359 |    "source": [
2360 |     "kmeans_data_pred = kmeans.predict(data_pred)"
2361 |    ]
2362 |   },
2363 |   {
2364 |    "cell_type": "code",
2365 |    "execution_count": 240,
2366 |    "metadata": {},
2367 |    "outputs": [
2368 |     {
2369 |      "data": {
2370 |       "text/plain": [
2371 |        "(646858,)"
2372 |       ]
2373 |      },
2374 |      "execution_count": 240,
2375 |      "metadata": {},
2376 |      "output_type": "execute_result"
2377 |     }
2378 |    ],
2379 |    "source": [
2380 |     "kmeans_data_pred.shape"
2381 |    ]
2382 |   },
2383 |   {
2384 |    "cell_type": "code",
2385 |    "execution_count": 241,
2386 |    "metadata": {},
2387 |    "outputs": [],
2388 |    "source": [
2389 |     "with open('kmeans_model.pickle', 'wb') as f:\n",
2390 |     "     pickle.dump(kmeans_fit, f)"
2391 |    ]
2392 |   },
2393 |   {
2394 |    "cell_type": "code",
2395 |    "execution_count": 242,
2396 |    "metadata": {},
2397 |    "outputs": [],
2398 |    "source": [
2399 |     "for i in range(20):\n",
2400 |     "    index = np.where(kmeans_data_pred == i)\n",
2401 |     "    np.array(cluster_centers[i]).dump(open('clusters/cluster_center[%s].npy' % i, 'wb'))"
2402 |    ]
2403 |   },
2404 |   {
2405 |    "cell_type": "code",
2406 |    "execution_count": 261,
2407 |    "metadata": {},
2408 |    "outputs": [],
2409 |    "source": [
2410 |     "array = np.zeros((1000,3))"
2411 |    ]
2412 |   },
2413 |   {
2414 |    "cell_type": "code",
2415 |    "execution_count": 262,
2416 |    "metadata": {},
2417 |    "outputs": [],
2418 |    "source": [
2419 |     "def compare(cluster,j):\n",
2420 |     "    #print (cluster)\n",
2421 |     "    assign = dict()\n",
2422 |     "    a = np.where(kmeans_data_pred == cluster)[0]\n",
2423 |     "    for i in a:\n",
2424 |     "        dist = np.linalg.norm(query_pred[j] - data_pred[i])\n",
2425 |     "        assign[i] = dist\n",
2426 |     "    sorted_by_value = sorted(assign.items(), key=lambda kv: kv[1])\n",
2427 |     "    a = [i for i, v in (sorted_by_value)]\n",
2428 |     "    return (a[:3])"
2429 |    ]
2430 |   },
2431 |   {
2432 |    "cell_type": "code",
2433 |    "execution_count": 263,
2434 |    "metadata": {},
2435 |    "outputs": [
2436 |     {
2437 |      "name": "stdout",
2438 |      "output_type": "stream",
2439 |      "text": [
2440 |       "201.52732800000013\n"
2441 |      ]
2442 |     }
2443 |    ],
2444 |    "source": [
2445 |     "import time\n",
2446 |     "start = time.clock()\n",
2447 |     "for i in range(1000):\n",
2448 |     "    array[i,:] = compare(kmeans.predict(query_pred[i].reshape(-1,128)),i)\n",
2449 |     "print (time.clock() - start)    "
2450 |    ]
2451 |   },
2452 |   {
2453 |    "cell_type": "code",
2454 |    "execution_count": 265,
2455 |    "metadata": {},
2456 |    "outputs": [],
2457 |    "source": [
2458 |     "array = array.astype(np.int64)"
2459 |    ]
2460 |   },
2461 |   {
2462 |    "cell_type": "code",
2463 |    "execution_count": 269,
2464 |    "metadata": {
2465 |     "scrolled": true
2466 |    },
2467 |    "outputs": [],
2468 |    "source": [
2469 |     "import csv\n",
2470 |     "import os\n",
2471 |     "filename = 'output_kmeans.csv'\n",
2472 |     "a = open(filename, 'w')\n",
2473 |     "\n",
2474 |     "headers = ['Query', 'Closest-1','Closest-2','Closest-3']\n",
2475 |     "writer = csv.DictWriter(a, delimiter='\\t', lineterminator='\\n',fieldnames=headers)\n",
2476 |     "fileEmpty = os.stat(filename).st_size == 0\n",
2477 |     "writer.writeheader()\n",
2478 |     "\n",
2479 |     "for i in range(len(array)):\n",
2480 |     "    a.write((str(query_questions.iloc[i])).split('\\n')[0].split('    ')[1] + '\\t' + str(data_questions.iloc[array[i][0]]).split('\\n')[0].split('    ')[1] + '\\t' + str(data_questions.iloc[array[i][1]]).split('\\n')[0].split('    ')[1] + '\\t' + str(data_questions.iloc[array[i][2]]).split('\\n')[0].split('    ')[1] + \"\\n\")\n",
2481 |     "a.close()    "
2482 |    ]
2483 |   },
2484 |   {
2485 |    "cell_type": "code",
2486 |    "execution_count": 272,
2487 |    "metadata": {},
2488 |    "outputs": [],
2489 |    "source": [
2490 |     "data_kmeans = pd.read_csv('output_kmeans.csv',sep='\\t')"
2491 |    ]
2492 |   },
2493 |   {
2494 |    "cell_type": "code",
2495 |    "execution_count": 273,
2496 |    "metadata": {
2497 |     "scrolled": true
2498 |    },
2499 |    "outputs": [
2500 |     {
2501 |      "data": {
2502 |       "text/html": [
2503 |        "<div>\n",
2504 |        "<style scoped>\n",
2505 |        "    .dataframe tbody tr th:only-of-type {\n",
2506 |        "        vertical-align: middle;\n",
2507 |        "    }\n",
2508 |        "\n",
2509 |        "    .dataframe tbody tr th {\n",
2510 |        "        vertical-align: top;\n",
2511 |        "    }\n",
2512 |        "\n",
2513 |        "    .dataframe thead th {\n",
2514 |        "        text-align: right;\n",
2515 |        "    }\n",
2516 |        "</style>\n",
2517 |        "<table border=\"1\" class=\"dataframe\">\n",
2518 |        "  <thead>\n",
2519 |        "    <tr style=\"text-align: right;\">\n",
2520 |        "      <th></th>\n",
2521 |        "      <th>Query</th>\n",
2522 |        "      <th>Closest-1</th>\n",
2523 |        "      <th>Closest-2</th>\n",
2524 |        "      <th>Closest-3</th>\n",
2525 |        "    </tr>\n",
2526 |        "  </thead>\n",
2527 |        "  <tbody>\n",
2528 |        "    <tr>\n",
2529 |        "      <th>0</th>\n",
2530 |        "      <td>What are the things that we can do to bring change in Indian education system?</td>\n",
2531 |        "      <td>What are the things that we can do to bring change in Indian education system?</td>\n",
2532 |        "      <td>What are the things that we can do to bring change in Indian education system?</td>\n",
2533 |        "      <td>What are the things that we can do to bring change in Indian education system?</td>\n",
2534 |        "    </tr>\n",
2535 |        "    <tr>\n",
2536 |        "      <th>1</th>\n",
2537 |        "      <td>Is Donald Trump in league with Putin?</td>\n",
2538 |        "      <td>Is Donald Trump in league with Putin?</td>\n",
2539 |        "      <td>Is Donald Trump in league with Putin?</td>\n",
2540 |        "      <td>What are some ways to contact Jesse Ventura?</td>\n",
2541 |        "    </tr>\n",
2542 |        "    <tr>\n",
2543 |        "      <th>2</th>\n",
2544 |        "      <td>What does Bootstrap do?</td>\n",
2545 |        "      <td>What is Kairos like?</td>\n",
2546 |        "      <td>What is frugal?</td>\n",
2547 |        "      <td>What is a Discord chat?</td>\n",
2548 |        "    </tr>\n",
2549 |        "    <tr>\n",
2550 |        "      <th>3</th>\n",
2551 |        "      <td>Why did Sanskrit fail to become a suitable language for computers?</td>\n",
2552 |        "      <td>Why did Sanskrit fail to become a suitable language for computers?</td>\n",
2553 |        "      <td>Why did Sanskrit fail to become a suitable language for computers?</td>\n",
2554 |        "      <td>Technology: Are we compromising on wisdom over information?</td>\n",
2555 |        "    </tr>\n",
2556 |        "    <tr>\n",
2557 |        "      <th>4</th>\n",
2558 |        "      <td>How can I increase my typing speed fast?</td>\n",
2559 |        "      <td>How can I increase my typing speed fast?</td>\n",
2560 |        "      <td>How can I increase my typing speed fast?</td>\n",
2561 |        "      <td>How can I increase my typing speed fast?</td>\n",
2562 |        "    </tr>\n",
2563 |        "  </tbody>\n",
2564 |        "</table>\n",
2565 |        "</div>"
2566 |       ],
2567 |       "text/plain": [
2568 |        "                                                                            Query  \\\n",
2569 |        "0  What are the things that we can do to bring change in Indian education system?   \n",
2570 |        "1  Is Donald Trump in league with Putin?                                            \n",
2571 |        "2  What does Bootstrap do?                                                          \n",
2572 |        "3  Why did Sanskrit fail to become a suitable language for computers?               \n",
2573 |        "4  How can I increase my typing speed fast?                                         \n",
2574 |        "\n",
2575 |        "                                                                        Closest-1  \\\n",
2576 |        "0  What are the things that we can do to bring change in Indian education system?   \n",
2577 |        "1  Is Donald Trump in league with Putin?                                            \n",
2578 |        "2  What is Kairos like?                                                             \n",
2579 |        "3  Why did Sanskrit fail to become a suitable language for computers?               \n",
2580 |        "4  How can I increase my typing speed fast?                                         \n",
2581 |        "\n",
2582 |        "                                                                        Closest-2  \\\n",
2583 |        "0  What are the things that we can do to bring change in Indian education system?   \n",
2584 |        "1  Is Donald Trump in league with Putin?                                            \n",
2585 |        "2  What is frugal?                                                                  \n",
2586 |        "3  Why did Sanskrit fail to become a suitable language for computers?               \n",
2587 |        "4  How can I increase my typing speed fast?                                         \n",
2588 |        "\n",
2589 |        "                                                                        Closest-3  \n",
2590 |        "0  What are the things that we can do to bring change in Indian education system?  \n",
2591 |        "1  What are some ways to contact Jesse Ventura?                                    \n",
2592 |        "2  What is a Discord chat?                                                         \n",
2593 |        "3  Technology: Are we compromising on wisdom over information?                     \n",
2594 |        "4  How can I increase my typing speed fast?                                        "
2595 |       ]
2596 |      },
2597 |      "execution_count": 273,
2598 |      "metadata": {},
2599 |      "output_type": "execute_result"
2600 |     }
2601 |    ],
2602 |    "source": [
2603 |     "data_kmeans.head()"
2604 |    ]
2605 |   }
2606 |  ],
2607 |  "metadata": {
2608 |   "kernelspec": {
2609 |    "display_name": "Python 3",
2610 |    "language": "python",
2611 |    "name": "python3"
2612 |   },
2613 |   "language_info": {
2614 |    "codemirror_mode": {
2615 |     "name": "ipython",
2616 |     "version": 3
2617 |    },
2618 |    "file_extension": ".py",
2619 |    "mimetype": "text/x-python",
2620 |    "name": "python",
2621 |    "nbconvert_exporter": "python",
2622 |    "pygments_lexer": "ipython3",
2623 |    "version": "3.5.2"
2624 |   }
2625 |  },
2626 |  "nbformat": 4,
2627 |  "nbformat_minor": 2
2628 | }
2629 | 


--------------------------------------------------------------------------------
/pipeline.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/aditya-AI/Information-Retrieval-System/973bddf49efef539df9ea8f582fc30b451b79cfd/pipeline.png


--------------------------------------------------------------------------------
/siamese.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/aditya-AI/Information-Retrieval-System/973bddf49efef539df9ea8f582fc30b451b79cfd/siamese.png


--------------------------------------------------------------------------------