├── NLP using Deep Learning in Python.ipynb
└── README.md


/NLP using Deep Learning in Python.ipynb:
--------------------------------------------------------------------------------
   1 | {
   2 |  "cells": [
   3 |   {
   4 |    "cell_type": "markdown",
   5 |    "metadata": {},
   6 |    "source": [
   7 |     "# NLP using Deep Learning in Python - Quora Duplicate Questions"
   8 |    ]
   9 |   },
  10 |   {
  11 |    "cell_type": "markdown",
  12 |    "metadata": {},
  13 |    "source": [
  14 |     "## Problem Statement:\n",
  15 |     "\n",
  16 |     "Over 100 million people visit Quora every month, so it's no surprise that **many people ask similarly worded questions**. Multiple questions with the same intent **can cause seekers to spend more time finding the best answer to their question**, and **make writers feel they need to answer multiple versions of the same question**. Quora values canonical questions because they **provide a better experience to active seekers and writers**, and offer more value to both of these groups in the long term.\n",
  17 |     "\n",
  18 |     "**Reference:** https://www.kaggle.com/c/quora-question-pairs"
  19 |    ]
  20 |   },
  21 |   {
  22 |    "cell_type": "code",
  23 |    "execution_count": 322,
  24 |    "metadata": {},
  25 |    "outputs": [],
  26 |    "source": [
  27 |     "import pandas as pd\n",
  28 |     "import numpy as np\n",
  29 |     "import sklearn"
  30 |    ]
  31 |   },
  32 |   {
  33 |    "cell_type": "code",
  34 |    "execution_count": 323,
  35 |    "metadata": {},
  36 |    "outputs": [
  37 |     {
  38 |      "data": {
  39 |       "text/html": [
  40 |        "<div>\n",
  41 |        "<style scoped>\n",
  42 |        "    .dataframe tbody tr th:only-of-type {\n",
  43 |        "        vertical-align: middle;\n",
  44 |        "    }\n",
  45 |        "\n",
  46 |        "    .dataframe tbody tr th {\n",
  47 |        "        vertical-align: top;\n",
  48 |        "    }\n",
  49 |        "\n",
  50 |        "    .dataframe thead th {\n",
  51 |        "        text-align: right;\n",
  52 |        "    }\n",
  53 |        "</style>\n",
  54 |        "<table border=\"1\" class=\"dataframe\">\n",
  55 |        "  <thead>\n",
  56 |        "    <tr style=\"text-align: right;\">\n",
  57 |        "      <th></th>\n",
  58 |        "      <th>id</th>\n",
  59 |        "      <th>qid1</th>\n",
  60 |        "      <th>qid2</th>\n",
  61 |        "      <th>question1</th>\n",
  62 |        "      <th>question2</th>\n",
  63 |        "      <th>is_duplicate</th>\n",
  64 |        "    </tr>\n",
  65 |        "  </thead>\n",
  66 |        "  <tbody>\n",
  67 |        "    <tr>\n",
  68 |        "      <th>0</th>\n",
  69 |        "      <td>0</td>\n",
  70 |        "      <td>1</td>\n",
  71 |        "      <td>2</td>\n",
  72 |        "      <td>What is the step by step guide to invest in sh...</td>\n",
  73 |        "      <td>What is the step by step guide to invest in sh...</td>\n",
  74 |        "      <td>0</td>\n",
  75 |        "    </tr>\n",
  76 |        "    <tr>\n",
  77 |        "      <th>1</th>\n",
  78 |        "      <td>1</td>\n",
  79 |        "      <td>3</td>\n",
  80 |        "      <td>4</td>\n",
  81 |        "      <td>What is the story of Kohinoor (Koh-i-Noor) Dia...</td>\n",
  82 |        "      <td>What would happen if the Indian government sto...</td>\n",
  83 |        "      <td>0</td>\n",
  84 |        "    </tr>\n",
  85 |        "    <tr>\n",
  86 |        "      <th>2</th>\n",
  87 |        "      <td>2</td>\n",
  88 |        "      <td>5</td>\n",
  89 |        "      <td>6</td>\n",
  90 |        "      <td>How can I increase the speed of my internet co...</td>\n",
  91 |        "      <td>How can Internet speed be increased by hacking...</td>\n",
  92 |        "      <td>0</td>\n",
  93 |        "    </tr>\n",
  94 |        "    <tr>\n",
  95 |        "      <th>3</th>\n",
  96 |        "      <td>3</td>\n",
  97 |        "      <td>7</td>\n",
  98 |        "      <td>8</td>\n",
  99 |        "      <td>Why am I mentally very lonely? How can I solve...</td>\n",
 100 |        "      <td>Find the remainder when [math]23^{24}[/math] i...</td>\n",
 101 |        "      <td>0</td>\n",
 102 |        "    </tr>\n",
 103 |        "    <tr>\n",
 104 |        "      <th>4</th>\n",
 105 |        "      <td>4</td>\n",
 106 |        "      <td>9</td>\n",
 107 |        "      <td>10</td>\n",
 108 |        "      <td>Which one dissolve in water quikly sugar, salt...</td>\n",
 109 |        "      <td>Which fish would survive in salt water?</td>\n",
 110 |        "      <td>0</td>\n",
 111 |        "    </tr>\n",
 112 |        "  </tbody>\n",
 113 |        "</table>\n",
 114 |        "</div>"
 115 |       ],
 116 |       "text/plain": [
 117 |        "   id  qid1  qid2                                          question1  \\\n",
 118 |        "0   0     1     2  What is the step by step guide to invest in sh...   \n",
 119 |        "1   1     3     4  What is the story of Kohinoor (Koh-i-Noor) Dia...   \n",
 120 |        "2   2     5     6  How can I increase the speed of my internet co...   \n",
 121 |        "3   3     7     8  Why am I mentally very lonely? How can I solve...   \n",
 122 |        "4   4     9    10  Which one dissolve in water quikly sugar, salt...   \n",
 123 |        "\n",
 124 |        "                                           question2  is_duplicate  \n",
 125 |        "0  What is the step by step guide to invest in sh...             0  \n",
 126 |        "1  What would happen if the Indian government sto...             0  \n",
 127 |        "2  How can Internet speed be increased by hacking...             0  \n",
 128 |        "3  Find the remainder when [math]23^{24}[/math] i...             0  \n",
 129 |        "4            Which fish would survive in salt water?             0  "
 130 |       ]
 131 |      },
 132 |      "execution_count": 323,
 133 |      "metadata": {},
 134 |      "output_type": "execute_result"
 135 |     }
 136 |    ],
 137 |    "source": [
 138 |     "question_pairs = pd.read_csv(\"../../raw_data/questions.csv\")\n",
 139 |     "question_pairs.head()"
 140 |    ]
 141 |   },
 142 |   {
 143 |    "cell_type": "code",
 144 |    "execution_count": 325,
 145 |    "metadata": {},
 146 |    "outputs": [
 147 |     {
 148 |      "data": {
 149 |       "text/plain": [
 150 |        "(808702, 2)"
 151 |       ]
 152 |      },
 153 |      "execution_count": 325,
 154 |      "metadata": {},
 155 |      "output_type": "execute_result"
 156 |     }
 157 |    ],
 158 |    "source": [
 159 |     "question_pairs_1 = question_pairs[['qid1', 'question1']]\n",
 160 |     "question_pairs_1.columns = ['id', 'question']\n",
 161 |     "question_pairs_2 = question_pairs[['qid2', 'question2']]\n",
 162 |     "question_pairs_2.columns = ['id', 'question']\n",
 163 |     "questions_list = pd.concat([question_pairs_1,question_pairs_2]).sort_values('id')\n",
 164 |     "questions_list.shape"
 165 |    ]
 166 |   },
 167 |   {
 168 |    "cell_type": "code",
 169 |    "execution_count": 326,
 170 |    "metadata": {},
 171 |    "outputs": [
 172 |     {
 173 |      "data": {
 174 |       "text/plain": [
 175 |        "['What is the step by step guide to invest in share market in india?',\n",
 176 |        " 'What is the step by step guide to invest in share market?',\n",
 177 |        " 'What is the story of Kohinoor (Koh-i-Noor) Diamond?',\n",
 178 |        " 'What would happen if the Indian government stole the Kohinoor (Koh-i-Noor) diamond back?',\n",
 179 |        " 'How can I increase the speed of my internet connection while using a VPN?',\n",
 180 |        " 'How can Internet speed be increased by hacking through DNS?',\n",
 181 |        " 'Why am I mentally very lonely? How can I solve it?',\n",
 182 |        " 'Find the remainder when [math]23^{24}[/math] is divided by 24,23?',\n",
 183 |        " 'Which one dissolve in water quikly sugar, salt, methane and carbon di oxide?',\n",
 184 |        " 'Which fish would survive in salt water?']"
 185 |       ]
 186 |      },
 187 |      "execution_count": 326,
 188 |      "metadata": {},
 189 |      "output_type": "execute_result"
 190 |     }
 191 |    ],
 192 |    "source": [
 193 |     "corpus = questions_list['question'].tolist()\n",
 194 |     "corpus[:10]"
 195 |    ]
 196 |   },
 197 |   {
 198 |    "cell_type": "code",
 199 |    "execution_count": 327,
 200 |    "metadata": {},
 201 |    "outputs": [],
 202 |    "source": [
 203 |     "corpus = list(np.unique(corpus))"
 204 |    ]
 205 |   },
 206 |   {
 207 |    "cell_type": "markdown",
 208 |    "metadata": {},
 209 |    "source": [
 210 |     "-------\n",
 211 |     "## Feature Extraction"
 212 |    ]
 213 |   },
 214 |   {
 215 |    "cell_type": "markdown",
 216 |    "metadata": {},
 217 |    "source": [
 218 |     "### Count Vectorizer"
 219 |    ]
 220 |   },
 221 |   {
 222 |    "cell_type": "code",
 223 |    "execution_count": 79,
 224 |    "metadata": {},
 225 |    "outputs": [
 226 |     {
 227 |      "data": {
 228 |       "text/html": [
 229 |        "<div>\n",
 230 |        "<style scoped>\n",
 231 |        "    .dataframe tbody tr th:only-of-type {\n",
 232 |        "        vertical-align: middle;\n",
 233 |        "    }\n",
 234 |        "\n",
 235 |        "    .dataframe tbody tr th {\n",
 236 |        "        vertical-align: top;\n",
 237 |        "    }\n",
 238 |        "\n",
 239 |        "    .dataframe thead th {\n",
 240 |        "        text-align: right;\n",
 241 |        "    }\n",
 242 |        "</style>\n",
 243 |        "<table border=\"1\" class=\"dataframe\">\n",
 244 |        "  <thead>\n",
 245 |        "    <tr style=\"text-align: right;\">\n",
 246 |        "      <th></th>\n",
 247 |        "      <th>1000</th>\n",
 248 |        "      <th>2000</th>\n",
 249 |        "      <th>500</th>\n",
 250 |        "      <th>about</th>\n",
 251 |        "      <th>add</th>\n",
 252 |        "      <th>all</th>\n",
 253 |        "      <th>am</th>\n",
 254 |        "      <th>and</th>\n",
 255 |        "      <th>any</th>\n",
 256 |        "      <th>are</th>\n",
 257 |        "      <th>...</th>\n",
 258 |        "      <th>was</th>\n",
 259 |        "      <th>we</th>\n",
 260 |        "      <th>what</th>\n",
 261 |        "      <th>who</th>\n",
 262 |        "      <th>will</th>\n",
 263 |        "      <th>win</th>\n",
 264 |        "      <th>with</th>\n",
 265 |        "      <th>world</th>\n",
 266 |        "      <th>you</th>\n",
 267 |        "      <th>your</th>\n",
 268 |        "    </tr>\n",
 269 |        "  </thead>\n",
 270 |        "  <tbody>\n",
 271 |        "    <tr>\n",
 272 |        "      <th>0</th>\n",
 273 |        "      <td>0</td>\n",
 274 |        "      <td>0</td>\n",
 275 |        "      <td>0</td>\n",
 276 |        "      <td>0</td>\n",
 277 |        "      <td>0</td>\n",
 278 |        "      <td>0</td>\n",
 279 |        "      <td>0</td>\n",
 280 |        "      <td>0</td>\n",
 281 |        "      <td>1</td>\n",
 282 |        "      <td>0</td>\n",
 283 |        "      <td>...</td>\n",
 284 |        "      <td>1</td>\n",
 285 |        "      <td>0</td>\n",
 286 |        "      <td>1</td>\n",
 287 |        "      <td>0</td>\n",
 288 |        "      <td>0</td>\n",
 289 |        "      <td>0</td>\n",
 290 |        "      <td>1</td>\n",
 291 |        "      <td>0</td>\n",
 292 |        "      <td>0</td>\n",
 293 |        "      <td>0</td>\n",
 294 |        "    </tr>\n",
 295 |        "    <tr>\n",
 296 |        "      <th>1</th>\n",
 297 |        "      <td>1</td>\n",
 298 |        "      <td>0</td>\n",
 299 |        "      <td>1</td>\n",
 300 |        "      <td>0</td>\n",
 301 |        "      <td>0</td>\n",
 302 |        "      <td>0</td>\n",
 303 |        "      <td>0</td>\n",
 304 |        "      <td>2</td>\n",
 305 |        "      <td>0</td>\n",
 306 |        "      <td>0</td>\n",
 307 |        "      <td>...</td>\n",
 308 |        "      <td>0</td>\n",
 309 |        "      <td>0</td>\n",
 310 |        "      <td>1</td>\n",
 311 |        "      <td>0</td>\n",
 312 |        "      <td>1</td>\n",
 313 |        "      <td>0</td>\n",
 314 |        "      <td>0</td>\n",
 315 |        "      <td>0</td>\n",
 316 |        "      <td>0</td>\n",
 317 |        "      <td>0</td>\n",
 318 |        "    </tr>\n",
 319 |        "    <tr>\n",
 320 |        "      <th>2</th>\n",
 321 |        "      <td>0</td>\n",
 322 |        "      <td>0</td>\n",
 323 |        "      <td>0</td>\n",
 324 |        "      <td>0</td>\n",
 325 |        "      <td>1</td>\n",
 326 |        "      <td>0</td>\n",
 327 |        "      <td>0</td>\n",
 328 |        "      <td>0</td>\n",
 329 |        "      <td>0</td>\n",
 330 |        "      <td>0</td>\n",
 331 |        "      <td>...</td>\n",
 332 |        "      <td>0</td>\n",
 333 |        "      <td>0</td>\n",
 334 |        "      <td>0</td>\n",
 335 |        "      <td>0</td>\n",
 336 |        "      <td>0</td>\n",
 337 |        "      <td>0</td>\n",
 338 |        "      <td>0</td>\n",
 339 |        "      <td>0</td>\n",
 340 |        "      <td>0</td>\n",
 341 |        "      <td>0</td>\n",
 342 |        "    </tr>\n",
 343 |        "    <tr>\n",
 344 |        "      <th>3</th>\n",
 345 |        "      <td>0</td>\n",
 346 |        "      <td>0</td>\n",
 347 |        "      <td>0</td>\n",
 348 |        "      <td>0</td>\n",
 349 |        "      <td>0</td>\n",
 350 |        "      <td>1</td>\n",
 351 |        "      <td>0</td>\n",
 352 |        "      <td>0</td>\n",
 353 |        "      <td>0</td>\n",
 354 |        "      <td>0</td>\n",
 355 |        "      <td>...</td>\n",
 356 |        "      <td>0</td>\n",
 357 |        "      <td>0</td>\n",
 358 |        "      <td>0</td>\n",
 359 |        "      <td>1</td>\n",
 360 |        "      <td>1</td>\n",
 361 |        "      <td>1</td>\n",
 362 |        "      <td>0</td>\n",
 363 |        "      <td>0</td>\n",
 364 |        "      <td>1</td>\n",
 365 |        "      <td>0</td>\n",
 366 |        "    </tr>\n",
 367 |        "    <tr>\n",
 368 |        "      <th>4</th>\n",
 369 |        "      <td>1</td>\n",
 370 |        "      <td>0</td>\n",
 371 |        "      <td>0</td>\n",
 372 |        "      <td>0</td>\n",
 373 |        "      <td>0</td>\n",
 374 |        "      <td>0</td>\n",
 375 |        "      <td>1</td>\n",
 376 |        "      <td>0</td>\n",
 377 |        "      <td>0</td>\n",
 378 |        "      <td>0</td>\n",
 379 |        "      <td>...</td>\n",
 380 |        "      <td>0</td>\n",
 381 |        "      <td>0</td>\n",
 382 |        "      <td>0</td>\n",
 383 |        "      <td>0</td>\n",
 384 |        "      <td>0</td>\n",
 385 |        "      <td>0</td>\n",
 386 |        "      <td>0</td>\n",
 387 |        "      <td>0</td>\n",
 388 |        "      <td>0</td>\n",
 389 |        "      <td>0</td>\n",
 390 |        "    </tr>\n",
 391 |        "    <tr>\n",
 392 |        "      <th>5</th>\n",
 393 |        "      <td>0</td>\n",
 394 |        "      <td>1</td>\n",
 395 |        "      <td>1</td>\n",
 396 |        "      <td>0</td>\n",
 397 |        "      <td>0</td>\n",
 398 |        "      <td>0</td>\n",
 399 |        "      <td>0</td>\n",
 400 |        "      <td>1</td>\n",
 401 |        "      <td>0</td>\n",
 402 |        "      <td>1</td>\n",
 403 |        "      <td>...</td>\n",
 404 |        "      <td>0</td>\n",
 405 |        "      <td>0</td>\n",
 406 |        "      <td>0</td>\n",
 407 |        "      <td>0</td>\n",
 408 |        "      <td>0</td>\n",
 409 |        "      <td>0</td>\n",
 410 |        "      <td>0</td>\n",
 411 |        "      <td>0</td>\n",
 412 |        "      <td>0</td>\n",
 413 |        "      <td>0</td>\n",
 414 |        "    </tr>\n",
 415 |        "    <tr>\n",
 416 |        "      <th>6</th>\n",
 417 |        "      <td>0</td>\n",
 418 |        "      <td>1</td>\n",
 419 |        "      <td>0</td>\n",
 420 |        "      <td>0</td>\n",
 421 |        "      <td>0</td>\n",
 422 |        "      <td>0</td>\n",
 423 |        "      <td>0</td>\n",
 424 |        "      <td>0</td>\n",
 425 |        "      <td>0</td>\n",
 426 |        "      <td>1</td>\n",
 427 |        "      <td>...</td>\n",
 428 |        "      <td>0</td>\n",
 429 |        "      <td>0</td>\n",
 430 |        "      <td>0</td>\n",
 431 |        "      <td>0</td>\n",
 432 |        "      <td>0</td>\n",
 433 |        "      <td>0</td>\n",
 434 |        "      <td>1</td>\n",
 435 |        "      <td>0</td>\n",
 436 |        "      <td>0</td>\n",
 437 |        "      <td>0</td>\n",
 438 |        "    </tr>\n",
 439 |        "    <tr>\n",
 440 |        "      <th>7</th>\n",
 441 |        "      <td>0</td>\n",
 442 |        "      <td>0</td>\n",
 443 |        "      <td>0</td>\n",
 444 |        "      <td>1</td>\n",
 445 |        "      <td>0</td>\n",
 446 |        "      <td>0</td>\n",
 447 |        "      <td>0</td>\n",
 448 |        "      <td>0</td>\n",
 449 |        "      <td>0</td>\n",
 450 |        "      <td>1</td>\n",
 451 |        "      <td>...</td>\n",
 452 |        "      <td>0</td>\n",
 453 |        "      <td>0</td>\n",
 454 |        "      <td>0</td>\n",
 455 |        "      <td>1</td>\n",
 456 |        "      <td>0</td>\n",
 457 |        "      <td>0</td>\n",
 458 |        "      <td>1</td>\n",
 459 |        "      <td>0</td>\n",
 460 |        "      <td>1</td>\n",
 461 |        "      <td>1</td>\n",
 462 |        "    </tr>\n",
 463 |        "    <tr>\n",
 464 |        "      <th>8</th>\n",
 465 |        "      <td>0</td>\n",
 466 |        "      <td>0</td>\n",
 467 |        "      <td>0</td>\n",
 468 |        "      <td>0</td>\n",
 469 |        "      <td>0</td>\n",
 470 |        "      <td>0</td>\n",
 471 |        "      <td>0</td>\n",
 472 |        "      <td>0</td>\n",
 473 |        "      <td>0</td>\n",
 474 |        "      <td>1</td>\n",
 475 |        "      <td>...</td>\n",
 476 |        "      <td>0</td>\n",
 477 |        "      <td>1</td>\n",
 478 |        "      <td>0</td>\n",
 479 |        "      <td>0</td>\n",
 480 |        "      <td>0</td>\n",
 481 |        "      <td>0</td>\n",
 482 |        "      <td>0</td>\n",
 483 |        "      <td>1</td>\n",
 484 |        "      <td>0</td>\n",
 485 |        "      <td>0</td>\n",
 486 |        "    </tr>\n",
 487 |        "    <tr>\n",
 488 |        "      <th>9</th>\n",
 489 |        "      <td>0</td>\n",
 490 |        "      <td>0</td>\n",
 491 |        "      <td>0</td>\n",
 492 |        "      <td>0</td>\n",
 493 |        "      <td>0</td>\n",
 494 |        "      <td>0</td>\n",
 495 |        "      <td>0</td>\n",
 496 |        "      <td>0</td>\n",
 497 |        "      <td>0</td>\n",
 498 |        "      <td>1</td>\n",
 499 |        "      <td>...</td>\n",
 500 |        "      <td>0</td>\n",
 501 |        "      <td>1</td>\n",
 502 |        "      <td>0</td>\n",
 503 |        "      <td>0</td>\n",
 504 |        "      <td>0</td>\n",
 505 |        "      <td>0</td>\n",
 506 |        "      <td>0</td>\n",
 507 |        "      <td>1</td>\n",
 508 |        "      <td>0</td>\n",
 509 |        "      <td>0</td>\n",
 510 |        "    </tr>\n",
 511 |        "  </tbody>\n",
 512 |        "</table>\n",
 513 |        "<p>10 rows × 96 columns</p>\n",
 514 |        "</div>"
 515 |       ],
 516 |       "text/plain": [
 517 |        "   1000  2000  500  about  add  all  am  and  any  are  ...  was  we  what  \\\n",
 518 |        "0     0     0    0      0    0    0   0    0    1    0  ...    1   0     1   \n",
 519 |        "1     1     0    1      0    0    0   0    2    0    0  ...    0   0     1   \n",
 520 |        "2     0     0    0      0    1    0   0    0    0    0  ...    0   0     0   \n",
 521 |        "3     0     0    0      0    0    1   0    0    0    0  ...    0   0     0   \n",
 522 |        "4     1     0    0      0    0    0   1    0    0    0  ...    0   0     0   \n",
 523 |        "5     0     1    1      0    0    0   0    1    0    1  ...    0   0     0   \n",
 524 |        "6     0     1    0      0    0    0   0    0    0    1  ...    0   0     0   \n",
 525 |        "7     0     0    0      1    0    0   0    0    0    1  ...    0   0     0   \n",
 526 |        "8     0     0    0      0    0    0   0    0    0    1  ...    0   1     0   \n",
 527 |        "9     0     0    0      0    0    0   0    0    0    1  ...    0   1     0   \n",
 528 |        "\n",
 529 |        "   who  will  win  with  world  you  your  \n",
 530 |        "0    0     0    0     1      0    0     0  \n",
 531 |        "1    0     1    0     0      0    0     0  \n",
 532 |        "2    0     0    0     0      0    0     0  \n",
 533 |        "3    1     1    1     0      0    1     0  \n",
 534 |        "4    0     0    0     0      0    0     0  \n",
 535 |        "5    0     0    0     0      0    0     0  \n",
 536 |        "6    0     0    0     1      0    0     0  \n",
 537 |        "7    1     0    0     1      0    1     1  \n",
 538 |        "8    0     0    0     0      1    0     0  \n",
 539 |        "9    0     0    0     0      1    0     0  \n",
 540 |        "\n",
 541 |        "[10 rows x 96 columns]"
 542 |       ]
 543 |      },
 544 |      "execution_count": 79,
 545 |      "metadata": {},
 546 |      "output_type": "execute_result"
 547 |     }
 548 |    ],
 549 |    "source": [
 550 |     "from sklearn.feature_extraction.text import CountVectorizer\n",
 551 |     "count_vect = CountVectorizer()\n",
 552 |     "\n",
 553 |     "X_train_counts = count_vect.fit_transform(corpus[:10])\n",
 554 |     "X_train_counts = pd.DataFrame(X_train_counts.toarray())\n",
 555 |     "X_train_counts.columns = count_vect.get_feature_names()\n",
 556 |     "X_train_counts"
 557 |    ]
 558 |   },
 559 |   {
 560 |    "cell_type": "code",
 561 |    "execution_count": 80,
 562 |    "metadata": {},
 563 |    "outputs": [
 564 |     {
 565 |      "data": {
 566 |       "text/plain": [
 567 |        "'\"The question was marked as needing improvement\" how to deal with this, what ever I do still this error pops up? Is it Quora bot or any user?'"
 568 |       ]
 569 |      },
 570 |      "execution_count": 80,
 571 |      "metadata": {},
 572 |      "output_type": "execute_result"
 573 |     }
 574 |    ],
 575 |    "source": [
 576 |     "corpus[0]"
 577 |    ]
 578 |   },
 579 |   {
 580 |    "cell_type": "code",
 581 |    "execution_count": 81,
 582 |    "metadata": {},
 583 |    "outputs": [
 584 |     {
 585 |      "data": {
 586 |       "text/plain": [
 587 |        "1000             0\n",
 588 |        "2000             0\n",
 589 |        "500              0\n",
 590 |        "about            0\n",
 591 |        "add              0\n",
 592 |        "all              0\n",
 593 |        "am               0\n",
 594 |        "and              0\n",
 595 |        "any              1\n",
 596 |        "are              0\n",
 597 |        "as               1\n",
 598 |        "aside            0\n",
 599 |        "at               0\n",
 600 |        "banned           0\n",
 601 |        "based            0\n",
 602 |        "be               0\n",
 603 |        "been             0\n",
 604 |        "biases           0\n",
 605 |        "big              0\n",
 606 |        "bot              1\n",
 607 |        "can              0\n",
 608 |        "chip             0\n",
 609 |        "closer           0\n",
 610 |        "currency         0\n",
 611 |        "deal             1\n",
 612 |        "distance         0\n",
 613 |        "do               1\n",
 614 |        "election         0\n",
 615 |        "embedded         0\n",
 616 |        "error            1\n",
 617 |        "                ..\n",
 618 |        "really           0\n",
 619 |        "relationship     0\n",
 620 |        "relationships    0\n",
 621 |        "rs               0\n",
 622 |        "see              0\n",
 623 |        "short            0\n",
 624 |        "starting         0\n",
 625 |        "still            1\n",
 626 |        "successful       0\n",
 627 |        "tell             0\n",
 628 |        "term             0\n",
 629 |        "the              1\n",
 630 |        "there            0\n",
 631 |        "think            0\n",
 632 |        "this             2\n",
 633 |        "time             0\n",
 634 |        "to               1\n",
 635 |        "up               1\n",
 636 |        "user             1\n",
 637 |        "war              0\n",
 638 |        "was              1\n",
 639 |        "we               0\n",
 640 |        "what             1\n",
 641 |        "who              0\n",
 642 |        "will             0\n",
 643 |        "win              0\n",
 644 |        "with             1\n",
 645 |        "world            0\n",
 646 |        "you              0\n",
 647 |        "your             0\n",
 648 |        "Name: 0, Length: 96, dtype: int64"
 649 |       ]
 650 |      },
 651 |      "execution_count": 81,
 652 |      "metadata": {},
 653 |      "output_type": "execute_result"
 654 |     }
 655 |    ],
 656 |    "source": [
 657 |     "X_train_counts.loc[0]"
 658 |    ]
 659 |   },
 660 |   {
 661 |    "cell_type": "markdown",
 662 |    "metadata": {},
 663 |    "source": [
 664 |     "### Tf-Idf (Term Frequency - Inverse Document Frequency)"
 665 |    ]
 666 |   },
 667 |   {
 668 |    "cell_type": "code",
 669 |    "execution_count": 82,
 670 |    "metadata": {},
 671 |    "outputs": [
 672 |     {
 673 |      "data": {
 674 |       "text/html": [
 675 |        "<div>\n",
 676 |        "<style scoped>\n",
 677 |        "    .dataframe tbody tr th:only-of-type {\n",
 678 |        "        vertical-align: middle;\n",
 679 |        "    }\n",
 680 |        "\n",
 681 |        "    .dataframe tbody tr th {\n",
 682 |        "        vertical-align: top;\n",
 683 |        "    }\n",
 684 |        "\n",
 685 |        "    .dataframe thead th {\n",
 686 |        "        text-align: right;\n",
 687 |        "    }\n",
 688 |        "</style>\n",
 689 |        "<table border=\"1\" class=\"dataframe\">\n",
 690 |        "  <thead>\n",
 691 |        "    <tr style=\"text-align: right;\">\n",
 692 |        "      <th></th>\n",
 693 |        "      <th>1000</th>\n",
 694 |        "      <th>2000</th>\n",
 695 |        "      <th>500</th>\n",
 696 |        "      <th>about</th>\n",
 697 |        "      <th>add</th>\n",
 698 |        "      <th>all</th>\n",
 699 |        "      <th>am</th>\n",
 700 |        "      <th>and</th>\n",
 701 |        "      <th>any</th>\n",
 702 |        "      <th>are</th>\n",
 703 |        "      <th>...</th>\n",
 704 |        "      <th>was</th>\n",
 705 |        "      <th>we</th>\n",
 706 |        "      <th>what</th>\n",
 707 |        "      <th>who</th>\n",
 708 |        "      <th>will</th>\n",
 709 |        "      <th>win</th>\n",
 710 |        "      <th>with</th>\n",
 711 |        "      <th>world</th>\n",
 712 |        "      <th>you</th>\n",
 713 |        "      <th>your</th>\n",
 714 |        "    </tr>\n",
 715 |        "  </thead>\n",
 716 |        "  <tbody>\n",
 717 |        "    <tr>\n",
 718 |        "      <th>0</th>\n",
 719 |        "      <td>0.000000</td>\n",
 720 |        "      <td>0.000000</td>\n",
 721 |        "      <td>0.000000</td>\n",
 722 |        "      <td>0.000000</td>\n",
 723 |        "      <td>0.000000</td>\n",
 724 |        "      <td>0.000000</td>\n",
 725 |        "      <td>0.000000</td>\n",
 726 |        "      <td>0.000000</td>\n",
 727 |        "      <td>0.20204</td>\n",
 728 |        "      <td>0.000000</td>\n",
 729 |        "      <td>...</td>\n",
 730 |        "      <td>0.20204</td>\n",
 731 |        "      <td>0.000000</td>\n",
 732 |        "      <td>0.171753</td>\n",
 733 |        "      <td>0.000000</td>\n",
 734 |        "      <td>0.000000</td>\n",
 735 |        "      <td>0.000000</td>\n",
 736 |        "      <td>0.150263</td>\n",
 737 |        "      <td>0.000000</td>\n",
 738 |        "      <td>0.000000</td>\n",
 739 |        "      <td>0.000000</td>\n",
 740 |        "    </tr>\n",
 741 |        "    <tr>\n",
 742 |        "      <th>1</th>\n",
 743 |        "      <td>0.205776</td>\n",
 744 |        "      <td>0.000000</td>\n",
 745 |        "      <td>0.205776</td>\n",
 746 |        "      <td>0.000000</td>\n",
 747 |        "      <td>0.000000</td>\n",
 748 |        "      <td>0.000000</td>\n",
 749 |        "      <td>0.000000</td>\n",
 750 |        "      <td>0.411553</td>\n",
 751 |        "      <td>0.00000</td>\n",
 752 |        "      <td>0.000000</td>\n",
 753 |        "      <td>...</td>\n",
 754 |        "      <td>0.00000</td>\n",
 755 |        "      <td>0.000000</td>\n",
 756 |        "      <td>0.205776</td>\n",
 757 |        "      <td>0.000000</td>\n",
 758 |        "      <td>0.205776</td>\n",
 759 |        "      <td>0.000000</td>\n",
 760 |        "      <td>0.000000</td>\n",
 761 |        "      <td>0.000000</td>\n",
 762 |        "      <td>0.000000</td>\n",
 763 |        "      <td>0.000000</td>\n",
 764 |        "    </tr>\n",
 765 |        "    <tr>\n",
 766 |        "      <th>2</th>\n",
 767 |        "      <td>0.000000</td>\n",
 768 |        "      <td>0.000000</td>\n",
 769 |        "      <td>0.000000</td>\n",
 770 |        "      <td>0.000000</td>\n",
 771 |        "      <td>0.518291</td>\n",
 772 |        "      <td>0.000000</td>\n",
 773 |        "      <td>0.000000</td>\n",
 774 |        "      <td>0.000000</td>\n",
 775 |        "      <td>0.00000</td>\n",
 776 |        "      <td>0.000000</td>\n",
 777 |        "      <td>...</td>\n",
 778 |        "      <td>0.00000</td>\n",
 779 |        "      <td>0.000000</td>\n",
 780 |        "      <td>0.000000</td>\n",
 781 |        "      <td>0.000000</td>\n",
 782 |        "      <td>0.000000</td>\n",
 783 |        "      <td>0.000000</td>\n",
 784 |        "      <td>0.000000</td>\n",
 785 |        "      <td>0.000000</td>\n",
 786 |        "      <td>0.000000</td>\n",
 787 |        "      <td>0.000000</td>\n",
 788 |        "    </tr>\n",
 789 |        "    <tr>\n",
 790 |        "      <th>3</th>\n",
 791 |        "      <td>0.000000</td>\n",
 792 |        "      <td>0.000000</td>\n",
 793 |        "      <td>0.000000</td>\n",
 794 |        "      <td>0.000000</td>\n",
 795 |        "      <td>0.000000</td>\n",
 796 |        "      <td>0.263025</td>\n",
 797 |        "      <td>0.000000</td>\n",
 798 |        "      <td>0.000000</td>\n",
 799 |        "      <td>0.00000</td>\n",
 800 |        "      <td>0.000000</td>\n",
 801 |        "      <td>...</td>\n",
 802 |        "      <td>0.00000</td>\n",
 803 |        "      <td>0.000000</td>\n",
 804 |        "      <td>0.000000</td>\n",
 805 |        "      <td>0.223595</td>\n",
 806 |        "      <td>0.223595</td>\n",
 807 |        "      <td>0.263025</td>\n",
 808 |        "      <td>0.000000</td>\n",
 809 |        "      <td>0.000000</td>\n",
 810 |        "      <td>0.223595</td>\n",
 811 |        "      <td>0.000000</td>\n",
 812 |        "    </tr>\n",
 813 |        "    <tr>\n",
 814 |        "      <th>4</th>\n",
 815 |        "      <td>0.266593</td>\n",
 816 |        "      <td>0.000000</td>\n",
 817 |        "      <td>0.000000</td>\n",
 818 |        "      <td>0.000000</td>\n",
 819 |        "      <td>0.000000</td>\n",
 820 |        "      <td>0.000000</td>\n",
 821 |        "      <td>0.313605</td>\n",
 822 |        "      <td>0.000000</td>\n",
 823 |        "      <td>0.00000</td>\n",
 824 |        "      <td>0.000000</td>\n",
 825 |        "      <td>...</td>\n",
 826 |        "      <td>0.00000</td>\n",
 827 |        "      <td>0.000000</td>\n",
 828 |        "      <td>0.000000</td>\n",
 829 |        "      <td>0.000000</td>\n",
 830 |        "      <td>0.000000</td>\n",
 831 |        "      <td>0.000000</td>\n",
 832 |        "      <td>0.000000</td>\n",
 833 |        "      <td>0.000000</td>\n",
 834 |        "      <td>0.000000</td>\n",
 835 |        "      <td>0.000000</td>\n",
 836 |        "    </tr>\n",
 837 |        "    <tr>\n",
 838 |        "      <th>5</th>\n",
 839 |        "      <td>0.000000</td>\n",
 840 |        "      <td>0.251549</td>\n",
 841 |        "      <td>0.251549</td>\n",
 842 |        "      <td>0.000000</td>\n",
 843 |        "      <td>0.000000</td>\n",
 844 |        "      <td>0.000000</td>\n",
 845 |        "      <td>0.000000</td>\n",
 846 |        "      <td>0.251549</td>\n",
 847 |        "      <td>0.00000</td>\n",
 848 |        "      <td>0.175717</td>\n",
 849 |        "      <td>...</td>\n",
 850 |        "      <td>0.00000</td>\n",
 851 |        "      <td>0.000000</td>\n",
 852 |        "      <td>0.000000</td>\n",
 853 |        "      <td>0.000000</td>\n",
 854 |        "      <td>0.000000</td>\n",
 855 |        "      <td>0.000000</td>\n",
 856 |        "      <td>0.000000</td>\n",
 857 |        "      <td>0.000000</td>\n",
 858 |        "      <td>0.000000</td>\n",
 859 |        "      <td>0.000000</td>\n",
 860 |        "    </tr>\n",
 861 |        "    <tr>\n",
 862 |        "      <th>6</th>\n",
 863 |        "      <td>0.000000</td>\n",
 864 |        "      <td>0.313340</td>\n",
 865 |        "      <td>0.000000</td>\n",
 866 |        "      <td>0.000000</td>\n",
 867 |        "      <td>0.000000</td>\n",
 868 |        "      <td>0.000000</td>\n",
 869 |        "      <td>0.000000</td>\n",
 870 |        "      <td>0.000000</td>\n",
 871 |        "      <td>0.00000</td>\n",
 872 |        "      <td>0.218880</td>\n",
 873 |        "      <td>...</td>\n",
 874 |        "      <td>0.00000</td>\n",
 875 |        "      <td>0.000000</td>\n",
 876 |        "      <td>0.000000</td>\n",
 877 |        "      <td>0.000000</td>\n",
 878 |        "      <td>0.000000</td>\n",
 879 |        "      <td>0.000000</td>\n",
 880 |        "      <td>0.274136</td>\n",
 881 |        "      <td>0.000000</td>\n",
 882 |        "      <td>0.000000</td>\n",
 883 |        "      <td>0.000000</td>\n",
 884 |        "    </tr>\n",
 885 |        "    <tr>\n",
 886 |        "      <th>7</th>\n",
 887 |        "      <td>0.000000</td>\n",
 888 |        "      <td>0.000000</td>\n",
 889 |        "      <td>0.000000</td>\n",
 890 |        "      <td>0.204276</td>\n",
 891 |        "      <td>0.000000</td>\n",
 892 |        "      <td>0.000000</td>\n",
 893 |        "      <td>0.000000</td>\n",
 894 |        "      <td>0.000000</td>\n",
 895 |        "      <td>0.00000</td>\n",
 896 |        "      <td>0.121303</td>\n",
 897 |        "      <td>...</td>\n",
 898 |        "      <td>0.00000</td>\n",
 899 |        "      <td>0.000000</td>\n",
 900 |        "      <td>0.000000</td>\n",
 901 |        "      <td>0.173653</td>\n",
 902 |        "      <td>0.000000</td>\n",
 903 |        "      <td>0.000000</td>\n",
 904 |        "      <td>0.151926</td>\n",
 905 |        "      <td>0.000000</td>\n",
 906 |        "      <td>0.173653</td>\n",
 907 |        "      <td>0.204276</td>\n",
 908 |        "    </tr>\n",
 909 |        "    <tr>\n",
 910 |        "      <th>8</th>\n",
 911 |        "      <td>0.000000</td>\n",
 912 |        "      <td>0.000000</td>\n",
 913 |        "      <td>0.000000</td>\n",
 914 |        "      <td>0.000000</td>\n",
 915 |        "      <td>0.000000</td>\n",
 916 |        "      <td>0.000000</td>\n",
 917 |        "      <td>0.000000</td>\n",
 918 |        "      <td>0.000000</td>\n",
 919 |        "      <td>0.00000</td>\n",
 920 |        "      <td>0.263628</td>\n",
 921 |        "      <td>...</td>\n",
 922 |        "      <td>0.00000</td>\n",
 923 |        "      <td>0.377400</td>\n",
 924 |        "      <td>0.000000</td>\n",
 925 |        "      <td>0.000000</td>\n",
 926 |        "      <td>0.000000</td>\n",
 927 |        "      <td>0.000000</td>\n",
 928 |        "      <td>0.000000</td>\n",
 929 |        "      <td>0.377400</td>\n",
 930 |        "      <td>0.000000</td>\n",
 931 |        "      <td>0.000000</td>\n",
 932 |        "    </tr>\n",
 933 |        "    <tr>\n",
 934 |        "      <th>9</th>\n",
 935 |        "      <td>0.000000</td>\n",
 936 |        "      <td>0.000000</td>\n",
 937 |        "      <td>0.000000</td>\n",
 938 |        "      <td>0.000000</td>\n",
 939 |        "      <td>0.000000</td>\n",
 940 |        "      <td>0.000000</td>\n",
 941 |        "      <td>0.000000</td>\n",
 942 |        "      <td>0.000000</td>\n",
 943 |        "      <td>0.00000</td>\n",
 944 |        "      <td>0.235430</td>\n",
 945 |        "      <td>...</td>\n",
 946 |        "      <td>0.00000</td>\n",
 947 |        "      <td>0.337033</td>\n",
 948 |        "      <td>0.000000</td>\n",
 949 |        "      <td>0.000000</td>\n",
 950 |        "      <td>0.000000</td>\n",
 951 |        "      <td>0.000000</td>\n",
 952 |        "      <td>0.000000</td>\n",
 953 |        "      <td>0.337033</td>\n",
 954 |        "      <td>0.000000</td>\n",
 955 |        "      <td>0.000000</td>\n",
 956 |        "    </tr>\n",
 957 |        "  </tbody>\n",
 958 |        "</table>\n",
 959 |        "<p>10 rows × 96 columns</p>\n",
 960 |        "</div>"
 961 |       ],
 962 |       "text/plain": [
 963 |        "       1000      2000       500     about       add       all        am  \\\n",
 964 |        "0  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000   \n",
 965 |        "1  0.205776  0.000000  0.205776  0.000000  0.000000  0.000000  0.000000   \n",
 966 |        "2  0.000000  0.000000  0.000000  0.000000  0.518291  0.000000  0.000000   \n",
 967 |        "3  0.000000  0.000000  0.000000  0.000000  0.000000  0.263025  0.000000   \n",
 968 |        "4  0.266593  0.000000  0.000000  0.000000  0.000000  0.000000  0.313605   \n",
 969 |        "5  0.000000  0.251549  0.251549  0.000000  0.000000  0.000000  0.000000   \n",
 970 |        "6  0.000000  0.313340  0.000000  0.000000  0.000000  0.000000  0.000000   \n",
 971 |        "7  0.000000  0.000000  0.000000  0.204276  0.000000  0.000000  0.000000   \n",
 972 |        "8  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000   \n",
 973 |        "9  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000   \n",
 974 |        "\n",
 975 |        "        and      any       are  ...      was        we      what       who  \\\n",
 976 |        "0  0.000000  0.20204  0.000000  ...  0.20204  0.000000  0.171753  0.000000   \n",
 977 |        "1  0.411553  0.00000  0.000000  ...  0.00000  0.000000  0.205776  0.000000   \n",
 978 |        "2  0.000000  0.00000  0.000000  ...  0.00000  0.000000  0.000000  0.000000   \n",
 979 |        "3  0.000000  0.00000  0.000000  ...  0.00000  0.000000  0.000000  0.223595   \n",
 980 |        "4  0.000000  0.00000  0.000000  ...  0.00000  0.000000  0.000000  0.000000   \n",
 981 |        "5  0.251549  0.00000  0.175717  ...  0.00000  0.000000  0.000000  0.000000   \n",
 982 |        "6  0.000000  0.00000  0.218880  ...  0.00000  0.000000  0.000000  0.000000   \n",
 983 |        "7  0.000000  0.00000  0.121303  ...  0.00000  0.000000  0.000000  0.173653   \n",
 984 |        "8  0.000000  0.00000  0.263628  ...  0.00000  0.377400  0.000000  0.000000   \n",
 985 |        "9  0.000000  0.00000  0.235430  ...  0.00000  0.337033  0.000000  0.000000   \n",
 986 |        "\n",
 987 |        "       will       win      with     world       you      your  \n",
 988 |        "0  0.000000  0.000000  0.150263  0.000000  0.000000  0.000000  \n",
 989 |        "1  0.205776  0.000000  0.000000  0.000000  0.000000  0.000000  \n",
 990 |        "2  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  \n",
 991 |        "3  0.223595  0.263025  0.000000  0.000000  0.223595  0.000000  \n",
 992 |        "4  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  \n",
 993 |        "5  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  \n",
 994 |        "6  0.000000  0.000000  0.274136  0.000000  0.000000  0.000000  \n",
 995 |        "7  0.000000  0.000000  0.151926  0.000000  0.173653  0.204276  \n",
 996 |        "8  0.000000  0.000000  0.000000  0.377400  0.000000  0.000000  \n",
 997 |        "9  0.000000  0.000000  0.000000  0.337033  0.000000  0.000000  \n",
 998 |        "\n",
 999 |        "[10 rows x 96 columns]"
1000 |       ]
1001 |      },
1002 |      "execution_count": 82,
1003 |      "metadata": {},
1004 |      "output_type": "execute_result"
1005 |     }
1006 |    ],
1007 |    "source": [
1008 |     "from sklearn.feature_extraction.text import TfidfVectorizer\n",
1009 |     "vectorizer = TfidfVectorizer()\n",
1010 |     "\n",
1011 |     "X_train_tfidf = vectorizer.fit_transform(corpus[:10])\n",
1012 |     "X_train_tfidf = pd.DataFrame(X_train_tfidf.toarray())\n",
1013 |     "X_train_tfidf.columns = vectorizer.get_feature_names()\n",
1014 |     "X_train_tfidf"
1015 |    ]
1016 |   },
1017 |   {
1018 |    "cell_type": "markdown",
1019 |    "metadata": {},
1020 |    "source": [
1021 |     "$$Tf(w, d) = Number \\ of \\ times \\ word \\ `w` \\ appears \\ in \\ document \\ `d`$$\n",
1022 |     "\n",
1023 |     "$$IDF(w) = \\log \\frac{Total \\ number \\ of \\ documents}{Number \\ of \\ documents \\ with \\ word \\ `w`}$$\n",
1024 |     "\n",
1025 |     "$$Tfidf(w, d) = Tf(w, d) * IDF(w)$$"
1026 |    ]
1027 |   },
1028 |   {
1029 |    "cell_type": "markdown",
1030 |    "metadata": {},
1031 |    "source": [
1032 |     "### Word Vectors\n",
1033 |     "\n",
1034 |     "Word vectors - also called *word embeddings* - are mathematical descriptions of individual words such that words that appear frequently together in the language will have similar values. In this way we can mathematically derive *context*.\n",
1035 |     "\n",
1036 |     "**There are two possible approaches:**\n",
1037 |     "\n",
1038 |     "<img src=\"img/w2v_image.png\" alt=\"Drawing\" style=\"width: 600px;\"/>\n",
1039 |     "\n",
1040 |     "**CBOW (Continuous Bag Of Words):** It predicts the word, given context around the word as input\n",
1041 |     "\n",
1042 |     "**Skip-gram:** It predicts the context, given the word as input"
1043 |    ]
1044 |   },
1045 |   {
1046 |    "cell_type": "code",
1047 |    "execution_count": 83,
1048 |    "metadata": {},
1049 |    "outputs": [],
1050 |    "source": [
1051 |     "import spacy\n",
1052 |     "nlp = spacy.load('en_core_web_md')"
1053 |    ]
1054 |   },
1055 |   {
1056 |    "cell_type": "code",
1057 |    "execution_count": 84,
1058 |    "metadata": {},
1059 |    "outputs": [
1060 |     {
1061 |      "data": {
1062 |       "text/plain": [
1063 |        "300"
1064 |       ]
1065 |      },
1066 |      "execution_count": 84,
1067 |      "metadata": {},
1068 |      "output_type": "execute_result"
1069 |     }
1070 |    ],
1071 |    "source": [
1072 |     "len(nlp('dog').vector)"
1073 |    ]
1074 |   },
1075 |   {
1076 |    "cell_type": "code",
1077 |    "execution_count": 85,
1078 |    "metadata": {},
1079 |    "outputs": [],
1080 |    "source": [
1081 |     "def most_similar(word, topn=5):\n",
1082 |     "    word = nlp.vocab[str(word)]\n",
1083 |     "    queries = [\n",
1084 |     "      w for w in word.vocab \n",
1085 |     "      if w.is_lower == word.is_lower and w.prob >= -15 and np.count_nonzero(w.vector)\n",
1086 |     "    ]\n",
1087 |     "\n",
1088 |     "    by_similarity = sorted(queries, key=lambda w: word.similarity(w), reverse=True)\n",
1089 |     "    return [(w.lower_,w.similarity(word)) for w in by_similarity[:topn+1] if w.lower_ != word.lower_]"
1090 |    ]
1091 |   },
1092 |   {
1093 |    "cell_type": "code",
1094 |    "execution_count": 86,
1095 |    "metadata": {},
1096 |    "outputs": [
1097 |     {
1098 |      "data": {
1099 |       "text/plain": [
1100 |        "[('princes', 0.7876614),\n",
1101 |        " ('kings', 0.7876614),\n",
1102 |        " ('prince', 0.73377377),\n",
1103 |        " ('queen', 0.72526103),\n",
1104 |        " ('scepter', 0.6726005),\n",
1105 |        " ('throne', 0.6726005),\n",
1106 |        " ('kingdoms', 0.6604046),\n",
1107 |        " ('kingdom', 0.6604046),\n",
1108 |        " ('lord', 0.6439695),\n",
1109 |        " ('royal', 0.6168811)]"
1110 |       ]
1111 |      },
1112 |      "execution_count": 86,
1113 |      "metadata": {},
1114 |      "output_type": "execute_result"
1115 |     }
1116 |    ],
1117 |    "source": [
1118 |     "most_similar(\"king\", topn=10)"
1119 |    ]
1120 |   },
1121 |   {
1122 |    "cell_type": "code",
1123 |    "execution_count": 87,
1124 |    "metadata": {},
1125 |    "outputs": [
1126 |     {
1127 |      "data": {
1128 |       "text/plain": [
1129 |        "[('cheetah', 0.9999999),\n",
1130 |        " ('lions', 0.7758893),\n",
1131 |        " ('tiger', 0.7359829),\n",
1132 |        " ('panther', 0.7359829),\n",
1133 |        " ('leopard', 0.7359829),\n",
1134 |        " ('elephant', 0.71239567),\n",
1135 |        " ('hippo', 0.71239567),\n",
1136 |        " ('zebra', 0.71239567),\n",
1137 |        " ('rhino', 0.71239567),\n",
1138 |        " ('giraffe', 0.71239567)]"
1139 |       ]
1140 |      },
1141 |      "execution_count": 87,
1142 |      "metadata": {},
1143 |      "output_type": "execute_result"
1144 |     }
1145 |    ],
1146 |    "source": [
1147 |     "most_similar(\"lion\", topn=10)"
1148 |    ]
1149 |   },
1150 |   {
1151 |    "cell_type": "markdown",
1152 |    "metadata": {},
1153 |    "source": [
1154 |     "Sentence (or document) objects  have vectors, derived from the averages of individual token vectors. This makes it possible to compare similarities between whole documents."
1155 |    ]
1156 |   },
1157 |   {
1158 |    "cell_type": "code",
1159 |    "execution_count": 88,
1160 |    "metadata": {},
1161 |    "outputs": [
1162 |     {
1163 |      "data": {
1164 |       "text/plain": [
1165 |        "300"
1166 |       ]
1167 |      },
1168 |      "execution_count": 88,
1169 |      "metadata": {},
1170 |      "output_type": "execute_result"
1171 |     }
1172 |    ],
1173 |    "source": [
1174 |     "doc = nlp('The quick brown fox jumped over the lazy dogs.')\n",
1175 |     "len(doc.vector)"
1176 |    ]
1177 |   },
1178 |   {
1179 |    "cell_type": "markdown",
1180 |    "metadata": {},
1181 |    "source": [
1182 |     "### Bert Sentence Transformer"
1183 |    ]
1184 |   },
1185 |   {
1186 |    "cell_type": "code",
1187 |    "execution_count": 89,
1188 |    "metadata": {},
1189 |    "outputs": [],
1190 |    "source": [
1191 |     "from sentence_transformers import SentenceTransformer\n",
1192 |     "import scipy.spatial\n",
1193 |     "embedder = SentenceTransformer('bert-base-nli-mean-tokens')"
1194 |    ]
1195 |   },
1196 |   {
1197 |    "cell_type": "code",
1198 |    "execution_count": 90,
1199 |    "metadata": {},
1200 |    "outputs": [
1201 |     {
1202 |      "name": "stdout",
1203 |      "output_type": "stream",
1204 |      "text": [
1205 |       "CPU times: user 1min 54s, sys: 3.71 s, total: 1min 58s\n",
1206 |       "Wall time: 34.9 s\n"
1207 |      ]
1208 |     }
1209 |    ],
1210 |    "source": [
1211 |     "%%time\n",
1212 |     "corpus_embeddings = embedder.encode(corpus)"
1213 |    ]
1214 |   },
1215 |   {
1216 |    "cell_type": "markdown",
1217 |    "metadata": {},
1218 |    "source": [
1219 |     "----\n",
1220 |     "## Candidate Genration using Faiss vector similarity search library"
1221 |    ]
1222 |   },
1223 |   {
1224 |    "cell_type": "markdown",
1225 |    "metadata": {},
1226 |    "source": [
1227 |     "Faiss is a library developed by Facebook AI Research. It is for effecient similarity search and clustering of dense vectors.\n",
1228 |     "\n",
1229 |     "**References:**\n",
1230 |     "\n",
1231 |     "1. [Tutorial](https://github.com/facebookresearch/faiss/wiki/Getting-started)\n",
1232 |     "2. [facebookresearch/faiss](https://github.com/facebookresearch/faiss)"
1233 |    ]
1234 |   },
1235 |   {
1236 |    "cell_type": "code",
1237 |    "execution_count": 91,
1238 |    "metadata": {},
1239 |    "outputs": [
1240 |     {
1241 |      "name": "stdout",
1242 |      "output_type": "stream",
1243 |      "text": [
1244 |       "True\n",
1245 |       "1362\n"
1246 |      ]
1247 |     }
1248 |    ],
1249 |    "source": [
1250 |     "import faiss\n",
1251 |     "d= 768\n",
1252 |     "index = faiss.IndexFlatL2(d)\n",
1253 |     "print(index.is_trained)\n",
1254 |     "index.add(np.stack(corpus_embeddings, axis=0))\n",
1255 |     "print(index.ntotal)"
1256 |    ]
1257 |   },
1258 |   {
1259 |    "cell_type": "code",
1260 |    "execution_count": 92,
1261 |    "metadata": {},
1262 |    "outputs": [],
1263 |    "source": [
1264 |     "# queries = ['What is the step by step guide to invest in share market in india?', 'How can Internet speed be increased by hacking through DNS?']\n",
1265 |     "queries = question_pairs['question1'][:3].tolist()\n",
1266 |     "query_embeddings = embedder.encode(queries)"
1267 |    ]
1268 |   },
1269 |   {
1270 |    "cell_type": "code",
1271 |    "execution_count": 93,
1272 |    "metadata": {},
1273 |    "outputs": [
1274 |     {
1275 |      "name": "stdout",
1276 |      "output_type": "stream",
1277 |      "text": [
1278 |       "[[ 858  998 1025  983   73]\n",
1279 |       " [ 771 1015  775 1014 1133]\n",
1280 |       " [ 436  455  457  463  462]]\n"
1281 |      ]
1282 |     }
1283 |    ],
1284 |    "source": [
1285 |     "k = 5                          # we want to see 4 nearest neighbors\n",
1286 |     "D, I = index.search(np.stack(query_embeddings, axis=0), k)     # actual search\n",
1287 |     "print(I)                   # neighbors of the 5 first queries"
1288 |    ]
1289 |   },
1290 |   {
1291 |    "cell_type": "code",
1292 |    "execution_count": 94,
1293 |    "metadata": {},
1294 |    "outputs": [
1295 |     {
1296 |      "name": "stdout",
1297 |      "output_type": "stream",
1298 |      "text": [
1299 |       "\n",
1300 |       "======================\n",
1301 |       "\n",
1302 |       "Query: What is purpose of life?\n",
1303 |       "\n",
1304 |       "Top 5 most similar sentences in corpus:\n",
1305 |       "What is purpose of life? (Distance: 0.0000)\n",
1306 |       "What is the purpose of life? (Distance: 7.9868)\n",
1307 |       "What is your purpose of life? (Distance: 12.3884)\n",
1308 |       "What is the meaning or purpose of life? (Distance: 13.6231)\n",
1309 |       "From your perspective, what is the purpose of life? (Distance: 17.5448)\n",
1310 |       "\n",
1311 |       "======================\n",
1312 |       "\n",
1313 |       "Query: What are your New Year's resolutions for 2017?\n",
1314 |       "\n",
1315 |       "Top 5 most similar sentences in corpus:\n",
1316 |       "What are your New Year's resolutions for 2017? (Distance: 0.0000)\n",
1317 |       "What is your New Year's resolutions for 2017? (Distance: 0.6093)\n",
1318 |       "What are your new year resolutions for 2017? (Distance: 5.8446)\n",
1319 |       "What is your New Year's resolution for 2017? (Distance: 6.6011)\n",
1320 |       "What's your New Year's resolution for 2017? (Distance: 8.1350)\n",
1321 |       "\n",
1322 |       "======================\n",
1323 |       "\n",
1324 |       "Query: How will Indian GDP be affected from banning 500 and 1000 rupees notes?\n",
1325 |       "\n",
1326 |       "Top 5 most similar sentences in corpus:\n",
1327 |       "How will Indian GDP be affected from banning 500 and 1000 rupees notes? (Distance: 0.0000)\n",
1328 |       "How will the ban of 1000 and 500 rupee notes affect the Indian economy? (Distance: 24.4171)\n",
1329 |       "How will the ban of Rs 500 and Rs 1000 notes affect Indian economy? (Distance: 26.4862)\n",
1330 |       "How will the ban on Rs 500 and 1000 notes impact the Indian economy? (Distance: 26.5522)\n",
1331 |       "How will the ban on 500₹ and 1000₹ notes impact the Indian economy? (Distance: 28.7938)\n"
1332 |      ]
1333 |     }
1334 |    ],
1335 |    "source": [
1336 |     "for query, query_embedding in zip(queries, query_embeddings):\n",
1337 |     "    distances, indices = index.search(np.asarray(query_embedding).reshape(1,768),k)\n",
1338 |     "    print(\"\\n======================\\n\")\n",
1339 |     "    print(\"Query:\", query)\n",
1340 |     "    print(\"\\nTop 5 most similar sentences in corpus:\")\n",
1341 |     "    for idx in range(0,5):\n",
1342 |     "        print(corpus[indices[0,idx]], \"(Distance: %.4f)\" % distances[0,idx])"
1343 |    ]
1344 |   },
1345 |   {
1346 |    "cell_type": "code",
1347 |    "execution_count": 122,
1348 |    "metadata": {},
1349 |    "outputs": [],
1350 |    "source": [
1351 |     "query = \"How will Indian GDP be affected from banning 500 and 1000 rupees notes?\"\n",
1352 |     "query_embed = embedder.encode(query)\n",
1353 |     "distances, indices = index.search(np.asarray(query_embedding).reshape(1,768),50)\n",
1354 |     "relevant_docs = [corpus[indices[0,idx]] for idx in range(50)]"
1355 |    ]
1356 |   },
1357 |   {
1358 |    "cell_type": "markdown",
1359 |    "metadata": {},
1360 |    "source": [
1361 |     "----\n",
1362 |     "## Reranking using Bidirectional LSTM model"
1363 |    ]
1364 |   },
1365 |   {
1366 |    "cell_type": "markdown",
1367 |    "metadata": {},
1368 |    "source": [
1369 |     "<img src=\"img/bi_lstm.png\" alt=\"Drawing\" style=\"width: 600px;\"/>\n",
1370 |     "\n",
1371 |     "**Reference:** https://mlwhiz.com/blog/2019/03/09/deeplearning_architectures_text_classification/"
1372 |    ]
1373 |   },
1374 |   {
1375 |    "cell_type": "code",
1376 |    "execution_count": 344,
1377 |    "metadata": {},
1378 |    "outputs": [],
1379 |    "source": [
1380 |     "import re\n",
1381 |     "import nltk\n",
1382 |     "from nltk.tokenize.toktok import ToktokTokenizer\n",
1383 |     "from nltk.stem import WordNetLemmatizer, SnowballStemmer\n",
1384 |     "toko_tokenizer = ToktokTokenizer()\n",
1385 |     "wordnet_lemmatizer = WordNetLemmatizer()\n",
1386 |     "\n",
1387 |     "def normalize_text(text):\n",
1388 |     "        puncts = ['/', ',', '.', '\"', ':', ')', '(', '-', '!', '?', '|', ';', '$', '&', '/', '[', ']', '>', '%', '=', '#', '*', '+', '\\\\', '•',  '~', '@', '£', \n",
1389 |     "         '·', '_', '{', '}', '©', '^', '®', '`',  '<', '→', '°', '€', '™', '›',  '♥', '←', '×', '§', '″', '′', 'Â', '█', '½', 'à', '…', \n",
1390 |     "         '“', '★', '”', '–', '●', 'â', '►', '−', '¢', '²', '¬', '░', '¶', '↑', '±', '¿', '▾', '═', '¦', '║', '―', '¥', '▓', '—', '‹', '─', \n",
1391 |     "         '▒', '：', '¼', '⊕', '▼', '▪', '†', '■', '’', '▀', '¨', '▄', '♫', '☆', 'é', '¯', '♦', '¤', '▲', 'è', '¸', '¾', 'Ã', '⋅', '‘', '∞', \n",
1392 |     "         '∙', '）', '↓', '、', '│', '（', '»', '，', '♪', '╩', '╚', '³', '・', '╦', '╣', '╔', '╗', '▬', '❤', 'ï', 'Ø', '¹', '≤', '‡', '√', ]\n",
1393 |     "\n",
1394 |     "        def clean_text(text):\n",
1395 |     "            text = str(text)\n",
1396 |     "            text = text.replace('\\n', '')\n",
1397 |     "            text = text.replace('\\r', '')\n",
1398 |     "            for punct in puncts:\n",
1399 |     "                if punct in text:\n",
1400 |     "                    text = text.replace(punct, '')\n",
1401 |     "            return text.lower()\n",
1402 |     "\n",
1403 |     "        def clean_numbers(text):\n",
1404 |     "            if bool(re.search(r'\\d', text)):\n",
1405 |     "                text = re.sub('[0-9]{5,}', '#####', text)\n",
1406 |     "                text = re.sub('[0-9]{4}', '####', text)\n",
1407 |     "                text = re.sub('[0-9]{3}', '###', text)\n",
1408 |     "                text = re.sub('[0-9]{2}', '##', text)\n",
1409 |     "            return text\n",
1410 |     "\n",
1411 |     "        contraction_dict = {\"ain't\": \"is not\", \"aren't\": \"are not\",\"can't\": \"cannot\", \"'cause\": \"because\", \"could've\": \"could have\", \"couldn't\": \"could not\", \"didn't\": \"did not\",  \"doesn't\": \"does not\", \"don't\": \"do not\", \"hadn't\": \"had not\", \"hasn't\": \"has not\", \"haven't\": \"have not\", \"he'd\": \"he would\",\"he'll\": \"he will\", \"he's\": \"he is\", \"how'd\": \"how did\", \"how'd'y\": \"how do you\", \"how'll\": \"how will\", \"how's\": \"how is\",  \"I'd\": \"I would\", \"I'd've\": \"I would have\", \"I'll\": \"I will\", \"I'll've\": \"I will have\",\"I'm\": \"I am\", \"I've\": \"I have\", \"i'd\": \"i would\", \"i'd've\": \"i would have\", \"i'll\": \"i will\",  \"i'll've\": \"i will have\",\"i'm\": \"i am\", \"i've\": \"i have\", \"isn't\": \"is not\", \"it'd\": \"it would\", \"it'd've\": \"it would have\", \"it'll\": \"it will\", \"it'll've\": \"it will have\",\"it's\": \"it is\", \"let's\": \"let us\", \"ma'am\": \"madam\", \"mayn't\": \"may not\", \"might've\": \"might have\",\"mightn't\": \"might not\",\"mightn't've\": \"might not have\", \"must've\": \"must have\", \"mustn't\": \"must not\", \"mustn't've\": \"must not have\", \"needn't\": \"need not\", \"needn't've\": \"need not have\",\"o'clock\": \"of the clock\", \"oughtn't\": \"ought not\", \"oughtn't've\": \"ought not have\", \"shan't\": \"shall not\", \"sha'n't\": \"shall not\", \"shan't've\": \"shall not have\", \"she'd\": \"she would\", \"she'd've\": \"she would have\", \"she'll\": \"she will\", \"she'll've\": \"she will have\", \"she's\": \"she is\", \"should've\": \"should have\", \"shouldn't\": \"should not\", \"shouldn't've\": \"should not have\", \"so've\": \"so have\",\"so's\": \"so as\", \"this's\": \"this is\",\"that'd\": \"that would\", \"that'd've\": \"that would have\", \"that's\": \"that is\", \"there'd\": \"there would\", \"there'd've\": \"there would have\", \"there's\": \"there is\", \"here's\": \"here is\",\"they'd\": \"they would\", \"they'd've\": \"they would have\", \"they'll\": \"they will\", \"they'll've\": \"they will have\", \"they're\": \"they are\", \"they've\": \"they have\", \"to've\": \"to have\", \"wasn't\": \"was not\", \"we'd\": \"we would\", \"we'd've\": \"we would have\", \"we'll\": \"we will\", \"we'll've\": \"we will have\", \"we're\": \"we are\", \"we've\": \"we have\", \"weren't\": \"were not\", \"what'll\": \"what will\", \"what'll've\": \"what will have\", \"what're\": \"what are\",  \"what's\": \"what is\", \"what've\": \"what have\", \"when's\": \"when is\", \"when've\": \"when have\", \"where'd\": \"where did\", \"where's\": \"where is\", \"where've\": \"where have\", \"who'll\": \"who will\", \"who'll've\": \"who will have\", \"who's\": \"who is\", \"who've\": \"who have\", \"why's\": \"why is\", \"why've\": \"why have\", \"will've\": \"will have\", \"won't\": \"will not\", \"won't've\": \"will not have\", \"would've\": \"would have\", \"wouldn't\": \"would not\", \"wouldn't've\": \"would not have\", \"y'all\": \"you all\", \"y'all'd\": \"you all would\",\"y'all'd've\": \"you all would have\",\"y'all're\": \"you all are\",\"y'all've\": \"you all have\",\"you'd\": \"you would\", \"you'd've\": \"you would have\", \"you'll\": \"you will\", \"you'll've\": \"you will have\", \"you're\": \"you are\", \"you've\": \"you have\"}\n",
1412 |     "\n",
1413 |     "        def _get_contractions(contraction_dict):\n",
1414 |     "            contraction_re = re.compile('(%s)' % '|'.join(contraction_dict.keys()))\n",
1415 |     "            return contraction_dict, contraction_re\n",
1416 |     "\n",
1417 |     "        contractions, contractions_re = _get_contractions(contraction_dict)\n",
1418 |     "\n",
1419 |     "        def replace_contractions(text):\n",
1420 |     "            def replace(match):\n",
1421 |     "                return contractions[match.group(0)]\n",
1422 |     "            return contractions_re.sub(replace, text)\n",
1423 |     "\n",
1424 |     "        stopword_list = nltk.corpus.stopwords.words('english')\n",
1425 |     "\n",
1426 |     "        def remove_stopwords(text, is_lower_case=True):\n",
1427 |     "            tokens = toko_tokenizer.tokenize(text)\n",
1428 |     "            tokens = [token.strip() for token in tokens]\n",
1429 |     "            if is_lower_case:\n",
1430 |     "                filtered_tokens = [token for token in tokens if token not in stopword_list]\n",
1431 |     "            else:\n",
1432 |     "                filtered_tokens = [token for token in tokens if token.lower() not in stopword_list]\n",
1433 |     "            filtered_text = ' '.join(filtered_tokens)    \n",
1434 |     "            return filtered_text\n",
1435 |     "\n",
1436 |     "        def lemmatizer(text):\n",
1437 |     "            tokens = toko_tokenizer.tokenize(text)\n",
1438 |     "            tokens = [token.strip() for token in tokens]\n",
1439 |     "            tokens = [wordnet_lemmatizer.lemmatize(token) for token in tokens]\n",
1440 |     "            return ' '.join(tokens)\n",
1441 |     "\n",
1442 |     "        def trim_text(text):\n",
1443 |     "            tokens = toko_tokenizer.tokenize(text)\n",
1444 |     "            tokens = [token.strip() for token in tokens]\n",
1445 |     "            return ' '.join(tokens)\n",
1446 |     "        \n",
1447 |     "        def remove_non_english(text):\n",
1448 |     "            tokens = toko_tokenizer.tokenize(text)\n",
1449 |     "            tokens = [token.strip() for token in tokens]\n",
1450 |     "            tokens = [token for token in tokens if d.check(token)]\n",
1451 |     "            eng_text = ' '.join(tokens)\n",
1452 |     "            return eng_text\n",
1453 |     "\n",
1454 |     "        text_norm = clean_text(text)\n",
1455 |     "        text_norm = clean_numbers(text_norm)\n",
1456 |     "        text_norm = replace_contractions(text_norm)\n",
1457 |     "#         text_norm = remove_stopwords(text_norm)\n",
1458 |     "#         text_norm = remove_non_english(text_norm)\n",
1459 |     "        text_norm = lemmatizer(text_norm)\n",
1460 |     "        text_norm = trim_text(text_norm)\n",
1461 |     "        return text_norm"
1462 |    ]
1463 |   },
1464 |   {
1465 |    "cell_type": "code",
1466 |    "execution_count": 328,
1467 |    "metadata": {},
1468 |    "outputs": [
1469 |     {
1470 |      "data": {
1471 |       "text/html": [
1472 |        "<div>\n",
1473 |        "<style scoped>\n",
1474 |        "    .dataframe tbody tr th:only-of-type {\n",
1475 |        "        vertical-align: middle;\n",
1476 |        "    }\n",
1477 |        "\n",
1478 |        "    .dataframe tbody tr th {\n",
1479 |        "        vertical-align: top;\n",
1480 |        "    }\n",
1481 |        "\n",
1482 |        "    .dataframe thead th {\n",
1483 |        "        text-align: right;\n",
1484 |        "    }\n",
1485 |        "</style>\n",
1486 |        "<table border=\"1\" class=\"dataframe\">\n",
1487 |        "  <thead>\n",
1488 |        "    <tr style=\"text-align: right;\">\n",
1489 |        "      <th></th>\n",
1490 |        "      <th>id</th>\n",
1491 |        "      <th>qid1</th>\n",
1492 |        "      <th>qid2</th>\n",
1493 |        "      <th>question1</th>\n",
1494 |        "      <th>question2</th>\n",
1495 |        "      <th>is_duplicate</th>\n",
1496 |        "    </tr>\n",
1497 |        "  </thead>\n",
1498 |        "  <tbody>\n",
1499 |        "    <tr>\n",
1500 |        "      <th>0</th>\n",
1501 |        "      <td>0</td>\n",
1502 |        "      <td>1</td>\n",
1503 |        "      <td>2</td>\n",
1504 |        "      <td>What is the step by step guide to invest in sh...</td>\n",
1505 |        "      <td>What is the step by step guide to invest in sh...</td>\n",
1506 |        "      <td>0</td>\n",
1507 |        "    </tr>\n",
1508 |        "    <tr>\n",
1509 |        "      <th>1</th>\n",
1510 |        "      <td>1</td>\n",
1511 |        "      <td>3</td>\n",
1512 |        "      <td>4</td>\n",
1513 |        "      <td>What is the story of Kohinoor (Koh-i-Noor) Dia...</td>\n",
1514 |        "      <td>What would happen if the Indian government sto...</td>\n",
1515 |        "      <td>0</td>\n",
1516 |        "    </tr>\n",
1517 |        "    <tr>\n",
1518 |        "      <th>2</th>\n",
1519 |        "      <td>2</td>\n",
1520 |        "      <td>5</td>\n",
1521 |        "      <td>6</td>\n",
1522 |        "      <td>How can I increase the speed of my internet co...</td>\n",
1523 |        "      <td>How can Internet speed be increased by hacking...</td>\n",
1524 |        "      <td>0</td>\n",
1525 |        "    </tr>\n",
1526 |        "    <tr>\n",
1527 |        "      <th>3</th>\n",
1528 |        "      <td>3</td>\n",
1529 |        "      <td>7</td>\n",
1530 |        "      <td>8</td>\n",
1531 |        "      <td>Why am I mentally very lonely? How can I solve...</td>\n",
1532 |        "      <td>Find the remainder when [math]23^{24}[/math] i...</td>\n",
1533 |        "      <td>0</td>\n",
1534 |        "    </tr>\n",
1535 |        "    <tr>\n",
1536 |        "      <th>4</th>\n",
1537 |        "      <td>4</td>\n",
1538 |        "      <td>9</td>\n",
1539 |        "      <td>10</td>\n",
1540 |        "      <td>Which one dissolve in water quikly sugar, salt...</td>\n",
1541 |        "      <td>Which fish would survive in salt water?</td>\n",
1542 |        "      <td>0</td>\n",
1543 |        "    </tr>\n",
1544 |        "  </tbody>\n",
1545 |        "</table>\n",
1546 |        "</div>"
1547 |       ],
1548 |       "text/plain": [
1549 |        "   id  qid1  qid2                                          question1  \\\n",
1550 |        "0   0     1     2  What is the step by step guide to invest in sh...   \n",
1551 |        "1   1     3     4  What is the story of Kohinoor (Koh-i-Noor) Dia...   \n",
1552 |        "2   2     5     6  How can I increase the speed of my internet co...   \n",
1553 |        "3   3     7     8  Why am I mentally very lonely? How can I solve...   \n",
1554 |        "4   4     9    10  Which one dissolve in water quikly sugar, salt...   \n",
1555 |        "\n",
1556 |        "                                           question2  is_duplicate  \n",
1557 |        "0  What is the step by step guide to invest in sh...             0  \n",
1558 |        "1  What would happen if the Indian government sto...             0  \n",
1559 |        "2  How can Internet speed be increased by hacking...             0  \n",
1560 |        "3  Find the remainder when [math]23^{24}[/math] i...             0  \n",
1561 |        "4            Which fish would survive in salt water?             0  "
1562 |       ]
1563 |      },
1564 |      "execution_count": 328,
1565 |      "metadata": {},
1566 |      "output_type": "execute_result"
1567 |     }
1568 |    ],
1569 |    "source": [
1570 |     "question_pairs.head()"
1571 |    ]
1572 |   },
1573 |   {
1574 |    "cell_type": "code",
1575 |    "execution_count": 329,
1576 |    "metadata": {},
1577 |    "outputs": [
1578 |     {
1579 |      "data": {
1580 |       "text/plain": [
1581 |        "(404351, 6)"
1582 |       ]
1583 |      },
1584 |      "execution_count": 329,
1585 |      "metadata": {},
1586 |      "output_type": "execute_result"
1587 |     }
1588 |    ],
1589 |    "source": [
1590 |     "question_pairs.shape"
1591 |    ]
1592 |   },
1593 |   {
1594 |    "cell_type": "code",
1595 |    "execution_count": 293,
1596 |    "metadata": {},
1597 |    "outputs": [],
1598 |    "source": [
1599 |     "embedding_path = \"./../../Embeddings/glove.twitter.27B/glove.twitter.27B.200d.txt\"\n",
1600 |     "def get_word2vec(file_path):\n",
1601 |     "    file = open(embedding_path, \"r\")\n",
1602 |     "    if (file):\n",
1603 |     "        word2vec = dict()\n",
1604 |     "        split = file.read().splitlines()\n",
1605 |     "        for line in split:\n",
1606 |     "            key = line.split(' ',1)[0]\n",
1607 |     "            value = np.array([float(val) for val in line.split(' ')[1:]])\n",
1608 |     "            word2vec[key] = value\n",
1609 |     "        return (word2vec)\n",
1610 |     "    else:\n",
1611 |     "        print(\"invalid fiel path\")\n",
1612 |     "w2v = get_word2vec(embedding_path)"
1613 |    ]
1614 |   },
1615 |   {
1616 |    "cell_type": "code",
1617 |    "execution_count": 352,
1618 |    "metadata": {},
1619 |    "outputs": [],
1620 |    "source": [
1621 |     "total_text = pd.concat([question_pairs['question1'], question_pairs['question2']]).reset_index(drop=True)\n",
1622 |     "total_text = total_text.apply(lambda x: str(x))\n",
1623 |     "total_text = total_text.apply(lambda x: normalize_text(x))\n",
1624 |     "max_features = 6000\n",
1625 |     "tokenizer = Tokenizer(num_words=max_features)\n",
1626 |     "tokenizer.fit_on_texts(total_text)\n",
1627 |     "question_1_sequenced = tokenizer.texts_to_sequences(question_pairs['question1'].apply(lambda x: normalize_text(x)))\n",
1628 |     "question_2_sequenced = tokenizer.texts_to_sequences(question_pairs['question2'].apply(lambda x: normalize_text(x)))\n",
1629 |     "vocab_size = len(tokenizer.word_index) + 1"
1630 |    ]
1631 |   },
1632 |   {
1633 |    "cell_type": "code",
1634 |    "execution_count": 353,
1635 |    "metadata": {},
1636 |    "outputs": [
1637 |     {
1638 |      "data": {
1639 |       "text/plain": [
1640 |        "92423"
1641 |       ]
1642 |      },
1643 |      "execution_count": 353,
1644 |      "metadata": {},
1645 |      "output_type": "execute_result"
1646 |     }
1647 |    ],
1648 |    "source": [
1649 |     "vocab_size"
1650 |    ]
1651 |   },
1652 |   {
1653 |    "cell_type": "code",
1654 |    "execution_count": 354,
1655 |    "metadata": {},
1656 |    "outputs": [],
1657 |    "source": [
1658 |     "maxlen = 100\n",
1659 |     "question_1_padded = pad_sequences(question_1_sequenced, maxlen=maxlen)\n",
1660 |     "question_2_padded = pad_sequences(question_2_sequenced, maxlen=maxlen)"
1661 |    ]
1662 |   },
1663 |   {
1664 |    "cell_type": "code",
1665 |    "execution_count": 359,
1666 |    "metadata": {},
1667 |    "outputs": [],
1668 |    "source": [
1669 |     "y = question_pairs['is_duplicate']"
1670 |    ]
1671 |   },
1672 |   {
1673 |    "cell_type": "code",
1674 |    "execution_count": 381,
1675 |    "metadata": {},
1676 |    "outputs": [],
1677 |    "source": [
1678 |     "from tqdm import tqdm"
1679 |    ]
1680 |   },
1681 |   {
1682 |    "cell_type": "code",
1683 |    "execution_count": 384,
1684 |    "metadata": {},
1685 |    "outputs": [],
1686 |    "source": [
1687 |     "from numpy import zeros\n",
1688 |     "embedding_matrix = zeros((vocab_size, 768))\n",
1689 |     "for word, i in tokenizer.word_index.items():\n",
1690 |     "    embedding_vector = w2v.get(word)\n",
1691 |     "    if embedding_vector is not None:\n",
1692 |     "        embedding_matrix[i] = embedding_vector[0]"
1693 |    ]
1694 |   },
1695 |   {
1696 |    "cell_type": "code",
1697 |    "execution_count": 357,
1698 |    "metadata": {},
1699 |    "outputs": [],
1700 |    "source": [
1701 |     "embedding_size = 128\n",
1702 |     "max_len = 100\n",
1703 |     "\n",
1704 |     "inp1 = Input(shape=(100,))\n",
1705 |     "inp2 = Input(shape=(100,))\n",
1706 |     "\n",
1707 |     "x1 = Embedding(vocab_size, 200, weights=[embedding_matrix], input_length=max_len)(inp1)\n",
1708 |     "x2 = Embedding(vocab_size, 200, weights=[embedding_matrix], input_length=max_len)(inp2)\n",
1709 |     "\n",
1710 |     "x3 = Bidirectional(LSTM(32, return_sequences = True))(x1)\n",
1711 |     "x4 = Bidirectional(LSTM(32, return_sequences = True))(x2)\n",
1712 |     "\n",
1713 |     "x5 = GlobalMaxPool1D()(x3)\n",
1714 |     "x6 = GlobalMaxPool1D()(x4)\n",
1715 |     "\n",
1716 |     "x7 =  dot([x5, x6], axes=1)\n",
1717 |     "\n",
1718 |     "x8 = Dense(40, activation='relu')(x7)\n",
1719 |     "x9 = Dropout(0.05)(x8)\n",
1720 |     "x10 = Dense(10, activation='relu')(x9)\n",
1721 |     "output = Dense(1, activation=\"sigmoid\")(x10)\n",
1722 |     "\n",
1723 |     "model = Model(inputs=[inp1, inp2], outputs=output)\n",
1724 |     "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
1725 |     "batch_size = 256\n",
1726 |     "epochs = 4"
1727 |    ]
1728 |   },
1729 |   {
1730 |    "cell_type": "code",
1731 |    "execution_count": 360,
1732 |    "metadata": {},
1733 |    "outputs": [
1734 |     {
1735 |      "name": "stdout",
1736 |      "output_type": "stream",
1737 |      "text": [
1738 |       "Train on 323480 samples, validate on 80871 samples\n",
1739 |       "Epoch 1/4\n",
1740 |       "323480/323480 [==============================] - 1096s 3ms/step - loss: 0.5212 - acc: 0.7372 - val_loss: 0.4596 - val_acc: 0.7802\n",
1741 |       "Epoch 2/4\n",
1742 |       "323480/323480 [==============================] - 1101s 3ms/step - loss: 0.4169 - acc: 0.8038 - val_loss: 0.4300 - val_acc: 0.7978\n",
1743 |       "Epoch 3/4\n",
1744 |       "323480/323480 [==============================] - 1135s 4ms/step - loss: 0.3536 - acc: 0.8400 - val_loss: 0.4340 - val_acc: 0.7976\n",
1745 |       "Epoch 4/4\n",
1746 |       "323480/323480 [==============================] - 1077s 3ms/step - loss: 0.2963 - acc: 0.8701 - val_loss: 0.4411 - val_acc: 0.8035\n"
1747 |      ]
1748 |     },
1749 |     {
1750 |      "data": {
1751 |       "text/plain": [
1752 |        "<keras.callbacks.History at 0x1bd352da0>"
1753 |       ]
1754 |      },
1755 |      "execution_count": 360,
1756 |      "metadata": {},
1757 |      "output_type": "execute_result"
1758 |     }
1759 |    ],
1760 |    "source": [
1761 |     "model.fit([question_1_padded, question_2_padded], y, batch_size=batch_size, epochs=epochs, validation_split=0.2, )"
1762 |    ]
1763 |   },
1764 |   {
1765 |    "cell_type": "markdown",
1766 |    "metadata": {},
1767 |    "source": [
1768 |     "--------\n",
1769 |     "## Combining candidate generation and reranking"
1770 |    ]
1771 |   },
1772 |   {
1773 |    "cell_type": "code",
1774 |    "execution_count": 361,
1775 |    "metadata": {},
1776 |    "outputs": [
1777 |     {
1778 |      "data": {
1779 |       "text/plain": [
1780 |        "'How will Indian GDP be affected from banning 500 and 1000 rupees notes?'"
1781 |       ]
1782 |      },
1783 |      "execution_count": 361,
1784 |      "metadata": {},
1785 |      "output_type": "execute_result"
1786 |     }
1787 |    ],
1788 |    "source": [
1789 |     "query"
1790 |    ]
1791 |   },
1792 |   {
1793 |    "cell_type": "code",
1794 |    "execution_count": 363,
1795 |    "metadata": {},
1796 |    "outputs": [],
1797 |    "source": [
1798 |     "query_copy = [query]*len(relevant_docs)\n",
1799 |     "question_1_sequenced_final = tokenizer.texts_to_sequences(query_copy)\n",
1800 |     "question_2_sequenced_final = tokenizer.texts_to_sequences(relevant_docs)"
1801 |    ]
1802 |   },
1803 |   {
1804 |    "cell_type": "code",
1805 |    "execution_count": 364,
1806 |    "metadata": {},
1807 |    "outputs": [],
1808 |    "source": [
1809 |     "maxlen = 100\n",
1810 |     "question_1_padded_final = pad_sequences(question_1_sequenced_final, maxlen=maxlen)\n",
1811 |     "question_2_padded_final = pad_sequences(question_2_sequenced_final, maxlen=maxlen)"
1812 |    ]
1813 |   },
1814 |   {
1815 |    "cell_type": "code",
1816 |    "execution_count": 365,
1817 |    "metadata": {},
1818 |    "outputs": [],
1819 |    "source": [
1820 |     "preds_test = model.predict([question_1_padded_final, question_2_padded_final])\n",
1821 |     "preds_test = np.array([x[0] for x in preds_test])"
1822 |    ]
1823 |   },
1824 |   {
1825 |    "cell_type": "code",
1826 |    "execution_count": 390,
1827 |    "metadata": {},
1828 |    "outputs": [
1829 |     {
1830 |      "data": {
1831 |       "text/plain": [
1832 |        "['What do you think about banning 500 and 1000 rupee notes in India?',\n",
1833 |        " 'What will be the implications of banning 500 and 1000 rupees currency notes on Indian economy?',\n",
1834 |        " 'What will be the consequences of 500 and 1000 rupee notes banning?',\n",
1835 |        " 'What will be the effects after banning on 500 and 1000 rupee notes?',\n",
1836 |        " 'What will be the impact on real estate by banning 500 and 1000 rupee notes from India?',\n",
1837 |        " 'How is banning 500 and 1000 INR going to help Indian economy?',\n",
1838 |        " 'What are your views on India banning 500 and 1000 notes? In what way it will affect Indian economy?',\n",
1839 |        " 'How is discontinuing 500 and 1000 rupee note going to put a hold on black money in India?',\n",
1840 |        " 'What will be the result of banning 500 and 1000 rupees note in India?',\n",
1841 |        " 'What are the economic implications of banning 500 and 1000 rupee notes?']"
1842 |       ]
1843 |      },
1844 |      "execution_count": 390,
1845 |      "metadata": {},
1846 |      "output_type": "execute_result"
1847 |     }
1848 |    ],
1849 |    "source": [
1850 |     "[relevant_docs[x] for x in preds_test.argsort()[::-1]][:10]"
1851 |    ]
1852 |   },
1853 |   {
1854 |    "cell_type": "code",
1855 |    "execution_count": null,
1856 |    "metadata": {},
1857 |    "outputs": [],
1858 |    "source": []
1859 |   }
1860 |  ],
1861 |  "metadata": {
1862 |   "kernelspec": {
1863 |    "display_name": "Python 3",
1864 |    "language": "python",
1865 |    "name": "python3"
1866 |   },
1867 |   "language_info": {
1868 |    "codemirror_mode": {
1869 |     "name": "ipython",
1870 |     "version": 3
1871 |    },
1872 |    "file_extension": ".py",
1873 |    "mimetype": "text/x-python",
1874 |    "name": "python",
1875 |    "nbconvert_exporter": "python",
1876 |    "pygments_lexer": "ipython3",
1877 |    "version": "3.6.10"
1878 |   }
1879 |  },
1880 |  "nbformat": 4,
1881 |  "nbformat_minor": 4
1882 | }
1883 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Deep-Learning-for-Semantic-Text-Matching
2 | 
3 | [Deep Learning for Semantic Text Matching](https://medium.com/swlh/deep-learning-for-semantic-text-matching-d4df6c2cf4c5)
4 | 


--------------------------------------------------------------------------------