├── README.md ├── article_5.ipynb ├── images └── article_5 │ ├── ai.gif │ ├── benchmark.png │ ├── cnn_model_multi_channel.png │ ├── cnn_text.png │ ├── cover.jpg │ ├── cover.xcf │ ├── cover_resized.png │ ├── deep.jpg │ ├── heatmap.png │ ├── investigate.jpg │ ├── ngrams.png │ ├── ngrams_char.jpg │ ├── rnn_cnn_model.png │ ├── rnn_simple_model.png │ ├── rnn_unrolled.png │ └── tfidf.jpg ├── models ├── lr_char_ngram.pkl ├── lr_word_char_ngram.pkl └── lr_word_ngram.pkl └── predictions ├── lr_char_ngram.csv ├── lr_word_char_ngram.csv ├── lr_word_ngram.csv ├── y_pred_cnn_multi_channel.csv ├── y_pred_rnn_cnn.csv ├── y_pred_rnn_simple.csv ├── y_pred_rnn_with_glove_embeddings.csv └── y_true.csv /README.md: -------------------------------------------------------------------------------- 1 | # Overview and benchmark of traditional and deep learning models in text classification 2 | 3 | Original post: https://ahmedbesbes.com/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification.html 4 | 5 | This article is an extension of a previous one I wrote when I was experimenting sentiment analysis on twitter data. Back in the time, I explored a simple model: a two-layer feed-forward neural network trained on keras. The input tweets were represented as document vectors resulting from a weighted average of the embeddings of the words composing the tweet. 6 | 7 | The embedding I used was a word2vec model I trained from scratch on the corpus using gensim. The task was a binary classification and I was able with this setting to achieve 79% accuracy. 8 | 9 | The goal of this post is to explore other NLP models trained on the same dataset and then benchmark their respective performance on a given test set. 10 | 11 | We'll go through different models: from simple ones relying on a bag-of-word representation to a heavy machinery deploying convolutional/recurrent networks: We'll see if we'll score more than 79% accuracy! 12 | 13 | ---------- 14 | 15 | Here are the models that have been tested: 16 | 17 | - Logistic regression with word ngrams 18 | - Logistic regression with character ngrams 19 | - Logistic regression with word and character ngrams 20 | - Recurrent neural network (bidirectional GRU) without pre-trained embeddings 21 | - Recurrent neural network (bidirectional GRU) with GloVe pre-trained embeddings 22 | - Multi channel Convolutional Neural Network 23 | - RNN (Bidirectional GRU) + CNN model 24 | 25 | By the end of this post, you will have a boilerplate code for each of these NLP techniques. It'll help you kickstart your NLP project and eventually achieve state-of-the art results (some of these models are really powerful). 26 | 27 | Here's a sneak peak of the final result: 28 | 29 | ![benchmark](images/article_5/benchmark.png) 30 | -------------------------------------------------------------------------------- /article_5.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "This article is an extension of a previous one I wrote when I was experimenting sentiment analysis on twitter data. Back in the time, I explored a simple model: a two-layer feed-forward neural network trained on **keras**. The input tweets were represented as document vectors resulting from a weighted average of the embeddings of the words composing the tweet.\n", 8 | "\n", 9 | "The embedding I used was a word2vec model I trained from scratch on the corpus using **gensim**. The task was a binary classification and I was able with this setting to achieve 79% accuracy.\n", 10 | "\n", 11 | "The goal of this post is to explore other NLP models trained on the same dataset and then benchmark their respective performance on a given test set.\n", 12 | "\n", 13 | "We'll go through different models: from simple ones relying on a bag-of-word representation to a heavy machinery deploying convolutional/recurrent networks: We'll see if we'll score more than 79% accuracy! " 14 | ] 15 | }, 16 | { 17 | "cell_type": "markdown", 18 | "metadata": {}, 19 | "source": [ 20 | "" 21 | ] 22 | }, 23 | { 24 | "cell_type": "markdown", 25 | "metadata": {}, 26 | "source": [ 27 | "I will start from simple models and add up complexity progressively. The goal is also to show that ** simple models work well too**.\n", 28 | "\n", 29 | "So I'm going to try out these:\n", 30 | "\n", 31 | "- Logistic regression with word ngrams\n", 32 | "- Logistic regression with character ngrams\n", 33 | "- Logistic regression with word and character ngrams\n", 34 | "- Recurrent neural network (bidirectional GRU) without pre-trained embeddings\n", 35 | "- Recurrent neural network (bidirectional GRU) with GloVe pre-trained embeddings\n", 36 | "- Multi channel Convolutional Neural Network \n", 37 | "- RNN (Bidirectional GRU) + CNN model \n", 38 | "\n", 39 | "By the end of this post, you will have a boilerplate code for each of these NLP techniques. It'll help you kickstart your NLP project and eventually achieve state-of-the art results (some of these models are really powerful).\n", 40 | "\n", 41 | "We'll also provide a comprehensive benchmark from which we'll tell which model is best suited for predicting the sentiment of a tweet.\n", 42 | "\n", 43 | "In the related git repo, I will publish the different models, their predictions, as well as the test set. You can try them yourself and get confident about the results\n", 44 | "\n", 45 | "Let's get started!" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 32, 51 | "metadata": {}, 52 | "outputs": [ 53 | { 54 | "data": { 55 | "text/html": [ 56 | "" 59 | ], 60 | "text/plain": [ 61 | "" 62 | ] 63 | }, 64 | "metadata": {}, 65 | "output_type": "display_data" 66 | } 67 | ], 68 | "source": [ 69 | "import os\n", 70 | "import re\n", 71 | "\n", 72 | "import warnings\n", 73 | "warnings.simplefilter(\"ignore\", UserWarning)\n", 74 | "from matplotlib import pyplot as plt\n", 75 | "%matplotlib inline\n", 76 | "\n", 77 | "\n", 78 | "import pandas as pd\n", 79 | "pd.options.mode.chained_assignment = None\n", 80 | "import numpy as np \n", 81 | "from string import punctuation\n", 82 | "\n", 83 | "from nltk.tokenize import word_tokenize\n", 84 | "\n", 85 | "from sklearn.model_selection import train_test_split\n", 86 | "from sklearn.feature_extraction.text import TfidfVectorizer\n", 87 | "from sklearn.linear_model import LogisticRegression\n", 88 | "from sklearn.metrics import accuracy_score, auc, roc_auc_score\n", 89 | "from sklearn.externals import joblib\n", 90 | "\n", 91 | "import scipy\n", 92 | "from scipy.sparse import hstack" 93 | ] 94 | }, 95 | { 96 | "cell_type": "markdown", 97 | "metadata": {}, 98 | "source": [ 99 | "# 0 - Data pre-processing" 100 | ] 101 | }, 102 | { 103 | "cell_type": "markdown", 104 | "metadata": {}, 105 | "source": [ 106 | "The dataset can be downloaded from this link.\n", 107 | "\n", 108 | "We'll load it and restrict ourselves to the variables we need (Sentiment and SentimentText).\n", 109 | "\n", 110 | "It contains 1578614 classified tweets, each row is marked as 1 for positive sentiment and 0 for negative sentiment.\n", 111 | "\n", 112 | "The author recommend using 1/10 for testing the algorithm and the rest for training." 113 | ] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "execution_count": 3, 118 | "metadata": {}, 119 | "outputs": [ 120 | { 121 | "name": "stdout", 122 | "output_type": "stream", 123 | "text": [ 124 | "(1578614, 2)\n" 125 | ] 126 | } 127 | ], 128 | "source": [ 129 | "data = pd.read_csv('./data/tweets.csv', encoding='latin1', usecols=['Sentiment', 'SentimentText'])\n", 130 | "data.columns = ['sentiment', 'text']\n", 131 | "data = data.sample(frac=1, random_state=42)\n", 132 | "print(data.shape)" 133 | ] 134 | }, 135 | { 136 | "cell_type": "code", 137 | "execution_count": 4, 138 | "metadata": {}, 139 | "outputs": [ 140 | { 141 | "name": "stdout", 142 | "output_type": "stream", 143 | "text": [ 144 | "1 http://www.popsugar.com/2999655 keep voting for robert pattinson in the popsugar100 as well!! \n", 145 | "1 @GamrothTaylor I am starting to worry about you, only I have Navy Seal type sleep hours. \n", 146 | "0 sunburned...no sunbaked! ow. it hurts to sit.\n", 147 | "1 Celebrating my 50th birthday by doing exactly the same as I do every other day - working on our websites. It's just another day. \n", 148 | "1 Leah and Aiden Gosselin are the cutest kids on the face of the Earth \n", 149 | "1 @MissHell23 Oh. I didn't even notice. \n", 150 | "0 WTF is wrong with me?!!! I'm completely miserable. I need to snap out of this \n", 151 | "0 Was having the best time in the gym until I got to the car and had messages waiting for me... back to the down stage! \n", 152 | "1 @JENTSYY oh what happened?? \n", 153 | "0 @catawu Ghod forbid he should feel responsible for anything! \n" 154 | ] 155 | } 156 | ], 157 | "source": [ 158 | "for row in data.head(10).iterrows():\n", 159 | " print(row[1]['sentiment'], row[1]['text']) " 160 | ] 161 | }, 162 | { 163 | "cell_type": "markdown", 164 | "metadata": {}, 165 | "source": [ 166 | "Tweets are noisy, let's clean them by removing urls, hashtags and user mentions." 167 | ] 168 | }, 169 | { 170 | "cell_type": "code", 171 | "execution_count": 5, 172 | "metadata": {}, 173 | "outputs": [], 174 | "source": [ 175 | "def tokenize(tweet):\n", 176 | " tweet = re.sub(r'http\\S+', '', tweet)\n", 177 | " tweet = re.sub(r\"#(\\w+)\", '', tweet)\n", 178 | " tweet = re.sub(r\"@(\\w+)\", '', tweet)\n", 179 | " tweet = re.sub(r'[^\\w\\s]', '', tweet)\n", 180 | " tweet = tweet.strip().lower()\n", 181 | " tokens = word_tokenize(tweet)\n", 182 | " return tokens" 183 | ] 184 | }, 185 | { 186 | "cell_type": "markdown", 187 | "metadata": {}, 188 | "source": [ 189 | "Once the data is clean, we save it on disk." 190 | ] 191 | }, 192 | { 193 | "cell_type": "code", 194 | "execution_count": 6, 195 | "metadata": {}, 196 | "outputs": [ 197 | { 198 | "name": "stdout", 199 | "output_type": "stream", 200 | "text": [ 201 | "(1575026, 2)\n" 202 | ] 203 | } 204 | ], 205 | "source": [ 206 | "data['tokens'] = data.text.progress_map(tokenize)\n", 207 | "data['cleaned_text'] = data['tokens'].map(lambda tokens: ' '.join(tokens))\n", 208 | "data[['sentiment', 'cleaned_text']].to_csv('./data/cleaned_text.csv')\n", 209 | "\n", 210 | "data = pd.read_csv('./data/cleaned_text.csv')\n", 211 | "print(data.shape)" 212 | ] 213 | }, 214 | { 215 | "cell_type": "code", 216 | "execution_count": 7, 217 | "metadata": {}, 218 | "outputs": [ 219 | { 220 | "data": { 221 | "text/html": [ 222 | "
\n", 223 | "\n", 236 | "\n", 237 | " \n", 238 | " \n", 239 | " \n", 240 | " \n", 241 | " \n", 242 | " \n", 243 | " \n", 244 | " \n", 245 | " \n", 246 | " \n", 247 | " \n", 248 | " \n", 249 | " \n", 250 | " \n", 251 | " \n", 252 | " \n", 253 | " \n", 254 | " \n", 255 | " \n", 256 | " \n", 257 | " \n", 258 | " \n", 259 | " \n", 260 | " \n", 261 | " \n", 262 | " \n", 263 | " \n", 264 | " \n", 265 | " \n", 266 | " \n", 267 | " \n", 268 | " \n", 269 | " \n", 270 | " \n", 271 | "
sentimentcleaned_text
00playing with my routers looks like i might hav...
11sleeeep agh im so tired and they wrote gay on ...
20alan ignored me during the concert boo
31really want some mini eggs why are they only a...
40thanks guys sorry i had to miss your show at m...
\n", 272 | "
" 273 | ], 274 | "text/plain": [ 275 | " sentiment cleaned_text\n", 276 | "0 0 playing with my routers looks like i might hav...\n", 277 | "1 1 sleeeep agh im so tired and they wrote gay on ...\n", 278 | "2 0 alan ignored me during the concert boo\n", 279 | "3 1 really want some mini eggs why are they only a...\n", 280 | "4 0 thanks guys sorry i had to miss your show at m..." 281 | ] 282 | }, 283 | "execution_count": 7, 284 | "metadata": {}, 285 | "output_type": "execute_result" 286 | } 287 | ], 288 | "source": [ 289 | "data.head()" 290 | ] 291 | }, 292 | { 293 | "cell_type": "markdown", 294 | "metadata": {}, 295 | "source": [ 296 | "Now that the dataset is cleaned, let's prepare a train/test split to build our models. \n", 297 | "\n", 298 | "We'll use this split throughout all the notebook." 299 | ] 300 | }, 301 | { 302 | "cell_type": "code", 303 | "execution_count": 8, 304 | "metadata": {}, 305 | "outputs": [ 306 | { 307 | "name": "stdout", 308 | "output_type": "stream", 309 | "text": [ 310 | "(1417523,) (157503,) (1417523,) (157503,)\n" 311 | ] 312 | } 313 | ], 314 | "source": [ 315 | "x_train, x_test, y_train, y_test = train_test_split(data['cleaned_text'], \n", 316 | " data['sentiment'], \n", 317 | " test_size=0.1, \n", 318 | " random_state=42,\n", 319 | " stratify=data['sentiment'])\n", 320 | "\n", 321 | "print(x_train.shape, x_test.shape, y_train.shape, y_test.shape)" 322 | ] 323 | }, 324 | { 325 | "cell_type": "markdown", 326 | "metadata": {}, 327 | "source": [ 328 | "I'm saving the test labels on disk for later use." 329 | ] 330 | }, 331 | { 332 | "cell_type": "code", 333 | "execution_count": 9, 334 | "metadata": {}, 335 | "outputs": [], 336 | "source": [ 337 | "pd.DataFrame(y_test).to_csv('./predictions/y_true.csv', index=False, encoding='utf-8')" 338 | ] 339 | }, 340 | { 341 | "cell_type": "markdown", 342 | "metadata": {}, 343 | "source": [ 344 | "Let's start applying some machine learning now:" 345 | ] 346 | }, 347 | { 348 | "cell_type": "markdown", 349 | "metadata": {}, 350 | "source": [ 351 | "# 1 - Bag of word model based on word ngrams\n", 352 | "\n", 353 | "So. What's an **n-gram** ?" 354 | ] 355 | }, 356 | { 357 | "cell_type": "markdown", 358 | "metadata": {}, 359 | "source": [ 360 | " ![](./images/article_5/ngrams.png)" 361 | ] 362 | }, 363 | { 364 | "cell_type": "markdown", 365 | "metadata": {}, 366 | "source": [ 367 | "As we see it in this figure, n-grams are simply all combinations of adjacent words (in this case) of length n that you can find in your source text. \n", 368 | "\n", 369 | "In our model, we are going to use unigrams (n=1) and bigrams (n=2) as features. \n", 370 | "\n", 371 | "The dataset will therefore be represented as a matrix where each row refers to a tweet and each column refers to a feature (unigram or bigram) extracted from the text (after tokenization and cleaning). Each cell will be the **tf-idf** score. (we could also use a simple count but tf-idf is more used in general and usually works better). We call this matrix the **document-term matrix**. \n", 372 | "\n", 373 | "As you can imagine, the number of unique unigrams and bigrams of 1.5 million-tweet corpus is huge. In practice, we will set this number to a fixed value, for a computational reason. You can determine this value by using a cross-validation.\n", 374 | "\n", 375 | "This is what the corpus should look like after vectorization." 376 | ] 377 | }, 378 | { 379 | "cell_type": "markdown", 380 | "metadata": {}, 381 | "source": [ 382 | "![](./images/article_5/tfidf.jpg)" 383 | ] 384 | }, 385 | { 386 | "cell_type": "markdown", 387 | "metadata": {}, 388 | "source": [ 389 | "I like pizza a lot\n", 390 | "===\n", 391 | " \n", 392 | "Let's say we want to feed this sentence to a predictive model using the features described above.\n", 393 | "\n", 394 | "Given that we're using unigrams and bigrams, the model will extract the following features:\n", 395 | "\n", 396 | "i, like, pizza, a, lot, i like, like pizza, pizza a, a lot\n", 397 | "===\n", 398 | "\n", 399 | "Therefore, the sentence will be formed by a vector of size N (= total number of tokens) containing lots of zeros and the tf-idf scores of these ngrams.\n", 400 | "So you can clearly see that we will be dealing with large and **sparse** vectors.\n", 401 | "\n", 402 | "When dealing with large and sparse data, linear models generally perform quite well. Besides, they are faster to train than other types of models (e.g. tree-based models).\n", 403 | "\n", 404 | "I can tell from past experience that **logistic regression** works well on top of sparse tfidf matrices." 405 | ] 406 | }, 407 | { 408 | "cell_type": "code", 409 | "execution_count": 11, 410 | "metadata": {}, 411 | "outputs": [], 412 | "source": [ 413 | "vectorizer_word = TfidfVectorizer(max_features=40000,\n", 414 | " min_df=5, \n", 415 | " max_df=0.5, \n", 416 | " analyzer='word', \n", 417 | " stop_words='english', \n", 418 | " ngram_range=(1, 2))\n", 419 | "\n", 420 | "vectorizer_word.fit(x_train, leave=False)\n", 421 | "\n", 422 | "tfidf_matrix_word_train = vectorizer_word.transform(x_train)\n", 423 | "tfidf_matrix_word_test = vectorizer_word.transform(x_test)" 424 | ] 425 | }, 426 | { 427 | "cell_type": "markdown", 428 | "metadata": {}, 429 | "source": [ 430 | "After generating tfidf matrices for both train and test sets, we can build our first model an test it.\n", 431 | "\n", 432 | "The tifidf matrices are the features of the logistic regression." 433 | ] 434 | }, 435 | { 436 | "cell_type": "code", 437 | "execution_count": null, 438 | "metadata": {}, 439 | "outputs": [], 440 | "source": [ 441 | "lr_word = LogisticRegression(solver='sag', verbose=2)\n", 442 | "lr_word.fit(tfidf_matrix_word_train, y_train)" 443 | ] 444 | }, 445 | { 446 | "cell_type": "markdown", 447 | "metadata": {}, 448 | "source": [ 449 | "Once the model is trained, we apply it on the test data to get predictions. Then we save these values as well as the model on disk." 450 | ] 451 | }, 452 | { 453 | "cell_type": "code", 454 | "execution_count": null, 455 | "metadata": {}, 456 | "outputs": [], 457 | "source": [ 458 | "joblib.dump(lr_word, './models/lr_word_ngram.pkl')\n", 459 | "\n", 460 | "y_pred_word = lr_word.predict(tfidf_matrix_word_test)\n", 461 | "pd.DataFrame(y_pred_word, columns=['y_pred']).to_csv('./predictions/lr_word_ngram.csv', index=False)" 462 | ] 463 | }, 464 | { 465 | "cell_type": "markdown", 466 | "metadata": {}, 467 | "source": [ 468 | "Let's see what accuracy score we have:" 469 | ] 470 | }, 471 | { 472 | "cell_type": "code", 473 | "execution_count": 12, 474 | "metadata": {}, 475 | "outputs": [ 476 | { 477 | "name": "stdout", 478 | "output_type": "stream", 479 | "text": [ 480 | "0.782042246814\n" 481 | ] 482 | } 483 | ], 484 | "source": [ 485 | "y_pred_word = pd.read_csv('./predictions/lr_word_ngram.csv')\n", 486 | "print(accuracy_score(y_test, y_pred_word))" 487 | ] 488 | }, 489 | { 490 | "cell_type": "markdown", 491 | "metadata": {}, 492 | "source": [ 493 | "**78.2% accuracy** for a first model ! Quite not bad. Let's move to the next model." 494 | ] 495 | }, 496 | { 497 | "cell_type": "markdown", 498 | "metadata": {}, 499 | "source": [ 500 | "# 2 - Bag of word model based on character ngrams" 501 | ] 502 | }, 503 | { 504 | "cell_type": "markdown", 505 | "metadata": {}, 506 | "source": [ 507 | "We never said that ngrams were for words only. We can apply them at a character level as well." 508 | ] 509 | }, 510 | { 511 | "cell_type": "markdown", 512 | "metadata": {}, 513 | "source": [ 514 | "\n", 515 | " " 516 | ] 517 | }, 518 | { 519 | "cell_type": "markdown", 520 | "metadata": {}, 521 | "source": [ 522 | "You see it coming, right? We're going to apply the same code above to character ngrams instead, and we will go up to 4-grams.\n", 523 | "\n", 524 | "This basically means that a sentence like \"I like this movie\" will have these features:\n", 525 | "\n", 526 | "I, l, i, k, e, ..., I li, lik, like, ..., this, ... , is m, s mo, movi, ...\n", 527 | "===\n", 528 | "\n", 529 | "Character ngrams are surprisingly very effective. They can even outperform word tokens in modeling a language task. For example spam filters or Native Language Identification heavily rely on character ngrams.\n", 530 | "\n", 531 | "Unlike the previous model which learns combinations of words, this model learns combinations of letters, which can handle the morphological makeup of a word.\n", 532 | "\n", 533 | "One of the advantages of the character-based representation is the better handling of **misspelled words**.\n", 534 | "\n", 535 | "Let's run the same pipeline :" 536 | ] 537 | }, 538 | { 539 | "cell_type": "code", 540 | "execution_count": null, 541 | "metadata": {}, 542 | "outputs": [], 543 | "source": [ 544 | "vectorizer_char = TfidfVectorizer(max_features=40000,\n", 545 | " min_df=5, \n", 546 | " max_df=0.5, \n", 547 | " analyzer='char', \n", 548 | " ngram_range=(1, 4))\n", 549 | "\n", 550 | "vectorizer_char.fit(tqdm_notebook(x_train, leave=False));\n", 551 | "\n", 552 | "tfidf_matrix_char_train = vectorizer_char.transform(x_train)\n", 553 | "tfidf_matrix_char_test = vectorizer_char.transform(x_test)\n", 554 | "\n", 555 | "lr_char = LogisticRegression(solver='sag', verbose=2)\n", 556 | "lr_char.fit(tfidf_matrix_char_train, y_train)\n", 557 | "\n", 558 | "y_pred_char = lr_char.predict(tfidf_matrix_char_test)\n", 559 | "joblib.dump(lr_char, './models/lr_char_ngram.pkl')\n", 560 | "\n", 561 | "pd.DataFrame(y_pred_char, columns=['y_pred']).to_csv('./predictions/lr_char_ngram.csv', index=False)" 562 | ] 563 | }, 564 | { 565 | "cell_type": "code", 566 | "execution_count": 13, 567 | "metadata": {}, 568 | "outputs": [ 569 | { 570 | "name": "stdout", 571 | "output_type": "stream", 572 | "text": [ 573 | "0.80420055491\n" 574 | ] 575 | } 576 | ], 577 | "source": [ 578 | "y_pred_char = pd.read_csv('./predictions/lr_char_ngram.csv')\n", 579 | "print(accuracy_score(y_test, y_pred_char))" 580 | ] 581 | }, 582 | { 583 | "cell_type": "markdown", 584 | "metadata": {}, 585 | "source": [ 586 | "** 80.4% accuracy ! ** Character-ngrams perfom better that word-ngrams." 587 | ] 588 | }, 589 | { 590 | "cell_type": "markdown", 591 | "metadata": {}, 592 | "source": [ 593 | "# 3 - Bag of word model based on word and character ngrams" 594 | ] 595 | }, 596 | { 597 | "cell_type": "markdown", 598 | "metadata": {}, 599 | "source": [ 600 | "Character ngram features seem to provide a better accuracy than word ngrams. But what about the combination of the two: word + character ngrams?\n", 601 | "\n", 602 | "Let's concatenate the two tfidf matrices we generated and build a new, *hybrid* tfidf matrix.\n", 603 | "\n", 604 | "This model will help us learn the indentity of a word and its possible neighbors as well as its morphological structure.\n", 605 | "\n", 606 | "These properties are combined." 607 | ] 608 | }, 609 | { 610 | "cell_type": "code", 611 | "execution_count": null, 612 | "metadata": { 613 | "scrolled": true 614 | }, 615 | "outputs": [], 616 | "source": [ 617 | "tfidf_matrix_word_char_train = hstack((tfidf_matrix_word_train, tfidf_matrix_char_train))\n", 618 | "tfidf_matrix_word_char_test = hstack((tfidf_matrix_word_test, tfidf_matrix_char_test))\n", 619 | "\n", 620 | "lr_word_char = LogisticRegression(solver='sag', verbose=2)\n", 621 | "lr_word_char.fit(tfidf_matrix_word_char_train, y_train)\n", 622 | "\n", 623 | "y_pred_word_char = lr_word_char.predict(tfidf_matrix_word_char_test)\n", 624 | "joblib.dump(lr_word_char, './models/lr_word_char_ngram.pkl')\n", 625 | "\n", 626 | "pd.DataFrame(y_pred_word_char, columns=['y_pred']).to_csv('./predictions/lr_word_char_ngram.csv', index=False)" 627 | ] 628 | }, 629 | { 630 | "cell_type": "code", 631 | "execution_count": 14, 632 | "metadata": {}, 633 | "outputs": [ 634 | { 635 | "name": "stdout", 636 | "output_type": "stream", 637 | "text": [ 638 | "0.81423845895\n" 639 | ] 640 | } 641 | ], 642 | "source": [ 643 | "y_pred_word_char = pd.read_csv('./predictions/lr_word_char_ngram.csv')\n", 644 | "print(accuracy_score(y_test, y_pred_word_char))" 645 | ] 646 | }, 647 | { 648 | "cell_type": "markdown", 649 | "metadata": {}, 650 | "source": [ 651 | "Awesome: **81.4% accuracy**. We just increased by one whole unit and outperformed the two previous settings." 652 | ] 653 | }, 654 | { 655 | "cell_type": "markdown", 656 | "metadata": {}, 657 | "source": [ 658 | "## What can we say about bag-of-word models before we move on?\n", 659 | "\n", 660 | "\n", 661 | "- **Pros**: They can be surprisingly powerful given their simplicity, they are fast to train, and easy to understand. \n", 662 | "- **Cons**: Even though ngrams bring some context between words, bag of word models fail in modeling long-term dependencies between words in a sequence.\n" 663 | ] 664 | }, 665 | { 666 | "cell_type": "markdown", 667 | "metadata": {}, 668 | "source": [ 669 | "Now we're going to dive into deep learning models. The reason deep learning outperform bag of word models is the ability to capture the sequencial dependency between words in a sentence. This has been possible thanks to the invention of special neural network architectures called **Recurrent Neural Networks**.\n", 670 | "\n", 671 | "I won't cover the theoretical foundations of RNNs, but here's a link I find worth reading. It's from Cristopher Olah's blog. It details LSTM: Long Short Term Memory. A special kind of RNN. \n", 672 | "\n", 673 | "Before starting, we have to setup a deep learning dedicated environment that uses Keras on top of Tensorflow. I honestly tried to run everything on my personal laptop but given the important size of the dataset and the complexity of RNN architectures, this has not been practical. At all.\n", 674 | "\n", 675 | "One good option is AWS. I generally use this deep learning AMI on an EC2 p2.xlarge instance. Amazon AMI are pre-configured VM images, where all the packages (Tensorflow, PyTocrh, Keras, etc. ) are installed. I highly recommend this one which I have been using for a while." 676 | ] 677 | }, 678 | { 679 | "cell_type": "code", 680 | "execution_count": 16, 681 | "metadata": {}, 682 | "outputs": [ 683 | { 684 | "name": "stderr", 685 | "output_type": "stream", 686 | "text": [ 687 | "Using TensorFlow backend.\n" 688 | ] 689 | } 690 | ], 691 | "source": [ 692 | "from keras.preprocessing.text import Tokenizer\n", 693 | "from keras.preprocessing.text import text_to_word_sequence\n", 694 | "from keras.preprocessing.sequence import pad_sequences\n", 695 | "\n", 696 | "from keras.models import Model\n", 697 | "from keras.models import Sequential\n", 698 | "\n", 699 | "from keras.layers import Input, Dense, Embedding, Conv1D, Conv2D, MaxPooling1D, MaxPool2D\n", 700 | "from keras.layers import Reshape, Flatten, Dropout, Concatenate\n", 701 | "from keras.layers import SpatialDropout1D, concatenate\n", 702 | "from keras.layers import GRU, Bidirectional, GlobalAveragePooling1D, GlobalMaxPooling1D\n", 703 | "\n", 704 | "from keras.callbacks import Callback\n", 705 | "from keras.optimizers import Adam\n", 706 | "\n", 707 | "from keras.callbacks import ModelCheckpoint, EarlyStopping\n", 708 | "from keras.models import load_model\n", 709 | "from keras.utils.vis_utils import plot_model" 710 | ] 711 | }, 712 | { 713 | "cell_type": "markdown", 714 | "metadata": {}, 715 | "source": [ 716 | "# 4 - Recurrent Neural Network without pre-trained embedding" 717 | ] 718 | }, 719 | { 720 | "cell_type": "markdown", 721 | "metadata": {}, 722 | "source": [ 723 | "RNNs may look scary. Although they're complex to understand, they're quite interesting. They encapsulate a very beautiful design that overcomes traditional neural networks' shortcomings that rise when dealing with sequence data: text, time series, videos, DNA sequences, etc.\n", 724 | "\n", 725 | "An RNN is a sequence of neural network blocks that are linked to each others like a chain. Each one is passing a message to a successor.\n", 726 | "\n", 727 | "Again if you want to dive into the internal mechanics, I highly recommend Colah's blog out of which the diagram below is taken." 728 | ] 729 | }, 730 | { 731 | "cell_type": "markdown", 732 | "metadata": {}, 733 | "source": [ 734 | "" 735 | ] 736 | }, 737 | { 738 | "cell_type": "markdown", 739 | "metadata": {}, 740 | "source": [ 741 | "We will process text data, which is a sequence type. The order of words is very important to the meaning. Hopefully RNNs take care of this and can capture long-term dependencies.\n", 742 | "\n", 743 | "To use Keras on text data, we firt have to preprocess it. For this, we can use Keras' Tokenizer class. This object takes as argument **num_words** which is the maximum number of words kept after tokenization based on their word frequency." 744 | ] 745 | }, 746 | { 747 | "cell_type": "code", 748 | "execution_count": 17, 749 | "metadata": {}, 750 | "outputs": [], 751 | "source": [ 752 | "MAX_NB_WORDS = 80000\n", 753 | "tokenizer = Tokenizer(num_words=MAX_NB_WORDS)\n", 754 | "\n", 755 | "tokenizer.fit_on_texts(data['cleaned_text'])" 756 | ] 757 | }, 758 | { 759 | "cell_type": "markdown", 760 | "metadata": {}, 761 | "source": [ 762 | "Once the tokenizer is fitted on the data, we can use it to convert text strings to sequences of numbers.\n", 763 | "\n", 764 | "These numbers represent the position of each word in the dictionary (think of it as mapping). \n", 765 | "\n", 766 | "Let's see an example:" 767 | ] 768 | }, 769 | { 770 | "cell_type": "code", 771 | "execution_count": 18, 772 | "metadata": {}, 773 | "outputs": [ 774 | { 775 | "data": { 776 | "text/plain": [ 777 | "'breakfast time happy time'" 778 | ] 779 | }, 780 | "execution_count": 18, 781 | "metadata": {}, 782 | "output_type": "execute_result" 783 | } 784 | ], 785 | "source": [ 786 | "x_train[15]" 787 | ] 788 | }, 789 | { 790 | "cell_type": "markdown", 791 | "metadata": {}, 792 | "source": [ 793 | "Here's how the tokenizer turns it into a sequence of digits. " 794 | ] 795 | }, 796 | { 797 | "cell_type": "code", 798 | "execution_count": 19, 799 | "metadata": {}, 800 | "outputs": [ 801 | { 802 | "data": { 803 | "text/plain": [ 804 | "[[530, 50, 119, 50]]" 805 | ] 806 | }, 807 | "execution_count": 19, 808 | "metadata": {}, 809 | "output_type": "execute_result" 810 | } 811 | ], 812 | "source": [ 813 | "tokenizer.texts_to_sequences([x_train[15]])" 814 | ] 815 | }, 816 | { 817 | "cell_type": "markdown", 818 | "metadata": {}, 819 | "source": [ 820 | "Let's now apply this tokenizer on the train and test sequences:" 821 | ] 822 | }, 823 | { 824 | "cell_type": "code", 825 | "execution_count": 21, 826 | "metadata": {}, 827 | "outputs": [], 828 | "source": [ 829 | "train_sequences = tokenizer.texts_to_sequences(x_train)\n", 830 | "test_sequences = tokenizer.texts_to_sequences(x_test)" 831 | ] 832 | }, 833 | { 834 | "cell_type": "markdown", 835 | "metadata": {}, 836 | "source": [ 837 | "Now the tweets are mapped to lists of integers. However, we still cannot stack them together in a matrix since they have different lengths.\n", 838 | "Hopefully Keras allows to **pad** sequences with **0s** to a maximum length. We'll set this length to 35. (which is the maximum numbers of tokens in the tweets)." 839 | ] 840 | }, 841 | { 842 | "cell_type": "code", 843 | "execution_count": 22, 844 | "metadata": {}, 845 | "outputs": [], 846 | "source": [ 847 | "MAX_LENGTH = 35\n", 848 | "padded_train_sequences = pad_sequences(train_sequences, maxlen=MAX_LENGTH)\n", 849 | "padded_test_sequences = pad_sequences(test_sequences, maxlen=MAX_LENGTH)" 850 | ] 851 | }, 852 | { 853 | "cell_type": "code", 854 | "execution_count": 23, 855 | "metadata": {}, 856 | "outputs": [ 857 | { 858 | "data": { 859 | "text/plain": [ 860 | "array([[ 0, 0, 0, ..., 2383, 284, 9],\n", 861 | " [ 0, 0, 0, ..., 13, 30, 76],\n", 862 | " [ 0, 0, 0, ..., 19, 37, 45231],\n", 863 | " ..., \n", 864 | " [ 0, 0, 0, ..., 43, 502, 1653],\n", 865 | " [ 0, 0, 0, ..., 5, 1045, 890],\n", 866 | " [ 0, 0, 0, ..., 13748, 38750, 154]])" 867 | ] 868 | }, 869 | "execution_count": 23, 870 | "metadata": {}, 871 | "output_type": "execute_result" 872 | } 873 | ], 874 | "source": [ 875 | "padded_train_sequences" 876 | ] 877 | }, 878 | { 879 | "cell_type": "code", 880 | "execution_count": 24, 881 | "metadata": {}, 882 | "outputs": [ 883 | { 884 | "data": { 885 | "text/plain": [ 886 | "(1417523, 35)" 887 | ] 888 | }, 889 | "execution_count": 24, 890 | "metadata": {}, 891 | "output_type": "execute_result" 892 | } 893 | ], 894 | "source": [ 895 | "padded_train_sequences.shape" 896 | ] 897 | }, 898 | { 899 | "cell_type": "markdown", 900 | "metadata": {}, 901 | "source": [ 902 | "Now the data is ready to be fed to an RNN.\n", 903 | "\n", 904 | "Here are some elements of the architecture I'll be using:\n", 905 | "\n", 906 | "- An embedding dimension of 300. This means that each word from the 80000 that we'll be using is mapped to a 300-dimension dense vector (of float numbers). The mapping will adjust throughout the training.\n", 907 | "\n", 908 | "- A spatial dropout is applied on the embedding layer to reduce overfitting: it basically looks at batches of 35x300 matrices and randomly drop (set to 0) word vectors (i.e rows) in each matrix. This helps not to focus on specific words in an attempt to generalize well.\n", 909 | "\n", 910 | "- A bidirectional Gated Recurrent Unit (GRU): this is the recurrent network part. It's a faster variant of the LSTM architecture. Think of it as a combination of two recurrent networks that scan the text sequence in both directions: from left to right and from right to left. This allows the network, when reading a given word, to understand it by using the context from both past and future information. The GRU takes as parameter a number of units which is the dimension of the output h_t of each network block. We will set this number to 100. And since we are using a bidirectional version of the GRU, the final output per RNN block will be of dimension 200.\n", 911 | " \n", 912 | "The output of the bidirectional GRU has the dimension (batch_size, timesteps, units). This means that if we use a typical batch size of 256, this dimension will be (256, 35, 200)\n", 913 | "\n", 914 | "- On top of every batch, we apply a global average pooling that consists in averaging the output vectors corresponding to the each time step (i.e the words)\n", 915 | "\n", 916 | "- We apply the same operation with max pooling. \n", 917 | "\n", 918 | "- We concatenate the outputs of the two previous operations." 919 | ] 920 | }, 921 | { 922 | "cell_type": "code", 923 | "execution_count": null, 924 | "metadata": {}, 925 | "outputs": [], 926 | "source": [ 927 | "def get_simple_rnn_model():\n", 928 | " embedding_dim = 300\n", 929 | " embedding_matrix = np.random.random((MAX_NB_WORDS, embedding_dim))\n", 930 | " \n", 931 | " inp = Input(shape=(MAX_LENGTH, ))\n", 932 | " x = Embedding(input_dim=MAX_NB_WORDS, output_dim=embedding_dim, input_length=MAX_LENGTH, \n", 933 | " weights=[embedding_matrix], trainable=True)(inp)\n", 934 | " x = SpatialDropout1D(0.3)(x)\n", 935 | " x = Bidirectional(GRU(100, return_sequences=True))(x)\n", 936 | " avg_pool = GlobalAveragePooling1D()(x)\n", 937 | " max_pool = GlobalMaxPooling1D()(x)\n", 938 | " conc = concatenate([avg_pool, max_pool])\n", 939 | " outp = Dense(1, activation=\"sigmoid\")(conc)\n", 940 | " \n", 941 | " model = Model(inputs=inp, outputs=outp)\n", 942 | " model.compile(loss='binary_crossentropy',\n", 943 | " optimizer='adam',\n", 944 | " metrics=['accuracy'])\n", 945 | " return model\n", 946 | "\n", 947 | "rnn_simple_model = get_simple_rnn_model()" 948 | ] 949 | }, 950 | { 951 | "cell_type": "markdown", 952 | "metadata": {}, 953 | "source": [ 954 | "Let's look at the different layers of this model:" 955 | ] 956 | }, 957 | { 958 | "cell_type": "code", 959 | "execution_count": null, 960 | "metadata": {}, 961 | "outputs": [], 962 | "source": [ 963 | "plot_model(rnn_simple_model, \n", 964 | " to_file='./images/article_5/rnn_simple_model.png', \n", 965 | " show_shapes=True, \n", 966 | " show_layer_names=True)" 967 | ] 968 | }, 969 | { 970 | "cell_type": "markdown", 971 | "metadata": {}, 972 | "source": [ 973 | "" 974 | ] 975 | }, 976 | { 977 | "cell_type": "markdown", 978 | "metadata": {}, 979 | "source": [ 980 | "During the training, model checkpoint is used. It allows to automatically save (on disk) the best models (w.r.t accuracy measure) at the end of each epoch." 981 | ] 982 | }, 983 | { 984 | "cell_type": "code", 985 | "execution_count": null, 986 | "metadata": {}, 987 | "outputs": [], 988 | "source": [ 989 | "filepath=\"./models/rnn_no_embeddings/weights-improvement-{epoch:02d}-{val_acc:.4f}.hdf5\"\n", 990 | "checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')\n", 991 | "\n", 992 | "batch_size = 256\n", 993 | "epochs = 2\n", 994 | "\n", 995 | "history = rnn_simple_model.fit(x=padded_train_sequences, \n", 996 | " y=y_train, \n", 997 | " validation_data=(padded_test_sequences, y_test), \n", 998 | " batch_size=batch_size, \n", 999 | " callbacks=[checkpoint], \n", 1000 | " epochs=epochs, \n", 1001 | " verbose=1)\n", 1002 | "\n", 1003 | "best_rnn_simple_model = load_model('./models/rnn_no_embeddings/weights-improvement-01-0.8262.hdf5')\n", 1004 | "\n", 1005 | "y_pred_rnn_simple = best_rnn_simple_model.predict(padded_test_sequences, verbose=1, batch_size=2048)\n", 1006 | "\n", 1007 | "y_pred_rnn_simple = pd.DataFrame(y_pred_rnn_simple, columns=['prediction'])\n", 1008 | "y_pred_rnn_simple['prediction'] = y_pred_rnn_simple['prediction'].map(lambda p: 1 if p >= 0.5 else 0)\n", 1009 | "y_pred_rnn_simple.to_csv('./predictions/y_pred_rnn_simple.csv', index=False)" 1010 | ] 1011 | }, 1012 | { 1013 | "cell_type": "code", 1014 | "execution_count": 25, 1015 | "metadata": { 1016 | "scrolled": true 1017 | }, 1018 | "outputs": [ 1019 | { 1020 | "name": "stdout", 1021 | "output_type": "stream", 1022 | "text": [ 1023 | "0.826219183127\n" 1024 | ] 1025 | } 1026 | ], 1027 | "source": [ 1028 | "y_pred_rnn_simple = pd.read_csv('./predictions/y_pred_rnn_simple.csv')\n", 1029 | "print(accuracy_score(y_test, y_pred_rnn_simple))" 1030 | ] 1031 | }, 1032 | { 1033 | "cell_type": "markdown", 1034 | "metadata": {}, 1035 | "source": [ 1036 | "** 82.6% accuracy ! ** Quite not bad ! We are now performing better than the previous bag-of-word models because we are taking into account the sequence nature of the text. \n", 1037 | "\n", 1038 | "Can we do better?" 1039 | ] 1040 | }, 1041 | { 1042 | "cell_type": "markdown", 1043 | "metadata": {}, 1044 | "source": [ 1045 | "# 5 - Recurrent Neural Network with GloVe pre-trained embeddings" 1046 | ] 1047 | }, 1048 | { 1049 | "cell_type": "markdown", 1050 | "metadata": {}, 1051 | "source": [ 1052 | "In the last model, the embedding matrix was initialized randomly. What if we could use pre-trained word embeddings to intialize it instead?\n", 1053 | "\n", 1054 | "Let's take an example: imagine that you have the word *pizza* in your corpus. Following the previous architecture, you would initialize it to a 300 dimension vector of random float values. This is perfectly fine. You can do that, and this embedding will adjust an evolve throughout training. However, what you could do instead of randomly picking a vector for pizza is using an embedding for this word that has been learnt from another model on a very large corpus. This is a special kind of **transfer learning**.\n", 1055 | "\n", 1056 | "Using the knowledge from an external embedding can enhance the precision of your RNN because it integrates new information (lexical and semantic) about the words, an information that has been trained and distilled on a very large corpus of data. \n", 1057 | "\n", 1058 | "The pre-trained embedding we'll be using is GloVe.\n", 1059 | "\n", 1060 | "Official documentation: *GloVe is an unsupervised learning algorithm for obtaining vector representations for words. Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space.* \n", 1061 | "\n", 1062 | "The GloVe embeddings I'll be using are trained on a ** very large ** common internet crawl that includes:\n", 1063 | "\n", 1064 | "- 840 Billion tokens, \n", 1065 | "- 2.2 million size vocab\n", 1066 | "\n", 1067 | "The zipped file is 2.03 GB download. Beware, this file cannot be easily loaded on a standard laptop.\n", 1068 | "\n", 1069 | "The dimension of GloVe embeddings is 300. " 1070 | ] 1071 | }, 1072 | { 1073 | "cell_type": "markdown", 1074 | "metadata": {}, 1075 | "source": [ 1076 | "GloVe embeddings come in raw text data, where each line contains a word and 300 floats (the corresponding embedding). So the first thing to do is convert this structure to a python dictionary." 1077 | ] 1078 | }, 1079 | { 1080 | "cell_type": "code", 1081 | "execution_count": null, 1082 | "metadata": {}, 1083 | "outputs": [], 1084 | "source": [ 1085 | "def get_coefs(word, *arr):\n", 1086 | " try:\n", 1087 | " return word, np.asarray(arr, dtype='float32')\n", 1088 | " except:\n", 1089 | " return None, None\n", 1090 | " \n", 1091 | "embeddings_index = dict(get_coefs(*o.strip().split()) for o in tqdm_notebook(open('./embeddings/glove.840B.300d.txt')))\n", 1092 | "\n", 1093 | "embed_size=300\n", 1094 | "for k in tqdm_notebook(list(embeddings_index.keys())):\n", 1095 | " v = embeddings_index[k]\n", 1096 | " try:\n", 1097 | " if v.shape != (embed_size, ):\n", 1098 | " embeddings_index.pop(k)\n", 1099 | " except:\n", 1100 | " pass\n", 1101 | " \n", 1102 | "embeddings_index.pop(None)" 1103 | ] 1104 | }, 1105 | { 1106 | "cell_type": "markdown", 1107 | "metadata": {}, 1108 | "source": [ 1109 | "Once the embedding index in created, we extract all the vectors, we stack them together and compute their mean and standard deviation. " 1110 | ] 1111 | }, 1112 | { 1113 | "cell_type": "code", 1114 | "execution_count": null, 1115 | "metadata": {}, 1116 | "outputs": [], 1117 | "source": [ 1118 | "values = list(embeddings_index.values())\n", 1119 | "all_embs = np.stack(values)\n", 1120 | "\n", 1121 | "emb_mean, emb_std = all_embs.mean(), all_embs.std()" 1122 | ] 1123 | }, 1124 | { 1125 | "cell_type": "markdown", 1126 | "metadata": {}, 1127 | "source": [ 1128 | "Now we generate the embedding matrix. We will initialize it following a normal distribution of mean=emb_mean and std=emb_std.\n", 1129 | "\n", 1130 | "Then we go through the 80000 words of our corpus. For each word, if it is contained in GloVe, we pick its embedding. \n", 1131 | "\n", 1132 | "Otherwise, we pass." 1133 | ] 1134 | }, 1135 | { 1136 | "cell_type": "code", 1137 | "execution_count": null, 1138 | "metadata": {}, 1139 | "outputs": [], 1140 | "source": [ 1141 | "word_index = tokenizer.word_index\n", 1142 | "nb_words = MAX_NB_WORDS\n", 1143 | "embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n", 1144 | "\n", 1145 | "oov = 0\n", 1146 | "for word, i in tqdm_notebook(word_index.items()):\n", 1147 | " if i >= MAX_NB_WORDS: continue\n", 1148 | " embedding_vector = embeddings_index.get(word)\n", 1149 | " if embedding_vector is not None:\n", 1150 | " embedding_matrix[i] = embedding_vector\n", 1151 | " else:\n", 1152 | " oov += 1\n", 1153 | "\n", 1154 | "print(oov)\n", 1155 | "\n", 1156 | "def get_rnn_model_with_glove_embeddings():\n", 1157 | " embedding_dim = 300\n", 1158 | " inp = Input(shape=(MAX_LENGTH, ))\n", 1159 | " x = Embedding(MAX_NB_WORDS, embedding_dim, weights=[embedding_matrix], input_length=MAX_LENGTH, trainable=True)(inp)\n", 1160 | " x = SpatialDropout1D(0.3)(x)\n", 1161 | " x = Bidirectional(GRU(100, return_sequences=True))(x)\n", 1162 | " avg_pool = GlobalAveragePooling1D()(x)\n", 1163 | " max_pool = GlobalMaxPooling1D()(x)\n", 1164 | " conc = concatenate([avg_pool, max_pool])\n", 1165 | " outp = Dense(1, activation=\"sigmoid\")(conc)\n", 1166 | " \n", 1167 | " model = Model(inputs=inp, outputs=outp)\n", 1168 | " model.compile(loss='binary_crossentropy',\n", 1169 | " optimizer='adam',\n", 1170 | " metrics=['accuracy'])\n", 1171 | " return model\n", 1172 | "\n", 1173 | "rnn_model_with_embeddings = get_rnn_model_with_glove_embeddings()\n", 1174 | "\n", 1175 | "filepath=\"./models/rnn_with_embeddings/weights-improvement-{epoch:02d}-{val_acc:.4f}.hdf5\"\n", 1176 | "checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')\n", 1177 | "\n", 1178 | "batch_size = 256\n", 1179 | "epochs = 4\n", 1180 | "\n", 1181 | "history = rnn_model_with_embeddings.fit(x=padded_train_sequences, \n", 1182 | " y=y_train, \n", 1183 | " validation_data=(padded_test_sequences, y_test), \n", 1184 | " batch_size=batch_size, \n", 1185 | " callbacks=[checkpoint], \n", 1186 | " epochs=epochs, \n", 1187 | " verbose=1)\n", 1188 | "\n", 1189 | "best_rnn_model_with_glove_embeddings = load_model('./models/rnn_with_embeddings/weights-improvement-03-0.8372.hdf5')\n", 1190 | "\n", 1191 | "y_pred_rnn_with_glove_embeddings = best_rnn_model_with_glove_embeddings.predict(\n", 1192 | " padded_test_sequences, verbose=1, batch_size=2048)\n", 1193 | "\n", 1194 | "y_pred_rnn_with_glove_embeddings = pd.DataFrame(y_pred_rnn_with_glove_embeddings, columns=['prediction'])\n", 1195 | "y_pred_rnn_with_glove_embeddings['prediction'] = y_pred_rnn_with_glove_embeddings['prediction'].map(lambda p: \n", 1196 | " 1 if p >= 0.5 else 0)\n", 1197 | "y_pred_rnn_with_glove_embeddings.to_csv('./predictions/y_pred_rnn_with_glove_embeddings.csv', index=False)" 1198 | ] 1199 | }, 1200 | { 1201 | "cell_type": "code", 1202 | "execution_count": 26, 1203 | "metadata": {}, 1204 | "outputs": [ 1205 | { 1206 | "name": "stdout", 1207 | "output_type": "stream", 1208 | "text": [ 1209 | "0.837203100893\n" 1210 | ] 1211 | } 1212 | ], 1213 | "source": [ 1214 | "y_pred_rnn_with_glove_embeddings = pd.read_csv('./predictions/y_pred_rnn_with_glove_embeddings.csv')\n", 1215 | "print(accuracy_score(y_test, y_pred_rnn_with_glove_embeddings))" 1216 | ] 1217 | }, 1218 | { 1219 | "cell_type": "markdown", 1220 | "metadata": {}, 1221 | "source": [ 1222 | "**83.7% accuracy ! ** Transfer learning from external word embeddings works! For the rest of the tutorial, I'll be using GloVe embeddings in the embedding matrix." 1223 | ] 1224 | }, 1225 | { 1226 | "cell_type": "markdown", 1227 | "metadata": {}, 1228 | "source": [ 1229 | "# 6 - Multi-channel Convolutional Neural Network" 1230 | ] 1231 | }, 1232 | { 1233 | "cell_type": "markdown", 1234 | "metadata": {}, 1235 | "source": [ 1236 | "In this section, I'm experimenting a convolutional neural network architecture I read about here. CNNs are generally used in computer vision. However, we've recently started applying them to NLP tasks and the results were promising. \n", 1237 | "\n", 1238 | "Let's briefly see what happens when we use convnets on text data. To explain this, I'm borrowing this famous diagram (below) from wildm.com (a very good blog!)\n", 1239 | "\n", 1240 | "Let's consider the example it used: I like this movie very much ! (7 tokens)\n", 1241 | "\n", 1242 | "- The embedding dimension of each word is 5. Therefore this sentence is represented by a matrix of dimension (7,5). You can think of it as an \"image\" (~ a matrix of digits/floats). \n", 1243 | "- 6 Filters, 2 of size (2, 5) (3, 5) and (4, 5) are applied on this matrix. The particularity of these filters is that they are not square matrices and their width is equal to the embedding matrix's width. So the result of each convolution will be a column vector.\n", 1244 | "\n", 1245 | "- Each column vector resulting from the convolution is subsampled using a maxpooling operation.\n", 1246 | "\n", 1247 | "- The results of the maxpooling operations are concatenated in a final vector that is passed to a softmax function for classification.\n", 1248 | "\n", 1249 | "\n", 1250 | "What is the intuition behind ?\n", 1251 | "==\n", 1252 | "\n", 1253 | "The result of each convolution will fire when a special pattern is detected. By varying the size of the kernels and concatenating their outputs, you're allowing yourself to detect patterns of multiples sizes (2, 3, or 5 adjacent words).\n", 1254 | "\n", 1255 | "Patterns could be expressions (word ngrams?) like \"I hate\", \"very good\" and therefore CNNs can identify them in the sentence regardless of their position." 1256 | ] 1257 | }, 1258 | { 1259 | "cell_type": "markdown", 1260 | "metadata": {}, 1261 | "source": [ 1262 | "" 1263 | ] 1264 | }, 1265 | { 1266 | "cell_type": "code", 1267 | "execution_count": null, 1268 | "metadata": {}, 1269 | "outputs": [], 1270 | "source": [ 1271 | "def get_cnn_model():\n", 1272 | " embedding_dim = 300\n", 1273 | " \n", 1274 | " filter_sizes = [2, 3, 5]\n", 1275 | " num_filters = 256\n", 1276 | " drop = 0.3\n", 1277 | "\n", 1278 | " inputs = Input(shape=(MAX_LENGTH,), dtype='int32')\n", 1279 | " embedding = Embedding(input_dim=MAX_NB_WORDS,\n", 1280 | " output_dim=embedding_dim,\n", 1281 | " weights=[embedding_matrix],\n", 1282 | " input_length=MAX_LENGTH,\n", 1283 | " trainable=True)(inputs)\n", 1284 | "\n", 1285 | " reshape = Reshape((MAX_LENGTH, embedding_dim, 1))(embedding)\n", 1286 | " conv_0 = Conv2D(num_filters, \n", 1287 | " kernel_size=(filter_sizes[0], embedding_dim), \n", 1288 | " padding='valid', kernel_initializer='normal', \n", 1289 | " activation='relu')(reshape)\n", 1290 | "\n", 1291 | " conv_1 = Conv2D(num_filters, \n", 1292 | " kernel_size=(filter_sizes[1], embedding_dim), \n", 1293 | " padding='valid', kernel_initializer='normal', \n", 1294 | " activation='relu')(reshape)\n", 1295 | " conv_2 = Conv2D(num_filters, \n", 1296 | " kernel_size=(filter_sizes[2], embedding_dim), \n", 1297 | " padding='valid', kernel_initializer='normal', \n", 1298 | " activation='relu')(reshape)\n", 1299 | "\n", 1300 | " maxpool_0 = MaxPool2D(pool_size=(MAX_LENGTH - filter_sizes[0] + 1, 1), \n", 1301 | " strides=(1,1), padding='valid')(conv_0)\n", 1302 | "\n", 1303 | " maxpool_1 = MaxPool2D(pool_size=(MAX_LENGTH - filter_sizes[1] + 1, 1), \n", 1304 | " strides=(1,1), padding='valid')(conv_1)\n", 1305 | "\n", 1306 | " maxpool_2 = MaxPool2D(pool_size=(MAX_LENGTH - filter_sizes[2] + 1, 1), \n", 1307 | " strides=(1,1), padding='valid')(conv_2)\n", 1308 | " concatenated_tensor = Concatenate(axis=1)(\n", 1309 | " [maxpool_0, maxpool_1, maxpool_2])\n", 1310 | " flatten = Flatten()(concatenated_tensor)\n", 1311 | " dropout = Dropout(drop)(flatten)\n", 1312 | " output = Dense(units=1, activation='sigmoid')(dropout)\n", 1313 | "\n", 1314 | " model = Model(inputs=inputs, outputs=output)\n", 1315 | " adam = Adam(lr=1e-4, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)\n", 1316 | "\n", 1317 | " model.compile(optimizer=adam, loss='binary_crossentropy', metrics=['accuracy'])\n", 1318 | " \n", 1319 | " return model\n", 1320 | "\n", 1321 | "cnn_model_multi_channel = get_cnn_model()\n", 1322 | "\n", 1323 | "plot_model(cnn_model_multi_channel, \n", 1324 | " to_file='./images/article_5/cnn_model_multi_channel.png', \n", 1325 | " show_shapes=True, \n", 1326 | " show_layer_names=True)" 1327 | ] 1328 | }, 1329 | { 1330 | "cell_type": "markdown", 1331 | "metadata": {}, 1332 | "source": [ 1333 | "" 1334 | ] 1335 | }, 1336 | { 1337 | "cell_type": "code", 1338 | "execution_count": null, 1339 | "metadata": {}, 1340 | "outputs": [], 1341 | "source": [ 1342 | "filepath=\"./models/cnn_multi_channel/weights-improvement-{epoch:02d}-{val_acc:.4f}.hdf5\"\n", 1343 | "checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')\n", 1344 | "\n", 1345 | "batch_size = 256\n", 1346 | "epochs = 4\n", 1347 | "\n", 1348 | "history = cnn_model_multi_channel.fit(x=padded_train_sequences, \n", 1349 | " y=y_train, \n", 1350 | " validation_data=(padded_test_sequences, y_test), \n", 1351 | " batch_size=batch_size, \n", 1352 | " callbacks=[checkpoint], \n", 1353 | " epochs=epochs, \n", 1354 | " verbose=1)\n", 1355 | "\n", 1356 | "best_cnn_model = load_model('./models/cnn_multi_channel/weights-improvement-04-0.8264.hdf5')\n", 1357 | "\n", 1358 | "y_pred_cnn_multi_channel = best_cnn_model.predict(padded_test_sequences, verbose=1, batch_size=2048)\n", 1359 | "\n", 1360 | "y_pred_cnn_multi_channel = pd.DataFrame(y_pred_cnn_multi_channel, columns=['prediction'])\n", 1361 | "y_pred_cnn_multi_channel['prediction'] = y_pred_cnn_multi_channel['prediction'].map(lambda p: 1 if p >= 0.5 else 0)\n", 1362 | "y_pred_cnn_multi_channel.to_csv('./predictions/y_pred_cnn_multi_channel.csv', index=False)" 1363 | ] 1364 | }, 1365 | { 1366 | "cell_type": "code", 1367 | "execution_count": 27, 1368 | "metadata": {}, 1369 | "outputs": [ 1370 | { 1371 | "name": "stdout", 1372 | "output_type": "stream", 1373 | "text": [ 1374 | "0.826409655689\n" 1375 | ] 1376 | } 1377 | ], 1378 | "source": [ 1379 | "y_pred_cnn_multi_channel = pd.read_csv('./predictions/y_pred_cnn_multi_channel.csv')\n", 1380 | "print(accuracy_score(y_test, y_pred_cnn_multi_channel))" 1381 | ] 1382 | }, 1383 | { 1384 | "cell_type": "markdown", 1385 | "metadata": {}, 1386 | "source": [ 1387 | "** 82.6% accuracy **, we're less precise than RNNs but still better than BOW models. Maybe an investigation of the hyperparameters (number of filters, and size) gives an edge?" 1388 | ] 1389 | }, 1390 | { 1391 | "cell_type": "markdown", 1392 | "metadata": {}, 1393 | "source": [ 1394 | "# 7 - Recurrent + Convolutional neural network" 1395 | ] 1396 | }, 1397 | { 1398 | "cell_type": "markdown", 1399 | "metadata": {}, 1400 | "source": [ 1401 | "RNNs are powerful. Howerer, some people found out that they could maek them more robust by adding a convolutional layer on top of the reccurrent layer.\n", 1402 | "\n", 1403 | "The rational behind is that RNNs allow you to embed the information about the sequence and previous words and CNN takes this embedding and extract local features from it. Having these two layers working together is a winning combination.\n", 1404 | "\n", 1405 | "More about this here." 1406 | ] 1407 | }, 1408 | { 1409 | "cell_type": "code", 1410 | "execution_count": null, 1411 | "metadata": {}, 1412 | "outputs": [], 1413 | "source": [ 1414 | "def get_rnn_cnn_model():\n", 1415 | " embedding_dim = 300\n", 1416 | " inp = Input(shape=(MAX_LENGTH, ))\n", 1417 | " x = Embedding(MAX_NB_WORDS, embedding_dim, weights=[embedding_matrix], input_length=MAX_LENGTH, trainable=True)(inp)\n", 1418 | " x = SpatialDropout1D(0.3)(x)\n", 1419 | " x = Bidirectional(GRU(100, return_sequences=True))(x)\n", 1420 | " x = Conv1D(64, kernel_size = 2, padding = \"valid\", kernel_initializer = \"he_uniform\")(x)\n", 1421 | " avg_pool = GlobalAveragePooling1D()(x)\n", 1422 | " max_pool = GlobalMaxPooling1D()(x)\n", 1423 | " conc = concatenate([avg_pool, max_pool])\n", 1424 | " outp = Dense(1, activation=\"sigmoid\")(conc)\n", 1425 | " \n", 1426 | " model = Model(inputs=inp, outputs=outp)\n", 1427 | " model.compile(loss='binary_crossentropy',\n", 1428 | " optimizer='adam',\n", 1429 | " metrics=['accuracy'])\n", 1430 | " return model\n", 1431 | "\n", 1432 | "rnn_cnn_model = get_rnn_cnn_model()\n", 1433 | "\n", 1434 | "plot_model(rnn_cnn_model, to_file='./images/article_5/rnn_cnn_model.png', show_shapes=True, show_layer_names=True)" 1435 | ] 1436 | }, 1437 | { 1438 | "cell_type": "markdown", 1439 | "metadata": {}, 1440 | "source": [ 1441 | "" 1442 | ] 1443 | }, 1444 | { 1445 | "cell_type": "code", 1446 | "execution_count": null, 1447 | "metadata": {}, 1448 | "outputs": [], 1449 | "source": [ 1450 | "filepath=\"./models/rnn_cnn/weights-improvement-{epoch:02d}-{val_acc:.4f}.hdf5\"\n", 1451 | "checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')\n", 1452 | "\n", 1453 | "batch_size = 256\n", 1454 | "epochs = 4\n", 1455 | "\n", 1456 | "history = rnn_cnn_model.fit(x=padded_train_sequences, \n", 1457 | " y=y_train, \n", 1458 | " validation_data=(padded_test_sequences, y_test), \n", 1459 | " batch_size=batch_size, \n", 1460 | " callbacks=[checkpoint], \n", 1461 | " epochs=epochs, \n", 1462 | " verbose=1)\n", 1463 | "\n", 1464 | "best_rnn_cnn_model = load_model('./models/rnn_cnn/weights-improvement-03-0.8379.hdf5')\n", 1465 | "\n", 1466 | "y_pred_rnn_cnn = best_rnn_cnn_model.predict(padded_test_sequences, verbose=1, batch_size=2048)\n", 1467 | "\n", 1468 | "y_pred_rnn_cnn = pd.DataFrame(y_pred_rnn_cnn, columns=['prediction'])\n", 1469 | "y_pred_rnn_cnn['prediction'] = y_pred_rnn_cnn['prediction'].map(lambda p: 1 if p >= 0.5 else 0)\n", 1470 | "y_pred_rnn_cnn.to_csv('./predictions/y_pred_rnn_cnn.csv', index=False)" 1471 | ] 1472 | }, 1473 | { 1474 | "cell_type": "code", 1475 | "execution_count": 28, 1476 | "metadata": {}, 1477 | "outputs": [ 1478 | { 1479 | "name": "stdout", 1480 | "output_type": "stream", 1481 | "text": [ 1482 | "0.837882453033\n" 1483 | ] 1484 | } 1485 | ], 1486 | "source": [ 1487 | "y_pred_rnn_cnn = pd.read_csv('./predictions/y_pred_rnn_cnn.csv')\n", 1488 | "print(accuracy_score(y_test, y_pred_rnn_cnn))" 1489 | ] 1490 | }, 1491 | { 1492 | "cell_type": "markdown", 1493 | "metadata": {}, 1494 | "source": [ 1495 | "**83.8% accuracy .** Best model so far." 1496 | ] 1497 | }, 1498 | { 1499 | "cell_type": "markdown", 1500 | "metadata": {}, 1501 | "source": [ 1502 | "# 8 - Conclusion" 1503 | ] 1504 | }, 1505 | { 1506 | "cell_type": "markdown", 1507 | "metadata": {}, 1508 | "source": [ 1509 | "We've run seven different models. Let's see how they compare:" 1510 | ] 1511 | }, 1512 | { 1513 | "cell_type": "code", 1514 | "execution_count": null, 1515 | "metadata": {}, 1516 | "outputs": [], 1517 | "source": [ 1518 | "import seaborn as sns\n", 1519 | "from sklearn.metrics import roc_auc_score\n", 1520 | "sns.set_style(\"whitegrid\")\n", 1521 | "sns.set_palette(\"pastel\")\n", 1522 | "\n", 1523 | "predictions_files = os.listdir('./predictions/')\n", 1524 | "\n", 1525 | "predictions_dfs = []\n", 1526 | "for f in predictions_files:\n", 1527 | " aux = pd.read_csv('./predictions/{0}'.format(f))\n", 1528 | " aux.columns = [f.strip('.csv')]\n", 1529 | " predictions_dfs.append(aux)\n", 1530 | "\n", 1531 | "predictions = pd.concat(predictions_dfs, axis=1)\n", 1532 | "\n", 1533 | "scores = {}\n", 1534 | "\n", 1535 | "for column in tqdm_notebook(predictions.columns, leave=False):\n", 1536 | " if column != 'y_true':\n", 1537 | " s = accuracy_score(predictions['y_true'].values, predictions[column].values)\n", 1538 | " scores[column] = s\n", 1539 | "\n", 1540 | "scores = pd.DataFrame([scores], index=['accuracy'])\n", 1541 | "\n", 1542 | "mapping_name = dict(zip(list(scores.columns), \n", 1543 | " ['Char ngram + LR', '(Word + Char ngram) + LR', \n", 1544 | " 'Word ngram + LR', 'CNN (multi channel)',\n", 1545 | " 'RNN + CNN', 'RNN no embd.', 'RNN + GloVe embds.']))\n", 1546 | "\n", 1547 | "scores = scores.rename(columns=mapping_name)\n", 1548 | "scores = scores[['Word ngram + LR', 'Char ngram + LR', '(Word + Char ngram) + LR',\n", 1549 | " 'RNN no embd.', 'RNN + GloVe embds.', 'CNN (multi channel)',\n", 1550 | " 'RNN + CNN']]\n", 1551 | "\n", 1552 | "scores = scores.T\n", 1553 | "\n", 1554 | "ax = scores['accuracy'].plot(kind='bar', \n", 1555 | " figsize=(16, 5), \n", 1556 | " ylim=(scores.accuracy.min()*0.97, scores.accuracy.max() * 1.01), \n", 1557 | " color='red', \n", 1558 | " alpha=0.75, \n", 1559 | " rot=45, \n", 1560 | " fontsize=13)\n", 1561 | "ax.set_title('Comparative accuracy of the different models')\n", 1562 | "\n", 1563 | "for i in ax.patches:\n", 1564 | " ax.annotate(str(round(i.get_height(), 3)), \n", 1565 | " (i.get_x() + 0.1, i.get_height() * 1.002), color='dimgrey', fontsize=14)" 1566 | ] 1567 | }, 1568 | { 1569 | "cell_type": "markdown", 1570 | "metadata": {}, 1571 | "source": [ 1572 | " ![](./images/article_5/benchmark.png)" 1573 | ] 1574 | }, 1575 | { 1576 | "cell_type": "markdown", 1577 | "metadata": {}, 1578 | "source": [ 1579 | "Let's quickly check the correlations between the predictions of the models." 1580 | ] 1581 | }, 1582 | { 1583 | "cell_type": "code", 1584 | "execution_count": null, 1585 | "metadata": {}, 1586 | "outputs": [], 1587 | "source": [ 1588 | "fig = plt.figure(figsize=(10, 5))\n", 1589 | "sns.heatmap(predictions.drop('y_true', axis=1).corr(method='kendall'), cmap=\"Blues\", annot=True);" 1590 | ] 1591 | }, 1592 | { 1593 | "cell_type": "markdown", 1594 | "metadata": {}, 1595 | "source": [ 1596 | " ![](./images/article_5/heatmap.png)" 1597 | ] 1598 | }, 1599 | { 1600 | "cell_type": "markdown", 1601 | "metadata": {}, 1602 | "source": [ 1603 | "# Conclusion\n", 1604 | "\n", 1605 | "Here are rapid findings I think are worth sharing:\n", 1606 | "\n", 1607 | "- Bag of word models using character ngrams can be very efficient. **Do not underestimate them!**. They are relatively cheap to compute, and also easy to interpret.\n", 1608 | "\n", 1609 | "- RNNs are powerful. However, you can sometimes pump them with external pre-trained embeddings like GloVe. You can also use other popular embeddings such as word2vec and FastText.\n", 1610 | "\n", 1611 | "- CNNs can be applied to text. They have the main advantage of being very fast to train. Besides, their ability to extract local features out of text is particularly interesting to nlp tasks.\n", 1612 | "\n", 1613 | "- RNNs and CNNs can be stacked together to take advantags of both architectures.\n", 1614 | "\n", 1615 | "This post was quite long, I hope you've enjoyed it. Don't hesitate to comment if you have any question or recommendation." 1616 | ] 1617 | }, 1618 | { 1619 | "cell_type": "markdown", 1620 | "metadata": {}, 1621 | "source": [ 1622 | "# Some helpful links to explore\n", 1623 | "\n", 1624 | "Here are great resources I used when writing this post:\n", 1625 | "\n", 1626 | "- http://colah.github.io/posts/2015-08-Understanding-LSTMs/\n", 1627 | "- http://wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/" 1628 | ] 1629 | } 1630 | ], 1631 | "metadata": { 1632 | "anaconda-cloud": {}, 1633 | "kernelspec": { 1634 | "display_name": "Python [conda env:tf]", 1635 | "language": "python", 1636 | "name": "conda-env-tf-py" 1637 | }, 1638 | "language_info": { 1639 | "codemirror_mode": { 1640 | "name": "ipython", 1641 | "version": 3 1642 | }, 1643 | "file_extension": ".py", 1644 | "mimetype": "text/x-python", 1645 | "name": "python", 1646 | "nbconvert_exporter": "python", 1647 | "pygments_lexer": "ipython3", 1648 | "version": "3.5.5" 1649 | }, 1650 | "widgets": { 1651 | "state": { 1652 | "dc45800408dc4d6680485931afacfa80": { 1653 | "views": [ 1654 | { 1655 | "cell_index": 4 1656 | } 1657 | ] 1658 | } 1659 | }, 1660 | "version": "1.2.0" 1661 | } 1662 | }, 1663 | "nbformat": 4, 1664 | "nbformat_minor": 2 1665 | } 1666 | -------------------------------------------------------------------------------- /images/article_5/ai.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahmedbesbes/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification/40f521d495dc89d7744aa932d451324b727e914d/images/article_5/ai.gif -------------------------------------------------------------------------------- /images/article_5/benchmark.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahmedbesbes/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification/40f521d495dc89d7744aa932d451324b727e914d/images/article_5/benchmark.png -------------------------------------------------------------------------------- /images/article_5/cnn_model_multi_channel.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahmedbesbes/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification/40f521d495dc89d7744aa932d451324b727e914d/images/article_5/cnn_model_multi_channel.png -------------------------------------------------------------------------------- /images/article_5/cnn_text.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahmedbesbes/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification/40f521d495dc89d7744aa932d451324b727e914d/images/article_5/cnn_text.png -------------------------------------------------------------------------------- /images/article_5/cover.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahmedbesbes/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification/40f521d495dc89d7744aa932d451324b727e914d/images/article_5/cover.jpg -------------------------------------------------------------------------------- /images/article_5/cover.xcf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahmedbesbes/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification/40f521d495dc89d7744aa932d451324b727e914d/images/article_5/cover.xcf -------------------------------------------------------------------------------- /images/article_5/cover_resized.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahmedbesbes/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification/40f521d495dc89d7744aa932d451324b727e914d/images/article_5/cover_resized.png -------------------------------------------------------------------------------- /images/article_5/deep.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahmedbesbes/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification/40f521d495dc89d7744aa932d451324b727e914d/images/article_5/deep.jpg -------------------------------------------------------------------------------- /images/article_5/heatmap.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahmedbesbes/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification/40f521d495dc89d7744aa932d451324b727e914d/images/article_5/heatmap.png -------------------------------------------------------------------------------- /images/article_5/investigate.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahmedbesbes/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification/40f521d495dc89d7744aa932d451324b727e914d/images/article_5/investigate.jpg -------------------------------------------------------------------------------- /images/article_5/ngrams.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahmedbesbes/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification/40f521d495dc89d7744aa932d451324b727e914d/images/article_5/ngrams.png -------------------------------------------------------------------------------- /images/article_5/ngrams_char.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahmedbesbes/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification/40f521d495dc89d7744aa932d451324b727e914d/images/article_5/ngrams_char.jpg -------------------------------------------------------------------------------- /images/article_5/rnn_cnn_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahmedbesbes/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification/40f521d495dc89d7744aa932d451324b727e914d/images/article_5/rnn_cnn_model.png -------------------------------------------------------------------------------- /images/article_5/rnn_simple_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahmedbesbes/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification/40f521d495dc89d7744aa932d451324b727e914d/images/article_5/rnn_simple_model.png -------------------------------------------------------------------------------- /images/article_5/rnn_unrolled.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahmedbesbes/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification/40f521d495dc89d7744aa932d451324b727e914d/images/article_5/rnn_unrolled.png -------------------------------------------------------------------------------- /images/article_5/tfidf.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahmedbesbes/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification/40f521d495dc89d7744aa932d451324b727e914d/images/article_5/tfidf.jpg -------------------------------------------------------------------------------- /models/lr_char_ngram.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahmedbesbes/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification/40f521d495dc89d7744aa932d451324b727e914d/models/lr_char_ngram.pkl -------------------------------------------------------------------------------- /models/lr_word_char_ngram.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahmedbesbes/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification/40f521d495dc89d7744aa932d451324b727e914d/models/lr_word_char_ngram.pkl -------------------------------------------------------------------------------- /models/lr_word_ngram.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahmedbesbes/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification/40f521d495dc89d7744aa932d451324b727e914d/models/lr_word_ngram.pkl --------------------------------------------------------------------------------