![\"The](\"https://s3.amazonaws.com/assets.datacamp.com/production/project_158/img/nips_logo.png\"){kind=logo}
\n",
96 | "\n",
97 | "Let’s start by looking at the content of the file\n",
98 | "\n",
99 | "** **\n",
100 | "#### Step 1: Loading Data\n",
101 | "** **\n",
102 | "\n",
103 | "For this tutorial, we’ll use the dataset of papers published in NIPS conference. The NIPS conference (Neural Information Processing Systems) is one of the most prestigious yearly events in the machine learning community. The CSV data file contains information on the different NIPS papers that were published from 1987 until 2016 (29 years!). These papers discuss a wide variety of topics in machine learning, from neural networks to optimization methods, and many more.\n",
104 | "\n",
105 | "Let’s start by looking at the content of the file"
106 | ]
107 | },
108 | {
109 | "cell_type": "code",
110 | "execution_count": null,
111 | "metadata": {},
112 | "outputs": [],
113 | "source": [
114 | "import zipfile\n",
115 | "import pandas as pd\n",
116 | "import os\n",
117 | "\n",
118 | "# Open the zip file\n",
119 | "with zipfile.ZipFile(\"./data/NIPS Papers.zip\", \"r\") as zip_ref:\n",
120 | " # Extract the file to a temporary directory\n",
121 | " zip_ref.extractall(\"temp\")\n",
122 | "\n",
123 | "# Read the CSV file into a pandas DataFrame\n",
124 | "papers = pd.read_csv(\"temp/NIPS Papers/papers.csv\")\n",
125 | "\n",
126 | "# Print head\n",
127 | "papers.head()"
128 | ]
129 | },
130 | {
131 | "cell_type": "markdown",
132 | "metadata": {},
133 | "source": [
134 | "** **\n",
135 | "#### Step 2: Data Cleaning\n",
136 | "** **\n",
137 | "\n",
138 | "Since the goal of this analysis is to perform topic modeling, we will solely focus on the text data from each paper, and drop other metadata columns"
139 | ]
140 | },
141 | {
142 | "cell_type": "code",
143 | "execution_count": null,
144 | "metadata": {},
145 | "outputs": [],
146 | "source": [
147 | "# Remove the columns\n",
148 | "papers = papers.drop(columns=['id', 'title', 'abstract', \n",
149 | " 'event_type', 'pdf_name', 'year'], axis=1)\n",
150 | "\n",
151 | "# sample only 100 papers\n",
152 | "papers = papers.sample(100)\n",
153 | "\n",
154 | "# Print out the first rows of papers\n",
155 | "papers.head()"
156 | ]
157 | },
158 | {
159 | "cell_type": "markdown",
160 | "metadata": {},
161 | "source": [
162 | "##### Remove punctuation/lower casing\n",
163 | "\n",
164 | "Next, let’s perform a simple preprocessing on the content of paper_text column to make them more amenable for analysis, and reliable results. To do that, we’ll use a regular expression to remove any punctuation, and then lowercase the text"
165 | ]
166 | },
167 | {
168 | "cell_type": "code",
169 | "execution_count": null,
170 | "metadata": {},
171 | "outputs": [],
172 | "source": [
173 | "# Load the regular expression library\n",
174 | "import re\n",
175 | "\n",
176 | "# Remove punctuation\n",
177 | "papers['paper_text_processed'] = papers['paper_text'].map(lambda x: re.sub('[,\\.!?]', '', x))\n",
178 | "\n",
179 | "# Convert the titles to lowercase\n",
180 | "papers['paper_text_processed'] = papers['paper_text_processed'].map(lambda x: x.lower())\n",
181 | "\n",
182 | "# Print out the first rows of papers\n",
183 | "papers['paper_text_processed'].head()"
184 | ]
185 | },
186 | {
187 | "cell_type": "markdown",
188 | "metadata": {},
189 | "source": [
190 | "##### Tokenize words and further clean-up text\n",
191 | "\n",
192 | "Let’s tokenize each sentence into a list of words, removing punctuations and unnecessary characters altogether."
193 | ]
194 | },
195 | {
196 | "cell_type": "code",
197 | "execution_count": null,
198 | "metadata": {},
199 | "outputs": [],
200 | "source": [
201 | "import gensim\n",
202 | "from gensim.utils import simple_preprocess\n",
203 | "\n",
204 | "def sent_to_words(sentences):\n",
205 | " for sentence in sentences:\n",
206 | " yield(gensim.utils.simple_preprocess(str(sentence), deacc=True)) # deacc=True removes punctuations\n",
207 | "\n",
208 | "data = papers.paper_text_processed.values.tolist()\n",
209 | "data_words = list(sent_to_words(data))\n",
210 | "\n",
211 | "print(data_words[:1][0][:30])"
212 | ]
213 | },
214 | {
215 | "cell_type": "markdown",
216 | "metadata": {},
217 | "source": [
218 | "** **\n",
219 | "#### Step 3: Phrase Modeling: Bigram and Trigram Models\n",
220 | "** **\n",
221 | "\n",
222 | "Bigrams are two words frequently occurring together in the document. Trigrams are 3 words frequently occurring. Some examples in our example are: 'back_bumper', 'oil_leakage', 'maryland_college_park' etc.\n",
223 | "\n",
224 | "Gensim's Phrases model can build and implement the bigrams, trigrams, quadgrams and more. The two important arguments to Phrases are min_count and threshold.\n",
225 | "\n",
226 | "*The higher the values of these param, the harder it is for words to be combined.*"
227 | ]
228 | },
229 | {
230 | "cell_type": "code",
231 | "execution_count": null,
232 | "metadata": {},
233 | "outputs": [],
234 | "source": [
235 | "# Build the bigram and trigram models\n",
236 | "bigram = gensim.models.Phrases(data_words, min_count=5, threshold=100) # higher threshold fewer phrases.\n",
237 | "trigram = gensim.models.Phrases(bigram[data_words], threshold=100) \n",
238 | "\n",
239 | "# Faster way to get a sentence clubbed as a trigram/bigram\n",
240 | "bigram_mod = gensim.models.phrases.Phraser(bigram)\n",
241 | "trigram_mod = gensim.models.phrases.Phraser(trigram)"
242 | ]
243 | },
244 | {
245 | "cell_type": "markdown",
246 | "metadata": {},
247 | "source": [
248 | "#### Remove Stopwords, Make Bigrams and Lemmatize\n",
249 | "\n",
250 | "The phrase models are ready. Let’s define the functions to remove the stopwords, make trigrams and lemmatization and call them sequentially."
251 | ]
252 | },
253 | {
254 | "cell_type": "code",
255 | "execution_count": null,
256 | "metadata": {},
257 | "outputs": [],
258 | "source": [
259 | "# NLTK Stop words\n",
260 | "import nltk\n",
261 | "nltk.download('stopwords')\n",
262 | "from nltk.corpus import stopwords\n",
263 | "\n",
264 | "stop_words = stopwords.words('english')\n",
265 | "stop_words.extend(['from', 'subject', 're', 'edu', 'use'])"
266 | ]
267 | },
268 | {
269 | "cell_type": "code",
270 | "execution_count": null,
271 | "metadata": {},
272 | "outputs": [],
273 | "source": [
274 | "# Define functions for stopwords, bigrams, trigrams and lemmatization\n",
275 | "def remove_stopwords(texts):\n",
276 | " return [[word for word in simple_preprocess(str(doc)) if word not in stop_words] for doc in texts]\n",
277 | "\n",
278 | "def make_bigrams(texts):\n",
279 | " return [bigram_mod[doc] for doc in texts]\n",
280 | "\n",
281 | "def make_trigrams(texts):\n",
282 | " return [trigram_mod[bigram_mod[doc]] for doc in texts]\n",
283 | "\n",
284 | "def lemmatization(texts, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV']):\n",
285 | " \"\"\"https://spacy.io/api/annotation\"\"\"\n",
286 | " texts_out = []\n",
287 | " for sent in texts:\n",
288 | " doc = nlp(\" \".join(sent)) \n",
289 | " texts_out.append([token.lemma_ for token in doc if token.pos_ in allowed_postags])\n",
290 | " return texts_out"
291 | ]
292 | },
293 | {
294 | "cell_type": "markdown",
295 | "metadata": {},
296 | "source": [
297 | "Let's call the functions in order."
298 | ]
299 | },
300 | {
301 | "cell_type": "code",
302 | "execution_count": null,
303 | "metadata": {},
304 | "outputs": [],
305 | "source": [
306 | "!python -m spacy download en_core_web_sm"
307 | ]
308 | },
309 | {
310 | "cell_type": "code",
311 | "execution_count": null,
312 | "metadata": {},
313 | "outputs": [],
314 | "source": [
315 | "import spacy\n",
316 | "\n",
317 | "# Remove Stop Words\n",
318 | "data_words_nostops = remove_stopwords(data_words)\n",
319 | "\n",
320 | "# Form Bigrams\n",
321 | "data_words_bigrams = make_bigrams(data_words_nostops)\n",
322 | "\n",
323 | "# Initialize spacy 'en' model, keeping only tagger component (for efficiency)\n",
324 | "nlp = spacy.load(\"en_core_web_sm\", disable=['parser', 'ner'])\n",
325 | "\n",
326 | "# Do lemmatization keeping only noun, adj, vb, adv\n",
327 | "data_lemmatized = lemmatization(data_words_bigrams, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV'])\n",
328 | "\n",
329 | "print(data_lemmatized[:1][0][:30])"
330 | ]
331 | },
332 | {
333 | "cell_type": "markdown",
334 | "metadata": {},
335 | "source": [
336 | "** **\n",
337 | "#### Step 4: Data transformation: Corpus and Dictionary\n",
338 | "** **\n",
339 | "\n",
340 | "The two main inputs to the LDA topic model are the dictionary(id2word) and the corpus. Let’s create them."
341 | ]
342 | },
343 | {
344 | "cell_type": "code",
345 | "execution_count": null,
346 | "metadata": {},
347 | "outputs": [],
348 | "source": [
349 | "import gensim.corpora as corpora\n",
350 | "\n",
351 | "# Create Dictionary\n",
352 | "id2word = corpora.Dictionary(data_lemmatized)\n",
353 | "\n",
354 | "# Create Corpus\n",
355 | "texts = data_lemmatized\n",
356 | "\n",
357 | "# Term Document Frequency\n",
358 | "corpus = [id2word.doc2bow(text) for text in texts]\n",
359 | "\n",
360 | "# View\n",
361 | "print(corpus[:1][0][:30])"
362 | ]
363 | },
364 | {
365 | "cell_type": "markdown",
366 | "metadata": {},
367 | "source": [
368 | "** **\n",
369 | "#### Step 5: Base Model \n",
370 | "** **\n",
371 | "\n",
372 | "We have everything required to train the base LDA model. In addition to the corpus and dictionary, you need to provide the number of topics as well. Apart from that, alpha and eta are hyperparameters that affect sparsity of the topics. According to the Gensim docs, both defaults to 1.0/num_topics prior (we'll use default for the base model).\n",
373 | "\n",
374 | "chunksize controls how many documents are processed at a time in the training algorithm. Increasing chunksize will speed up training, at least as long as the chunk of documents easily fit into memory.\n",
375 | "\n",
376 | "passes controls how often we train the model on the entire corpus (set to 10). Another word for passes might be \"epochs\". iterations is somewhat technical, but essentially it controls how often we repeat a particular loop over each document. It is important to set the number of \"passes\" and \"iterations\" high enough."
377 | ]
378 | },
379 | {
380 | "cell_type": "code",
381 | "execution_count": null,
382 | "metadata": {},
383 | "outputs": [],
384 | "source": [
385 | "# Build LDA model\n",
386 | "lda_model = gensim.models.LdaMulticore(corpus=corpus,\n",
387 | " id2word=id2word,\n",
388 | " num_topics=10, \n",
389 | " random_state=100,\n",
390 | " chunksize=100,\n",
391 | " passes=10,\n",
392 | " per_word_topics=True)"
393 | ]
394 | },
395 | {
396 | "cell_type": "markdown",
397 | "metadata": {},
398 | "source": [
399 | "** **\n",
400 | "The above LDA model is built with 10 different topics where each topic is a combination of keywords and each keyword contributes a certain weightage to the topic.\n",
401 | "\n",
402 | "You can see the keywords for each topic and the weightage(importance) of each keyword using `lda_model.print_topics()`"
403 | ]
404 | },
405 | {
406 | "cell_type": "code",
407 | "execution_count": null,
408 | "metadata": {},
409 | "outputs": [],
410 | "source": [
411 | "from pprint import pprint\n",
412 | "\n",
413 | "# Print the Keyword in the 10 topics\n",
414 | "pprint(lda_model.print_topics())\n",
415 | "doc_lda = lda_model[corpus]"
416 | ]
417 | },
418 | {
419 | "cell_type": "markdown",
420 | "metadata": {},
421 | "source": [
422 | "#### Compute Model Perplexity and Coherence Score\n",
423 | "\n",
424 | "Let's calculate the baseline coherence score"
425 | ]
426 | },
427 | {
428 | "cell_type": "code",
429 | "execution_count": null,
430 | "metadata": {},
431 | "outputs": [],
432 | "source": [
433 | "from gensim.models import CoherenceModel\n",
434 | "\n",
435 | "# Compute Coherence Score\n",
436 | "coherence_model_lda = CoherenceModel(model=lda_model, texts=data_lemmatized, dictionary=id2word, coherence='c_v')\n",
437 | "coherence_lda = coherence_model_lda.get_coherence()\n",
438 | "print('Coherence Score: ', coherence_lda)"
439 | ]
440 | },
441 | {
442 | "cell_type": "markdown",
443 | "metadata": {},
444 | "source": [
445 | "** **\n",
446 | "#### Step 6: Hyperparameter tuning\n",
447 | "** **\n",
448 | "First, let's differentiate between model hyperparameters and model parameters :\n",
449 | "\n",
450 | "- `Model hyperparameters` can be thought of as settings for a machine learning algorithm that are tuned by the data scientist before training. Examples would be the number of trees in the random forest, or in our case, number of topics K\n",
451 | "\n",
452 | "- `Model parameters` can be thought of as what the model learns during training, such as the weights for each word in a given topic.\n",
453 | "\n",
454 | "Now that we have the baseline coherence score for the default LDA model, let's perform a series of sensitivity tests to help determine the following model hyperparameters: \n",
455 | "- Number of Topics (K)\n",
456 | "- Dirichlet hyperparameter alpha: Document-Topic Density\n",
457 | "- Dirichlet hyperparameter beta: Word-Topic Density\n",
458 | "\n",
459 | "We'll perform these tests in sequence, one parameter at a time by keeping others constant and run them over the two difference validation corpus sets. We'll use `C_v` as our choice of metric for performance comparison "
460 | ]
461 | },
462 | {
463 | "cell_type": "code",
464 | "execution_count": null,
465 | "metadata": {},
466 | "outputs": [],
467 | "source": [
468 | "# supporting function\n",
469 | "def compute_coherence_values(corpus, dictionary, k, a, b):\n",
470 | " \n",
471 | " lda_model = gensim.models.LdaMulticore(corpus=corpus,\n",
472 | " id2word=dictionary,\n",
473 | " num_topics=k, \n",
474 | " random_state=100,\n",
475 | " chunksize=100,\n",
476 | " passes=10,\n",
477 | " alpha=a,\n",
478 | " eta=b)\n",
479 | " \n",
480 | " coherence_model_lda = CoherenceModel(model=lda_model, texts=data_lemmatized, dictionary=id2word, coherence='c_v')\n",
481 | " \n",
482 | " return coherence_model_lda.get_coherence()"
483 | ]
484 | },
485 | {
486 | "cell_type": "markdown",
487 | "metadata": {},
488 | "source": [
489 | "Let's call the function, and iterate it over the range of topics, alpha, and beta parameter values"
490 | ]
491 | },
492 | {
493 | "cell_type": "code",
494 | "execution_count": null,
495 | "metadata": {},
496 | "outputs": [],
497 | "source": [
498 | "import numpy as np\n",
499 | "import tqdm\n",
500 | "\n",
501 | "grid = {}\n",
502 | "grid['Validation_Set'] = {}\n",
503 | "\n",
504 | "# Topics range\n",
505 | "min_topics = 2\n",
506 | "max_topics = 11\n",
507 | "step_size = 1\n",
508 | "topics_range = range(min_topics, max_topics, step_size)\n",
509 | "\n",
510 | "# Alpha parameter\n",
511 | "alpha = list(np.arange(0.01, 1, 0.3))\n",
512 | "alpha.append('symmetric')\n",
513 | "alpha.append('asymmetric')\n",
514 | "\n",
515 | "# Beta parameter\n",
516 | "beta = list(np.arange(0.01, 1, 0.3))\n",
517 | "beta.append('symmetric')\n",
518 | "\n",
519 | "# Validation sets\n",
520 | "num_of_docs = len(corpus)\n",
521 | "corpus_sets = [gensim.utils.ClippedCorpus(corpus, int(num_of_docs*0.75)), \n",
522 | " corpus]\n",
523 | "\n",
524 | "corpus_title = ['75% Corpus', '100% Corpus']\n",
525 | "\n",
526 | "model_results = {'Validation_Set': [],\n",
527 | " 'Topics': [],\n",
528 | " 'Alpha': [],\n",
529 | " 'Beta': [],\n",
530 | " 'Coherence': []\n",
531 | " }\n",
532 | "\n",
533 | "# Can take a long time to run\n",
534 | "if 1 == 1:\n",
535 | " pbar = tqdm.tqdm(total=(len(beta)*len(alpha)*len(topics_range)*len(corpus_title)))\n",
536 | " \n",
537 | " # iterate through validation corpuses\n",
538 | " for i in range(len(corpus_sets)):\n",
539 | " # iterate through number of topics\n",
540 | " for k in topics_range:\n",
541 | " # iterate through alpha values\n",
542 | " for a in alpha:\n",
543 | " # iterare through beta values\n",
544 | " for b in beta:\n",
545 | " # get the coherence score for the given parameters\n",
546 | " cv = compute_coherence_values(corpus=corpus_sets[i], dictionary=id2word, \n",
547 | " k=k, a=a, b=b)\n",
548 | " # Save the model results\n",
549 | " model_results['Validation_Set'].append(corpus_title[i])\n",
550 | " model_results['Topics'].append(k)\n",
551 | " model_results['Alpha'].append(a)\n",
552 | " model_results['Beta'].append(b)\n",
553 | " model_results['Coherence'].append(cv)\n",
554 | " \n",
555 | " pbar.update(1)\n",
556 | " pd.DataFrame(model_results).to_csv('./results/lda_tuning_results.csv', index=False)\n",
557 | " pbar.close()"
558 | ]
559 | },
560 | {
561 | "cell_type": "markdown",
562 | "metadata": {},
563 | "source": [
564 | "** **\n",
565 | "#### Step 7: Final Model\n",
566 | "** **\n",
567 | "\n",
568 | "Based on external evaluation (Code to be added from Excel based analysis), let's train the final model with parameters yielding highest coherence score"
569 | ]
570 | },
571 | {
572 | "cell_type": "code",
573 | "execution_count": null,
574 | "metadata": {},
575 | "outputs": [],
576 | "source": [
577 | "num_topics = 8\n",
578 | "\n",
579 | "lda_model = gensim.models.LdaMulticore(corpus=corpus,\n",
580 | " id2word=id2word,\n",
581 | " num_topics=num_topics, \n",
582 | " random_state=100,\n",
583 | " chunksize=100,\n",
584 | " passes=10,\n",
585 | " alpha=0.01,\n",
586 | " eta=0.9)"
587 | ]
588 | },
589 | {
590 | "cell_type": "code",
591 | "execution_count": null,
592 | "metadata": {},
593 | "outputs": [],
594 | "source": [
595 | "from pprint import pprint\n",
596 | "\n",
597 | "# Print the Keyword in the 10 topics\n",
598 | "pprint(lda_model.print_topics())\n",
599 | "doc_lda = lda_model[corpus]"
600 | ]
601 | },
602 | {
603 | "cell_type": "markdown",
604 | "metadata": {},
605 | "source": [
606 | "** **\n",
607 | "#### Step 8: Visualize Results\n",
608 | "** **"
609 | ]
610 | },
611 | {
612 | "cell_type": "code",
613 | "execution_count": null,
614 | "metadata": {},
615 | "outputs": [],
616 | "source": [
617 | "import pyLDAvis.gensim_models as gensimvis\n",
618 | "import pickle \n",
619 | "import pyLDAvis\n",
620 | "\n",
621 | "# Visualize the topics\n",
622 | "pyLDAvis.enable_notebook()\n",
623 | "\n",
624 | "LDAvis_data_filepath = os.path.join('./results/ldavis_tuned_'+str(num_topics))\n",
625 | "\n",
626 | "# # this is a bit time consuming - make the if statement True\n",
627 | "# # if you want to execute visualization prep yourself\n",
628 | "if 1 == 1:\n",
629 | " LDAvis_prepared = gensimvis.prepare(lda_model, corpus, id2word)\n",
630 | " with open(LDAvis_data_filepath, 'wb') as f:\n",
631 | " pickle.dump(LDAvis_prepared, f)\n",
632 | "\n",
633 | "# load the pre-prepared pyLDAvis data from disk\n",
634 | "with open(LDAvis_data_filepath, 'rb') as f:\n",
635 | " LDAvis_prepared = pickle.load(f)\n",
636 | "\n",
637 | "pyLDAvis.save_html(LDAvis_prepared, './results/ldavis_tuned_'+ str(num_topics) +'.html')\n",
638 | "\n",
639 | "LDAvis_prepared"
640 | ]
641 | },
642 | {
643 | "cell_type": "markdown",
644 | "metadata": {},
645 | "source": [
646 | "** **\n",
647 | "#### Closing Notes\n",
648 | "\n",
649 | "We started with understanding why evaluating the topic model is essential. Next, we reviewed existing methods and scratched the surface of topic coherence, along with the available coherence measures. Then we built a default LDA model using Gensim implementation to establish the baseline coherence score and reviewed practical ways to optimize the LDA hyperparameters.\n",
650 | "\n",
651 | "Hopefully, this article has managed to shed light on the underlying topic evaluation strategies, and intuitions behind it.\n",
652 | "\n",
653 | "** **\n",
654 | "#### References:\n",
655 | "1. http://qpleple.com/perplexity-to-evaluate-topic-models/\n",
656 | "2. https://www.amazon.com/Machine-Learning-Probabilistic-Perspective-Computation/dp/0262018020\n",
657 | "3. https://papers.nips.cc/paper/3700-reading-tea-leaves-how-humans-interpret-topic-models.pdf\n",
658 | "4. https://github.com/mattilyra/pydataberlin-2017/blob/master/notebook/EvaluatingUnsupervisedModels.ipynb\n",
659 | "5. https://www.machinelearningplus.com/nlp/topic-modeling-gensim-python/\n",
660 | "6. http://svn.aksw.org/papers/2015/WSDM_Topic_Evaluation/public.pdf\n",
661 | "7. http://palmetto.aksw.org/palmetto-webapp/"
662 | ]
663 | }
664 | ],
665 | "metadata": {
666 | "kernelspec": {
667 | "display_name": "Python 3 (ipykernel)",
668 | "language": "python",
669 | "name": "python3"
670 | },
671 | "language_info": {
672 | "codemirror_mode": {
673 | "name": "ipython",
674 | "version": 3
675 | },
676 | "file_extension": ".py",
677 | "mimetype": "text/x-python",
678 | "name": "python",
679 | "nbconvert_exporter": "python",
680 | "pygments_lexer": "ipython3",
681 | "version": "3.9.16"
682 | }
683 | },
684 | "nbformat": 4,
685 | "nbformat_minor": 4
686 | }
687 |
--------------------------------------------------------------------------------
/natural-language-processing/topic-modeling/Introduction to Topic Modeling.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "## Introduction\n",
8 | "##### How to get started with topic modeling using LDA in Python\n",
9 | "** **\n",
10 | "Topic Models, in a nutshell, are a type of statistical language models used for uncovering hidden structure in a collection of texts. In a practical and more intuitively, you can think of it as a task of:\n",
11 | "\n",
12 | "- **Dimensionality Reduction**, where rather than representing a text T in its feature space as {Word_i: count(Word_i, T) for Word_i in Vocabulary}, you can represent it in a topic space as {Topic_i: Weight(Topic_i, T) for Topic_i in Topics}\n",
13 | "- **Unsupervised Learning**, where it can be compared to clustering, as in the case of clustering, the number of topics, like the number of clusters, is an output parameter. By doing topic modeling, we build clusters of words rather than clusters of texts. A text is thus a mixture of all the topics, each having a specific weight\n",
14 | "- **Tagging**, abstract “topics” that occur in a collection of documents that best represents the information in them.\n",
15 | "\n",
16 | "There are several existing algorithms you can use to perform the topic modeling. The most common of it are, Latent Semantic Analysis (LSA/LSI), Probabilistic Latent Semantic Analysis (pLSA), and Latent Dirichlet Allocation (LDA)\n",
17 | "\n",
18 | "In this tutorial, we’ll take a closer look at LDA, and implement our first topic model using the sklearn implementation in python 2.7\n",
19 | "\n",
20 | "### Theoretical Overview\n",
21 | "LDA is a generative probabilistic model that assumes each topic is a mixture over an underlying set of words, and each document is a mixture of over a set of topic probabilities.\n",
22 | "\n",
23 | "\n",
24 | "\n",
25 | "We can describe the generative process of LDA as, given the M number of documents, N number of words, and prior K number of topics, the model trains to output:\n",
26 | "\n",
27 | "- `psi`, the distribution of words for each topic K\n",
28 | "- `phi`, the distribution of topics for each document i\n",
29 | "\n",
30 | "#### Parameters of LDA\n",
31 | "\n",
32 | "- `Alpha parameter` is Dirichlet prior concentration parameter that represents document-topic density — with a higher alpha, documents are assumed to be made up of more topics and result in more specific topic distribution per document.\n",
33 | "- `Beta parameter` is the same prior concentration parameter that represents topic-word density — with high beta, topics are assumed to made of up most of the words and result in a more specific word distribution per topic.\n",
34 | "\n",
35 | "**To read more: https://towardsdatascience.com/end-to-end-topic-modeling-in-python-latent-dirichlet-allocation-lda-35ce4ed6b3e0**"
36 | ]
37 | },
38 | {
39 | "cell_type": "markdown",
40 | "metadata": {},
41 | "source": [
42 | "** **\n",
43 | "### LDA Implementation\n",
44 | "\n",
45 | "1. [Loading data](#load_data)\n",
46 | "2. [Data cleaning](#clean_data)\n",
47 | "3. [Exploratory analysis](#eda)\n",
48 | "4. [Prepare data for LDA analysis](#data_preparation)\n",
49 | "5. [LDA model training](#train_model)\n",
50 | "6. [Analyzing LDA model results](#results)"
51 | ]
52 | },
53 | {
54 | "cell_type": "markdown",
55 | "metadata": {},
56 | "source": [
57 | "** **\n",
58 | "For this tutorial, we’ll use the dataset of papers published in NeurIPS (NIPS) conference which is one of the most prestigious yearly events in the machine learning community. The CSV data file contains information on the different NeurIPS papers that were published from 1987 until 2016 (29 years!). These papers discuss a wide variety of topics in machine learning, from neural networks to optimization methods, and many more.\n",
59 | "\n",
60 | "\n",
61 | "\n",
62 | "Let’s start by looking at the content of the file"
63 | ]
64 | },
65 | {
66 | "cell_type": "markdown",
67 | "metadata": {},
68 | "source": [
69 | "** **\n",
70 | "#### Step 1: Loading Data \n",
71 | "** **\n",
72 | "For this tutorial, we’ll use the dataset of papers published in NeurIPS (NIPS) conference which is one of the most prestigious yearly events in the machine learning community. The CSV data file contains information on the different NeurIPS papers that were published from 1987 until 2016 (29 years!). These papers discuss a wide variety of topics in machine learning, from neural networks to optimization methods, and many more.\n",
73 | "\n",
74 | "Let’s start by looking at the content of the file"
75 | ]
76 | },
77 | {
78 | "cell_type": "code",
79 | "execution_count": null,
80 | "metadata": {},
81 | "outputs": [],
82 | "source": [
83 | "import zipfile\n",
84 | "import pandas as pd\n",
85 | "import os\n",
86 | "\n",
87 | "# Open the zip file\n",
88 | "with zipfile.ZipFile(\"./data/NIPS Papers.zip\", \"r\") as zip_ref:\n",
89 | " # Extract the file to a temporary directory\n",
90 | " zip_ref.extractall(\"temp\")\n",
91 | "\n",
92 | "# Read the CSV file into a pandas DataFrame\n",
93 | "papers = pd.read_csv(\"temp/NIPS Papers/papers.csv\")\n",
94 | "\n",
95 | "# Print head\n",
96 | "papers.head()"
97 | ]
98 | },
99 | {
100 | "cell_type": "markdown",
101 | "metadata": {},
102 | "source": [
103 | "** **\n",
104 | "#### Step 2: Data Cleaning \n",
105 | "** **\n",
106 | "\n",
107 | "Since the goal of this analysis is to perform topic modeling, let's focus only on the text data from each paper, and drop other metadata columns. Also, for the demonstration, we'll only look at 100 papers"
108 | ]
109 | },
110 | {
111 | "cell_type": "code",
112 | "execution_count": null,
113 | "metadata": {},
114 | "outputs": [],
115 | "source": [
116 | "# Remove the columns\n",
117 | "papers = papers.drop(columns=['id', 'event_type', 'pdf_name'], axis=1).sample(100)\n",
118 | "\n",
119 | "# Print out the first rows of papers\n",
120 | "papers.head()"
121 | ]
122 | },
123 | {
124 | "cell_type": "markdown",
125 | "metadata": {},
126 | "source": [
127 | "##### Remove punctuation/lower casing\n",
128 | "\n",
129 | "Next, let’s perform a simple preprocessing on the content of `paper_text` column to make them more amenable for analysis, and reliable results. To do that, we’ll use a regular expression to remove any punctuation, and then lowercase the text"
130 | ]
131 | },
132 | {
133 | "cell_type": "code",
134 | "execution_count": null,
135 | "metadata": {},
136 | "outputs": [],
137 | "source": [
138 | "# Load the regular expression library\n",
139 | "import re\n",
140 | "\n",
141 | "# Remove punctuation\n",
142 | "papers['paper_text_processed'] = \\\n",
143 | "papers['paper_text'].map(lambda x: re.sub('[,\\.!?]', '', x))\n",
144 | "\n",
145 | "# Convert the titles to lowercase\n",
146 | "papers['paper_text_processed'] = \\\n",
147 | "papers['paper_text_processed'].map(lambda x: x.lower())\n",
148 | "\n",
149 | "# Print out the first rows of papers\n",
150 | "papers['paper_text_processed'].head()"
151 | ]
152 | },
153 | {
154 | "cell_type": "markdown",
155 | "metadata": {},
156 | "source": [
157 | "** **\n",
158 | "#### Step 3: Exploratory Analysis \n",
159 | "** **\n",
160 | "\n",
161 | "To verify whether the preprocessing, we’ll make a simple word cloud using the `wordcloud` package to get a visual representation of most common words. It is key to understanding the data and ensuring we are on the right track, and if any more preprocessing is necessary before training the model."
162 | ]
163 | },
164 | {
165 | "cell_type": "code",
166 | "execution_count": null,
167 | "metadata": {},
168 | "outputs": [],
169 | "source": [
170 | "# Import the wordcloud library\n",
171 | "from wordcloud import WordCloud\n",
172 | "\n",
173 | "# Join the different processed titles together.\n",
174 | "long_string = ','.join(list(papers['paper_text_processed'].values))\n",
175 | "\n",
176 | "# Create a WordCloud object\n",
177 | "wordcloud = WordCloud(background_color=\"white\", max_words=1000, contour_width=3, contour_color='steelblue')\n",
178 | "\n",
179 | "# Generate a word cloud\n",
180 | "wordcloud.generate(long_string)\n",
181 | "\n",
182 | "# Visualize the word cloud\n",
183 | "wordcloud.to_image()"
184 | ]
185 | },
186 | {
187 | "cell_type": "markdown",
188 | "metadata": {},
189 | "source": [
190 | "** **\n",
191 | "#### Step 4: Prepare text for LDA analysis \n",
192 | "** **\n",
193 | "\n",
194 | "Next, let’s work to transform the textual data in a format that will serve as an input for training LDA model. We start by tokenizing the text and removing stopwords. Next, we convert the tokenized object into a corpus and dictionary."
195 | ]
196 | },
197 | {
198 | "cell_type": "code",
199 | "execution_count": null,
200 | "metadata": {},
201 | "outputs": [],
202 | "source": [
203 | "import gensim\n",
204 | "from gensim.utils import simple_preprocess\n",
205 | "import nltk\n",
206 | "nltk.download('stopwords')\n",
207 | "from nltk.corpus import stopwords\n",
208 | "\n",
209 | "stop_words = stopwords.words('english')\n",
210 | "stop_words.extend(['from', 'subject', 're', 'edu', 'use'])\n",
211 | "\n",
212 | "def sent_to_words(sentences):\n",
213 | " for sentence in sentences:\n",
214 | " # deacc=True removes punctuations\n",
215 | " yield(gensim.utils.simple_preprocess(str(sentence), deacc=True))\n",
216 | "\n",
217 | "def remove_stopwords(texts):\n",
218 | " return [[word for word in simple_preprocess(str(doc)) \n",
219 | " if word not in stop_words] for doc in texts]\n",
220 | "\n",
221 | "\n",
222 | "data = papers.paper_text_processed.values.tolist()\n",
223 | "data_words = list(sent_to_words(data))\n",
224 | "\n",
225 | "# remove stop words\n",
226 | "data_words = remove_stopwords(data_words)\n",
227 | "\n",
228 | "print(data_words[:1][0][:30])"
229 | ]
230 | },
231 | {
232 | "cell_type": "code",
233 | "execution_count": null,
234 | "metadata": {},
235 | "outputs": [],
236 | "source": [
237 | "import gensim.corpora as corpora\n",
238 | "\n",
239 | "# Create Dictionary\n",
240 | "id2word = corpora.Dictionary(data_words)\n",
241 | "\n",
242 | "# Create Corpus\n",
243 | "texts = data_words\n",
244 | "\n",
245 | "# Term Document Frequency\n",
246 | "corpus = [id2word.doc2bow(text) for text in texts]\n",
247 | "\n",
248 | "# View\n",
249 | "print(corpus[:1][0][:30])"
250 | ]
251 | },
252 | {
253 | "cell_type": "markdown",
254 | "metadata": {},
255 | "source": [
256 | "** **\n",
257 | "#### Step 5: LDA model tranining \n",
258 | "** **\n",
259 | "\n",
260 | "To keep things simple, we'll keep all the parameters to default except for inputting the number of topics. For this tutorial, we will build a model with 10 topics where each topic is a combination of keywords, and each keyword contributes a certain weightage to the topic."
261 | ]
262 | },
263 | {
264 | "cell_type": "code",
265 | "execution_count": null,
266 | "metadata": {},
267 | "outputs": [],
268 | "source": [
269 | "from pprint import pprint\n",
270 | "\n",
271 | "# number of topics\n",
272 | "num_topics = 10\n",
273 | "\n",
274 | "# Build LDA model\n",
275 | "lda_model = gensim.models.LdaMulticore(corpus=corpus,\n",
276 | " id2word=id2word,\n",
277 | " num_topics=num_topics)\n",
278 | "\n",
279 | "# Print the Keyword in the 10 topics\n",
280 | "pprint(lda_model.print_topics())\n",
281 | "doc_lda = lda_model[corpus]"
282 | ]
283 | },
284 | {
285 | "cell_type": "markdown",
286 | "metadata": {},
287 | "source": [
288 | "** **\n",
289 | "#### Step 6: Analyzing our LDA model \n",
290 | "** **\n",
291 | "\n",
292 | "Now that we have a trained model let’s visualize the topics for interpretability. To do so, we’ll use a popular visualization package, pyLDAvis which is designed to help interactively with:\n",
293 | "\n",
294 | "1. Better understanding and interpreting individual topics, and\n",
295 | "2. Better understanding the relationships between the topics.\n",
296 | "\n",
297 | "For (1), you can manually select each topic to view its top most frequent and/or “relevant” terms, using different values of the λ parameter. This can help when you’re trying to assign a human interpretable name or “meaning” to each topic.\n",
298 | "\n",
299 | "For (2), exploring the Intertopic Distance Plot can help you learn about how topics relate to each other, including potential higher-level structure between groups of topics."
300 | ]
301 | },
302 | {
303 | "cell_type": "code",
304 | "execution_count": null,
305 | "metadata": {},
306 | "outputs": [],
307 | "source": [
308 | "import pyLDAvis.gensim_models as gensimvis\n",
309 | "import pickle \n",
310 | "import pyLDAvis\n",
311 | "\n",
312 | "# Visualize the topics\n",
313 | "pyLDAvis.enable_notebook()\n",
314 | "\n",
315 | "LDAvis_data_filepath = os.path.join('./results/ldavis_prepared_'+str(num_topics))\n",
316 | "\n",
317 | "# # this is a bit time consuming - make the if statement True\n",
318 | "# # if you want to execute visualization prep yourself\n",
319 | "if 1 == 1:\n",
320 | " LDAvis_prepared = gensimvis.prepare(lda_model, corpus, id2word)\n",
321 | " with open(LDAvis_data_filepath, 'wb') as f:\n",
322 | " pickle.dump(LDAvis_prepared, f)\n",
323 | "\n",
324 | "# load the pre-prepared pyLDAvis data from disk\n",
325 | "with open(LDAvis_data_filepath, 'rb') as f:\n",
326 | " LDAvis_prepared = pickle.load(f)\n",
327 | "\n",
328 | "pyLDAvis.save_html(LDAvis_prepared, './results/ldavis_prepared_'+ str(num_topics) +'.html')\n",
329 | "\n",
330 | "LDAvis_prepared"
331 | ]
332 | },
333 | {
334 | "cell_type": "markdown",
335 | "metadata": {},
336 | "source": [
337 | "** **\n",
338 | "#### Closing Notes\n",
339 | "Machine learning has become increasingly popular over the past decade, and recent advances in computational availability have led to exponential growth to people looking for ways how new methods can be incorporated to advance the field of Natural Language Processing.\n",
340 | "\n",
341 | "Often, we treat topic models as black-box algorithms, but hopefully, this article addressed to shed light on the underlying math, and intuitions behind it, and high-level code to get you started with any textual data.\n",
342 | "\n",
343 | "In the next article, we’ll go one step deeper into understanding how you can evaluate the performance of topic models, tune its hyper-parameters to get more intuitive and reliable results.\n",
344 | "\n",
345 | "** **\n",
346 | "#### References:\n",
347 | "1. Topic model — Wikipedia. https://en.wikipedia.org/wiki/Topic_model\n",
348 | "2. Distributed Strategies for Topic Modeling. https://www.ideals.illinois.edu/bitstream/handle/2142/46405/ParallelTopicModels.pdf?sequence=2&isAllowed=y\n",
349 | "3. Topic Mapping — Software — Resources — Amaral Lab. https://amaral.northwestern.edu/resources/software/topic-mapping\n",
350 | "4. A Survey of Topic Modeling in Text Mining. https://thesai.org/Downloads/Volume6No1/Paper_21-A_Survey_of_Topic_Modeling_in_Text_Mining.pdf\n"
351 | ]
352 | }
353 | ],
354 | "metadata": {
355 | "kernelspec": {
356 | "display_name": "Python 3 (ipykernel)",
357 | "language": "python",
358 | "name": "python3"
359 | },
360 | "language_info": {
361 | "codemirror_mode": {
362 | "name": "ipython",
363 | "version": 3
364 | },
365 | "file_extension": ".py",
366 | "mimetype": "text/x-python",
367 | "name": "python",
368 | "nbconvert_exporter": "python",
369 | "pygments_lexer": "ipython3",
370 | "version": "3.9.16"
371 | }
372 | },
373 | "nbformat": 4,
374 | "nbformat_minor": 4
375 | }
376 |
--------------------------------------------------------------------------------
/natural-language-processing/topic-modeling/data/NIPS Papers.zip:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/kapadias/medium-articles/749c93d1e6373c7911d3076fa8bcc38997cef884/natural-language-processing/topic-modeling/data/NIPS Papers.zip
--------------------------------------------------------------------------------
/natural-language-processing/topic-modeling/pyproject.toml:
--------------------------------------------------------------------------------
1 | [tool.poetry]
2 | name = "topic-modeling"
3 | version = "0.1.0"
4 | description = ""
5 | authors = ["Shashank Kapadia
236 |
756 |
757 |
--------------------------------------------------------------------------------
237 |
239 | Overview
238 |
240 |
299 |
241 |
243 |
244 |
266 |
267 | Dataset info
242 || Number of variables | 246 |2 | 247 |
|---|---|
| Number of observations | 250 |2734350 | 251 |
| Total Missing (%) | 254 |0.0% | 255 |
| Total size in memory | 258 |41.7 MiB | 259 |
| Average record size in memory | 262 |16.0 B | 263 |
268 |
270 |
271 |
293 |
294 | Variables types
269 || Numeric | 273 |1 | 274 |
|---|---|
| Categorical | 277 |1 | 278 |
| Date | 281 |0 | 282 |
| Text (Unique) | 285 |0 | 286 |
| Rejected | 289 |0 | 290 |
295 |
298 | Warnings
296 |ratinghas 1505291 / 55.1% zeros- Dataset has 2734343 duplicate rows Warning
300 |
302 | Variables
301 |
303 |
304 |
is_read
305 | Categorical
306 |
308 |
309 |
326 |
327 | | Distinct count | 311 |2 | 312 |
|---|---|
| Unique (%) | 315 |0.0% | 316 |
| Missing (%) | 319 |0.0% | 320 |
| Missing (n) | 323 |0 | 324 |
328 |
329 |
349 |
350 | | false | 331 |332 | 336 | 337 | | 338 |
|---|---|
| true | 340 |341 | 345 | 346 | | 347 |
351 |
353 | Toggle details
354 |
355 |
356 |
357 |
358 |
359 |
360 |
383 |
384 | | Value | 362 |Count | 363 |Frequency (%) | 364 |365 | |
| false | 369 |1420740 | 370 |52.0% | 371 |372 | 373 | | 374 |
| true | 376 |1313610 | 377 |48.0% | 378 |379 | 380 | | 381 |
385 |
712 |
386 |
rating
387 | Numeric
388 |
390 |
443 |
391 |
442 |
392 |
393 |
418 |
419 |
420 | | Distinct count | 395 |6 | 396 |
|---|---|
| Unique (%) | 399 |0.0% | 400 |
| Missing (%) | 403 |0.0% | 404 |
| Missing (n) | 407 |0 | 408 |
| Infinite (%) | 411 |0.0% | 412 |
| Infinite (n) | 415 |0 | 416 |
421 |
422 |
423 |
440 |
441 | | Mean | 425 |1.8248 | 426 |
|---|---|
| Minimum | 429 |0 | 430 |
| Maximum | 433 |5 | 434 |
| Zeros (%) | 437 |55.1% | 438 |
444 | 
445 |
446 |
447 |
445 |
446 |
448 |
450 | Toggle details
451 |
452 |
453 |
454 |
466 |
467 |
711 |
468 |
710 |
469 |
552 |
470 |
472 |
509 |
510 | Quantile statistics
471 || Minimum | 474 |0 | 475 |
|---|---|
| 5-th percentile | 478 |0 | 479 |
| Q1 | 482 |0 | 483 |
| Median | 486 |0 | 487 |
| Q3 | 490 |4 | 491 |
| 95-th percentile | 494 |5 | 495 |
| Maximum | 498 |5 | 499 |
| Range | 502 |5 | 503 |
| Interquartile range | 506 |4 | 507 |
511 |
513 |
550 |
551 | Descriptive statistics
512 || Standard deviation | 515 |2.1232 | 516 |
|---|---|
| Coef of variation | 519 |1.1635 | 520 |
| Kurtosis | 523 |-1.6116 | 524 |
| Mean | 527 |1.8248 | 528 |
| MAD | 531 |2.0215 | 532 |
| Skewness | 535 |0.4371 | 536 |
| Sum | 539 |4989606 | 540 |
| Variance | 543 |4.5081 | 544 |
| Memory size | 547 |20.9 MiB | 548 |
553 | 
554 |
555 |
554 |
556 |
557 |
558 |
559 |
610 |
611 | | Value | 561 |Count | 562 |Frequency (%) | 563 |564 | |
| 0 | 568 |1505291 | 569 |55.1% | 570 |571 | 572 | | 573 |
| 5 | 575 |500971 | 576 |18.3% | 577 |578 | 579 | | 580 |
| 4 | 582 |405565 | 583 |14.8% | 584 |585 | 586 | | 587 |
| 3 | 589 |237942 | 590 |8.7% | 591 |592 | 593 | | 594 |
| 2 | 596 |64084 | 597 |2.3% | 598 |599 | 600 | | 601 |
| 1 | 603 |20497 | 604 |0.7% | 605 |606 | 607 | | 608 |
612 |
615 |
616 |
660 |
663 |
664 |
708 |
709 | Minimum 5 values
613 | 614 || Value | 618 |Count | 619 |Frequency (%) | 620 |621 | |
| 0 | 625 |1505291 | 626 |55.1% | 627 |628 | 629 | | 630 |
| 1 | 632 |20497 | 633 |0.7% | 634 |635 | 636 | | 637 |
| 2 | 639 |64084 | 640 |2.3% | 641 |642 | 643 | | 644 |
| 3 | 646 |237942 | 647 |8.7% | 648 |649 | 650 | | 651 |
| 4 | 653 |405565 | 654 |14.8% | 655 |656 | 657 | | 658 |
Maximum 5 values
661 | 662 || Value | 666 |Count | 667 |Frequency (%) | 668 |669 | |
| 1 | 673 |20497 | 674 |0.7% | 675 |676 | 677 | | 678 |
| 2 | 680 |64084 | 681 |2.3% | 682 |683 | 684 | | 685 |
| 3 | 687 |237942 | 688 |8.7% | 689 |690 | 691 | | 692 |
| 4 | 694 |405565 | 695 |14.8% | 696 |697 | 698 | | 699 |
| 5 | 701 |500971 | 702 |18.3% | 703 |704 | 705 | | 706 |
713 |
715 | Sample
714 |
716 |
754 |
755 |