├── NLP - Text processing pipeline.ipynb
├── README.md
└── text-processing-pipeline-classification-models.ipynb


/NLP - Text processing pipeline.ipynb:
--------------------------------------------------------------------------------
1 | {"cells":[{"metadata":{},"cell_type":"markdown","source":"This notebook contains functions required for text cleaning and processing pipeline in NLP problems.\nThese are ready-to-use functions and use NLTK and SKlearn packages."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport string as st\nimport re\nimport nltk\nfrom nltk import PorterStemmer, WordNetLemmatizer\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":2,"outputs":[{"output_type":"stream","text":"/kaggle/input/spam-text-message-classification/SPAM text message 20170820 - Data.csv\n","name":"stdout"}]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Read the data. Here it is already in .csv format.\ndata = pd.read_csv('../input/spam-text-message-classification/SPAM text message 20170820 - Data.csv')\ndata.head()","execution_count":3,"outputs":[{"output_type":"execute_result","execution_count":3,"data":{"text/plain":"  Category                                            Message\n0      ham  Go until jurong point, crazy.. Available only ...\n1      ham                      Ok lar... Joking wif u oni...\n2     spam  Free entry in 2 a wkly comp to win FA Cup fina...\n3      ham  U dun say so early hor... U c already then say...\n4      ham  Nah I don't think he goes to usf, he lives aro...","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Category</th>\n      <th>Message</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>ham</td>\n      <td>Go until jurong point, crazy.. Available only ...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>ham</td>\n      <td>Ok lar... Joking wif u oni...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>spam</td>\n      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>ham</td>\n      <td>U dun say so early hor... U c already then say...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>ham</td>\n      <td>Nah I don't think he goes to usf, he lives aro...</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.shape","execution_count":4,"outputs":[{"output_type":"execute_result","execution_count":4,"data":{"text/plain":"(5572, 2)"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"# Text cleaning and processing steps-\n* Remove punctuations\n* Convert text to tokens\n* Remove tokens of length less than or equal to 3\n* Remove stopwords using NLTK corpus stopwords list to match\n* Apply stemming\n* Apply lemmatization\n* Convert words to feature vectors"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Remove all punctuations from the text\n\ndef remove_punct(text):\n    return (\"\".join([ch for ch in text if ch not in st.punctuation]))","execution_count":5,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data['removed_punc'] = data['Message'].apply(lambda x: remove_punct(x))\ndata.head()","execution_count":6,"outputs":[{"output_type":"execute_result","execution_count":6,"data":{"text/plain":"  Category                                            Message  \\\n0      ham  Go until jurong point, crazy.. Available only ...   \n1      ham                      Ok lar... Joking wif u oni...   \n2     spam  Free entry in 2 a wkly comp to win FA Cup fina...   \n3      ham  U dun say so early hor... U c already then say...   \n4      ham  Nah I don't think he goes to usf, he lives aro...   \n\n                                        removed_punc  \n0  Go until jurong point crazy Available only in ...  \n1                            Ok lar Joking wif u oni  \n2  Free entry in 2 a wkly comp to win FA Cup fina...  \n3        U dun say so early hor U c already then say  \n4  Nah I dont think he goes to usf he lives aroun...  ","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Category</th>\n      <th>Message</th>\n      <th>removed_punc</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>ham</td>\n      <td>Go until jurong point, crazy.. Available only ...</td>\n      <td>Go until jurong point crazy Available only in ...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>ham</td>\n      <td>Ok lar... Joking wif u oni...</td>\n      <td>Ok lar Joking wif u oni</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>spam</td>\n      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>ham</td>\n      <td>U dun say so early hor... U c already then say...</td>\n      <td>U dun say so early hor U c already then say</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>ham</td>\n      <td>Nah I don't think he goes to usf, he lives aro...</td>\n      <td>Nah I dont think he goes to usf he lives aroun...</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"''' Convert text to lower case tokens. Here, split() is applied on white-spaces. But, it could be applied\n    on special characters, tabs or any other string based on which text is to be seperated into tokens.\n'''\ndef tokenize(text):\n    text = re.split('\\s+' ,text)\n    return [x.lower() for x in text]","execution_count":7,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data['tokens'] = data['removed_punc'].apply(lambda msg : tokenize(msg))\ndata.head()","execution_count":8,"outputs":[{"output_type":"execute_result","execution_count":8,"data":{"text/plain":"  Category                                            Message  \\\n0      ham  Go until jurong point, crazy.. Available only ...   \n1      ham                      Ok lar... Joking wif u oni...   \n2     spam  Free entry in 2 a wkly comp to win FA Cup fina...   \n3      ham  U dun say so early hor... U c already then say...   \n4      ham  Nah I don't think he goes to usf, he lives aro...   \n\n                                        removed_punc  \\\n0  Go until jurong point crazy Available only in ...   \n1                            Ok lar Joking wif u oni   \n2  Free entry in 2 a wkly comp to win FA Cup fina...   \n3        U dun say so early hor U c already then say   \n4  Nah I dont think he goes to usf he lives aroun...   \n\n                                              tokens  \n0  [go, until, jurong, point, crazy, available, o...  \n1                     [ok, lar, joking, wif, u, oni]  \n2  [free, entry, in, 2, a, wkly, comp, to, win, f...  \n3  [u, dun, say, so, early, hor, u, c, already, t...  \n4  [nah, i, dont, think, he, goes, to, usf, he, l...  ","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Category</th>\n      <th>Message</th>\n      <th>removed_punc</th>\n      <th>tokens</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>ham</td>\n      <td>Go until jurong point, crazy.. Available only ...</td>\n      <td>Go until jurong point crazy Available only in ...</td>\n      <td>[go, until, jurong, point, crazy, available, o...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>ham</td>\n      <td>Ok lar... Joking wif u oni...</td>\n      <td>Ok lar Joking wif u oni</td>\n      <td>[ok, lar, joking, wif, u, oni]</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>spam</td>\n      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n      <td>[free, entry, in, 2, a, wkly, comp, to, win, f...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>ham</td>\n      <td>U dun say so early hor... U c already then say...</td>\n      <td>U dun say so early hor U c already then say</td>\n      <td>[u, dun, say, so, early, hor, u, c, already, t...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>ham</td>\n      <td>Nah I don't think he goes to usf, he lives aro...</td>\n      <td>Nah I dont think he goes to usf he lives aroun...</td>\n      <td>[nah, i, dont, think, he, goes, to, usf, he, l...</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Remove tokens of length less than 3\ndef remove_small_words(text):\n    return [x for x in text if len(x) > 3 ]","execution_count":9,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data['larger_tokens'] = data['tokens'].apply(lambda x : remove_small_words(x))\ndata.head()","execution_count":10,"outputs":[{"output_type":"execute_result","execution_count":10,"data":{"text/plain":"  Category                                            Message  \\\n0      ham  Go until jurong point, crazy.. Available only ...   \n1      ham                      Ok lar... Joking wif u oni...   \n2     spam  Free entry in 2 a wkly comp to win FA Cup fina...   \n3      ham  U dun say so early hor... U c already then say...   \n4      ham  Nah I don't think he goes to usf, he lives aro...   \n\n                                        removed_punc  \\\n0  Go until jurong point crazy Available only in ...   \n1                            Ok lar Joking wif u oni   \n2  Free entry in 2 a wkly comp to win FA Cup fina...   \n3        U dun say so early hor U c already then say   \n4  Nah I dont think he goes to usf he lives aroun...   \n\n                                              tokens  \\\n0  [go, until, jurong, point, crazy, available, o...   \n1                     [ok, lar, joking, wif, u, oni]   \n2  [free, entry, in, 2, a, wkly, comp, to, win, f...   \n3  [u, dun, say, so, early, hor, u, c, already, t...   \n4  [nah, i, dont, think, he, goes, to, usf, he, l...   \n\n                                       larger_tokens  \n0  [until, jurong, point, crazy, available, only,...  \n1                                           [joking]  \n2  [free, entry, wkly, comp, final, tkts, 21st, 2...  \n3                             [early, already, then]  \n4   [dont, think, goes, lives, around, here, though]  ","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Category</th>\n      <th>Message</th>\n      <th>removed_punc</th>\n      <th>tokens</th>\n      <th>larger_tokens</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>ham</td>\n      <td>Go until jurong point, crazy.. Available only ...</td>\n      <td>Go until jurong point crazy Available only in ...</td>\n      <td>[go, until, jurong, point, crazy, available, o...</td>\n      <td>[until, jurong, point, crazy, available, only,...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>ham</td>\n      <td>Ok lar... Joking wif u oni...</td>\n      <td>Ok lar Joking wif u oni</td>\n      <td>[ok, lar, joking, wif, u, oni]</td>\n      <td>[joking]</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>spam</td>\n      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n      <td>[free, entry, in, 2, a, wkly, comp, to, win, f...</td>\n      <td>[free, entry, wkly, comp, final, tkts, 21st, 2...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>ham</td>\n      <td>U dun say so early hor... U c already then say...</td>\n      <td>U dun say so early hor U c already then say</td>\n      <td>[u, dun, say, so, early, hor, u, c, already, t...</td>\n      <td>[early, already, then]</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>ham</td>\n      <td>Nah I don't think he goes to usf, he lives aro...</td>\n      <td>Nah I dont think he goes to usf he lives aroun...</td>\n      <td>[nah, i, dont, think, he, goes, to, usf, he, l...</td>\n      <td>[dont, think, goes, lives, around, here, though]</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"''' Remove stopwords. Here, NLTK corpus list is used for a match. However, a customized user-defined \n    list could be created and used to limit the matches in input text. \n'''\ndef remove_stopwords(text):\n    return [word for word in text if word not in nltk.corpus.stopwords.words('english')]","execution_count":11,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data['clean_tokens'] = data['larger_tokens'].apply(lambda x : remove_stopwords(x))\ndata.head()","execution_count":12,"outputs":[{"output_type":"execute_result","execution_count":12,"data":{"text/plain":"  Category                                            Message  \\\n0      ham  Go until jurong point, crazy.. Available only ...   \n1      ham                      Ok lar... Joking wif u oni...   \n2     spam  Free entry in 2 a wkly comp to win FA Cup fina...   \n3      ham  U dun say so early hor... U c already then say...   \n4      ham  Nah I don't think he goes to usf, he lives aro...   \n\n                                        removed_punc  \\\n0  Go until jurong point crazy Available only in ...   \n1                            Ok lar Joking wif u oni   \n2  Free entry in 2 a wkly comp to win FA Cup fina...   \n3        U dun say so early hor U c already then say   \n4  Nah I dont think he goes to usf he lives aroun...   \n\n                                              tokens  \\\n0  [go, until, jurong, point, crazy, available, o...   \n1                     [ok, lar, joking, wif, u, oni]   \n2  [free, entry, in, 2, a, wkly, comp, to, win, f...   \n3  [u, dun, say, so, early, hor, u, c, already, t...   \n4  [nah, i, dont, think, he, goes, to, usf, he, l...   \n\n                                       larger_tokens  \\\n0  [until, jurong, point, crazy, available, only,...   \n1                                           [joking]   \n2  [free, entry, wkly, comp, final, tkts, 21st, 2...   \n3                             [early, already, then]   \n4   [dont, think, goes, lives, around, here, though]   \n\n                                        clean_tokens  \n0  [jurong, point, crazy, available, bugis, great...  \n1                                           [joking]  \n2  [free, entry, wkly, comp, final, tkts, 21st, 2...  \n3                                   [early, already]  \n4         [dont, think, goes, lives, around, though]  ","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Category</th>\n      <th>Message</th>\n      <th>removed_punc</th>\n      <th>tokens</th>\n      <th>larger_tokens</th>\n      <th>clean_tokens</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>ham</td>\n      <td>Go until jurong point, crazy.. Available only ...</td>\n      <td>Go until jurong point crazy Available only in ...</td>\n      <td>[go, until, jurong, point, crazy, available, o...</td>\n      <td>[until, jurong, point, crazy, available, only,...</td>\n      <td>[jurong, point, crazy, available, bugis, great...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>ham</td>\n      <td>Ok lar... Joking wif u oni...</td>\n      <td>Ok lar Joking wif u oni</td>\n      <td>[ok, lar, joking, wif, u, oni]</td>\n      <td>[joking]</td>\n      <td>[joking]</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>spam</td>\n      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n      <td>[free, entry, in, 2, a, wkly, comp, to, win, f...</td>\n      <td>[free, entry, wkly, comp, final, tkts, 21st, 2...</td>\n      <td>[free, entry, wkly, comp, final, tkts, 21st, 2...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>ham</td>\n      <td>U dun say so early hor... U c already then say...</td>\n      <td>U dun say so early hor U c already then say</td>\n      <td>[u, dun, say, so, early, hor, u, c, already, t...</td>\n      <td>[early, already, then]</td>\n      <td>[early, already]</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>ham</td>\n      <td>Nah I don't think he goes to usf, he lives aro...</td>\n      <td>Nah I dont think he goes to usf he lives aroun...</td>\n      <td>[nah, i, dont, think, he, goes, to, usf, he, l...</td>\n      <td>[dont, think, goes, lives, around, here, though]</td>\n      <td>[dont, think, goes, lives, around, though]</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"### Apply stemming to convert tokens to their root form. This is a rule-based process of word form conversion where word-suffixes are truncated irrespective of whether the root word is an actual word in the language dictionary. \n##### Note that this step is optional and depends on problem type.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Apply stemming to get root words \ndef stemming(text):\n    ps = PorterStemmer()\n    return [ps.stem(word) for word in text]","execution_count":13,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data['stem_words'] = data['clean_tokens'].apply(lambda wrd: stemming(wrd))\ndata.head()","execution_count":14,"outputs":[{"output_type":"execute_result","execution_count":14,"data":{"text/plain":"  Category                                            Message  \\\n0      ham  Go until jurong point, crazy.. Available only ...   \n1      ham                      Ok lar... Joking wif u oni...   \n2     spam  Free entry in 2 a wkly comp to win FA Cup fina...   \n3      ham  U dun say so early hor... U c already then say...   \n4      ham  Nah I don't think he goes to usf, he lives aro...   \n\n                                        removed_punc  \\\n0  Go until jurong point crazy Available only in ...   \n1                            Ok lar Joking wif u oni   \n2  Free entry in 2 a wkly comp to win FA Cup fina...   \n3        U dun say so early hor U c already then say   \n4  Nah I dont think he goes to usf he lives aroun...   \n\n                                              tokens  \\\n0  [go, until, jurong, point, crazy, available, o...   \n1                     [ok, lar, joking, wif, u, oni]   \n2  [free, entry, in, 2, a, wkly, comp, to, win, f...   \n3  [u, dun, say, so, early, hor, u, c, already, t...   \n4  [nah, i, dont, think, he, goes, to, usf, he, l...   \n\n                                       larger_tokens  \\\n0  [until, jurong, point, crazy, available, only,...   \n1                                           [joking]   \n2  [free, entry, wkly, comp, final, tkts, 21st, 2...   \n3                             [early, already, then]   \n4   [dont, think, goes, lives, around, here, though]   \n\n                                        clean_tokens  \\\n0  [jurong, point, crazy, available, bugis, great...   \n1                                           [joking]   \n2  [free, entry, wkly, comp, final, tkts, 21st, 2...   \n3                                   [early, already]   \n4         [dont, think, goes, lives, around, though]   \n\n                                          stem_words  \n0  [jurong, point, crazi, avail, bugi, great, wor...  \n1                                             [joke]  \n2  [free, entri, wkli, comp, final, tkt, 21st, 20...  \n3                                   [earli, alreadi]  \n4           [dont, think, goe, live, around, though]  ","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Category</th>\n      <th>Message</th>\n      <th>removed_punc</th>\n      <th>tokens</th>\n      <th>larger_tokens</th>\n      <th>clean_tokens</th>\n      <th>stem_words</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>ham</td>\n      <td>Go until jurong point, crazy.. Available only ...</td>\n      <td>Go until jurong point crazy Available only in ...</td>\n      <td>[go, until, jurong, point, crazy, available, o...</td>\n      <td>[until, jurong, point, crazy, available, only,...</td>\n      <td>[jurong, point, crazy, available, bugis, great...</td>\n      <td>[jurong, point, crazi, avail, bugi, great, wor...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>ham</td>\n      <td>Ok lar... Joking wif u oni...</td>\n      <td>Ok lar Joking wif u oni</td>\n      <td>[ok, lar, joking, wif, u, oni]</td>\n      <td>[joking]</td>\n      <td>[joking]</td>\n      <td>[joke]</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>spam</td>\n      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n      <td>[free, entry, in, 2, a, wkly, comp, to, win, f...</td>\n      <td>[free, entry, wkly, comp, final, tkts, 21st, 2...</td>\n      <td>[free, entry, wkly, comp, final, tkts, 21st, 2...</td>\n      <td>[free, entri, wkli, comp, final, tkt, 21st, 20...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>ham</td>\n      <td>U dun say so early hor... U c already then say...</td>\n      <td>U dun say so early hor U c already then say</td>\n      <td>[u, dun, say, so, early, hor, u, c, already, t...</td>\n      <td>[early, already, then]</td>\n      <td>[early, already]</td>\n      <td>[earli, alreadi]</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>ham</td>\n      <td>Nah I don't think he goes to usf, he lives aro...</td>\n      <td>Nah I dont think he goes to usf he lives aroun...</td>\n      <td>[nah, i, dont, think, he, goes, to, usf, he, l...</td>\n      <td>[dont, think, goes, lives, around, here, though]</td>\n      <td>[dont, think, goes, lives, around, though]</td>\n      <td>[dont, think, goe, live, around, though]</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"### Lemmatization converts word to it's dictionary base form. This process takes language grammar and vocabulary into consideration while conversion. Hence, it is different from Stemming in that it does not merely truncate the suffixes to get the root word.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Apply lemmatization on tokens\ndef lemmatize(text):\n    word_net = WordNetLemmatizer()\n    return [word_net.lemmatize(word) for word in text]","execution_count":15,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data['lemma_words'] = data['clean_tokens'].apply(lambda x : lemmatize(x))\ndata.head()","execution_count":16,"outputs":[{"output_type":"execute_result","execution_count":16,"data":{"text/plain":"  Category                                            Message  \\\n0      ham  Go until jurong point, crazy.. Available only ...   \n1      ham                      Ok lar... Joking wif u oni...   \n2     spam  Free entry in 2 a wkly comp to win FA Cup fina...   \n3      ham  U dun say so early hor... U c already then say...   \n4      ham  Nah I don't think he goes to usf, he lives aro...   \n\n                                        removed_punc  \\\n0  Go until jurong point crazy Available only in ...   \n1                            Ok lar Joking wif u oni   \n2  Free entry in 2 a wkly comp to win FA Cup fina...   \n3        U dun say so early hor U c already then say   \n4  Nah I dont think he goes to usf he lives aroun...   \n\n                                              tokens  \\\n0  [go, until, jurong, point, crazy, available, o...   \n1                     [ok, lar, joking, wif, u, oni]   \n2  [free, entry, in, 2, a, wkly, comp, to, win, f...   \n3  [u, dun, say, so, early, hor, u, c, already, t...   \n4  [nah, i, dont, think, he, goes, to, usf, he, l...   \n\n                                       larger_tokens  \\\n0  [until, jurong, point, crazy, available, only,...   \n1                                           [joking]   \n2  [free, entry, wkly, comp, final, tkts, 21st, 2...   \n3                             [early, already, then]   \n4   [dont, think, goes, lives, around, here, though]   \n\n                                        clean_tokens  \\\n0  [jurong, point, crazy, available, bugis, great...   \n1                                           [joking]   \n2  [free, entry, wkly, comp, final, tkts, 21st, 2...   \n3                                   [early, already]   \n4         [dont, think, goes, lives, around, though]   \n\n                                          stem_words  \\\n0  [jurong, point, crazi, avail, bugi, great, wor...   \n1                                             [joke]   \n2  [free, entri, wkli, comp, final, tkt, 21st, 20...   \n3                                   [earli, alreadi]   \n4           [dont, think, goe, live, around, though]   \n\n                                         lemma_words  \n0  [jurong, point, crazy, available, bugis, great...  \n1                                           [joking]  \n2  [free, entry, wkly, comp, final, tkts, 21st, 2...  \n3                                   [early, already]  \n4            [dont, think, go, life, around, though]  ","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Category</th>\n      <th>Message</th>\n      <th>removed_punc</th>\n      <th>tokens</th>\n      <th>larger_tokens</th>\n      <th>clean_tokens</th>\n      <th>stem_words</th>\n      <th>lemma_words</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>ham</td>\n      <td>Go until jurong point, crazy.. Available only ...</td>\n      <td>Go until jurong point crazy Available only in ...</td>\n      <td>[go, until, jurong, point, crazy, available, o...</td>\n      <td>[until, jurong, point, crazy, available, only,...</td>\n      <td>[jurong, point, crazy, available, bugis, great...</td>\n      <td>[jurong, point, crazi, avail, bugi, great, wor...</td>\n      <td>[jurong, point, crazy, available, bugis, great...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>ham</td>\n      <td>Ok lar... Joking wif u oni...</td>\n      <td>Ok lar Joking wif u oni</td>\n      <td>[ok, lar, joking, wif, u, oni]</td>\n      <td>[joking]</td>\n      <td>[joking]</td>\n      <td>[joke]</td>\n      <td>[joking]</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>spam</td>\n      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n      <td>[free, entry, in, 2, a, wkly, comp, to, win, f...</td>\n      <td>[free, entry, wkly, comp, final, tkts, 21st, 2...</td>\n      <td>[free, entry, wkly, comp, final, tkts, 21st, 2...</td>\n      <td>[free, entri, wkli, comp, final, tkt, 21st, 20...</td>\n      <td>[free, entry, wkly, comp, final, tkts, 21st, 2...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>ham</td>\n      <td>U dun say so early hor... U c already then say...</td>\n      <td>U dun say so early hor U c already then say</td>\n      <td>[u, dun, say, so, early, hor, u, c, already, t...</td>\n      <td>[early, already, then]</td>\n      <td>[early, already]</td>\n      <td>[earli, alreadi]</td>\n      <td>[early, already]</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>ham</td>\n      <td>Nah I don't think he goes to usf, he lives aro...</td>\n      <td>Nah I dont think he goes to usf he lives aroun...</td>\n      <td>[nah, i, dont, think, he, goes, to, usf, he, l...</td>\n      <td>[dont, think, goes, lives, around, here, though]</td>\n      <td>[dont, think, goes, lives, around, though]</td>\n      <td>[dont, think, goe, live, around, though]</td>\n      <td>[dont, think, go, life, around, though]</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create sentences to get clean text as input for vectors\n\ndef return_sentences(tokens):\n    return \" \".join([word for word in tokens])","execution_count":18,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data['clean_text'] = data['lemma_words'].apply(lambda x : return_sentences(x))\ndata.head()","execution_count":19,"outputs":[{"output_type":"execute_result","execution_count":19,"data":{"text/plain":"  Category                                            Message  \\\n0      ham  Go until jurong point, crazy.. Available only ...   \n1      ham                      Ok lar... Joking wif u oni...   \n2     spam  Free entry in 2 a wkly comp to win FA Cup fina...   \n3      ham  U dun say so early hor... U c already then say...   \n4      ham  Nah I don't think he goes to usf, he lives aro...   \n\n                                        removed_punc  \\\n0  Go until jurong point crazy Available only in ...   \n1                            Ok lar Joking wif u oni   \n2  Free entry in 2 a wkly comp to win FA Cup fina...   \n3        U dun say so early hor U c already then say   \n4  Nah I dont think he goes to usf he lives aroun...   \n\n                                              tokens  \\\n0  [go, until, jurong, point, crazy, available, o...   \n1                     [ok, lar, joking, wif, u, oni]   \n2  [free, entry, in, 2, a, wkly, comp, to, win, f...   \n3  [u, dun, say, so, early, hor, u, c, already, t...   \n4  [nah, i, dont, think, he, goes, to, usf, he, l...   \n\n                                       larger_tokens  \\\n0  [until, jurong, point, crazy, available, only,...   \n1                                           [joking]   \n2  [free, entry, wkly, comp, final, tkts, 21st, 2...   \n3                             [early, already, then]   \n4   [dont, think, goes, lives, around, here, though]   \n\n                                        clean_tokens  \\\n0  [jurong, point, crazy, available, bugis, great...   \n1                                           [joking]   \n2  [free, entry, wkly, comp, final, tkts, 21st, 2...   \n3                                   [early, already]   \n4         [dont, think, goes, lives, around, though]   \n\n                                          stem_words  \\\n0  [jurong, point, crazi, avail, bugi, great, wor...   \n1                                             [joke]   \n2  [free, entri, wkli, comp, final, tkt, 21st, 20...   \n3                                   [earli, alreadi]   \n4           [dont, think, goe, live, around, though]   \n\n                                         lemma_words  \\\n0  [jurong, point, crazy, available, bugis, great...   \n1                                           [joking]   \n2  [free, entry, wkly, comp, final, tkts, 21st, 2...   \n3                                   [early, already]   \n4            [dont, think, go, life, around, though]   \n\n                                          clean_text  \n0  jurong point crazy available bugis great world...  \n1                                             joking  \n2  free entry wkly comp final tkts 21st 2005 text...  \n3                                      early already  \n4                   dont think go life around though  ","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Category</th>\n      <th>Message</th>\n      <th>removed_punc</th>\n      <th>tokens</th>\n      <th>larger_tokens</th>\n      <th>clean_tokens</th>\n      <th>stem_words</th>\n      <th>lemma_words</th>\n      <th>clean_text</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>ham</td>\n      <td>Go until jurong point, crazy.. Available only ...</td>\n      <td>Go until jurong point crazy Available only in ...</td>\n      <td>[go, until, jurong, point, crazy, available, o...</td>\n      <td>[until, jurong, point, crazy, available, only,...</td>\n      <td>[jurong, point, crazy, available, bugis, great...</td>\n      <td>[jurong, point, crazi, avail, bugi, great, wor...</td>\n      <td>[jurong, point, crazy, available, bugis, great...</td>\n      <td>jurong point crazy available bugis great world...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>ham</td>\n      <td>Ok lar... Joking wif u oni...</td>\n      <td>Ok lar Joking wif u oni</td>\n      <td>[ok, lar, joking, wif, u, oni]</td>\n      <td>[joking]</td>\n      <td>[joking]</td>\n      <td>[joke]</td>\n      <td>[joking]</td>\n      <td>joking</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>spam</td>\n      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n      <td>[free, entry, in, 2, a, wkly, comp, to, win, f...</td>\n      <td>[free, entry, wkly, comp, final, tkts, 21st, 2...</td>\n      <td>[free, entry, wkly, comp, final, tkts, 21st, 2...</td>\n      <td>[free, entri, wkli, comp, final, tkt, 21st, 20...</td>\n      <td>[free, entry, wkly, comp, final, tkts, 21st, 2...</td>\n      <td>free entry wkly comp final tkts 21st 2005 text...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>ham</td>\n      <td>U dun say so early hor... U c already then say...</td>\n      <td>U dun say so early hor U c already then say</td>\n      <td>[u, dun, say, so, early, hor, u, c, already, t...</td>\n      <td>[early, already, then]</td>\n      <td>[early, already]</td>\n      <td>[earli, alreadi]</td>\n      <td>[early, already]</td>\n      <td>early already</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>ham</td>\n      <td>Nah I don't think he goes to usf, he lives aro...</td>\n      <td>Nah I dont think he goes to usf he lives aroun...</td>\n      <td>[nah, i, dont, think, he, goes, to, usf, he, l...</td>\n      <td>[dont, think, goes, lives, around, here, though]</td>\n      <td>[dont, think, goes, lives, around, though]</td>\n      <td>[dont, think, goe, live, around, though]</td>\n      <td>[dont, think, go, life, around, though]</td>\n      <td>dont think go life around though</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"### TF-IDF : Term Frequency - Inverse Document Frequency\n#### The term frequency is the number of times a term occurs in a document. Inverse document frequency is an inverse function of the number of documents in which   that a given word occurs.\n#### The product of these two terms gives tf-idf weight for a word in the corpus. The higher the frequency of occurrence of a word, lower is it's weight and vice-versa. This gives more weightage to rare terms in the corpus and penalizes more commonly occuring terms.\n#### Other widely used vectorizer is Count vectorizer which only considers the frequency of occurrence of a word across the corpus.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Convert lemmatized words to Tf-Idf feature vectors\n\nfrom sklearn.feature_extraction.text import TfidfVectorizer\n\ntfidf = TfidfVectorizer()\ntfidf_vect = tfidf.fit_transform(data['clean_text'])\ntfidf_vect.shape","execution_count":24,"outputs":[{"output_type":"execute_result","execution_count":24,"data":{"text/plain":"(5572, 7912)"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get feature names in the vector\ntfidf.get_feature_names()","execution_count":25,"outputs":[{"output_type":"execute_result","execution_count":25,"data":{"text/plain":"['008704050406',\n '0089my',\n '0121',\n '01223585236',\n '01223585334',\n '0125698789',\n '020603',\n '0207',\n '02070836089',\n '02072069400',\n '02073162414',\n '02085076972',\n '020903',\n '050703',\n '0578',\n '060505',\n '061104',\n '07008009200',\n '07046744435',\n '07090201529',\n '07090298926',\n '07099833605',\n '071104',\n '07123456789',\n '0721072',\n '07732584351',\n '07734396839',\n '07742676969',\n '07753741225',\n '0776xxxxxxx',\n '07786200117',\n '077xxx',\n '07801543489',\n '07808',\n '07808247860',\n '07808726822',\n '07815296484',\n '07821230901',\n '0784987',\n '0789xxxxxxx',\n '0794674629107880867867',\n '0796xxxxxx',\n '07973788240',\n '07xxxxxxxxx',\n '0800',\n '08000407165',\n '08000776320',\n '08000839402',\n '08000930705',\n '08000938767',\n '08001950382',\n '08002888812',\n '08002986030',\n '08002986906',\n '08002988890',\n '08006344447',\n '0808',\n '08081263000',\n '08081560665',\n '0825',\n '0844',\n '08448350055',\n '08448714184',\n '0845',\n '08450542832',\n '08452810071',\n '08452810073',\n '08452810075over18s',\n '0870',\n '08700621170150p',\n '08701213186',\n '08701237397',\n '08701417012',\n '08701417012150p',\n '0870141701216',\n '087016248',\n '08701752560',\n '087018728737',\n '0870241182716',\n '08702490080',\n '08702840625',\n '08702840625comuk',\n '08704439680',\n '08704439680tscs',\n '08706091795',\n '0870737910216yrs',\n '08707500020',\n '08707509020',\n '0870753331018',\n '08707808226',\n '08708034412',\n '08708800282',\n '08709222922',\n '08709501522',\n '0870k',\n '087104711148',\n '08712101358',\n '08712103738',\n '0871212025016',\n '08712300220',\n '087123002209am7pm',\n '08712317606',\n '08712400200',\n '08712400603',\n '08712402050',\n '08712402578',\n '08712402779',\n '08712402902',\n '08712402972',\n '08712404000',\n '08712405020',\n '08712405022',\n '08712460324',\n '08712460324nat',\n '08712466669',\n '0871277810710pmin',\n '0871277810810',\n '0871277810910pmin',\n '087143423992stop',\n '087147123779am7pm',\n '08714712379',\n '08714712388',\n '08714712394',\n '08714712412',\n '08714714011',\n '08714719523',\n '08715203028',\n '08715203649',\n '08715203652',\n '08715203656',\n '08715203677',\n '08715203685',\n '08715203694',\n '08715205273',\n '08715500022',\n '08715705022',\n '08717111821',\n '08717168528',\n '08717205546',\n '08717507382',\n '08717507711',\n '08717509990',\n '08717890890',\n '08717895698',\n '08717898035',\n '08718711108',\n '08718720201',\n '08718723815',\n '08718725756',\n '08718726270',\n '08718726270150gbpmtmsg18',\n '08718726970',\n '08718726971',\n '08718726978',\n '087187272008',\n '08718727870',\n '08718729755',\n '08718729758',\n '08718730555',\n '08718730666',\n '08718738001',\n '08718738002',\n '08718738034',\n '08719180219',\n '08719180248',\n '08719181259',\n '08719181503',\n '08719181513',\n '08719839835',\n '08719899217',\n '08719899229',\n '08719899230',\n '09041940223',\n '09050000301',\n '09050000332',\n '09050000460',\n '09050000555',\n '09050000878',\n '09050000928',\n '09050001295',\n '09050001808',\n '09050003091',\n '09050005321',\n '09050090044',\n '09050280520',\n '09053750005',\n '09056242159',\n '09057039994',\n '09058091854',\n '09058091870',\n '09058094454',\n '09058094455',\n '09058094507',\n '09058094565',\n '09058094583',\n '09058094594',\n '09058094597',\n '09058094599',\n '09058095107',\n '09058095201',\n '09058097189',\n '09058097218',\n '09058098002',\n '09058099801',\n '09061104276',\n '09061104283',\n '09061209465',\n '09061213237',\n '09061221061',\n '09061221066',\n '09061701444',\n '09061701461',\n '09061701851',\n '09061701939',\n '09061702893',\n '09061743386',\n '09061743806',\n '09061743810',\n '09061743811',\n '09061744553',\n '09061749602',\n '09061790121',\n '09061790125',\n '09061790126',\n '09063440451',\n '09063442151',\n '09063458130',\n '0906346330',\n '09064011000',\n '09064012103',\n '09064012160',\n '09064015307',\n '09064017295',\n '09064017305',\n '09064018838',\n '09064019014',\n '09064019788',\n '09065069120',\n '09065069154',\n '09065171142stopsms08',\n '09065171142stopsms08718727870150ppm',\n '09065174042',\n '09065394514',\n '09065394973',\n '09065989180',\n '09065989182',\n '09066350750',\n '09066358152',\n '09066358361',\n '09066361921',\n '09066362206',\n '09066362220',\n '09066362231',\n '09066364311',\n '09066364349',\n '09066364589',\n '09066368327',\n '09066368470',\n '09066368753',\n '09066380611',\n '09066382422',\n '09066612661',\n '09066649731from',\n '09066660100',\n '09071512432',\n '09071512433',\n '09071517866',\n '09077818151',\n '09090204448',\n '09090900040',\n '09094100151',\n '09094646631',\n '09094646899',\n '09095350301',\n '09096102316',\n '09099725823',\n '09099726395',\n '09099726429',\n '09099726481',\n '09099726553',\n '09111030116',\n '09111032124',\n '09701213186',\n '0anetworks',\n '100',\n '1000',\n '10000',\n '100000',\n '1000call',\n '100603',\n '100psms',\n '1010',\n '1013',\n '101mega',\n '1030',\n '10803',\n '10am',\n '10am7pm',\n '10am9pm',\n '10pmin',\n '10ppm',\n '10th',\n '1120',\n '1131',\n '11414',\n '1146',\n '1148',\n '1172',\n '118pmsg',\n '11mths',\n '12000pes',\n '1205',\n '1225',\n '1230',\n '125',\n '1250',\n '125gift',\n '12hours',\n '12hrs',\n '12mths',\n '12price',\n '131004',\n '1327',\n '13404',\n '1405',\n '140ppm',\n '1450',\n '146tf150p',\n '14thmarch',\n '150',\n '1500',\n '150ea',\n '150morefrmmob',\n '150msg',\n '150mtmsgrcvd18',\n '150p',\n '150pday',\n '150perweeksub',\n '150perwksub',\n '150pm',\n '150pmeg',\n '150pmin',\n '150pmmorefrommobile2bremovedmobypobox734ls27yf',\n '150pmsg',\n '150pmsgrcvd',\n '150pmsgrcvdhgsuite3422landsroww1j6hl',\n '150pmt',\n '150pmtmsg',\n '150pmtmsgrcvd18',\n '150ppermesssubscription',\n '150ppm',\n '150ppmpobox10183bhamb64xe',\n '150ppmsg',\n '150prcvd',\n '150psms',\n '150ptext',\n '150ptone',\n '150pw',\n '150pwk',\n '150rcvd',\n '150week',\n '150wk',\n '1526',\n '15541',\n '15pmin',\n '1680',\n '16only',\n '181104',\n '1843',\n '186',\n '18only',\n '18ptxt',\n '18yrs',\n '1956669',\n '1appledayno',\n '1childish',\n '1cup',\n '1hanuman',\n '1his',\n '1lemondayno',\n '1mcflyall',\n '1million',\n '1minmobsmore',\n '1minmobsmorelkpobox177hp51fl',\n '1minmoremobsemspobox45po139wa',\n '1month',\n '1st4terms',\n '1stchoicecouk',\n '1stone',\n '1tulsi',\n '1unbreakable',\n '1winaweek',\n '1winawk',\n '1x150pwk',\n '200',\n '2000',\n '20000',\n '2003',\n '2004',\n '2005',\n '2006',\n '2007',\n '2025050',\n '20m12aq',\n '20pmin',\n '211104',\n '21870000hi',\n '21st',\n '220cm2',\n '2309',\n '230ish',\n '241004',\n '247mp',\n '24hrs',\n '24th',\n '250',\n '250k',\n '260305',\n '261004',\n '261104',\n '2667',\n '26th',\n '2703',\n '27603',\n '2814032',\n '28days',\n '28th',\n '28thfebtcs',\n '290305',\n '29100',\n '2bajarangabali',\n '2bold',\n '2channel',\n '2day',\n '2daylove',\n '2docdplease',\n '2end',\n '2exit',\n '2getha',\n '2geva',\n '2godid',\n '2gthr',\n '2hook',\n '2hrs',\n '2kbsubject',\n '2marrow',\n '2moro',\n '2morow',\n '2morro',\n '2morrow',\n '2morrowxxxx',\n '2mro',\n '2mrw',\n '2mwen',\n '2naughty',\n '2nhite',\n '2nights',\n '2nite',\n '2nitetell',\n '2optout',\n '2optoutd3wv',\n '2police',\n '2rcv',\n '2stop',\n '2stoptx',\n '2stoptxt',\n '2untamed',\n '2watershd',\n '2waxsto',\n '2when',\n '2wks',\n '2years',\n '2yrs',\n '300',\n '3000',\n '300603',\n '300603tcsbcm4235wc1n3xxcallcost150ppmmobilesvary',\n '300p',\n '3030',\n '30apr',\n '30pptxt',\n '30th',\n '3100',\n '310303',\n '311004',\n '31pmsg150p',\n '32000',\n '3230',\n '32323',\n '3350',\n '3365',\n '350',\n '3510i',\n '3650',\n '36504',\n '3680',\n '3680offer',\n '3750',\n '375max',\n '38',\n '391784',\n '399',\n '3cover',\n '3days',\n '3gbp',\n '3hrs',\n '3lions',\n '3maruti',\n '3miles',\n '3mins',\n '3mobile',\n '3optical',\n '3pound',\n '3qxj9',\n '3sentiment',\n '3unkempt',\n '3wife',\n '3wk',\n '3wks',\n '3x',\n '400',\n '400minscall',\n '4041',\n '40411',\n '40533',\n '40gb',\n '40mph',\n '41685',\n '41782',\n '42049',\n '4217',\n '42478',\n '42810',\n '4403ldnw1a7rw18',\n '447797706009',\n '447801259231',\n '447per',\n '448712404000please',\n '449050000301',\n '449month',\n '450',\n '450p',\n '450ppw',\n '450pw',\n '45239',\n '4712',\n '4742',\n '4882',\n '48922',\n '49557',\n '4brekkie',\n '4cook',\n '4eva',\n '4few',\n '4fil',\n '4get',\n '4give',\n '4got',\n '4goten',\n '4info',\n '4lux',\n '4mths',\n '4pavanaputra',\n '4press',\n '4rowdy',\n '4some1',\n '4tctxt',\n '4the',\n '4thnovbehind',\n '4txt120p',\n '4txtú120',\n '4utxt',\n '4ward',\n '4wrd',\n '4years',\n '50',\n '500',\n '5000',\n '500000',\n '505060',\n '50award',\n '515pm',\n '5226',\n '5249',\n '5903',\n '5digital',\n '5free',\n '5ful',\n '5gardener',\n '5gently',\n '5i',\n '5ish',\n '5min',\n '5mls',\n '5month',\n '5sankatmochan',\n '5terror',\n '5wkg',\n '5years',\n '600',\n '6031',\n '60400thousadi',\n '60pmin',\n '61200',\n '61610',\n '62220cncl',\n '6230',\n '62468',\n '62735',\n '63miles',\n '645pm',\n '6669',\n '67441233',\n '68866',\n '69101',\n '69200',\n '69669',\n '69696',\n '69698',\n '69855',\n '6986618',\n '69876',\n '69888',\n '69888nyt',\n '69911',\n '69969',\n '69988',\n '6cruel',\n '6days',\n '6housemaid',\n '6hrs',\n '6ish',\n '6missed',\n '6months',\n '6ramaduth',\n '6romantic',\n '6times',\n '7250',\n '7250i',\n '730ish',\n '730pm',\n '74355',\n '750',\n '75000',\n '7548',\n '7634',\n '7684',\n '7732584351',\n '7876150ppm',\n '78pmin',\n '7cfca1a',\n '7children',\n '7ish',\n '7mahaveer',\n '7romantic',\n '7shy',\n '800',\n '8000930705',\n '80062',\n '8007',\n '80082',\n '80086',\n '80122300pwk',\n '80155',\n '80160',\n '80182',\n '8027',\n '80488',\n '80488biz',\n '80608',\n '8077',\n '80878',\n '81010',\n '81151',\n '81303',\n '81618',\n '816183',\n '82242',\n '82277',\n '82277unsub',\n '82324',\n '82468',\n '83021',\n '83039',\n '83049',\n '83110',\n '83118',\n '83222',\n '83332please',\n '83338',\n '83355',\n '83370',\n '83383',\n '83435',\n '83600',\n '83738',\n '84025',\n '84122',\n '84128',\n '84128custcare',\n '84199',\n '84484',\n '85023',\n '85069',\n '85222',\n '85233',\n '8552',\n '85555',\n '86021',\n '864233',\n '86688',\n '86888',\n '87021',\n '87066',\n '87070',\n '87077',\n '87121',\n '87131',\n '8714714',\n '87239',\n '87575',\n '8800',\n '88039',\n '88039skilgmetscs087147403231winawkage16',\n '88066',\n '88088',\n '88222',\n '8830',\n '88600',\n '88800',\n '8883',\n '88877',\n '88877free',\n '88888',\n '89034',\n '89070',\n '89080',\n '89105',\n '89123',\n '89545',\n '89555',\n '89693',\n '89938',\n '8attractive',\n '8ball',\n '8lovable',\n '8neighbour',\n '900',\n '9061100010',\n '9153',\n '95pax',\n '97n7qp',\n '98321561',\n '9996',\n '9am11pm',\n '9decent',\n '9funny',\n 'aaniye',\n 'aaooooright',\n 'aathilove',\n 'aathiwhere',\n 'abbey',\n 'abdomen',\n 'abeg',\n 'abelu',\n 'aberdeen',\n 'ability',\n 'abiola',\n 'able',\n 'abnormally',\n 'aboutas',\n 'abroad',\n 'absence',\n 'absolutely',\n 'abstract',\n 'abta',\n 'aburo',\n 'abuse',\n 'abuser',\n 'academic',\n 'accent',\n 'accenture',\n 'accept',\n 'access',\n 'accessible',\n 'accidant',\n 'accident',\n 'accidentally',\n 'accommodation',\n 'accommodationvouchers',\n 'accomodate',\n 'accomodations',\n 'accordin',\n 'accordingly',\n 'accordinglyor',\n 'account',\n 'accounting',\n 'accumulation',\n 'achanammarakheshqatar',\n 'ache',\n 'achieve',\n 'acid',\n 'acknowledgement',\n 'acl03530150pm',\n 'acnt',\n 'acoentry41',\n 'across',\n 'acsmsrewards',\n 'acted',\n 'actin',\n 'acting',\n 'action',\n 'activ8',\n 'activate',\n 'active',\n 'activity',\n 'actor',\n 'actual',\n 'actually',\n 'acwicmb3cktz8r74',\n 'adam',\n 'add',\n 'addamsfa',\n 'added',\n 'addicted',\n 'addie',\n 'adding',\n 'address',\n 'addressull',\n 'adewale',\n 'adjustable',\n 'admin',\n 'administrator',\n 'admirer',\n 'admission',\n 'admit',\n 'admiti',\n 'adore',\n 'adoring',\n 'adress',\n 'adrian',\n 'adrink',\n 'adsense',\n 'adult',\n 'advance',\n 'adventure',\n 'adventuring',\n 'advice',\n 'advise',\n 'advising',\n 'advisor',\n 'aeronautics',\n 'aeroplane',\n 'afew',\n 'affair',\n 'affection',\n 'affectionate',\n 'affectionsamp',\n 'affidavit',\n 'afford',\n 'afghanistan',\n 'afraid',\n 'africa',\n 'african',\n 'afternon',\n 'afternoon',\n 'afterwards',\n 'aftr',\n 'againcall',\n 'againloving',\n 'agalla',\n 'age',\n 'age16',\n 'age16150ppermesssubscription',\n 'age23',\n 'agency',\n 'agent',\n 'agesring',\n 'agidhane',\n 'aging',\n 'agocusoon',\n 'agree',\n 'agreen',\n 'ahead',\n 'ahgee',\n 'ahhh',\n 'ahhhhjust',\n 'ahmad',\n 'ahnow',\n 'ahold',\n 'ahsen',\n 'ahthe',\n 'ahwhat',\n 'aid',\n 'aight',\n 'aint',\n 'air1',\n 'airport',\n 'airtel',\n 'aiya',\n 'aiyah',\n 'aiyar',\n 'aiyo',\n 'ajith',\n 'akonlonely',\n 'alaikkumpride',\n 'alaipayuthe',\n 'albi',\n 'album',\n 'albumquite',\n 'alcohol',\n 'aldrine',\n 'alert',\n 'alertfrom',\n 'aletter',\n 'alex',\n 'alexs',\n 'alfie',\n 'algarve',\n 'algebra',\n 'algorithm',\n 'alian',\n 'alibi',\n 'alive',\n 'alivebetter',\n 'allah',\n 'allahmeet',\n 'allahrakhesh',\n 'allalo',\n 'allday',\n 'alle',\n 'allo',\n 'allow',\n 'allowed',\n 'allows',\n 'alls',\n 'almost',\n 'alone',\n 'along',\n 'alot',\n 'already',\n 'alreadysabarish',\n 'alright',\n 'alrightokay',\n 'alrite',\n 'alritehave',\n 'also',\n 'alsoor',\n 'alter',\n 'alternativehope',\n 'although',\n 'alwa',\n 'always',\n 'alwys',\n 'amanda',\n 'amazing',\n 'ambitious',\n 'ambrithmaduraimet',\n 'american',\n 'amigo',\n 'ammaelife',\n 'ammo',\n 'amnow',\n 'among',\n 'amongst',\n 'amore',\n 'amount',\n 'amplikater',\n 'amrca',\n 'amrita',\n 'amused',\n 'anal',\n 'analysis',\n 'anand',\n 'anderson',\n 'andor',\n 'andre',\n 'andres',\n 'andrewsboy',\n 'andros',\n 'angel',\n 'angry',\n 'animal',\n 'animation',\n 'anjie',\n 'anjolas',\n 'anna',\n 'annie',\n 'anniversary',\n 'annoncement',\n 'announced',\n 'announcement',\n 'annoyin',\n 'annoying',\n 'anonymous',\n 'anot',\n 'another',\n 'ansr',\n 'answer',\n 'answered',\n ...]"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"Next steps -\n* Feature Engineering\n* Model generation\n* Model evaluation"}],"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.7.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":4}


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | # NLP - Text cleaning and processing pipeline.
 2 | 
 3 | ## Text processing pipeline for NLP problems with ready-to-use functions and text classification models.
 4 | Code file environment - Jupyter notebook
 5 | 
 6 | Language - Python
 7 | 
 8 | Packages - NLTK (Natural Language ToolKit) and Sklearn
 9 | 
10 | Data source - https://www.kaggle.com/team-ai/spam-text-message-classification
11 | 
12 | Data format - .csv
13 | 
14 | The functions could be directly used in any other input data format.
15 | 


--------------------------------------------------------------------------------