\n",
15 | "[Source](http://www.purdue.edu/newsroom/releases/2014/Q1/smartphone-to-become-smarter-with-deep-learning-innovation.html)\n",
16 | "\n",
17 | "One of the most well known problems in machine learning regards how to categorize [handwritten numbers](http://en.wikipedia.org/wiki/MNIST_database) automatically. Basically, the idea is that you have 10 different digits (0-9) and you want a computer to be able to correctly identify each number. This would come in handy at the post office, along with many other applications such as identifying address numbers in images. \n",
18 | "\n",
19 | "Different machine learning experts have worked on this problem for several years trying [a variety of approaches](http://yann.lecun.com/exdb/mnist/). Ultimately, however, the best algorithm for the task of image classification would be an algorithm that can fit features that aren't easily described. This is where neural networks truly shine. \n",
20 | "\n",
21 | "Deep learning has become quite the trendy subject recently. Here is an [example](http://venturebeat.com/2015/02/09/microsoft-researchers-say-their-newest-deep-learning-system-beats-humans-and-google/). Essentially, researchers are trying to use Deep Learning to categorize images better than a human could, something humans are typically better at doing naturally than a computer. \n",
22 | "\n",
23 | "If you want an idea as to how Deep Learning is being used (and how it works) listening to a talk from Dr. Andrew Ng is always a good idea!"
24 | ]
25 | },
26 | {
27 | "cell_type": "code",
28 | "execution_count": 1,
29 | "metadata": {
30 | "collapsed": false
31 | },
32 | "outputs": [
33 | {
34 | "data": {
35 | "text/html": [
36 | "\n",
37 | " 
\n",
15 | "[Source](http://imgkid.com/movie-theater-wallpaper.shtml)\n",
16 | "\n",
17 | "I decided to try playing around with a Kaggle competition. In this case, I entered the [\"When bag of words meets bags of popcorn\"](http://www.kaggle.com/c/word2vec-nlp-tutorial) contest. This contest isn't for money; it is just a way to learn about various machine learning approaches. "
18 | ]
19 | },
20 | {
21 | "cell_type": "markdown",
22 | "metadata": {},
23 | "source": [
24 | "The competition was trying to showcase Google's [Word2Vec](https://code.google.com/p/word2vec/). This essentially uses deep learning to find features in text that can be used to help in classification tasks. Specifically, in the case of this contest, the goal involves labeling the sentiment of a movie review from IMDB. Ratings were on a 10 point scale, and any review of 7 or greater was considered a positive movie review."
25 | ]
26 | },
27 | {
28 | "cell_type": "markdown",
29 | "metadata": {},
30 | "source": [
31 | "Originally, I was going to try out Word2Vec and train it on unlabeled reviews, but then one of the competitors [pointed out](http://www.kaggle.com/c/word2vec-nlp-tutorial/forums/t/11261/beat-the-benchmark-with-shallow-learning-0-95-lb) that you could simply use a less complicated classifier to do this and still get a good result. "
32 | ]
33 | },
34 | {
35 | "cell_type": "markdown",
36 | "metadata": {},
37 | "source": [
38 | "I decided to take this basic inspiration and try a few various classifiers to see what I could come up with. The highest my score received was 6th place back in December of 2014, but then people started using [ensemble methods](http://sebastianraschka.com/Articles/2014_ensemble_classifier.html) to combine various models together and get a perfect score after a lot of fine tuning with the parameters of the ensemble weights. "
39 | ]
40 | },
41 | {
42 | "cell_type": "markdown",
43 | "metadata": {},
44 | "source": [
45 | "Hopefully, this notebook will help you understand some basic NLP (Natural Language Processing) techniques, along with some tips on using [scikit-learn](http://scikit-learn.org/stable/) to make your classification models."
46 | ]
47 | },
48 | {
49 | "cell_type": "markdown",
50 | "metadata": {},
51 | "source": [
52 | "## Cleaning the Reviews"
53 | ]
54 | },
55 | {
56 | "cell_type": "markdown",
57 | "metadata": {},
58 | "source": [
59 | "The first thing we need to do is create a simple function that will clean the reviews into a format we can use. We just want the raw text, not all of the other associated HTML, symbols, or other junk. \n",
60 | "\n",
61 | "We will need a couple of very nice libraries for this task: BeautifulSoup for taking care of anything HTML related and re for regular expressions. "
62 | ]
63 | },
64 | {
65 | "cell_type": "code",
66 | "execution_count": 2,
67 | "metadata": {
68 | "collapsed": false
69 | },
70 | "outputs": [],
71 | "source": [
72 | "import re\n",
73 | "from bs4 import BeautifulSoup "
74 | ]
75 | },
76 | {
77 | "cell_type": "markdown",
78 | "metadata": {},
79 | "source": [
80 | "Now set up our function. This will clean all of the reviews for us."
81 | ]
82 | },
83 | {
84 | "cell_type": "code",
85 | "execution_count": 3,
86 | "metadata": {
87 | "collapsed": false
88 | },
89 | "outputs": [],
90 | "source": [
91 | "def review_to_wordlist(review):\n",
92 | " '''\n",
93 | " Meant for converting each of the IMDB reviews into a list of words.\n",
94 | " '''\n",
95 | " # First remove the HTML.\n",
96 | " review_text = BeautifulSoup(review).get_text()\n",
97 | " \n",
98 | " # Use regular expressions to only include words.\n",
99 | " review_text = re.sub(\"[^a-zA-Z]\",\" \", review_text)\n",
100 | " \n",
101 | " # Convert words to lower case and split them into separate words.\n",
102 | " words = review_text.lower().split()\n",
103 | " \n",
104 | " # Return a list of words\n",
105 | " return(words)"
106 | ]
107 | },
108 | {
109 | "cell_type": "markdown",
110 | "metadata": {},
111 | "source": [
112 | "Great! Now it is time to go ahead and load our data in. For this, pandas is definitely the library of choice. If you want to follow along with a downloaded version of the notebook yourself, make sure you obtain the [data](http://www.kaggle.com/c/word2vec-nlp-tutorial/data) from Kaggle. You will need a Kaggle account in order to access it."
113 | ]
114 | },
115 | {
116 | "cell_type": "code",
117 | "execution_count": 4,
118 | "metadata": {
119 | "collapsed": false
120 | },
121 | "outputs": [],
122 | "source": [
123 | "import pandas as pd"
124 | ]
125 | },
126 | {
127 | "cell_type": "code",
128 | "execution_count": 5,
129 | "metadata": {
130 | "collapsed": false
131 | },
132 | "outputs": [],
133 | "source": [
134 | "train = pd.read_csv('labeledTrainData.tsv', header=0,\n",
135 | " delimiter=\"\\t\", quoting=3)\n",
136 | "test = pd.read_csv('testData.tsv', header=0, delimiter=\"\\t\",\n",
137 | " quoting=3 )\n",
138 | " \n",
139 | "# Import both the training and test data.\n"
140 | ]
141 | },
142 | {
143 | "cell_type": "markdown",
144 | "metadata": {},
145 | "source": [
146 | "Now it is time to get the labels from the training set for our reviews. That way, we can teach our classifier which reviews are positive vs. negative."
147 | ]
148 | },
149 | {
150 | "cell_type": "code",
151 | "execution_count": 6,
152 | "metadata": {
153 | "collapsed": false
154 | },
155 | "outputs": [],
156 | "source": [
157 | "y_train = train['sentiment']"
158 | ]
159 | },
160 | {
161 | "cell_type": "markdown",
162 | "metadata": {},
163 | "source": [
164 | "Now we need to clean both the train and test data to get it ready for the next part of our program. "
165 | ]
166 | },
167 | {
168 | "cell_type": "code",
169 | "execution_count": 8,
170 | "metadata": {
171 | "collapsed": false
172 | },
173 | "outputs": [],
174 | "source": [
175 | "traindata = []\n",
176 | "for i in xrange(0,len(train['review'])):\n",
177 | " traindata.append(\" \".join(review_to_wordlist(train['review'][i])))\n",
178 | "testdata = []\n",
179 | "for i in xrange(0,len(test['review'])):\n",
180 | " testdata.append(\" \".join(review_to_wordlist(test['review'][i])))"
181 | ]
182 | },
183 | {
184 | "cell_type": "markdown",
185 | "metadata": {},
186 | "source": [
187 | "## TF-IDF Vectorization"
188 | ]
189 | },
190 | {
191 | "cell_type": "markdown",
192 | "metadata": {},
193 | "source": [
194 | "The next thing we are going to do is make TF-IDF (term frequency-interdocument frequency) vectors of our reviews. In case you are not familiar with what this is doing, essentially we are going to evaluate how often a certain term occurs in a review, but normalize this somewhat by how many reviews a certain term also occurs in. [Wikipedia](http://en.wikipedia.org/wiki/Tf%E2%80%93idf) has an explanation that is sufficient if you want further information. \n",
195 | "\n",
196 | "This can be a great technique for helping to determine which words (or ngrams of words) will make good features to classify a review as positive or negative. "
197 | ]
198 | },
199 | {
200 | "cell_type": "markdown",
201 | "metadata": {},
202 | "source": [
203 | "To do this, we are going to use the TFIDF vectorizer from scikit-learn. Then, decide what settings to use. The documentation for the TFIDF class is available [here](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html).\n",
204 | "\n",
205 | "In the case of the example code on Kaggle, they decided to remove all stop words, along with ngrams up to a size of two (you could use more but this will require a LOT of memory, so be careful which settings you use!)"
206 | ]
207 | },
208 | {
209 | "cell_type": "code",
210 | "execution_count": 10,
211 | "metadata": {
212 | "collapsed": false
213 | },
214 | "outputs": [],
215 | "source": [
216 | "from sklearn.feature_extraction.text import TfidfVectorizer as TFIV"
217 | ]
218 | },
219 | {
220 | "cell_type": "code",
221 | "execution_count": 11,
222 | "metadata": {
223 | "collapsed": false
224 | },
225 | "outputs": [],
226 | "source": [
227 | "tfv = TFIV(min_df=3, max_features=None, \n",
228 | " strip_accents='unicode', analyzer='word',token_pattern=r'\\w{1,}',\n",
229 | " ngram_range=(1, 2), use_idf=1,smooth_idf=1,sublinear_tf=1,\n",
230 | " stop_words = 'english')"
231 | ]
232 | },
233 | {
234 | "cell_type": "markdown",
235 | "metadata": {},
236 | "source": [
237 | "Now that we have the vectorization object, we need to run this on all of the data (both training and testing) to make sure it is applied to both datasets. This could take some time on your computer!"
238 | ]
239 | },
240 | {
241 | "cell_type": "code",
242 | "execution_count": 12,
243 | "metadata": {
244 | "collapsed": false
245 | },
246 | "outputs": [],
247 | "source": [
248 | "X_all = traindata + testdata # Combine both to fit the TFIDF vectorization.\n",
249 | "lentrain = len(traindata)\n",
250 | "\n",
251 | "tfv.fit(X_all) # This is the slow part!\n",
252 | "X_all = tfv.transform(X_all)\n",
253 | "\n",
254 | "X = X_all[:lentrain] # Separate back into training and test sets. \n",
255 | "X_test = X_all[lentrain:]\n"
256 | ]
257 | },
258 | {
259 | "cell_type": "markdown",
260 | "metadata": {},
261 | "source": [
262 | "## Making Our Classifiers "
263 | ]
264 | },
265 | {
266 | "cell_type": "markdown",
267 | "metadata": {},
268 | "source": [
269 | "Because we are working with text data, and we just made feature vectors of every word (that isn't a stop word of course) in all of the reviews, we are going to have sparse matrices to deal with that are quite large in size. Just to show you what I mean, let's examine the shape of our training set. "
270 | ]
271 | },
272 | {
273 | "cell_type": "code",
274 | "execution_count": 14,
275 | "metadata": {
276 | "collapsed": false
277 | },
278 | "outputs": [
279 | {
280 | "data": {
281 | "text/plain": [
282 | "(25000, 309798)"
283 | ]
284 | },
285 | "execution_count": 14,
286 | "metadata": {},
287 | "output_type": "execute_result"
288 | }
289 | ],
290 | "source": [
291 | "X.shape"
292 | ]
293 | },
294 | {
295 | "cell_type": "markdown",
296 | "metadata": {},
297 | "source": [
298 | "That means we have 25,000 training examples (or rows) and 309,798 features (or columns). We need something that is going to be somewhat computationally efficient given how many features we have. Using something like a random forest to classify would be unwieldy (plus random forests can't work with sparse matrices anyway yet in scikit-learn). That means we need something lightweight and fast that scales to many dimensions well. Some possible candidates are:\n",
299 | "\n",
300 | "* Naive Bayes\n",
301 | "* Logistic Regression\n",
302 | "* SGD Classifier (utilizes Stochastic Gradient Descent for much faster runtime)\n",
303 | "\n",
304 | "Let's just try all three as submissions to Kaggle and see how they perform. \n",
305 | "\n",
306 | "First up: Logistic Regression (see the scikit-learn documentation [here](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html))."
307 | ]
308 | },
309 | {
310 | "cell_type": "markdown",
311 | "metadata": {},
312 | "source": [
313 | "While in theory L1 regularization should work well because p>>n (many more features than training examples), I actually found through a lot of testing that L2 regularization got better results. You could set up your own trials using scikit-learn's built-in GridSearch class, which makes things a lot easier to try. I found through my testing that using a parameter C of 30 got the best results."
314 | ]
315 | },
316 | {
317 | "cell_type": "code",
318 | "execution_count": 15,
319 | "metadata": {
320 | "collapsed": false
321 | },
322 | "outputs": [],
323 | "source": [
324 | "from sklearn.linear_model import LogisticRegression as LR\n",
325 | "from sklearn.grid_search import GridSearchCV"
326 | ]
327 | },
328 | {
329 | "cell_type": "code",
330 | "execution_count": 17,
331 | "metadata": {
332 | "collapsed": false
333 | },
334 | "outputs": [
335 | {
336 | "data": {
337 | "text/plain": [
338 | "GridSearchCV(cv=20,\n",
339 | " estimator=LogisticRegression(C=1.0, class_weight=None, dual=True, fit_intercept=True,\n",
340 | " intercept_scaling=1, penalty='L2', random_state=0, tol=0.0001),\n",
341 | " fit_params={}, iid=True, loss_func=None, n_jobs=1,\n",
342 | " param_grid={'C': [30]}, pre_dispatch='2*n_jobs', refit=True,\n",
343 | " score_func=None, scoring='roc_auc', verbose=0)"
344 | ]
345 | },
346 | "execution_count": 17,
347 | "metadata": {},
348 | "output_type": "execute_result"
349 | }
350 | ],
351 | "source": [
352 | "grid_values = {'C':[30]} # Decide which settings you want for the grid search. \n",
353 | "\n",
354 | "model_LR = GridSearchCV(LR(penalty = 'L2', dual = True, random_state = 0), \n",
355 | " grid_values, scoring = 'roc_auc', cv = 20) \n",
356 | "# Try to set the scoring on what the contest is asking for. \n",
357 | "# The contest says scoring is for area under the ROC curve, so use this.\n",
358 | " \n",
359 | "model_LR.fit(X,y_train) # Fit the model."
360 | ]
361 | },
362 | {
363 | "cell_type": "markdown",
364 | "metadata": {},
365 | "source": [
366 | "You can investigate which parameters did the best and what scores they received by looking at the model_LR object."
367 | ]
368 | },
369 | {
370 | "cell_type": "code",
371 | "execution_count": 18,
372 | "metadata": {
373 | "collapsed": false
374 | },
375 | "outputs": [
376 | {
377 | "data": {
378 | "text/plain": [
379 | "[mean: 0.96459, std: 0.00489, params: {'C': 30}]"
380 | ]
381 | },
382 | "execution_count": 18,
383 | "metadata": {},
384 | "output_type": "execute_result"
385 | }
386 | ],
387 | "source": [
388 | "model_LR.grid_scores_"
389 | ]
390 | },
391 | {
392 | "cell_type": "code",
393 | "execution_count": 19,
394 | "metadata": {
395 | "collapsed": false
396 | },
397 | "outputs": [
398 | {
399 | "data": {
400 | "text/plain": [
401 | "LogisticRegression(C=30, class_weight=None, dual=True, fit_intercept=True,\n",
402 | " intercept_scaling=1, penalty='L2', random_state=0, tol=0.0001)"
403 | ]
404 | },
405 | "execution_count": 19,
406 | "metadata": {},
407 | "output_type": "execute_result"
408 | }
409 | ],
410 | "source": [
411 | "model_LR.best_estimator_"
412 | ]
413 | },
414 | {
415 | "cell_type": "markdown",
416 | "metadata": {},
417 | "source": [
418 | "Feel free, if you have an interactive version of the notebook, to play around with various settings inside the grid_values object to optimize your ROC_AUC score. Otherwise, let's move on to the next classifier, Naive Bayes. \n",
419 | "\n",
420 | "Unlike Logistic Regression, Naive Bayes doesn't have a regularization parameter to tune. You just have to choose which \"flavor\" of Naive Bayes to use. \n",
421 | "\n",
422 | "According to the [documentation on Naive Bayes from scikit-learn](http://scikit-learn.org/0.13/modules/naive_bayes.html), Multinomial is our best version to use, since we no longer have just a 1 or 0 for a word feature: it has been normalized by TF-IDF, so our values will be BETWEEN 0 and 1 (most of the time, although having a few TF-IDF scores exceed 1 is technically possible). If we were just looking at word occurrence vectors (with no counting), Bernoulli would have been a better fit since it is based on binary values. \n",
423 | "\n",
424 | "Let's make our Multinomial Naive Bayes object, and train it."
425 | ]
426 | },
427 | {
428 | "cell_type": "code",
429 | "execution_count": 20,
430 | "metadata": {
431 | "collapsed": false
432 | },
433 | "outputs": [],
434 | "source": [
435 | "from sklearn.naive_bayes import MultinomialNB as MNB"
436 | ]
437 | },
438 | {
439 | "cell_type": "code",
440 | "execution_count": 21,
441 | "metadata": {
442 | "collapsed": false
443 | },
444 | "outputs": [
445 | {
446 | "data": {
447 | "text/plain": [
448 | "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
449 | ]
450 | },
451 | "execution_count": 21,
452 | "metadata": {},
453 | "output_type": "execute_result"
454 | }
455 | ],
456 | "source": [
457 | "model_NB = MNB()\n",
458 | "model_NB.fit(X, y_train)"
459 | ]
460 | },
461 | {
462 | "cell_type": "markdown",
463 | "metadata": {},
464 | "source": [
465 | "Pretty fast, right? This speed comes at a price, however. Naive Bayes assumes all of your features are ENTIRELY independent from each other. In the case of word vectors, that seems like a somewhat reasonable assumption but with the ngrams we included that probably isn't always the case. Because of this, Naive Bayes tends to be less accurate than other classification algorithms, especially if you have a smaller number of training examples. \n",
466 | "\n",
467 | "Why don't we see how Naive Bayes does (at least in a 20 fold CV comparison) so we have a rough idea of how well it performs compared to our Logistic Regression classifier?\n",
468 | "\n",
469 | "You could use GridSearch again, but that seems like overkill. There is a simpler method we can import from scikit-learn for this task."
470 | ]
471 | },
472 | {
473 | "cell_type": "code",
474 | "execution_count": 23,
475 | "metadata": {
476 | "collapsed": false
477 | },
478 | "outputs": [],
479 | "source": [
480 | "from sklearn.cross_validation import cross_val_score\n",
481 | "import numpy as np"
482 | ]
483 | },
484 | {
485 | "cell_type": "code",
486 | "execution_count": 24,
487 | "metadata": {
488 | "collapsed": false
489 | },
490 | "outputs": [
491 | {
492 | "name": "stdout",
493 | "output_type": "stream",
494 | "text": [
495 | "20 Fold CV Score for Multinomial Naive Bayes: 0.949631232\n"
496 | ]
497 | }
498 | ],
499 | "source": [
500 | "print \"20 Fold CV Score for Multinomial Naive Bayes: \", np.mean(cross_val_score\n",
501 | " (model_NB, X, y_train, cv=20, scoring='roc_auc'))\n",
502 | " # This will give us a 20-fold cross validation score that looks at ROC_AUC so we can compare with Logistic Regression. "
503 | ]
504 | },
505 | {
506 | "cell_type": "markdown",
507 | "metadata": {},
508 | "source": [
509 | "Well, it wasn't quite as good as our well-tuned Logistic Regression classifier, but that is a pretty good score considering how little we had to do!\n",
510 | "\n",
511 | "One last classifier to try is the [SGD classifier](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html), which comes in handy when you need speed on a really large number of training examples/features. \n",
512 | "\n",
513 | "Which machine learning algorithm it ends up using depends on what you set for the loss function. If we chose loss = 'log', it would essentially be identical to our previous logistic regression model. We want to try something different, but we also want a loss option that includes probabilities. We need those probabilities if we are going to be able to calculate the area under a ROC curve. Looking at the documentation, it seems a 'modified_huber' loss would do the trick! This will be a Support Vector Machine that uses a linear kernel. \n",
514 | "\n"
515 | ]
516 | },
517 | {
518 | "cell_type": "code",
519 | "execution_count": 25,
520 | "metadata": {
521 | "collapsed": false
522 | },
523 | "outputs": [],
524 | "source": [
525 | "from sklearn.linear_model import SGDClassifier as SGD"
526 | ]
527 | },
528 | {
529 | "cell_type": "code",
530 | "execution_count": 26,
531 | "metadata": {
532 | "collapsed": false
533 | },
534 | "outputs": [
535 | {
536 | "data": {
537 | "text/plain": [
538 | "GridSearchCV(cv=20,\n",
539 | " estimator=SGDClassifier(alpha=0.0001, class_weight=None, epsilon=0.1, eta0=0.0,\n",
540 | " fit_intercept=True, l1_ratio=0.15, learning_rate='optimal',\n",
541 | " loss='modified_huber', n_iter=5, n_jobs=1, penalty='l2',\n",
542 | " power_t=0.5, random_state=0, shuffle=True, verbose=0,\n",
543 | " warm_start=False),\n",
544 | " fit_params={}, iid=True, loss_func=None, n_jobs=1,\n",
545 | " param_grid={'alpha': [6e-05, 7e-05, 8e-05, 0.0001, 0.0005]},\n",
546 | " pre_dispatch='2*n_jobs', refit=True, score_func=None,\n",
547 | " scoring='roc_auc', verbose=0)"
548 | ]
549 | },
550 | "execution_count": 26,
551 | "metadata": {},
552 | "output_type": "execute_result"
553 | }
554 | ],
555 | "source": [
556 | "sgd_params = {'alpha': [0.00006, 0.00007, 0.00008, 0.0001, 0.0005]} # Regularization parameter\n",
557 | "\n",
558 | "model_SGD = GridSearchCV(SGD(random_state = 0, shuffle = True, loss = 'modified_huber'), \n",
559 | " sgd_params, scoring = 'roc_auc', cv = 20) # Find out which regularization parameter works the best. \n",
560 | " \n",
561 | "model_SGD.fit(X, y_train) # Fit the model."
562 | ]
563 | },
564 | {
565 | "cell_type": "markdown",
566 | "metadata": {},
567 | "source": [
568 | "Again, similar to the Logistic Regression model, we can see which parameter did the best."
569 | ]
570 | },
571 | {
572 | "cell_type": "code",
573 | "execution_count": 27,
574 | "metadata": {
575 | "collapsed": false
576 | },
577 | "outputs": [
578 | {
579 | "data": {
580 | "text/plain": [
581 | "[mean: 0.96477, std: 0.00484, params: {'alpha': 6e-05},\n",
582 | " mean: 0.96484, std: 0.00481, params: {'alpha': 7e-05},\n",
583 | " mean: 0.96486, std: 0.00480, params: {'alpha': 8e-05},\n",
584 | " mean: 0.96479, std: 0.00480, params: {'alpha': 0.0001},\n",
585 | " mean: 0.95869, std: 0.00484, params: {'alpha': 0.0005}]"
586 | ]
587 | },
588 | "execution_count": 27,
589 | "metadata": {},
590 | "output_type": "execute_result"
591 | }
592 | ],
593 | "source": [
594 | "model_SGD.grid_scores_"
595 | ]
596 | },
597 | {
598 | "cell_type": "markdown",
599 | "metadata": {},
600 | "source": [
601 | "Looks like this beat our previous Logistic Regression model by a very small amount. Now that we have our three models, we can work on submitting our final scores in the proper format. It was found that submitting predicted probabilities of each score instead of the final predicted score worked better for evaluation from the contest participants, so we want to output this instead. \n",
602 | "\n",
603 | "First, do our Logistic Regression submission. \n",
604 | "\n"
605 | ]
606 | },
607 | {
608 | "cell_type": "code",
609 | "execution_count": 28,
610 | "metadata": {
611 | "collapsed": false
612 | },
613 | "outputs": [],
614 | "source": [
615 | "LR_result = model_LR.predict_proba(X_test)[:,1] # We only need the probabilities that the movie review was a 7 or greater. \n",
616 | "LR_output = pd.DataFrame(data={\"id\":test[\"id\"], \"sentiment\":LR_result}) # Create our dataframe that will be written.\n",
617 | "LR_output.to_csv('Logistic_Reg_Proj2.csv', index=False, quoting=3) # Get the .csv file we will submit to Kaggle."
618 | ]
619 | },
620 | {
621 | "cell_type": "markdown",
622 | "metadata": {},
623 | "source": [
624 | "Repeat this with the other two."
625 | ]
626 | },
627 | {
628 | "cell_type": "code",
629 | "execution_count": 29,
630 | "metadata": {
631 | "collapsed": false
632 | },
633 | "outputs": [],
634 | "source": [
635 | "# Repeat this for Multinomial Naive Bayes\n",
636 | "\n",
637 | "MNB_result = model_NB.predict_proba(X_test)[:,1]\n",
638 | "MNB_output = pd.DataFrame(data={\"id\":test[\"id\"], \"sentiment\":MNB_result})\n",
639 | "MNB_output.to_csv('MNB_Proj2.csv', index = False, quoting = 3)\n",
640 | "\n",
641 | "# Last, do the Stochastic Gradient Descent model with modified Huber loss.\n",
642 | "\n",
643 | "SGD_result = model_SGD.predict_proba(X_test)[:,1]\n",
644 | "SGD_output = pd.DataFrame(data={\"id\":test[\"id\"], \"sentiment\":SGD_result})\n",
645 | "SGD_output.to_csv('SGD_Proj2.csv', index = False, quoting = 3)"
646 | ]
647 | },
648 | {
649 | "cell_type": "markdown",
650 | "metadata": {},
651 | "source": [
652 | "Submitting the SGD result (using the linear SVM with modified Huber loss), I received a score of 0.95673 on the Kaggle [leaderboard](http://www.kaggle.com/c/word2vec-nlp-tutorial/leaderboard). That was good enough for sixth place back in December of 2014. "
653 | ]
654 | },
655 | {
656 | "cell_type": "markdown",
657 | "metadata": {},
658 | "source": [
659 | "## Ideas for Improvement and Summary"
660 | ]
661 | },
662 | {
663 | "cell_type": "markdown",
664 | "metadata": {},
665 | "source": [
666 | "In this notebook, we examined a text classification problem and cleaned unstructured review data. Next, we created a vector of features using TF-IDF normalization on a Bag of Words. We then trained these features on three different classifiers, some of which were optimized using 20-fold cross-validation, and made a submission to a Kaggle competition.\n",
667 | "\n",
668 | "Possible ideas for improvement:\n",
669 | "\n",
670 | "- Try increasing the number of ngrams to 3 or 4 in the TF-IDF vectorization and see if this makes a difference\n",
671 | "- Blend the models together into an ensemble that uses a majority vote for the classifiers\n",
672 | "- Try utilizing Word2Vec and creating feature vectors from the unlabeled training data. More data usually helps!"
673 | ]
674 | }
675 | ],
676 | "metadata": {
677 | "kernelspec": {
678 | "display_name": "Python 2",
679 | "language": "python",
680 | "name": "python2"
681 | },
682 | "language_info": {
683 | "codemirror_mode": {
684 | "name": "ipython",
685 | "version": 2
686 | },
687 | "file_extension": ".py",
688 | "mimetype": "text/x-python",
689 | "name": "python",
690 | "nbconvert_exporter": "python",
691 | "pygments_lexer": "ipython2",
692 | "version": "2.7.6"
693 | }
694 | },
695 | "nbformat": 4,
696 | "nbformat_minor": 0
697 | }
698 |
--------------------------------------------------------------------------------
/Penn_State.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jmsteinw/Notebooks/76915d49c3ea61a88f68a016eadf83583741025d/Penn_State.jpg
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Notebooks
2 | This is where all of the IPython Notebooks will be kept from the blog
3 |
--------------------------------------------------------------------------------
/RecEngine_NB.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# A Gentle Introduction to Recommender Systems with Implicit Feedback"
8 | ]
9 | },
10 | {
11 | "cell_type": "markdown",
12 | "metadata": {
13 | "collapsed": true
14 | },
15 | "source": [
16 | ""
17 | ]
18 | },
19 | {
20 | "cell_type": "markdown",
21 | "metadata": {},
22 | "source": [
23 | "## Part 1: Introduction"
24 | ]
25 | },
26 | {
27 | "cell_type": "markdown",
28 | "metadata": {},
29 | "source": [
30 | "[Recommender systems](https://en.wikipedia.org/wiki/Recommender_system) have become a very important part of the retail, social networking, and entertainment industries. From providing advice on songs for you to try, suggesting books for you to read, or finding clothes to buy, recommender systems have greatly improved the ability of customers to make choices more easily. \n",
31 | "\n",
32 | "Why is it so important for customers to have support in decision making? A well-cited study (over 2000 citations so far!) by [Iyengar and Lepper](https://faculty.washington.edu/jdb/345/345%20Articles/Iyengar%20%26%20Lepper%20%282000%29.pdf) ran an experiment where they had two stands of jam on two different days. One stand had 24 varieties of jam while the second had only six. The stand with 24 varieties of jam only converted 3% of the customers to a sale, while the stand with only six varieties converted 30% of the customers. This was an increase in sales of nearly ten-fold!"
33 | ]
34 | },
35 | {
36 | "cell_type": "markdown",
37 | "metadata": {},
38 | "source": [
39 | "Given the number of possible choices available, especially for online shopping, having some extra guidance on these choices can really make a difference. Xavier Amatriain, now at Quora and previously at Netflix, gave an absolutely outstanding talk on recommender systems at Carnegie Mellon in 2014. I have included the talk below if you would like to see it."
40 | ]
41 | },
42 | {
43 | "cell_type": "code",
44 | "execution_count": 1,
45 | "metadata": {
46 | "collapsed": false
47 | },
48 | "outputs": [
49 | {
50 | "data": {
51 | "text/html": [
52 | "\n",
53 | " \n",
278 | "\n",
279 | " \n",
280 | "
\n",
350 | "
"
351 | ],
352 | "text/plain": [
353 | " InvoiceNo StockCode Description Quantity \\\n",
354 | "0 536365 85123A WHITE HANGING HEART T-LIGHT HOLDER 6 \n",
355 | "1 536365 71053 WHITE METAL LANTERN 6 \n",
356 | "2 536365 84406B CREAM CUPID HEARTS COAT HANGER 8 \n",
357 | "3 536365 84029G KNITTED UNION FLAG HOT WATER BOTTLE 6 \n",
358 | "4 536365 84029E RED WOOLLY HOTTIE WHITE HEART. 6 \n",
359 | "\n",
360 | " InvoiceDate UnitPrice CustomerID Country \n",
361 | "0 2010-12-01 08:26:00 2.55 17850.0 United Kingdom \n",
362 | "1 2010-12-01 08:26:00 3.39 17850.0 United Kingdom \n",
363 | "2 2010-12-01 08:26:00 2.75 17850.0 United Kingdom \n",
364 | "3 2010-12-01 08:26:00 3.39 17850.0 United Kingdom \n",
365 | "4 2010-12-01 08:26:00 3.39 17850.0 United Kingdom "
366 | ]
367 | },
368 | "execution_count": 4,
369 | "metadata": {},
370 | "output_type": "execute_result"
371 | }
372 | ],
373 | "source": [
374 | "retail_data.head()"
375 | ]
376 | },
377 | {
378 | "cell_type": "markdown",
379 | "metadata": {},
380 | "source": [
381 | "The dataset includes the invoice number for different purchases, along with the StockCode (or item ID), an item description, the number purchased, the date of purchase, the price of the items, a customer ID, and the country of origin for the customer. \n",
382 | "\n",
383 | "Let's check to see if there are any missing values in the data. "
384 | ]
385 | },
386 | {
387 | "cell_type": "code",
388 | "execution_count": 5,
389 | "metadata": {
390 | "collapsed": false
391 | },
392 | "outputs": [
393 | {
394 | "name": "stdout",
395 | "output_type": "stream",
396 | "text": [
397 | "| \n", 282 | " | InvoiceNo | \n", 283 | "StockCode | \n", 284 | "Description | \n", 285 | "Quantity | \n", 286 | "InvoiceDate | \n", 287 | "UnitPrice | \n", 288 | "CustomerID | \n", 289 | "Country | \n", 290 | "
|---|---|---|---|---|---|---|---|---|
| 0 | \n", 295 | "536365 | \n", 296 | "85123A | \n", 297 | "WHITE HANGING HEART T-LIGHT HOLDER | \n", 298 | "6 | \n", 299 | "2010-12-01 08:26:00 | \n", 300 | "2.55 | \n", 301 | "17850.0 | \n", 302 | "United Kingdom | \n", 303 | "
| 1 | \n", 306 | "536365 | \n", 307 | "71053 | \n", 308 | "WHITE METAL LANTERN | \n", 309 | "6 | \n", 310 | "2010-12-01 08:26:00 | \n", 311 | "3.39 | \n", 312 | "17850.0 | \n", 313 | "United Kingdom | \n", 314 | "
| 2 | \n", 317 | "536365 | \n", 318 | "84406B | \n", 319 | "CREAM CUPID HEARTS COAT HANGER | \n", 320 | "8 | \n", 321 | "2010-12-01 08:26:00 | \n", 322 | "2.75 | \n", 323 | "17850.0 | \n", 324 | "United Kingdom | \n", 325 | "
| 3 | \n", 328 | "536365 | \n", 329 | "84029G | \n", 330 | "KNITTED UNION FLAG HOT WATER BOTTLE | \n", 331 | "6 | \n", 332 | "2010-12-01 08:26:00 | \n", 333 | "3.39 | \n", 334 | "17850.0 | \n", 335 | "United Kingdom | \n", 336 | "
| 4 | \n", 339 | "536365 | \n", 340 | "84029E | \n", 341 | "RED WOOLLY HOTTIE WHITE HEART. | \n", 342 | "6 | \n", 343 | "2010-12-01 08:26:00 | \n", 344 | "3.39 | \n", 345 | "17850.0 | \n", 346 | "United Kingdom | \n", 347 | "
\n",
498 | "\n",
499 | " \n",
500 | "
\n",
534 | "
"
535 | ],
536 | "text/plain": [
537 | " StockCode Description\n",
538 | "0 85123A WHITE HANGING HEART T-LIGHT HOLDER\n",
539 | "1 71053 WHITE METAL LANTERN\n",
540 | "2 84406B CREAM CUPID HEARTS COAT HANGER\n",
541 | "3 84029G KNITTED UNION FLAG HOT WATER BOTTLE\n",
542 | "4 84029E RED WOOLLY HOTTIE WHITE HEART."
543 | ]
544 | },
545 | "execution_count": 9,
546 | "metadata": {},
547 | "output_type": "execute_result"
548 | }
549 | ],
550 | "source": [
551 | "item_lookup.head()"
552 | ]
553 | },
554 | {
555 | "cell_type": "markdown",
556 | "metadata": {},
557 | "source": [
558 | "This can tell us what each item is, such as that StockCode 71053 is a white metal lantern. Now that this has been created, we need to:\n",
559 | "\n",
560 | "- Group purchase quantities together by stock code and item ID\n",
561 | "- Change any sums that equal zero to one (this can happen if items were returned, but we want to indicate that the user actually purchased the item instead of assuming no interaction between the user and the item ever took place)\n",
562 | "- Only include customers with a positive purchase total to eliminate possible errors\n",
563 | "- Set up our sparse ratings matrix\n",
564 | "\n",
565 | "This last step is especially important if you don't want to have unnecessary memory issues! If you think about it, our matrix is going to contain thousands of items and thousands of users with a user/item value required for every possible combination. That is a LARGE matrix, so we can save a lot of memory by keeping the matrix sparse and only saving the locations and values of items that are not zero. \n",
566 | "\n",
567 | "The code below will finish the preprocessing steps necessary for our final ratings sparse matrix:"
568 | ]
569 | },
570 | {
571 | "cell_type": "code",
572 | "execution_count": 10,
573 | "metadata": {
574 | "collapsed": false
575 | },
576 | "outputs": [
577 | {
578 | "name": "stderr",
579 | "output_type": "stream",
580 | "text": [
581 | "//anaconda/lib/python3.5/site-packages/ipykernel/__main__.py:1: SettingWithCopyWarning: \n",
582 | "A value is trying to be set on a copy of a slice from a DataFrame.\n",
583 | "Try using .loc[row_indexer,col_indexer] = value instead\n",
584 | "\n",
585 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
586 | " if __name__ == '__main__':\n",
587 | "//anaconda/lib/python3.5/site-packages/pandas/core/indexing.py:128: SettingWithCopyWarning: \n",
588 | "A value is trying to be set on a copy of a slice from a DataFrame\n",
589 | "\n",
590 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
591 | " self._setitem_with_indexer(indexer, value)\n"
592 | ]
593 | }
594 | ],
595 | "source": [
596 | "cleaned_retail['CustomerID'] = cleaned_retail.CustomerID.astype(int) # Convert to int for customer ID\n",
597 | "cleaned_retail = cleaned_retail[['StockCode', 'Quantity', 'CustomerID']] # Get rid of unnecessary info\n",
598 | "grouped_cleaned = cleaned_retail.groupby(['CustomerID', 'StockCode']).sum().reset_index() # Group together\n",
599 | "grouped_cleaned.Quantity.loc[grouped_cleaned.Quantity == 0] = 1 # Replace a sum of zero purchases with a one to\n",
600 | "# indicate purchased\n",
601 | "grouped_purchased = grouped_cleaned.query('Quantity > 0') # Only get customers where purchase totals were positive"
602 | ]
603 | },
604 | {
605 | "cell_type": "markdown",
606 | "metadata": {},
607 | "source": [
608 | "If we look at our final resulting matrix of grouped purchases, we see the following:"
609 | ]
610 | },
611 | {
612 | "cell_type": "code",
613 | "execution_count": 11,
614 | "metadata": {
615 | "collapsed": false
616 | },
617 | "outputs": [
618 | {
619 | "data": {
620 | "text/html": [
621 | "| \n", 502 | " | StockCode | \n", 503 | "Description | \n", 504 | "
|---|---|---|
| 0 | \n", 509 | "85123A | \n", 510 | "WHITE HANGING HEART T-LIGHT HOLDER | \n", 511 | "
| 1 | \n", 514 | "71053 | \n", 515 | "WHITE METAL LANTERN | \n", 516 | "
| 2 | \n", 519 | "84406B | \n", 520 | "CREAM CUPID HEARTS COAT HANGER | \n", 521 | "
| 3 | \n", 524 | "84029G | \n", 525 | "KNITTED UNION FLAG HOT WATER BOTTLE | \n", 526 | "
| 4 | \n", 529 | "84029E | \n", 530 | "RED WOOLLY HOTTIE WHITE HEART. | \n", 531 | "
\n",
622 | "\n",
623 | " \n",
624 | "
\n",
664 | "
"
665 | ],
666 | "text/plain": [
667 | " CustomerID StockCode Quantity\n",
668 | "0 12346 23166 1\n",
669 | "1 12347 16008 24\n",
670 | "2 12347 17021 36\n",
671 | "3 12347 20665 6\n",
672 | "4 12347 20719 40"
673 | ]
674 | },
675 | "execution_count": 11,
676 | "metadata": {},
677 | "output_type": "execute_result"
678 | }
679 | ],
680 | "source": [
681 | "grouped_purchased.head()"
682 | ]
683 | },
684 | {
685 | "cell_type": "markdown",
686 | "metadata": {},
687 | "source": [
688 | "Instead of representing an explicit rating, the purchase quantity can represent a \"confidence\" in terms of how strong the interaction was. Items with a larger number of purchases by a customer can carry more weight in our ratings matrix of purchases. \n",
689 | "\n",
690 | "Our last step is to create the sparse ratings matrix of users and items utilizing the code below:"
691 | ]
692 | },
693 | {
694 | "cell_type": "code",
695 | "execution_count": 12,
696 | "metadata": {
697 | "collapsed": true
698 | },
699 | "outputs": [],
700 | "source": [
701 | "customers = list(np.sort(grouped_purchased.CustomerID.unique())) # Get our unique customers\n",
702 | "products = list(grouped_purchased.StockCode.unique()) # Get our unique products that were purchased\n",
703 | "quantity = list(grouped_purchased.Quantity) # All of our purchases\n",
704 | "\n",
705 | "rows = grouped_purchased.CustomerID.astype('category', categories = customers).cat.codes \n",
706 | "# Get the associated row indices\n",
707 | "cols = grouped_purchased.StockCode.astype('category', categories = products).cat.codes \n",
708 | "# Get the associated column indices\n",
709 | "purchases_sparse = sparse.csr_matrix((quantity, (rows, cols)), shape=(len(customers), len(products)))"
710 | ]
711 | },
712 | {
713 | "cell_type": "markdown",
714 | "metadata": {},
715 | "source": [
716 | "Let's check our final matrix object:"
717 | ]
718 | },
719 | {
720 | "cell_type": "code",
721 | "execution_count": 13,
722 | "metadata": {
723 | "collapsed": false
724 | },
725 | "outputs": [
726 | {
727 | "data": {
728 | "text/plain": [
729 | "<4338x3664 sparse matrix of type '| \n", 626 | " | CustomerID | \n", 627 | "StockCode | \n", 628 | "Quantity | \n", 629 | "
|---|---|---|---|
| 0 | \n", 634 | "12346 | \n", 635 | "23166 | \n", 636 | "1 | \n", 637 | "
| 1 | \n", 640 | "12347 | \n", 641 | "16008 | \n", 642 | "24 | \n", 643 | "
| 2 | \n", 646 | "12347 | \n", 647 | "17021 | \n", 648 | "36 | \n", 649 | "
| 3 | \n", 652 | "12347 | \n", 653 | "20665 | \n", 654 | "6 | \n", 655 | "
| 4 | \n", 658 | "12347 | \n", 659 | "20719 | \n", 660 | "40 | \n", 661 | "
\n",
1487 | "\n",
1488 | " \n",
1489 | "
\n",
1503 | "
"
1504 | ],
1505 | "text/plain": [
1506 | " StockCode Description\n",
1507 | "61619 23166 MEDIUM CERAMIC TOP STORAGE JAR"
1508 | ]
1509 | },
1510 | "execution_count": 30,
1511 | "metadata": {},
1512 | "output_type": "execute_result"
1513 | }
1514 | ],
1515 | "source": [
1516 | "get_items_purchased(12346, product_train, customers_arr, products_arr, item_lookup)"
1517 | ]
1518 | },
1519 | {
1520 | "cell_type": "markdown",
1521 | "metadata": {},
1522 | "source": [
1523 | "We can see that the customer purchased a ceramic jar for storage, medium size. What items does the recommender system say this customer should purchase? We need to create another function that does this. Let's also import the MinMaxScaler from scikit-learn to help with this."
1524 | ]
1525 | },
1526 | {
1527 | "cell_type": "code",
1528 | "execution_count": 31,
1529 | "metadata": {
1530 | "collapsed": true
1531 | },
1532 | "outputs": [],
1533 | "source": [
1534 | "from sklearn.preprocessing import MinMaxScaler"
1535 | ]
1536 | },
1537 | {
1538 | "cell_type": "code",
1539 | "execution_count": 32,
1540 | "metadata": {
1541 | "collapsed": true
1542 | },
1543 | "outputs": [],
1544 | "source": [
1545 | "def rec_items(customer_id, mf_train, user_vecs, item_vecs, customer_list, item_list, item_lookup, num_items = 10):\n",
1546 | " '''\n",
1547 | " This function will return the top recommended items to our users \n",
1548 | " \n",
1549 | " parameters:\n",
1550 | " \n",
1551 | " customer_id - Input the customer's id number that you want to get recommendations for\n",
1552 | " \n",
1553 | " mf_train - The training matrix you used for matrix factorization fitting\n",
1554 | " \n",
1555 | " user_vecs - the user vectors from your fitted matrix factorization\n",
1556 | " \n",
1557 | " item_vecs - the item vectors from your fitted matrix factorization\n",
1558 | " \n",
1559 | " customer_list - an array of the customer's ID numbers that make up the rows of your ratings matrix \n",
1560 | " (in order of matrix)\n",
1561 | " \n",
1562 | " item_list - an array of the products that make up the columns of your ratings matrix\n",
1563 | " (in order of matrix)\n",
1564 | " \n",
1565 | " item_lookup - A simple pandas dataframe of the unique product ID/product descriptions available\n",
1566 | " \n",
1567 | " num_items - The number of items you want to recommend in order of best recommendations. Default is 10. \n",
1568 | " \n",
1569 | " returns:\n",
1570 | " \n",
1571 | " - The top n recommendations chosen based on the user/item vectors for items never interacted with/purchased\n",
1572 | " '''\n",
1573 | " \n",
1574 | " cust_ind = np.where(customer_list == customer_id)[0][0] # Returns the index row of our customer id\n",
1575 | " pref_vec = mf_train[cust_ind,:].toarray() # Get the ratings from the training set ratings matrix\n",
1576 | " pref_vec = pref_vec.reshape(-1) + 1 # Add 1 to everything, so that items not purchased yet become equal to 1\n",
1577 | " pref_vec[pref_vec > 1] = 0 # Make everything already purchased zero\n",
1578 | " rec_vector = user_vecs[cust_ind,:].dot(item_vecs.T) # Get dot product of user vector and all item vectors\n",
1579 | " # Scale this recommendation vector between 0 and 1\n",
1580 | " min_max = MinMaxScaler()\n",
1581 | " rec_vector_scaled = min_max.fit_transform(rec_vector.reshape(-1,1))[:,0] \n",
1582 | " recommend_vector = pref_vec*rec_vector_scaled \n",
1583 | " # Items already purchased have their recommendation multiplied by zero\n",
1584 | " product_idx = np.argsort(recommend_vector)[::-1][:num_items] # Sort the indices of the items into order \n",
1585 | " # of best recommendations\n",
1586 | " rec_list = [] # start empty list to store items\n",
1587 | " for index in product_idx:\n",
1588 | " code = item_list[index]\n",
1589 | " rec_list.append([code, item_lookup.Description.loc[item_lookup.StockCode == code].iloc[0]]) \n",
1590 | " # Append our descriptions to the list\n",
1591 | " codes = [item[0] for item in rec_list]\n",
1592 | " descriptions = [item[1] for item in rec_list]\n",
1593 | " final_frame = pd.DataFrame({'StockCode': codes, 'Description': descriptions}) # Create a dataframe \n",
1594 | " return final_frame[['StockCode', 'Description']] # Switch order of columns around"
1595 | ]
1596 | },
1597 | {
1598 | "cell_type": "markdown",
1599 | "metadata": {},
1600 | "source": [
1601 | "Essentially, this will retrieve the $N$ highest ranking dot products between our user and item vectors for a particular user. Items already purchased are not recommended to the user. For now, let's use a default of 10 items and see what the recommender system decides to pick for our customer. "
1602 | ]
1603 | },
1604 | {
1605 | "cell_type": "code",
1606 | "execution_count": 33,
1607 | "metadata": {
1608 | "collapsed": false
1609 | },
1610 | "outputs": [
1611 | {
1612 | "data": {
1613 | "text/html": [
1614 | "| \n", 1491 | " | StockCode | \n", 1492 | "Description | \n", 1493 | "
|---|---|---|
| 61619 | \n", 1498 | "23166 | \n", 1499 | "MEDIUM CERAMIC TOP STORAGE JAR | \n", 1500 | "
\n",
1615 | "\n",
1616 | " \n",
1617 | "
\n",
1676 | "
"
1677 | ],
1678 | "text/plain": [
1679 | " StockCode Description\n",
1680 | "0 23167 SMALL CERAMIC TOP STORAGE JAR \n",
1681 | "1 23165 LARGE CERAMIC TOP STORAGE JAR\n",
1682 | "2 22963 JAM JAR WITH GREEN LID\n",
1683 | "3 23294 SET OF 6 SNACK LOAF BAKING CASES\n",
1684 | "4 22980 PANTRY SCRUBBING BRUSH\n",
1685 | "5 23296 SET OF 6 TEA TIME BAKING CASES\n",
1686 | "6 23293 SET OF 12 FAIRY CAKE BAKING CASES\n",
1687 | "7 22978 PANTRY ROLLING PIN\n",
1688 | "8 23295 SET OF 12 MINI LOAF BAKING CASES\n",
1689 | "9 22962 JAM JAR WITH PINK LID"
1690 | ]
1691 | },
1692 | "execution_count": 33,
1693 | "metadata": {},
1694 | "output_type": "execute_result"
1695 | }
1696 | ],
1697 | "source": [
1698 | "rec_items(12346, product_train, user_vecs, item_vecs, customers_arr, products_arr, item_lookup,\n",
1699 | " num_items = 10)"
1700 | ]
1701 | },
1702 | {
1703 | "cell_type": "markdown",
1704 | "metadata": {},
1705 | "source": [
1706 | "These recommendations seem quite good! Remember that the recommendation system has no real understanding of what a ceramic jar is. All it knows is the purchase history. It identified that people purchasing a medium sized jar may also want to purchase jars of a differing size. The recommender system also suggests jar magnets and a sugar dispenser, which is similar in use to a storage jar. I personally was blown away by how well the system seems to pick up on these sorts of shopping patterns. Let's try another user that hasn't made a large number of purchases. "
1707 | ]
1708 | },
1709 | {
1710 | "cell_type": "code",
1711 | "execution_count": 34,
1712 | "metadata": {
1713 | "collapsed": false
1714 | },
1715 | "outputs": [
1716 | {
1717 | "data": {
1718 | "text/html": [
1719 | "| \n", 1619 | " | StockCode | \n", 1620 | "Description | \n", 1621 | "
|---|---|---|
| 0 | \n", 1626 | "23167 | \n", 1627 | "SMALL CERAMIC TOP STORAGE JAR | \n", 1628 | "
| 1 | \n", 1631 | "23165 | \n", 1632 | "LARGE CERAMIC TOP STORAGE JAR | \n", 1633 | "
| 2 | \n", 1636 | "22963 | \n", 1637 | "JAM JAR WITH GREEN LID | \n", 1638 | "
| 3 | \n", 1641 | "23294 | \n", 1642 | "SET OF 6 SNACK LOAF BAKING CASES | \n", 1643 | "
| 4 | \n", 1646 | "22980 | \n", 1647 | "PANTRY SCRUBBING BRUSH | \n", 1648 | "
| 5 | \n", 1651 | "23296 | \n", 1652 | "SET OF 6 TEA TIME BAKING CASES | \n", 1653 | "
| 6 | \n", 1656 | "23293 | \n", 1657 | "SET OF 12 FAIRY CAKE BAKING CASES | \n", 1658 | "
| 7 | \n", 1661 | "22978 | \n", 1662 | "PANTRY ROLLING PIN | \n", 1663 | "
| 8 | \n", 1666 | "23295 | \n", 1667 | "SET OF 12 MINI LOAF BAKING CASES | \n", 1668 | "
| 9 | \n", 1671 | "22962 | \n", 1672 | "JAM JAR WITH PINK LID | \n", 1673 | "
\n",
1720 | "\n",
1721 | " \n",
1722 | "
\n",
1751 | "
"
1752 | ],
1753 | "text/plain": [
1754 | " StockCode Description\n",
1755 | "2148 37446 MINI CAKE STAND WITH HANGING CAKES\n",
1756 | "2149 37449 CERAMIC CAKE STAND + HANGING CAKES\n",
1757 | "4859 37450 CERAMIC CAKE BOWL + HANGING CAKES\n",
1758 | "5108 22890 NOVELTY BISCUITS CAKE STAND 3 TIER"
1759 | ]
1760 | },
1761 | "execution_count": 34,
1762 | "metadata": {},
1763 | "output_type": "execute_result"
1764 | }
1765 | ],
1766 | "source": [
1767 | "get_items_purchased(12353, product_train, customers_arr, products_arr, item_lookup)"
1768 | ]
1769 | },
1770 | {
1771 | "cell_type": "markdown",
1772 | "metadata": {},
1773 | "source": [
1774 | "This person seems like they want to make cakes. What kind of items does the recommender system think they would be interested in?"
1775 | ]
1776 | },
1777 | {
1778 | "cell_type": "code",
1779 | "execution_count": 35,
1780 | "metadata": {
1781 | "collapsed": false
1782 | },
1783 | "outputs": [
1784 | {
1785 | "data": {
1786 | "text/html": [
1787 | "| \n", 1724 | " | StockCode | \n", 1725 | "Description | \n", 1726 | "
|---|---|---|
| 2148 | \n", 1731 | "37446 | \n", 1732 | "MINI CAKE STAND WITH HANGING CAKES | \n", 1733 | "
| 2149 | \n", 1736 | "37449 | \n", 1737 | "CERAMIC CAKE STAND + HANGING CAKES | \n", 1738 | "
| 4859 | \n", 1741 | "37450 | \n", 1742 | "CERAMIC CAKE BOWL + HANGING CAKES | \n", 1743 | "
| 5108 | \n", 1746 | "22890 | \n", 1747 | "NOVELTY BISCUITS CAKE STAND 3 TIER | \n", 1748 | "
\n",
1788 | "\n",
1789 | " \n",
1790 | "
\n",
1849 | "
"
1850 | ],
1851 | "text/plain": [
1852 | " StockCode Description\n",
1853 | "0 22645 CERAMIC HEART FAIRY CAKE MONEY BANK\n",
1854 | "1 22055 MINI CAKE STAND HANGING STRAWBERY\n",
1855 | "2 22644 CERAMIC CHERRY CAKE MONEY BANK\n",
1856 | "3 37447 CERAMIC CAKE DESIGN SPOTTED PLATE\n",
1857 | "4 37448 CERAMIC CAKE DESIGN SPOTTED MUG\n",
1858 | "5 22059 CERAMIC STRAWBERRY DESIGN MUG\n",
1859 | "6 22063 CERAMIC BOWL WITH STRAWBERRY DESIGN\n",
1860 | "7 22649 STRAWBERRY FAIRY CAKE TEAPOT\n",
1861 | "8 22057 CERAMIC PLATE STRAWBERRY DESIGN\n",
1862 | "9 22646 CERAMIC STRAWBERRY CAKE MONEY BANK"
1863 | ]
1864 | },
1865 | "execution_count": 35,
1866 | "metadata": {},
1867 | "output_type": "execute_result"
1868 | }
1869 | ],
1870 | "source": [
1871 | "rec_items(12353, product_train, user_vecs, item_vecs, customers_arr, products_arr, item_lookup,\n",
1872 | " num_items = 10)"
1873 | ]
1874 | },
1875 | {
1876 | "cell_type": "markdown",
1877 | "metadata": {},
1878 | "source": [
1879 | "It certainly picked up on the cake theme along with ceramic items. Again, these recommendations seem very impressive given the system doesn't understand the content behind the recommendations. Let's try one more."
1880 | ]
1881 | },
1882 | {
1883 | "cell_type": "code",
1884 | "execution_count": 36,
1885 | "metadata": {
1886 | "collapsed": false
1887 | },
1888 | "outputs": [
1889 | {
1890 | "data": {
1891 | "text/html": [
1892 | "| \n", 1792 | " | StockCode | \n", 1793 | "Description | \n", 1794 | "
|---|---|---|
| 0 | \n", 1799 | "22645 | \n", 1800 | "CERAMIC HEART FAIRY CAKE MONEY BANK | \n", 1801 | "
| 1 | \n", 1804 | "22055 | \n", 1805 | "MINI CAKE STAND HANGING STRAWBERY | \n", 1806 | "
| 2 | \n", 1809 | "22644 | \n", 1810 | "CERAMIC CHERRY CAKE MONEY BANK | \n", 1811 | "
| 3 | \n", 1814 | "37447 | \n", 1815 | "CERAMIC CAKE DESIGN SPOTTED PLATE | \n", 1816 | "
| 4 | \n", 1819 | "37448 | \n", 1820 | "CERAMIC CAKE DESIGN SPOTTED MUG | \n", 1821 | "
| 5 | \n", 1824 | "22059 | \n", 1825 | "CERAMIC STRAWBERRY DESIGN MUG | \n", 1826 | "
| 6 | \n", 1829 | "22063 | \n", 1830 | "CERAMIC BOWL WITH STRAWBERRY DESIGN | \n", 1831 | "
| 7 | \n", 1834 | "22649 | \n", 1835 | "STRAWBERRY FAIRY CAKE TEAPOT | \n", 1836 | "
| 8 | \n", 1839 | "22057 | \n", 1840 | "CERAMIC PLATE STRAWBERRY DESIGN | \n", 1841 | "
| 9 | \n", 1844 | "22646 | \n", 1845 | "CERAMIC STRAWBERRY CAKE MONEY BANK | \n", 1846 | "
\n",
1893 | "\n",
1894 | " \n",
1895 | "
\n",
1949 | "
"
1950 | ],
1951 | "text/plain": [
1952 | " StockCode Description\n",
1953 | "34 22326 ROUND SNACK BOXES SET OF4 WOODLAND \n",
1954 | "35 22629 SPACEBOY LUNCH BOX \n",
1955 | "37 22631 CIRCUS PARADE LUNCH BOX \n",
1956 | "93 20725 LUNCH BAG RED RETROSPOT\n",
1957 | "369 22382 LUNCH BAG SPACEBOY DESIGN \n",
1958 | "547 22328 ROUND SNACK BOXES SET OF 4 FRUITS \n",
1959 | "549 22630 DOLLY GIRL LUNCH BOX\n",
1960 | "1241 22555 PLASTERS IN TIN STRONGMAN\n",
1961 | "58132 20725 LUNCH BAG RED SPOTTY"
1962 | ]
1963 | },
1964 | "execution_count": 36,
1965 | "metadata": {},
1966 | "output_type": "execute_result"
1967 | }
1968 | ],
1969 | "source": [
1970 | "get_items_purchased(12361, product_train, customers_arr, products_arr, item_lookup)"
1971 | ]
1972 | },
1973 | {
1974 | "cell_type": "markdown",
1975 | "metadata": {},
1976 | "source": [
1977 | "This customer seems like they are buying products suitable for lunch time. What other items does the recommender system think they might like?"
1978 | ]
1979 | },
1980 | {
1981 | "cell_type": "code",
1982 | "execution_count": 37,
1983 | "metadata": {
1984 | "collapsed": false
1985 | },
1986 | "outputs": [
1987 | {
1988 | "data": {
1989 | "text/html": [
1990 | "| \n", 1897 | " | StockCode | \n", 1898 | "Description | \n", 1899 | "
|---|---|---|
| 34 | \n", 1904 | "22326 | \n", 1905 | "ROUND SNACK BOXES SET OF4 WOODLAND | \n", 1906 | "
| 35 | \n", 1909 | "22629 | \n", 1910 | "SPACEBOY LUNCH BOX | \n", 1911 | "
| 37 | \n", 1914 | "22631 | \n", 1915 | "CIRCUS PARADE LUNCH BOX | \n", 1916 | "
| 93 | \n", 1919 | "20725 | \n", 1920 | "LUNCH BAG RED RETROSPOT | \n", 1921 | "
| 369 | \n", 1924 | "22382 | \n", 1925 | "LUNCH BAG SPACEBOY DESIGN | \n", 1926 | "
| 547 | \n", 1929 | "22328 | \n", 1930 | "ROUND SNACK BOXES SET OF 4 FRUITS | \n", 1931 | "
| 549 | \n", 1934 | "22630 | \n", 1935 | "DOLLY GIRL LUNCH BOX | \n", 1936 | "
| 1241 | \n", 1939 | "22555 | \n", 1940 | "PLASTERS IN TIN STRONGMAN | \n", 1941 | "
| 58132 | \n", 1944 | "20725 | \n", 1945 | "LUNCH BAG RED SPOTTY | \n", 1946 | "
\n",
1991 | "\n",
1992 | " \n",
1993 | "
\n",
2052 | "
"
2053 | ],
2054 | "text/plain": [
2055 | " StockCode Description\n",
2056 | "0 22662 LUNCH BAG DOLLY GIRL DESIGN\n",
2057 | "1 20726 LUNCH BAG WOODLAND\n",
2058 | "2 20719 WOODLAND CHARLOTTE BAG\n",
2059 | "3 22383 LUNCH BAG SUKI DESIGN \n",
2060 | "4 20728 LUNCH BAG CARS BLUE\n",
2061 | "5 23209 LUNCH BAG DOILEY PATTERN \n",
2062 | "6 22661 CHARLOTTE BAG DOLLY GIRL DESIGN\n",
2063 | "7 20724 RED RETROSPOT CHARLOTTE BAG\n",
2064 | "8 23206 LUNCH BAG APPLE DESIGN\n",
2065 | "9 22384 LUNCH BAG PINK POLKADOT"
2066 | ]
2067 | },
2068 | "execution_count": 37,
2069 | "metadata": {},
2070 | "output_type": "execute_result"
2071 | }
2072 | ],
2073 | "source": [
2074 | "rec_items(12361, product_train, user_vecs, item_vecs, customers_arr, products_arr, item_lookup,\n",
2075 | " num_items = 10)"
2076 | ]
2077 | },
2078 | {
2079 | "cell_type": "markdown",
2080 | "metadata": {},
2081 | "source": [
2082 | "Once again, the recommender system comes through! Definitely a lot of bags and lunch related items in this recommendation list. Feel free to play around with the recommendations for other users and see what the system came up with!"
2083 | ]
2084 | },
2085 | {
2086 | "cell_type": "markdown",
2087 | "metadata": {},
2088 | "source": [
2089 | "## Summary"
2090 | ]
2091 | },
2092 | {
2093 | "cell_type": "markdown",
2094 | "metadata": {},
2095 | "source": [
2096 | "In this post, we have learned about how to design a recommender system with implicit feedback and how to provide recommendations. We also covered how to test the recommender system. \n",
2097 | "\n",
2098 | "In real life, if the size of your ratings matrix will not fit on a single machine very easily, utilizing the implementation in [Spark](http://spark.apache.org/docs/latest/mllib-collaborative-filtering.html) is going to be more practical. If you are interested in taking recommender systems to the next level, a hybrid system would be best that incorporates information about your users/items along with the purchase history. A Python library called LightFM from Maciej Kula at Lyst looks very interesting for this sort of application. You can find it [here](https://github.com/lyst/lightfm). \n",
2099 | "\n",
2100 | "Last, there are several other advanced methods you can incorporate in recommender systems to get a bump in performance. Part 2 of Xavier Amatriain's lecture would be a great place to [start](https://www.youtube.com/watch?v=mRToFXlNBpQ). \n",
2101 | "\n",
2102 | "If you are more interested in recommender systems with explicit feedback (such as with movie reviews) there are a couple of great posts that cover this in detail:\n",
2103 | "\n",
2104 | "- Alternating Least Squares Method for Collaborative Filtering by [Bugra Akyildiz](http://bugra.github.io/work/notes/2014-04-19/alternating-least-squares-method-for-collaborative-filtering/)\n",
2105 | "\n",
2106 | "- Explicit Matrix Factorization: ALS, SGD, and All That Jazz by [Ethan Rosenthal](http://blog.ethanrosenthal.com/2016/01/09/explicit-matrix-factorization-sgd-als/)\n",
2107 | "\n",
2108 | "If you are looking for great datasets to try a recommendation system out on for yourself, I found [this gist](https://gist.github.com/entaroadun/1653794) helpful. Some of the links don't work anymore but it's a great place to start looking for data to try a system out on your own!"
2109 | ]
2110 | }
2111 | ],
2112 | "metadata": {
2113 | "kernelspec": {
2114 | "display_name": "Python 3",
2115 | "language": "python",
2116 | "name": "python3"
2117 | },
2118 | "language_info": {
2119 | "codemirror_mode": {
2120 | "name": "ipython",
2121 | "version": 3
2122 | },
2123 | "file_extension": ".py",
2124 | "mimetype": "text/x-python",
2125 | "name": "python",
2126 | "nbconvert_exporter": "python",
2127 | "pygments_lexer": "ipython3",
2128 | "version": "3.5.1"
2129 | }
2130 | },
2131 | "nbformat": 4,
2132 | "nbformat_minor": 0
2133 | }
2134 |
--------------------------------------------------------------------------------
/Rec_Engine_Image_Amazon.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jmsteinw/Notebooks/76915d49c3ea61a88f68a016eadf83583741025d/Rec_Engine_Image_Amazon.png
--------------------------------------------------------------------------------
/Test_Image.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jmsteinw/Notebooks/76915d49c3ea61a88f68a016eadf83583741025d/Test_Image.png
--------------------------------------------------------------------------------
/Traintest_ex.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jmsteinw/Notebooks/76915d49c3ea61a88f68a016eadf83583741025d/Traintest_ex.png
--------------------------------------------------------------------------------
/XG_Cover.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jmsteinw/Notebooks/76915d49c3ea61a88f68a016eadf83583741025d/XG_Cover.jpg
--------------------------------------------------------------------------------
| \n", 1995 | " | StockCode | \n", 1996 | "Description | \n", 1997 | "
|---|---|---|
| 0 | \n", 2002 | "22662 | \n", 2003 | "LUNCH BAG DOLLY GIRL DESIGN | \n", 2004 | "
| 1 | \n", 2007 | "20726 | \n", 2008 | "LUNCH BAG WOODLAND | \n", 2009 | "
| 2 | \n", 2012 | "20719 | \n", 2013 | "WOODLAND CHARLOTTE BAG | \n", 2014 | "
| 3 | \n", 2017 | "22383 | \n", 2018 | "LUNCH BAG SUKI DESIGN | \n", 2019 | "
| 4 | \n", 2022 | "20728 | \n", 2023 | "LUNCH BAG CARS BLUE | \n", 2024 | "
| 5 | \n", 2027 | "23209 | \n", 2028 | "LUNCH BAG DOILEY PATTERN | \n", 2029 | "
| 6 | \n", 2032 | "22661 | \n", 2033 | "CHARLOTTE BAG DOLLY GIRL DESIGN | \n", 2034 | "
| 7 | \n", 2037 | "20724 | \n", 2038 | "RED RETROSPOT CHARLOTTE BAG | \n", 2039 | "
| 8 | \n", 2042 | "23206 | \n", 2043 | "LUNCH BAG APPLE DESIGN | \n", 2044 | "
| 9 | \n", 2047 | "22384 | \n", 2048 | "LUNCH BAG PINK POLKADOT | \n", 2049 | "