├── LICENSE.txt
├── Lab 10 - Ridge Regression and the Lasso in Python.ipynb
├── Lab 11 - PCR and PLS Regression in Python.ipynb
├── Lab 12 - Polynomial Regression and Step Functions in Python.ipynb
├── Lab 13 - Splines in Python.ipynb
├── Lab 14 - Decision Trees in Python.ipynb
├── Lab 15 - Support Vector Machines in Python.ipynb
├── Lab 16 - Multiclass SVMs and Applications to Real Data in Python.ipynb
├── Lab 18 - PCA in Python.ipynb
├── Lab 2 - Linear Regression in Python.ipynb
├── Lab 3 - K-Nearest Neighbors in Python.ipynb
├── Lab 4 - Logistic Regression in Python.ipynb
├── Lab 5 - LDA and QDA in Python.ipynb
├── Lab 7 - Cross-Validation in Python.ipynb
├── Lab 8 - Subset Selection in Python.ipynb
├── Lab 9 - Linear Model Selection in Python.ipynb
├── README.md
└── data
├── Auto.csv
├── Boston.csv
├── Caravan.csv
├── Carseats.csv
├── Hitters.csv
├── Khan_xtest.csv
├── Khan_xtrain.csv
├── Khan_ytest.csv
├── Khan_ytrain.csv
├── OJ.csv
├── Smarket 2.csv
├── Smarket.csv
└── Wage.csv
/LICENSE.txt:
--------------------------------------------------------------------------------
1 | MIT License
2 |
3 | Copyright (c) 2019 R. Jordan Crouser.
4 |
5 | Permission is hereby granted, free of charge, to any person obtaining a copy
6 | of this software and associated documentation files (the "Software"), to deal
7 | in the Software without restriction, including without limitation the rights
8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 |
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 |
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 |
--------------------------------------------------------------------------------
/Lab 13 - Splines in Python.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "This lab on Splines and GAMs is a python adaptation of p. 293-297 of \"Introduction to Statistical Learning with Applications in R\" by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani. It was originally written by Jordi Warmenhoven, and was adapted by R. Jordan Crouser at Smith College in Spring 2016. "
8 | ]
9 | },
10 | {
11 | "cell_type": "code",
12 | "execution_count": null,
13 | "metadata": {
14 | "collapsed": false
15 | },
16 | "outputs": [],
17 | "source": [
18 | "import pandas as pd\n",
19 | "import numpy as np\n",
20 | "import matplotlib as mpl\n",
21 | "import matplotlib.pyplot as plt\n",
22 | "\n",
23 | "from sklearn.preprocessing import PolynomialFeatures\n",
24 | "import statsmodels.api as sm\n",
25 | "import statsmodels.formula.api as smf\n",
26 | "\n",
27 | "%matplotlib inline\n",
28 | "\n",
29 | "# Read in the data\n",
30 | "df = pd.read_csv('Wage.csv')\n",
31 | "\n",
32 | "# Generate a sequence of age values spanning the range\n",
33 | "age_grid = np.arange(df.age.min(), df.age.max()).reshape(-1,1)"
34 | ]
35 | },
36 | {
37 | "cell_type": "markdown",
38 | "metadata": {},
39 | "source": [
40 | "# 7.8.2 Splines\n",
41 | "\n",
42 | "In order to fit regression splines in python, we use the ${\\tt dmatrix}$ module from the ${\\tt patsy}$ library. In lecture, we saw that regression splines can be fit by constructing an appropriate matrix of basis functions. The ${\\tt bs()}$ function generates the entire matrix of basis functions for splines with the specified set of knots. Fitting ${\\tt wage}$ to ${\\tt age}$ using a regression spline is simple:"
43 | ]
44 | },
45 | {
46 | "cell_type": "code",
47 | "execution_count": null,
48 | "metadata": {
49 | "collapsed": false
50 | },
51 | "outputs": [],
52 | "source": [
53 | "from patsy import dmatrix\n",
54 | "\n",
55 | "# Specifying 3 knots\n",
56 | "transformed_x1 = dmatrix(\"bs(df.age, knots=(25,40,60), degree=3, include_intercept=False)\",\n",
57 | " {\"df.age\": df.age}, return_type='dataframe')\n",
58 | "\n",
59 | "# Build a regular linear model from the splines\n",
60 | "fit1 = sm.GLM(df.wage, transformed_x1).fit()\n",
61 | "fit1.params"
62 | ]
63 | },
64 | {
65 | "cell_type": "markdown",
66 | "metadata": {},
67 | "source": [
68 | "Here we have prespecified knots at ages 25, 40, and 60. This produces a\n",
69 | "spline with six basis functions. (Recall that a cubic spline with three knots\n",
70 | "has seven degrees of freedom; these degrees of freedom are used up by an\n",
71 | "intercept, plus six basis functions.) We could also use the ${\\tt df}$ option to\n",
72 | "produce a spline with knots at uniform quantiles of the data:"
73 | ]
74 | },
75 | {
76 | "cell_type": "code",
77 | "execution_count": null,
78 | "metadata": {
79 | "collapsed": false
80 | },
81 | "outputs": [],
82 | "source": [
83 | "# Specifying 6 degrees of freedom \n",
84 | "transformed_x2 = dmatrix(\"bs(df.age, df=6, include_intercept=False)\",\n",
85 | " {\"df.age\": df.age}, return_type='dataframe')\n",
86 | "fit2 = sm.GLM(df.wage, transformed_x2).fit()\n",
87 | "fit2.params"
88 | ]
89 | },
90 | {
91 | "cell_type": "markdown",
92 | "metadata": {},
93 | "source": [
94 | "In this case python chooses knots which correspond\n",
95 | "to the 25th, 50th, and 75th percentiles of ${\\tt age}$. The function ${\\tt bs()}$ also has\n",
96 | "a ${\\tt degree}$ argument, so we can fit splines of any degree, rather than the\n",
97 | "default degree of 3 (which yields a cubic spline).\n",
98 | "\n",
99 | "In order to instead fit a natural spline, we use the ${\\tt cr()}$ function. Here\n",
100 | "we fit a natural spline with four degrees of freedom:"
101 | ]
102 | },
103 | {
104 | "cell_type": "code",
105 | "execution_count": null,
106 | "metadata": {
107 | "collapsed": false
108 | },
109 | "outputs": [],
110 | "source": [
111 | "# Specifying 4 degrees of freedom\n",
112 | "transformed_x3 = dmatrix(\"cr(df.age, df=4)\", {\"df.age\": df.age}, return_type='dataframe')\n",
113 | "fit3 = sm.GLM(df.wage, transformed_x3).fit()\n",
114 | "fit3.params"
115 | ]
116 | },
117 | {
118 | "cell_type": "markdown",
119 | "metadata": {},
120 | "source": [
121 | "As with the ${\\tt bs()}$ function, we could instead specify the knots directly using\n",
122 | "the ${\\tt knots}$ option.\n",
123 | "\n",
124 | "Let's see how these three models stack up:"
125 | ]
126 | },
127 | {
128 | "cell_type": "code",
129 | "execution_count": null,
130 | "metadata": {
131 | "collapsed": false
132 | },
133 | "outputs": [],
134 | "source": [
135 | "# Generate a sequence of age values spanning the range\n",
136 | "age_grid = np.arange(df.age.min(), df.age.max()).reshape(-1,1)\n",
137 | "\n",
138 | "# Make some predictions\n",
139 | "pred1 = fit1.predict(dmatrix(\"bs(age_grid, knots=(25,40,60), include_intercept=False)\",\n",
140 | " {\"age_grid\": age_grid}, return_type='dataframe'))\n",
141 | "pred2 = fit2.predict(dmatrix(\"bs(age_grid, df=6, include_intercept=False)\",\n",
142 | " {\"age_grid\": age_grid}, return_type='dataframe'))\n",
143 | "pred3 = fit3.predict(dmatrix(\"cr(age_grid, df=4)\", {\"age_grid\": age_grid}, return_type='dataframe'))\n",
144 | "\n",
145 | "# Plot the splines and error bands\n",
146 | "plt.scatter(df.age, df.wage, facecolor='None', edgecolor='k', alpha=0.1)\n",
147 | "plt.plot(age_grid, pred1, color='b', label='Specifying three knots')\n",
148 | "plt.plot(age_grid, pred2, color='r', label='Specifying df=6')\n",
149 | "plt.plot(age_grid, pred3, color='g', label='Natural spline df=4')\n",
150 | "plt.legend()\n",
151 | "plt.xlim(15,85)\n",
152 | "plt.ylim(0,350)\n",
153 | "plt.xlabel('age')\n",
154 | "plt.ylabel('wage')"
155 | ]
156 | },
157 | {
158 | "cell_type": "markdown",
159 | "metadata": {},
160 | "source": [
161 | "To get credit for this lab, post your answer to the following question:\n",
162 | " - How would you choose whether to use a polynomial, step, or spline function for each predictor when building a GAM?"
163 | ]
164 | }
165 | ],
166 | "metadata": {
167 | "kernelspec": {
168 | "display_name": "Python [python3]",
169 | "language": "python",
170 | "name": "Python [python3]"
171 | },
172 | "language_info": {
173 | "codemirror_mode": {
174 | "name": "ipython",
175 | "version": 3
176 | },
177 | "file_extension": ".py",
178 | "mimetype": "text/x-python",
179 | "name": "python",
180 | "nbconvert_exporter": "python",
181 | "pygments_lexer": "ipython3",
182 | "version": "3.5.1"
183 | }
184 | },
185 | "nbformat": 4,
186 | "nbformat_minor": 0
187 | }
188 |
--------------------------------------------------------------------------------
/Lab 14 - Decision Trees in Python.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "This lab on Decision Trees is a Python adaptation of p. 324-331 of \"Introduction to Statistical Learning with\n",
8 | "Applications in R\" by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani. Original adaptation by J. Warmenhoven, updated by R. Jordan Crouser at Smith\n",
9 | "College for SDS293: Machine Learning (Spring 2016)."
10 | ]
11 | },
12 | {
13 | "cell_type": "code",
14 | "execution_count": null,
15 | "metadata": {
16 | "collapsed": false
17 | },
18 | "outputs": [],
19 | "source": [
20 | "import pandas as pd\n",
21 | "import numpy as np\n",
22 | "import matplotlib as mpl\n",
23 | "import matplotlib.pyplot as plt\n",
24 | "import graphviz\n",
25 | "\n",
26 | "%matplotlib inline"
27 | ]
28 | },
29 | {
30 | "cell_type": "markdown",
31 | "metadata": {},
32 | "source": [
33 | "# 8.3.1 Fitting Classification Trees\n",
34 | "\n",
35 | "The ${\\tt sklearn}$ library has a lot of useful tools for constructing classification and regression trees:"
36 | ]
37 | },
38 | {
39 | "cell_type": "code",
40 | "execution_count": null,
41 | "metadata": {
42 | "collapsed": false
43 | },
44 | "outputs": [],
45 | "source": [
46 | "from sklearn.cross_validation import train_test_split\n",
47 | "from sklearn.tree import DecisionTreeRegressor, DecisionTreeClassifier, export_graphviz\n",
48 | "from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor\n",
49 | "from sklearn.metrics import confusion_matrix, mean_squared_error"
50 | ]
51 | },
52 | {
53 | "cell_type": "markdown",
54 | "metadata": {},
55 | "source": [
56 | "We'll start by using **classification trees** to analyze the ${\\tt Carseats}$ data set. In these\n",
57 | "data, ${\\tt Sales}$ is a continuous variable, and so we begin by converting it to a\n",
58 | "binary variable. We use the ${\\tt ifelse()}$ function to create a variable, called\n",
59 | "${\\tt High}$, which takes on a value of ${\\tt Yes}$ if the ${\\tt Sales}$ variable exceeds 8, and\n",
60 | "takes on a value of ${\\tt No}$ otherwise. We'll append this onto our dataFrame using the ${\\tt .map()}$ function, and then do a little data cleaning to tidy things up:"
61 | ]
62 | },
63 | {
64 | "cell_type": "code",
65 | "execution_count": null,
66 | "metadata": {
67 | "collapsed": false
68 | },
69 | "outputs": [],
70 | "source": [
71 | "df3 = pd.read_csv('Carseats.csv').drop('Unnamed: 0', axis=1)\n",
72 | "df3['High'] = df3.Sales.map(lambda x: 1 if x>8 else 0)\n",
73 | "df3.ShelveLoc = pd.factorize(df3.ShelveLoc)[0]\n",
74 | "df3.Urban = df3.Urban.map({'No':0, 'Yes':1})\n",
75 | "df3.US = df3.US.map({'No':0, 'Yes':1})\n",
76 | "df3.info()"
77 | ]
78 | },
79 | {
80 | "cell_type": "markdown",
81 | "metadata": {},
82 | "source": [
83 | "In order to properly evaluate the performance of a classification tree on\n",
84 | "the data, we must estimate the test error rather than simply computing\n",
85 | "the training error. We first split the observations into a training set and a test\n",
86 | "set:"
87 | ]
88 | },
89 | {
90 | "cell_type": "code",
91 | "execution_count": null,
92 | "metadata": {
93 | "collapsed": false
94 | },
95 | "outputs": [],
96 | "source": [
97 | "X = df3.drop(['Sales', 'High'], axis=1)\n",
98 | "y = df3.High\n",
99 | "\n",
100 | "X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.5, random_state=0)"
101 | ]
102 | },
103 | {
104 | "cell_type": "markdown",
105 | "metadata": {},
106 | "source": [
107 | "We now use the ${\\tt DecisionTreeClassifier()}$ function to fit a classification tree in order to predict\n",
108 | "${\\tt High}$ using all variables but ${\\tt Sales}$ (that would be a little silly...). Unfortunately, manual pruning is not implemented in ${\\tt sklearn}$: http://scikit-learn.org/stable/modules/tree.html\n",
109 | "\n",
110 | "However, we can limit the depth of a tree using the ${\\tt max\\_depth}$ parameter:"
111 | ]
112 | },
113 | {
114 | "cell_type": "code",
115 | "execution_count": null,
116 | "metadata": {
117 | "collapsed": false
118 | },
119 | "outputs": [],
120 | "source": [
121 | "clf = DecisionTreeClassifier(max_depth=6)\n",
122 | "clf.fit(X_train, y_train)\n",
123 | "clf.score(X_train, y_train)"
124 | ]
125 | },
126 | {
127 | "cell_type": "markdown",
128 | "metadata": {},
129 | "source": [
130 | "We see that the training accuracy is 95.5%.\n",
131 | "\n",
132 | "One of the most attractive properties of trees is that they can be\n",
133 | "graphically displayed. Unfortunately, this is a bit of a roundabout process in ${\\tt sklearn}$. We use the ${\\tt export\\_graphviz()}$ function to export the tree structure to a temporary ${\\tt .dot}$ file,\n",
134 | "and the ${\\tt graphviz.Source()}$ function to display the image:"
135 | ]
136 | },
137 | {
138 | "cell_type": "code",
139 | "execution_count": null,
140 | "metadata": {
141 | "collapsed": false
142 | },
143 | "outputs": [],
144 | "source": [
145 | "export_graphviz(clf, out_file=\"mytree.dot\", feature_names=X_train.columns)\n",
146 | "with open(\"mytree.dot\") as f:\n",
147 | " dot_graph = f.read()\n",
148 | "graphviz.Source(dot_graph)"
149 | ]
150 | },
151 | {
152 | "cell_type": "markdown",
153 | "metadata": {},
154 | "source": [
155 | "The most important indicator of ${\\tt High}$ sales appears to be ${\\tt Price}$."
156 | ]
157 | },
158 | {
159 | "cell_type": "markdown",
160 | "metadata": {},
161 | "source": [
162 | "Finally, let's evaluate the tree's performance on\n",
163 | "the test data. The ${\\tt predict()}$ function can be used for this purpose. We can then build a confusion matrix, which shows that we are making correct predictions for\n",
164 | "around 74.5% of the test data set:"
165 | ]
166 | },
167 | {
168 | "cell_type": "code",
169 | "execution_count": null,
170 | "metadata": {
171 | "collapsed": false
172 | },
173 | "outputs": [],
174 | "source": [
175 | "pred = clf.predict(X_test)\n",
176 | "cm = pd.DataFrame(confusion_matrix(y_test, pred).T, index=['No', 'Yes'], columns=['No', 'Yes'])\n",
177 | "print(cm)\n",
178 | "# 99+50/200 = 0.745"
179 | ]
180 | },
181 | {
182 | "cell_type": "markdown",
183 | "metadata": {},
184 | "source": [
185 | "# 8.3.2 Fitting Regression Trees\n",
186 | "\n",
187 | "Now let's try fitting a **regression tree** to the ${\\tt Boston}$ data set from the ${\\tt MASS}$ library. First, we create a\n",
188 | "training set, and fit the tree to the training data using ${\\tt medv}$ (median home value) as our response:"
189 | ]
190 | },
191 | {
192 | "cell_type": "code",
193 | "execution_count": null,
194 | "metadata": {
195 | "collapsed": false
196 | },
197 | "outputs": [],
198 | "source": [
199 | "boston_df = pd.read_csv('Boston.csv')\n",
200 | "X = boston_df.drop('medv', axis=1)\n",
201 | "y = boston_df.medv\n",
202 | "X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.5, random_state=0)\n",
203 | "\n",
204 | "# Pruning not supported. Choosing max depth 2)\n",
205 | "regr2 = DecisionTreeRegressor(max_depth=2)\n",
206 | "regr2.fit(X_train, y_train)"
207 | ]
208 | },
209 | {
210 | "cell_type": "markdown",
211 | "metadata": {},
212 | "source": [
213 | "Let's take a look at the tree:"
214 | ]
215 | },
216 | {
217 | "cell_type": "code",
218 | "execution_count": null,
219 | "metadata": {
220 | "collapsed": false
221 | },
222 | "outputs": [],
223 | "source": [
224 | "export_graphviz(regr2, out_file=\"mytree.dot\", feature_names=X_train.columns)\n",
225 | "with open(\"mytree.dot\") as f:\n",
226 | " dot_graph = f.read()\n",
227 | "graphviz.Source(dot_graph)"
228 | ]
229 | },
230 | {
231 | "cell_type": "markdown",
232 | "metadata": {},
233 | "source": [
234 | "The variable ${\\tt lstat}$ measures the percentage of individuals with lower\n",
235 | "socioeconomic status. The tree indicates that lower values of ${\\tt lstat}$ correspond\n",
236 | "to more expensive houses. The tree predicts a median house price\n",
237 | "of \\$45,766 for larger homes (${\\tt rm}>=7.435$) in suburbs in which residents have high socioeconomic\n",
238 | "status (${\\tt lstat}<7.81$).\n",
239 | "\n",
240 | "Now let's see how it does on the test data:"
241 | ]
242 | },
243 | {
244 | "cell_type": "code",
245 | "execution_count": null,
246 | "metadata": {
247 | "collapsed": false
248 | },
249 | "outputs": [],
250 | "source": [
251 | "pred = regr2.predict(X_test)\n",
252 | "\n",
253 | "plt.scatter(pred, y_test, label='medv')\n",
254 | "plt.plot([0, 1], [0, 1], '--k', transform=plt.gca().transAxes)\n",
255 | "plt.xlabel('pred')\n",
256 | "plt.ylabel('y_test')\n",
257 | "\n",
258 | "mean_squared_error(y_test, pred)"
259 | ]
260 | },
261 | {
262 | "cell_type": "markdown",
263 | "metadata": {},
264 | "source": [
265 | "The test set MSE associated with the regression tree is\n",
266 | "28.8. The square root of the MSE is therefore around 5.37, indicating\n",
267 | "that this model leads to test predictions that are within around \\$5,370 of\n",
268 | "the true median home value for the suburb.\n",
269 | " \n",
270 | "# 8.3.3 Bagging and Random Forests\n",
271 | "\n",
272 | "Let's see if we can improve on this result using **bagging** and **random forests**. The exact results obtained in this section may\n",
273 | "depend on the version of ${\\tt python}$ and the version of the ${\\tt RandomForestRegressor}$ package\n",
274 | "installed on your computer, so don't stress out if you don't match up exactly with the book. Recall that **bagging** is simply a special case of\n",
275 | "a **random forest** with $m = p$. Therefore, the ${\\tt RandomForestRegressor()}$ function can\n",
276 | "be used to perform both random forests and bagging. Let's start with bagging:"
277 | ]
278 | },
279 | {
280 | "cell_type": "code",
281 | "execution_count": null,
282 | "metadata": {
283 | "collapsed": false
284 | },
285 | "outputs": [],
286 | "source": [
287 | "# Bagging: using all features\n",
288 | "regr1 = RandomForestRegressor(max_features=13, random_state=1)\n",
289 | "regr1.fit(X_train, y_train)"
290 | ]
291 | },
292 | {
293 | "cell_type": "markdown",
294 | "metadata": {},
295 | "source": [
296 | "The argument ${\\tt max\\_features=13}$ indicates that all 13 predictors should be considered\n",
297 | "for each split of the tree -- in other words, that bagging should be done. How\n",
298 | "well does this bagged model perform on the test set?"
299 | ]
300 | },
301 | {
302 | "cell_type": "code",
303 | "execution_count": null,
304 | "metadata": {
305 | "collapsed": false
306 | },
307 | "outputs": [],
308 | "source": [
309 | "pred = regr1.predict(X_test)\n",
310 | "plt.scatter(pred, y_test, label='medv')\n",
311 | "plt.plot([0, 1], [0, 1], '--k', transform=plt.gca().transAxes)\n",
312 | "plt.xlabel('pred')\n",
313 | "plt.ylabel('y_test')\n",
314 | "mean_squared_error(y_test, pred)"
315 | ]
316 | },
317 | {
318 | "cell_type": "markdown",
319 | "metadata": {},
320 | "source": [
321 | "The test setMSE associated with the bagged regression tree is significantly lower than our single tree!"
322 | ]
323 | },
324 | {
325 | "cell_type": "markdown",
326 | "metadata": {},
327 | "source": [
328 | "We can grow a random forest in exactly the same way, except that\n",
329 | "we'll use a smaller value of the ${\\tt max\\_features}$ argument. Here we'll\n",
330 | "use ${\\tt max\\_features = 6}$:"
331 | ]
332 | },
333 | {
334 | "cell_type": "code",
335 | "execution_count": null,
336 | "metadata": {
337 | "collapsed": false
338 | },
339 | "outputs": [],
340 | "source": [
341 | "# Random forests: using 6 features\n",
342 | "regr2 = RandomForestRegressor(max_features=6, random_state=1)\n",
343 | "regr2.fit(X_train, y_train)\n",
344 | "\n",
345 | "pred = regr2.predict(X_test)\n",
346 | "mean_squared_error(y_test, pred)"
347 | ]
348 | },
349 | {
350 | "cell_type": "markdown",
351 | "metadata": {},
352 | "source": [
353 | "The test set MSE is even lower; this indicates that random forests yielded an\n",
354 | "improvement over bagging in this case.\n",
355 | "\n",
356 | "Using the ${\\tt feature\\_importances\\_}$ attribute of the ${\\tt RandomForestRegressor}$, we can view the importance of each\n",
357 | "variable:"
358 | ]
359 | },
360 | {
361 | "cell_type": "code",
362 | "execution_count": null,
363 | "metadata": {
364 | "collapsed": false
365 | },
366 | "outputs": [],
367 | "source": [
368 | "Importance = pd.DataFrame({'Importance':regr2.feature_importances_*100}, index=X.columns)\n",
369 | "Importance.sort_values(by='Importance', axis=0, ascending=True).plot(kind='barh', color='r', )\n",
370 | "plt.xlabel('Variable Importance')\n",
371 | "plt.gca().legend_ = None"
372 | ]
373 | },
374 | {
375 | "cell_type": "markdown",
376 | "metadata": {},
377 | "source": [
378 | "The results indicate that across all of the trees considered in the random\n",
379 | "forest, the wealth level of the community (${\\tt lstat}$) and the house size (${\\tt rm}$)\n",
380 | "are by far the two most important variables."
381 | ]
382 | },
383 | {
384 | "cell_type": "markdown",
385 | "metadata": {},
386 | "source": [
387 | "# 8.3.4 Boosting\n",
388 | "\n",
389 | "Now we'll use the ${\\tt GradientBoostingRegressor}$ package to fit **boosted\n",
390 | "regression trees** to the ${\\tt Boston}$ data set. The\n",
391 | "argument ${\\tt n_estimators=500}$ indicates that we want 500 trees, and the option\n",
392 | "${\\tt interaction.depth=4}$ limits the depth of each tree:"
393 | ]
394 | },
395 | {
396 | "cell_type": "code",
397 | "execution_count": null,
398 | "metadata": {
399 | "collapsed": false
400 | },
401 | "outputs": [],
402 | "source": [
403 | "regr = GradientBoostingRegressor(n_estimators=500, learning_rate=0.01, max_depth=4, random_state=1)\n",
404 | "regr.fit(X_train, y_train)"
405 | ]
406 | },
407 | {
408 | "cell_type": "markdown",
409 | "metadata": {},
410 | "source": [
411 | "Let's check out the feature importances again:"
412 | ]
413 | },
414 | {
415 | "cell_type": "code",
416 | "execution_count": null,
417 | "metadata": {
418 | "collapsed": false
419 | },
420 | "outputs": [],
421 | "source": [
422 | "feature_importance = regr.feature_importances_*100\n",
423 | "rel_imp = pd.Series(feature_importance, index=X.columns).sort_values(inplace=False)\n",
424 | "rel_imp.T.plot(kind='barh', color='r', )\n",
425 | "plt.xlabel('Variable Importance')\n",
426 | "plt.gca().legend_ = None"
427 | ]
428 | },
429 | {
430 | "cell_type": "markdown",
431 | "metadata": {},
432 | "source": [
433 | "We see that ${\\tt lstat}$ and ${\\tt rm}$ are again the most important variables by far. Now let's use the boosted model to predict ${\\tt medv}$ on the test set:"
434 | ]
435 | },
436 | {
437 | "cell_type": "code",
438 | "execution_count": null,
439 | "metadata": {
440 | "collapsed": false
441 | },
442 | "outputs": [],
443 | "source": [
444 | "mean_squared_error(y_test, regr.predict(X_test))"
445 | ]
446 | },
447 | {
448 | "cell_type": "markdown",
449 | "metadata": {},
450 | "source": [
451 | "The test MSE obtained is similar to the test MSE for random forests\n",
452 | "and superior to that for bagging. If we want to, we can perform boosting\n",
453 | "with a different value of the shrinkage parameter $\\lambda$. Here we take $\\lambda = 0.2$:"
454 | ]
455 | },
456 | {
457 | "cell_type": "code",
458 | "execution_count": null,
459 | "metadata": {
460 | "collapsed": false
461 | },
462 | "outputs": [],
463 | "source": [
464 | "regr2 = GradientBoostingRegressor(n_estimators=500, learning_rate=0.2, max_depth=4, random_state=1)\n",
465 | "regr2.fit(X_train, y_train)\n",
466 | "mean_squared_error(y_test, regr2.predict(X_test))"
467 | ]
468 | },
469 | {
470 | "cell_type": "markdown",
471 | "metadata": {},
472 | "source": [
473 | "In this case, using $\\lambda = 0.2$ leads to a slightly lower test MSE than $\\lambda = 0.01$.\n",
474 | "\n",
475 | "To get credit for this lab, post your responses to the following questions:\n",
476 | " - What's one real-world scenario where you might try using Bagging?\n",
477 | " - What's one real-world scenario where you might try using Random Forests?\n",
478 | " - What's one real-world scenario where you might try using Boosting?\n",
479 | " \n",
480 | "to Piazza: https://piazza.com/class/igwiv4w3ctb6rg?cid=53"
481 | ]
482 | }
483 | ],
484 | "metadata": {
485 | "anaconda-cloud": {},
486 | "kernelspec": {
487 | "display_name": "Python [conda root]",
488 | "language": "python",
489 | "name": "conda-root-py"
490 | },
491 | "language_info": {
492 | "codemirror_mode": {
493 | "name": "ipython",
494 | "version": 2
495 | },
496 | "file_extension": ".py",
497 | "mimetype": "text/x-python",
498 | "name": "python",
499 | "nbconvert_exporter": "python",
500 | "pygments_lexer": "ipython2",
501 | "version": "2.7.11"
502 | }
503 | },
504 | "nbformat": 4,
505 | "nbformat_minor": 0
506 | }
507 |
--------------------------------------------------------------------------------
/Lab 18 - PCA in Python.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "This lab on Principal Components Analysis is a python adaptation of p. 401-404,\n",
8 | "408-410 of \"Introduction to Statistical Learning with Applications in R\" by Gareth James,\n",
9 | "Daniela Witten, Trevor Hastie and Robert Tibshirani. Original adaptation by J. Warmenhoven, updated by R. Jordan Crouser at Smith College for\n",
10 | "SDS293: Machine Learning (Spring 2016)."
11 | ]
12 | },
13 | {
14 | "cell_type": "code",
15 | "execution_count": null,
16 | "metadata": {
17 | "collapsed": true
18 | },
19 | "outputs": [],
20 | "source": [
21 | "import pandas as pd\n",
22 | "import numpy as np\n",
23 | "import matplotlib as mpl\n",
24 | "import matplotlib.pyplot as plt\n",
25 | "\n",
26 | "%matplotlib inline"
27 | ]
28 | },
29 | {
30 | "cell_type": "markdown",
31 | "metadata": {},
32 | "source": [
33 | "# 10.4: Principal Components Analysis\n",
34 | "\n",
35 | "In this lab, we perform PCA on the ${\\tt USArrests}$ data set. The rows of the data set contain the 50 states, in\n",
36 | "alphabetical order:"
37 | ]
38 | },
39 | {
40 | "cell_type": "code",
41 | "execution_count": null,
42 | "metadata": {
43 | "collapsed": false
44 | },
45 | "outputs": [],
46 | "source": [
47 | "df = pd.read_csv('USArrests.csv', index_col=0)\n",
48 | "df.head()"
49 | ]
50 | },
51 | {
52 | "cell_type": "markdown",
53 | "metadata": {},
54 | "source": [
55 | "The columns of the data set contain four variables relating to various crimes:"
56 | ]
57 | },
58 | {
59 | "cell_type": "code",
60 | "execution_count": null,
61 | "metadata": {
62 | "collapsed": false
63 | },
64 | "outputs": [],
65 | "source": [
66 | "df.info()"
67 | ]
68 | },
69 | {
70 | "cell_type": "markdown",
71 | "metadata": {},
72 | "source": [
73 | "Let's start by taking a quick look at the column means of the data:"
74 | ]
75 | },
76 | {
77 | "cell_type": "code",
78 | "execution_count": null,
79 | "metadata": {
80 | "collapsed": false
81 | },
82 | "outputs": [],
83 | "source": [
84 | "df.mean()"
85 | ]
86 | },
87 | {
88 | "cell_type": "markdown",
89 | "metadata": {},
90 | "source": [
91 | "We see right away the the data have **vastly** different means. We can also examine the variances of the four variables:"
92 | ]
93 | },
94 | {
95 | "cell_type": "code",
96 | "execution_count": null,
97 | "metadata": {
98 | "collapsed": false
99 | },
100 | "outputs": [],
101 | "source": [
102 | "df.var()"
103 | ]
104 | },
105 | {
106 | "cell_type": "markdown",
107 | "metadata": {},
108 | "source": [
109 | "Not surprisingly, the variables also have vastly different variances: the\n",
110 | "${\\tt UrbanPop}$ variable measures the percentage of the population in each state\n",
111 | "living in an urban area, which is not a comparable number to the number\n",
112 | "of crimes committeed in each state per 100,000 individuals. If we failed to scale the\n",
113 | "variables before performing PCA, then most of the principal components\n",
114 | "that we observed would be driven by the ${\\tt Assault}$ variable, since it has by\n",
115 | "far the largest mean and variance. \n",
116 | "\n",
117 | "Thus, it is important to standardize the\n",
118 | "variables to have mean zero and standard deviation 1 before performing\n",
119 | "PCA. We can do this using the ${\\tt scale()}$ function from ${\\tt sklearn}$:"
120 | ]
121 | },
122 | {
123 | "cell_type": "code",
124 | "execution_count": null,
125 | "metadata": {
126 | "collapsed": true
127 | },
128 | "outputs": [],
129 | "source": [
130 | "from sklearn.preprocessing import scale\n",
131 | "X = pd.DataFrame(scale(df), index=df.index, columns=df.columns)"
132 | ]
133 | },
134 | {
135 | "cell_type": "markdown",
136 | "metadata": {},
137 | "source": [
138 | "Now we'll use the ${\\tt PCA()}$ function from ${\\tt sklearn}$ to compute the loading vectors:"
139 | ]
140 | },
141 | {
142 | "cell_type": "code",
143 | "execution_count": null,
144 | "metadata": {
145 | "collapsed": false
146 | },
147 | "outputs": [],
148 | "source": [
149 | "from sklearn.decomposition import PCA\n",
150 | "\n",
151 | "pca_loadings = pd.DataFrame(PCA().fit(X).components_.T, index=df.columns, columns=['V1', 'V2', 'V3', 'V4'])\n",
152 | "pca_loadings"
153 | ]
154 | },
155 | {
156 | "cell_type": "markdown",
157 | "metadata": {},
158 | "source": [
159 | "We see that there are four distinct principal components. This is to be\n",
160 | "expected because there are in general ${\\tt min(n − 1, p)}$ informative principal\n",
161 | "components in a data set with $n$ observations and $p$ variables.\n",
162 | "\n",
163 | "Using the ${\\tt fit_transform()}$ function, we can get the principal component scores of the original data. We'll take a look at the first few states:"
164 | ]
165 | },
166 | {
167 | "cell_type": "code",
168 | "execution_count": null,
169 | "metadata": {
170 | "collapsed": false
171 | },
172 | "outputs": [],
173 | "source": [
174 | "# Fit the PCA model and transform X to get the principal components\n",
175 | "pca = PCA()\n",
176 | "df_plot = pd.DataFrame(pca.fit_transform(X), columns=['PC1', 'PC2', 'PC3', 'PC4'], index=X.index)\n",
177 | "df_plot.head()"
178 | ]
179 | },
180 | {
181 | "cell_type": "markdown",
182 | "metadata": {},
183 | "source": [
184 | "We can construct a **biplot** of the first two principal components using our loading vectors:"
185 | ]
186 | },
187 | {
188 | "cell_type": "code",
189 | "execution_count": null,
190 | "metadata": {
191 | "collapsed": false
192 | },
193 | "outputs": [],
194 | "source": [
195 | "fig , ax1 = plt.subplots(figsize=(9,7))\n",
196 | "\n",
197 | "ax1.set_xlim(-3.5,3.5)\n",
198 | "ax1.set_ylim(-3.5,3.5)\n",
199 | "\n",
200 | "# Plot Principal Components 1 and 2\n",
201 | "for i in df_plot.index:\n",
202 | " ax1.annotate(i, (-df_plot.PC1.loc[i], -df_plot.PC2.loc[i]), ha='center')\n",
203 | "\n",
204 | "# Plot reference lines\n",
205 | "ax1.hlines(0,-3.5,3.5, linestyles='dotted', colors='grey')\n",
206 | "ax1.vlines(0,-3.5,3.5, linestyles='dotted', colors='grey')\n",
207 | "\n",
208 | "ax1.set_xlabel('First Principal Component')\n",
209 | "ax1.set_ylabel('Second Principal Component')\n",
210 | " \n",
211 | "# Plot Principal Component loading vectors, using a second y-axis.\n",
212 | "ax2 = ax1.twinx().twiny() \n",
213 | "\n",
214 | "ax2.set_ylim(-1,1)\n",
215 | "ax2.set_xlim(-1,1)\n",
216 | "ax2.set_xlabel('Principal Component loading vectors', color='red')\n",
217 | "\n",
218 | "# Plot labels for vectors. Variable 'a' is a small offset parameter to separate arrow tip and text.\n",
219 | "a = 1.07 \n",
220 | "for i in pca_loadings[['V1', 'V2']].index:\n",
221 | " ax2.annotate(i, (-pca_loadings.V1.loc[i]*a, -pca_loadings.V2.loc[i]*a), color='red')\n",
222 | "\n",
223 | "# Plot vectors\n",
224 | "ax2.arrow(0,0,-pca_loadings.V1[0], -pca_loadings.V2[0])\n",
225 | "ax2.arrow(0,0,-pca_loadings.V1[1], -pca_loadings.V2[1])\n",
226 | "ax2.arrow(0,0,-pca_loadings.V1[2], -pca_loadings.V2[2])\n",
227 | "ax2.arrow(0,0,-pca_loadings.V1[3], -pca_loadings.V2[3])"
228 | ]
229 | },
230 | {
231 | "cell_type": "markdown",
232 | "metadata": {},
233 | "source": [
234 | "The ${\\tt PCA()}$ function also outputs the variance explained by of each principal\n",
235 | "component. We can access these values as follows:"
236 | ]
237 | },
238 | {
239 | "cell_type": "code",
240 | "execution_count": null,
241 | "metadata": {
242 | "collapsed": false
243 | },
244 | "outputs": [],
245 | "source": [
246 | "pca.explained_variance_"
247 | ]
248 | },
249 | {
250 | "cell_type": "markdown",
251 | "metadata": {},
252 | "source": [
253 | "We can also get the proportion of variance explained:"
254 | ]
255 | },
256 | {
257 | "cell_type": "code",
258 | "execution_count": null,
259 | "metadata": {
260 | "collapsed": false
261 | },
262 | "outputs": [],
263 | "source": [
264 | "pca.explained_variance_ratio_"
265 | ]
266 | },
267 | {
268 | "cell_type": "markdown",
269 | "metadata": {},
270 | "source": [
271 | "We see that the first principal component explains 62.0% of the variance\n",
272 | "in the data, the next principal component explains 24.7% of the variance,\n",
273 | "and so forth. We can plot the PVE explained by each component as follows:"
274 | ]
275 | },
276 | {
277 | "cell_type": "code",
278 | "execution_count": null,
279 | "metadata": {
280 | "collapsed": false
281 | },
282 | "outputs": [],
283 | "source": [
284 | "plt.figure(figsize=(7,5))\n",
285 | "plt.plot([1,2,3,4], pca.explained_variance_ratio_, '-o')\n",
286 | "plt.ylabel('Proportion of Variance Explained')\n",
287 | "plt.xlabel('Principal Component')\n",
288 | "plt.xlim(0.75,4.25)\n",
289 | "plt.ylim(0,1.05)\n",
290 | "plt.xticks([1,2,3,4])"
291 | ]
292 | },
293 | {
294 | "cell_type": "markdown",
295 | "metadata": {},
296 | "source": [
297 | "We can also use the function ${\\tt cumsum()}$, which computes the cumulative sum of the elements of a numeric vector, to plot the cumulative PVE:"
298 | ]
299 | },
300 | {
301 | "cell_type": "code",
302 | "execution_count": null,
303 | "metadata": {
304 | "collapsed": false
305 | },
306 | "outputs": [],
307 | "source": [
308 | "plt.figure(figsize=(7,5))\n",
309 | "plt.plot([1,2,3,4], np.cumsum(pca.explained_variance_ratio_), '-s')\n",
310 | "plt.ylabel('Proportion of Variance Explained')\n",
311 | "plt.xlabel('Principal Component')\n",
312 | "plt.xlim(0.75,4.25)\n",
313 | "plt.ylim(0,1.05)\n",
314 | "plt.xticks([1,2,3,4])\n",
315 | "\n"
316 | ]
317 | },
318 | {
319 | "cell_type": "markdown",
320 | "metadata": {},
321 | "source": [
322 | "# 10.6: NCI60 Data Example\n",
323 | "\n",
324 | "Let's return to the ${\\tt NCI60}$ cancer cell line microarray data, which\n",
325 | "consists of 6,830 gene expression measurements on 64 cancer cell lines:"
326 | ]
327 | },
328 | {
329 | "cell_type": "code",
330 | "execution_count": null,
331 | "metadata": {
332 | "collapsed": false
333 | },
334 | "outputs": [],
335 | "source": [
336 | "df2 = pd.read_csv('NCI60.csv').drop('Unnamed: 0', axis=1)\n",
337 | "df2.columns = np.arange(df2.columns.size)\n",
338 | "df2.info()"
339 | ]
340 | },
341 | {
342 | "cell_type": "code",
343 | "execution_count": null,
344 | "metadata": {
345 | "collapsed": false
346 | },
347 | "outputs": [],
348 | "source": [
349 | "# Read in the labels to check our work later\n",
350 | "y = pd.read_csv('NCI60_y.csv', usecols=[1], skiprows=1, names=['type'])"
351 | ]
352 | },
353 | {
354 | "cell_type": "markdown",
355 | "metadata": {},
356 | "source": [
357 | "# 10.6.1 PCA on the NCI60 Data\n",
358 | "\n",
359 | "We first perform PCA on the data after scaling the variables (genes) to\n",
360 | "have standard deviation one, although one could reasonably argue that it\n",
361 | "is better not to scale the genes:"
362 | ]
363 | },
364 | {
365 | "cell_type": "code",
366 | "execution_count": null,
367 | "metadata": {
368 | "collapsed": true
369 | },
370 | "outputs": [],
371 | "source": [
372 | "# Scale the data\n",
373 | "X = pd.DataFrame(scale(df2))\n",
374 | "X.shape\n",
375 | "\n",
376 | "# Fit the PCA model and transform X to get the principal components\n",
377 | "pca2 = PCA()\n",
378 | "df2_plot = pd.DataFrame(pca2.fit_transform(X))"
379 | ]
380 | },
381 | {
382 | "cell_type": "markdown",
383 | "metadata": {},
384 | "source": [
385 | "We now plot the first few principal component score vectors, in order to\n",
386 | "visualize the data. The observations (cell lines) corresponding to a given\n",
387 | "cancer type will be plotted in the same color, so that we can see to what\n",
388 | "extent the observations within a cancer type are similar to each other:"
389 | ]
390 | },
391 | {
392 | "cell_type": "code",
393 | "execution_count": null,
394 | "metadata": {
395 | "collapsed": false
396 | },
397 | "outputs": [],
398 | "source": [
399 | "fig, (ax1, ax2) = plt.subplots(1,2, figsize=(15,6))\n",
400 | "\n",
401 | "color_idx = pd.factorize(y.type)[0]\n",
402 | "cmap = mpl.cm.hsv\n",
403 | "\n",
404 | "# Left plot\n",
405 | "ax1.scatter(df2_plot.iloc[:,0], df2_plot.iloc[:,1], c=color_idx, cmap=cmap, alpha=0.5, s=50)\n",
406 | "ax1.set_ylabel('Principal Component 2')\n",
407 | "\n",
408 | "# Right plot\n",
409 | "ax2.scatter(df2_plot.iloc[:,0], df2_plot.iloc[:,2], c=color_idx, cmap=cmap, alpha=0.5, s=50)\n",
410 | "ax2.set_ylabel('Principal Component 3')\n",
411 | "\n",
412 | "# Custom legend for the classes (y) since we do not create scatter plots per class (which could have their own labels).\n",
413 | "handles = []\n",
414 | "labels = pd.factorize(y.type.unique())\n",
415 | "norm = mpl.colors.Normalize(vmin=0.0, vmax=14.0)\n",
416 | "\n",
417 | "for i, v in zip(labels[0], labels[1]):\n",
418 | " handles.append(mpl.patches.Patch(color=cmap(norm(i)), label=v, alpha=0.5))\n",
419 | "\n",
420 | "ax2.legend(handles=handles, bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)\n",
421 | "\n",
422 | "# xlabel for both plots\n",
423 | "for ax in fig.axes:\n",
424 | " ax.set_xlabel('Principal Component 1') "
425 | ]
426 | },
427 | {
428 | "cell_type": "markdown",
429 | "metadata": {},
430 | "source": [
431 | "On the whole, cell lines corresponding to a single cancer type do tend to have similar values on the\n",
432 | "first few principal component score vectors. This indicates that cell lines\n",
433 | "from the same cancer type tend to have pretty similar gene expression\n",
434 | "levels.\n",
435 | "\n",
436 | "We can generate a summary of the proportion of variance explained (PVE)\n",
437 | "of the first few principal components:"
438 | ]
439 | },
440 | {
441 | "cell_type": "code",
442 | "execution_count": null,
443 | "metadata": {
444 | "collapsed": false
445 | },
446 | "outputs": [],
447 | "source": [
448 | "pd.DataFrame([df2_plot.iloc[:,:5].std(axis=0, ddof=0).as_matrix(),\n",
449 | " pca2.explained_variance_ratio_[:5],\n",
450 | " np.cumsum(pca2.explained_variance_ratio_[:5])],\n",
451 | " index=['Standard Deviation', 'Proportion of Variance', 'Cumulative Proportion'],\n",
452 | " columns=['PC1', 'PC2', 'PC3', 'PC4', 'PC5'])"
453 | ]
454 | },
455 | {
456 | "cell_type": "markdown",
457 | "metadata": {},
458 | "source": [
459 | "Using the ${\\tt plot()}$ function, we can also plot the variance explained by the\n",
460 | "first few principal components:"
461 | ]
462 | },
463 | {
464 | "cell_type": "code",
465 | "execution_count": null,
466 | "metadata": {
467 | "collapsed": false
468 | },
469 | "outputs": [],
470 | "source": [
471 | "df2_plot.iloc[:,:10].var(axis=0, ddof=0).plot(kind='bar', rot=0)\n",
472 | "plt.ylabel('Variances')"
473 | ]
474 | },
475 | {
476 | "cell_type": "markdown",
477 | "metadata": {},
478 | "source": [
479 | "However, it is generally more informative to\n",
480 | "plot the PVE of each principal component (i.e. a **scree plot**) and the cumulative\n",
481 | "PVE of each principal component. This can be done with just a\n",
482 | "little tweaking:"
483 | ]
484 | },
485 | {
486 | "cell_type": "code",
487 | "execution_count": null,
488 | "metadata": {
489 | "collapsed": false
490 | },
491 | "outputs": [],
492 | "source": [
493 | "fig , (ax1,ax2) = plt.subplots(1,2, figsize=(15,5))\n",
494 | "\n",
495 | "# Left plot\n",
496 | "ax1.plot(pca2.explained_variance_ratio_, '-o')\n",
497 | "ax1.set_ylabel('Proportion of Variance Explained')\n",
498 | "ax1.set_ylim(ymin=-0.01)\n",
499 | "\n",
500 | "# Right plot\n",
501 | "ax2.plot(np.cumsum(pca2.explained_variance_ratio_), '-ro')\n",
502 | "ax2.set_ylabel('Cumulative Proportion of Variance Explained')\n",
503 | "ax2.set_ylim(ymax=1.05)\n",
504 | "\n",
505 | "for ax in fig.axes:\n",
506 | " ax.set_xlabel('Principal Component')\n",
507 | " ax.set_xlim(-1,65) "
508 | ]
509 | },
510 | {
511 | "cell_type": "markdown",
512 | "metadata": {},
513 | "source": [
514 | "We see that together, the first seven principal components\n",
515 | "explain around 40% of the variance in the data. This is not a huge amount\n",
516 | "of the variance. However, looking at the scree plot, we see that while each\n",
517 | "of the first seven principal components explain a substantial amount of\n",
518 | "variance, there is a marked decrease in the variance explained by further\n",
519 | "principal components. That is, there is an **elbow** in the plot after approximately\n",
520 | "the seventh principal component. This suggests that there may\n",
521 | "be little benefit to examining more than seven or so principal components\n",
522 | "(phew! even examining seven principal components may be difficult)."
523 | ]
524 | }
525 | ],
526 | "metadata": {
527 | "kernelspec": {
528 | "display_name": "Python 3",
529 | "language": "python",
530 | "name": "python3"
531 | },
532 | "language_info": {
533 | "codemirror_mode": {
534 | "name": "ipython",
535 | "version": 3
536 | },
537 | "file_extension": ".py",
538 | "mimetype": "text/x-python",
539 | "name": "python",
540 | "nbconvert_exporter": "python",
541 | "pygments_lexer": "ipython3",
542 | "version": "3.5.1"
543 | }
544 | },
545 | "nbformat": 4,
546 | "nbformat_minor": 0
547 | }
548 |
--------------------------------------------------------------------------------
/Lab 3 - K-Nearest Neighbors in Python.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "This lab on K-Nearest Neighbors is a python adaptation of p. 163-167 of \"Introduction to Statistical Learning with Applications in R\" by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani. Originally adapted by Jordi Warmenhoven (github.com/JWarmenhoven/ISLR-python), modified by R. Jordan Crouser at Smith College for SDS293: Machine Learning (Spring 2016)."
8 | ]
9 | },
10 | {
11 | "cell_type": "code",
12 | "execution_count": null,
13 | "metadata": {
14 | "collapsed": false
15 | },
16 | "outputs": [],
17 | "source": [
18 | "import pandas as pd\n",
19 | "import numpy as np"
20 | ]
21 | },
22 | {
23 | "cell_type": "markdown",
24 | "metadata": {},
25 | "source": [
26 | "# 4.6.5: K-Nearest Neighbors"
27 | ]
28 | },
29 | {
30 | "cell_type": "markdown",
31 | "metadata": {},
32 | "source": [
33 | "In this lab, we will perform KNN on the ${\\tt Smarket}$ dataset from ${\\tt ISLR}$. This data set consists of percentage returns for the S&P 500 stock index over 1,250 days, from the\n",
34 | "beginning of 2001 until the end of 2005. For each date, we have recorded\n",
35 | "the percentage returns for each of the five previous trading days, ${\\tt Lag1}$\n",
36 | "through ${\\tt Lag5}$. We have also recorded ${\\tt Volume}$ (the number of shares traded on the previous day, in billions), ${\\tt Today}$ (the percentage return on the date\n",
37 | "in question) and ${\\tt Direction}$ (whether the market was ${\\tt Up}$ or ${\\tt Down}$ on this\n",
38 | "date). We can use the ${\\tt head(...)}$ function to look at the first few rows:"
39 | ]
40 | },
41 | {
42 | "cell_type": "code",
43 | "execution_count": null,
44 | "metadata": {
45 | "collapsed": false
46 | },
47 | "outputs": [],
48 | "source": [
49 | "df = pd.read_csv('Smarket.csv', usecols=range(1,10), index_col=0, parse_dates=True)\n",
50 | "df.head()"
51 | ]
52 | },
53 | {
54 | "cell_type": "markdown",
55 | "metadata": {},
56 | "source": [
57 | "Today we're going to try to predict ${\\tt Direction}$ using percentage returns from the previous two days (${\\tt Lag1}$ and ${\\tt Lag2}$). We'll build our model using the ${\\tt KNeighborsClassifier()}$ function, which is part of the\n",
58 | "${\\tt neighbors}$ submodule of SciKitLearn (${\\tt sklearn}$). We'll also grab a couple of useful tools from the ${\\tt metrics}$ submodule:"
59 | ]
60 | },
61 | {
62 | "cell_type": "code",
63 | "execution_count": null,
64 | "metadata": {
65 | "collapsed": true
66 | },
67 | "outputs": [],
68 | "source": [
69 | "from sklearn import neighbors\n",
70 | "from sklearn.metrics import confusion_matrix, classification_report"
71 | ]
72 | },
73 | {
74 | "cell_type": "markdown",
75 | "metadata": {},
76 | "source": [
77 | "This function works rather differently from the other model-fitting\n",
78 | "functions that we have encountered thus far. Rather than a two-step\n",
79 | "approach in which we first fit the model and then we use the model to make\n",
80 | "predictions, ${\\tt knn()}$ forms predictions using a single command. The function\n",
81 | "requires four inputs.\n",
82 | " 1. A matrix containing the predictors associated with the training data,\n",
83 | "labeled ${\\tt X\\_train}$ below.\n",
84 | " 2. A matrix containing the predictors associated with the data for which\n",
85 | "we wish to make predictions, labeled ${\\tt X\\_test}$ below.\n",
86 | " 3. A vector containing the class labels for the training observations,\n",
87 | "labeled ${\\tt y\\_train}$ below.\n",
88 | " 4. A value for $K$, the number of nearest neighbors to be used by the\n",
89 | "classifier.\n",
90 | "\n",
91 | "We'll first create a vector corresponding to the observations from 2001 through 2004, which we'll use to train the model. We will then use this vector to create a held out data set of observations from 2005 on which we will test. We'll also pull out our training and test labels."
92 | ]
93 | },
94 | {
95 | "cell_type": "code",
96 | "execution_count": null,
97 | "metadata": {
98 | "collapsed": false
99 | },
100 | "outputs": [],
101 | "source": [
102 | "X_train = df[:'2004'][['Lag1','Lag2']]\n",
103 | "y_train = df[:'2004']['Direction']\n",
104 | "\n",
105 | "X_test = df['2005':][['Lag1','Lag2']]\n",
106 | "y_test = df['2005':]['Direction']"
107 | ]
108 | },
109 | {
110 | "cell_type": "markdown",
111 | "metadata": {},
112 | "source": [
113 | "Now the ${\\tt neighbors.KNeighborsClassifier()}$ function can be used to predict the market’s movement for\n",
114 | "the dates in 2005."
115 | ]
116 | },
117 | {
118 | "cell_type": "code",
119 | "execution_count": null,
120 | "metadata": {
121 | "collapsed": false
122 | },
123 | "outputs": [],
124 | "source": [
125 | "knn = neighbors.KNeighborsClassifier(n_neighbors=1)\n",
126 | "pred = knn.fit(X_train, y_train).predict(X_test)"
127 | ]
128 | },
129 | {
130 | "cell_type": "markdown",
131 | "metadata": {},
132 | "source": [
133 | "The ${\\tt confusion\\_matrix()}$ function can be used to produce a **confusion matrix** in order to determine how many observations were correctly or incorrectly classified. The ${\\tt classification\\_report()}$ function gives us some summary statistics on the classifier's performance:"
134 | ]
135 | },
136 | {
137 | "cell_type": "code",
138 | "execution_count": null,
139 | "metadata": {
140 | "collapsed": false
141 | },
142 | "outputs": [],
143 | "source": [
144 | "print(confusion_matrix(y_test, pred).T)\n",
145 | "print(classification_report(y_test, pred, digits=3))"
146 | ]
147 | },
148 | {
149 | "cell_type": "markdown",
150 | "metadata": {},
151 | "source": [
152 | "The results using $K = 1$ are not very good, since only 50% of the observations\n",
153 | "are correctly predicted. Of course, it may be that $K = 1$ results in an\n",
154 | "overly flexible fit to the data. Below, we repeat the analysis using $K = 3$."
155 | ]
156 | },
157 | {
158 | "cell_type": "code",
159 | "execution_count": null,
160 | "metadata": {
161 | "collapsed": false
162 | },
163 | "outputs": [],
164 | "source": [
165 | "knn = neighbors.KNeighborsClassifier(n_neighbors=3)\n",
166 | "pred = knn.fit(X_train, y_train).predict(X_test)\n",
167 | "print(confusion_matrix(y_test, pred).T)\n",
168 | "print(classification_report(y_test, pred, digits=3))"
169 | ]
170 | },
171 | {
172 | "cell_type": "markdown",
173 | "metadata": {},
174 | "source": [
175 | "The results have improved slightly. Try looping through a few other $K$ values to see if you can get any further improvement:"
176 | ]
177 | },
178 | {
179 | "cell_type": "code",
180 | "execution_count": null,
181 | "metadata": {
182 | "collapsed": false
183 | },
184 | "outputs": [],
185 | "source": [
186 | "for k_val in range(10):\n",
187 | " # Your code here"
188 | ]
189 | },
190 | {
191 | "cell_type": "markdown",
192 | "metadata": {},
193 | "source": [
194 | "It looks like for classifying this dataset, ${KNN}$ might not be the right approach."
195 | ]
196 | },
197 | {
198 | "cell_type": "markdown",
199 | "metadata": {},
200 | "source": [
201 | "# 4.6.6: An Application to Caravan Insurance Data\n",
202 | "Let's see how the ${\\tt KNN}$ approach performs on the ${\\tt Caravan}$ data set, which is\n",
203 | "part of the ${\\tt ISLR}$ library. This data set includes 85 predictors that measure demographic characteristics for 5,822 individuals. The response variable is\n",
204 | "${\\tt Purchase}$, which indicates whether or not a given individual purchases a\n",
205 | "caravan insurance policy. In this data set, only 6% of people purchased\n",
206 | "caravan insurance."
207 | ]
208 | },
209 | {
210 | "cell_type": "code",
211 | "execution_count": null,
212 | "metadata": {
213 | "collapsed": false
214 | },
215 | "outputs": [],
216 | "source": [
217 | "df2 = pd.read_csv('Caravan.csv')\n",
218 | "df2[\"Purchase\"].value_counts()"
219 | ]
220 | },
221 | {
222 | "cell_type": "markdown",
223 | "metadata": {},
224 | "source": [
225 | "Because the ${\\tt KNN}$ classifier predicts the class of a given test observation by\n",
226 | "identifying the observations that are nearest to it, the scale of the variables\n",
227 | "matters. Any variables that are on a large scale will have a much larger\n",
228 | "effect on the distance between the observations, and hence on the ${\\tt KNN}$\n",
229 | "classifier, than variables that are on a small scale. \n",
230 | "\n",
231 | "For instance, imagine a\n",
232 | "data set that contains two variables, salary and age (measured in dollars\n",
233 | "and years, respectively). As far as ${\\tt KNN}$ is concerned, a difference of \\$1,000\n",
234 | "in salary is enormous compared to a difference of 50 years in age. Consequently,\n",
235 | "salary will drive the ${\\tt KNN}$ classification results, and age will have\n",
236 | "almost no effect. \n",
237 | "\n",
238 | "This is contrary to our intuition that a salary difference\n",
239 | "of \\$1,000 is quite small compared to an age difference of 50 years. Furthermore,\n",
240 | "the importance of scale to the ${\\tt KNN}$ classifier leads to another issue:\n",
241 | "if we measured salary in Japanese yen, or if we measured age in minutes,\n",
242 | "then we’d get quite different classification results from what we get if these\n",
243 | "two variables are measured in dollars and years.\n",
244 | "\n",
245 | "A good way to handle this problem is to **standardize** the data so that all\n",
246 | "variables are given a mean of zero and a standard deviation of one. Then\n",
247 | "all variables will be on a comparable scale. The ${\\tt scale()}$ function from the ${\\tt preprocessing}$ submodule of SciKitLearn does just\n",
248 | "this. In standardizing the data, we exclude column 86, because that is the\n",
249 | "qualitative ${\\tt Purchase}$ variable."
250 | ]
251 | },
252 | {
253 | "cell_type": "code",
254 | "execution_count": null,
255 | "metadata": {
256 | "collapsed": false
257 | },
258 | "outputs": [],
259 | "source": [
260 | "from sklearn import preprocessing\n",
261 | "y = df2.Purchase\n",
262 | "X = df2.drop('Purchase', axis=1).astype('float64')\n",
263 | "X_scaled = preprocessing.scale(X)\n",
264 | "print(np.std(X_scaled))"
265 | ]
266 | },
267 | {
268 | "cell_type": "markdown",
269 | "metadata": {},
270 | "source": [
271 | "Now every column of ${\\tt X\\_scaled}$ has a standard deviation of one and\n",
272 | "a mean of zero.\n",
273 | "\n",
274 | "We'll now split the observations into a test set, containing the first 1,000\n",
275 | "observations, and a training set, containing the remaining observations."
276 | ]
277 | },
278 | {
279 | "cell_type": "code",
280 | "execution_count": null,
281 | "metadata": {
282 | "collapsed": false
283 | },
284 | "outputs": [],
285 | "source": [
286 | "X_train = X_scaled[1000:,:]\n",
287 | "y_train = y[1000:]\n",
288 | "X_test = X_scaled[:1000,:]\n",
289 | "y_test = y[:1000]"
290 | ]
291 | },
292 | {
293 | "cell_type": "markdown",
294 | "metadata": {},
295 | "source": [
296 | "Let's fit a ${\\tt KNN}$ model on the training data using $K = 1$, and evaluate its\n",
297 | "performance on the test data."
298 | ]
299 | },
300 | {
301 | "cell_type": "code",
302 | "execution_count": null,
303 | "metadata": {
304 | "collapsed": false
305 | },
306 | "outputs": [],
307 | "source": [
308 | "knn = neighbors.KNeighborsClassifier(n_neighbors=1)\n",
309 | "pred = knn.fit(X_train, y_train).predict(X_test)\n",
310 | "print(classification_report(y_test, pred, digits=3))"
311 | ]
312 | },
313 | {
314 | "cell_type": "markdown",
315 | "metadata": {},
316 | "source": [
317 | "The KNN error rate on the 1,000 test observations is just under 12%. At first glance, this may appear to be fairly good. However, since only 6% of customers purchased insurance, we could get the error rate down to 6% by always predicting ${\\tt No}$ regardless of the values of the predictors!\n",
318 | "\n",
319 | "Suppose that there is some non-trivial cost to trying to sell insurance\n",
320 | "to a given individual. For instance, perhaps a salesperson must visit each\n",
321 | "potential customer. If the company tries to sell insurance to a random\n",
322 | "selection of customers, then the success rate will be only 6%, which may\n",
323 | "be far too low given the costs involved. \n",
324 | "\n",
325 | "Instead, the company would like\n",
326 | "to try to sell insurance only to customers who are likely to buy it. So the\n",
327 | "overall error rate is not of interest. Instead, the fraction of individuals that\n",
328 | "are correctly predicted to buy insurance is of interest.\n",
329 | "\n",
330 | "It turns out that ${\\tt KNN}$ with $K = 1$ does far better than random guessing\n",
331 | "among the customers that are predicted to buy insurance:"
332 | ]
333 | },
334 | {
335 | "cell_type": "code",
336 | "execution_count": null,
337 | "metadata": {
338 | "collapsed": false
339 | },
340 | "outputs": [],
341 | "source": [
342 | "print(confusion_matrix(y_test, pred).T)"
343 | ]
344 | },
345 | {
346 | "cell_type": "markdown",
347 | "metadata": {},
348 | "source": [
349 | "Among 77 such\n",
350 | "customers, 9, or 11.7%, actually do purchase insurance. This is double the\n",
351 | "rate that one would obtain from random guessing. Let's see if increasing $K$ helps! Try out a few different $K$ values below. Feeling adventurous? Write a function that figures out the best value for $K$."
352 | ]
353 | },
354 | {
355 | "cell_type": "code",
356 | "execution_count": null,
357 | "metadata": {
358 | "collapsed": false
359 | },
360 | "outputs": [],
361 | "source": [
362 | "# Your code here"
363 | ]
364 | },
365 | {
366 | "cell_type": "markdown",
367 | "metadata": {},
368 | "source": [
369 | "It appears that ${\\tt KNN}$ is finding some real patterns in a difficult data set! To get credit for this lab, post a response to the Piazza prompt available at: https://piazza.com/class/igwiv4w3ctb6rg?cid=10"
370 | ]
371 | }
372 | ],
373 | "metadata": {
374 | "kernelspec": {
375 | "display_name": "Python 2",
376 | "language": "python",
377 | "name": "python2"
378 | },
379 | "language_info": {
380 | "codemirror_mode": {
381 | "name": "ipython",
382 | "version": 2
383 | },
384 | "file_extension": ".py",
385 | "mimetype": "text/x-python",
386 | "name": "python",
387 | "nbconvert_exporter": "python",
388 | "pygments_lexer": "ipython2",
389 | "version": "2.7.11"
390 | }
391 | },
392 | "nbformat": 4,
393 | "nbformat_minor": 0
394 | }
395 |
--------------------------------------------------------------------------------
/Lab 4 - Logistic Regression in Python.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "This lab on Logistic Regression is a Python adaptation from p. 154-161 of \"Introduction to Statistical Learning with Applications in R\" by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani. Adapted by R. Jordan Crouser at Smith College for SDS293: Machine Learning (Spring 2016)."
8 | ]
9 | },
10 | {
11 | "cell_type": "code",
12 | "execution_count": 1,
13 | "metadata": {
14 | "collapsed": true
15 | },
16 | "outputs": [],
17 | "source": [
18 | "import pandas as pd\n",
19 | "import numpy as np\n",
20 | "import statsmodels.api as sm"
21 | ]
22 | },
23 | {
24 | "cell_type": "markdown",
25 | "metadata": {},
26 | "source": [
27 | "# 4.6.2 Logistic Regression\n",
28 | "\n",
29 | "Let's return to the ${\\tt Smarket}$ data from ${\\tt ISLR}$. "
30 | ]
31 | },
32 | {
33 | "cell_type": "code",
34 | "execution_count": 2,
35 | "metadata": {
36 | "collapsed": false
37 | },
38 | "outputs": [
39 | {
40 | "data": {
41 | "text/html": [
42 | "\n",
43 | "
\n",
44 | " \n",
45 | " \n",
46 | " | \n",
47 | " Lag1 | \n",
48 | " Lag2 | \n",
49 | " Lag3 | \n",
50 | " Lag4 | \n",
51 | " Lag5 | \n",
52 | " Volume | \n",
53 | " Today | \n",
54 | "
\n",
55 | " \n",
56 | " \n",
57 | " \n",
58 | " | count | \n",
59 | " 1250.000000 | \n",
60 | " 1250.000000 | \n",
61 | " 1250.000000 | \n",
62 | " 1250.000000 | \n",
63 | " 1250.00000 | \n",
64 | " 1250.000000 | \n",
65 | " 1250.000000 | \n",
66 | "
\n",
67 | " \n",
68 | " | mean | \n",
69 | " 0.003834 | \n",
70 | " 0.003919 | \n",
71 | " 0.001716 | \n",
72 | " 0.001636 | \n",
73 | " 0.00561 | \n",
74 | " 1.478305 | \n",
75 | " 0.003138 | \n",
76 | "
\n",
77 | " \n",
78 | " | std | \n",
79 | " 1.136299 | \n",
80 | " 1.136280 | \n",
81 | " 1.138703 | \n",
82 | " 1.138774 | \n",
83 | " 1.14755 | \n",
84 | " 0.360357 | \n",
85 | " 1.136334 | \n",
86 | "
\n",
87 | " \n",
88 | " | min | \n",
89 | " -4.922000 | \n",
90 | " -4.922000 | \n",
91 | " -4.922000 | \n",
92 | " -4.922000 | \n",
93 | " -4.92200 | \n",
94 | " 0.356070 | \n",
95 | " -4.922000 | \n",
96 | "
\n",
97 | " \n",
98 | " | 25% | \n",
99 | " -0.639500 | \n",
100 | " -0.639500 | \n",
101 | " -0.640000 | \n",
102 | " -0.640000 | \n",
103 | " -0.64000 | \n",
104 | " 1.257400 | \n",
105 | " -0.639500 | \n",
106 | "
\n",
107 | " \n",
108 | " | 50% | \n",
109 | " 0.039000 | \n",
110 | " 0.039000 | \n",
111 | " 0.038500 | \n",
112 | " 0.038500 | \n",
113 | " 0.03850 | \n",
114 | " 1.422950 | \n",
115 | " 0.038500 | \n",
116 | "
\n",
117 | " \n",
118 | " | 75% | \n",
119 | " 0.596750 | \n",
120 | " 0.596750 | \n",
121 | " 0.596750 | \n",
122 | " 0.596750 | \n",
123 | " 0.59700 | \n",
124 | " 1.641675 | \n",
125 | " 0.596750 | \n",
126 | "
\n",
127 | " \n",
128 | " | max | \n",
129 | " 5.733000 | \n",
130 | " 5.733000 | \n",
131 | " 5.733000 | \n",
132 | " 5.733000 | \n",
133 | " 5.73300 | \n",
134 | " 3.152470 | \n",
135 | " 5.733000 | \n",
136 | "
\n",
137 | " \n",
138 | "
\n",
139 | "