18 |
18 |
19 |
20 | [](https://mybinder.org/v2/gh/intro-stat-learning/ISLP_labs/v2.2?labpath=Ch03-linreg-lab.ipynb)
21 |
22 |
23 |
24 | ## Importing packages
25 | We import our standard libraries at this top
26 | level.
27 |
28 | ```{python}
29 | import numpy as np
30 | import pandas as pd
31 | from matplotlib.pyplot import subplots
32 |
33 | ```
34 |
35 |
36 | ### New imports
37 | Throughout this lab we will introduce new functions and libraries. However,
38 | we will import them here to emphasize these are the new
39 | code objects in this lab. Keeping imports near the top
40 | of a notebook makes the code more readable, since scanning the first few
41 | lines tells us what libraries are used.
42 |
43 | ```{python}
44 | import statsmodels.api as sm
45 |
46 | ```
47 | We will provide relevant details about the
48 | functions below as they are needed.
49 |
50 | Besides importing whole modules, it is also possible
51 | to import only a few items from a given module. This
52 | will help keep the *namespace* clean.
53 | We will use a few specific objects from the `statsmodels` package
54 | which we import here.
55 |
56 | ```{python}
57 | from statsmodels.stats.outliers_influence \
58 | import variance_inflation_factor as VIF
59 | from statsmodels.stats.anova import anova_lm
60 |
61 | ```
62 |
63 | As one of the import statements above is quite a long line, we inserted a line break `\` to
64 | ease readability.
65 |
66 | We will also use some functions written for the labs in this book in the `ISLP`
67 | package.
68 |
69 | ```{python}
70 | from ISLP import load_data
71 | from ISLP.models import (ModelSpec as MS,
72 | summarize,
73 | poly)
74 |
75 | ```
76 |
77 | ### Inspecting Objects and Namespaces
78 | The
79 | function `dir()`
80 | provides a list of
81 | objects in a namespace.
82 |
83 | ```{python}
84 | dir()
85 |
86 | ```
87 | This shows you everything that `Python` can find at the top level.
88 | There are certain objects like `__builtins__` that contain references to built-in
89 | functions like `print()`.
90 |
91 | Every python object has its own notion of
92 | namespace, also accessible with `dir()`. This will include
93 | both the attributes of the object
94 | as well as any methods associated with it. For instance, we see `'sum'` in the listing for an
95 | array.
96 |
97 | ```{python}
98 | A = np.array([3,5,11])
99 | dir(A)
100 |
101 | ```
102 | This indicates that the object `A.sum` exists. In this case it is a method
103 | that can be used to compute the sum of the array `A` as can be seen by typing `A.sum?`.
104 |
105 | ```{python}
106 | A.sum()
107 |
108 | ```
109 |
110 |
111 |
112 | ## Simple Linear Regression
113 | In this section we will construct model
114 | matrices (also called design matrices) using the `ModelSpec()` transform from `ISLP.models`.
115 |
116 | We will use the `Boston` housing data set, which is contained in the `ISLP` package. The `Boston` dataset records `medv` (median house value) for $506$ neighborhoods
117 | around Boston. We will build a regression model to predict `medv` using $13$
118 | predictors such as `rm` (average number of rooms per house),
119 | `age` (proportion of owner-occupied units built prior to 1940), and `lstat` (percent of
120 | households with low socioeconomic status). We will use `statsmodels` for this
121 | task, a `Python` package that implements several commonly used
122 | regression methods.
123 |
124 | We have included a simple loading function `load_data()` in the
125 | `ISLP` package:
126 |
127 | ```{python}
128 | Boston = load_data("Boston")
129 | Boston.columns
130 |
131 | ```
132 |
133 | Type `Boston?` to find out more about these data.
134 |
135 | We start by using the `sm.OLS()` function to fit a
136 | simple linear regression model. Our response will be
137 | `medv` and `lstat` will be the single predictor.
138 | For this model, we can create the model matrix by hand.
139 |
140 |
141 | ```{python}
142 | X = pd.DataFrame({'intercept': np.ones(Boston.shape[0]),
143 | 'lstat': Boston['lstat']})
144 | X[:4]
145 |
146 | ```
147 |
148 | We extract the response, and fit the model.
149 |
150 | ```{python}
151 | y = Boston['medv']
152 | model = sm.OLS(y, X)
153 | results = model.fit()
154 |
155 | ```
156 | Note that `sm.OLS()` does
157 | not fit the model; it specifies the model, and then `model.fit()` does the actual fitting.
158 |
159 | Our `ISLP` function `summarize()` produces a simple table of the parameter estimates,
160 | their standard errors, t-statistics and p-values.
161 | The function takes a single argument, such as the object `results`
162 | returned here by the `fit`
163 | method, and returns such a summary.
164 |
165 | ```{python}
166 | summarize(results)
167 |
168 | ```
169 |
170 |
171 | Before we describe other methods for working with fitted models, we outline a more useful and general framework for constructing a model matrix~`X`.
172 | ### Using Transformations: Fit and Transform
173 | Our model above has a single predictor, and constructing `X` was straightforward.
174 | In practice we often fit models with more than one predictor, typically selected from an array or data frame.
175 | We may wish to introduce transformations to the variables before fitting the model, specify interactions between variables, and expand some particular variables into sets of variables (e.g. polynomials).
176 | The `sklearn` package has a particular notion
177 | for this type of task: a *transform*. A transform is an object
178 | that is created with some parameters as arguments. The
179 | object has two main methods: `fit()` and `transform()`.
180 |
181 | We provide a general approach for specifying models and constructing
182 | the model matrix through the transform `ModelSpec()` in the `ISLP` library.
183 | `ModelSpec()`
184 | (renamed `MS()` in the preamble) creates a
185 | transform object, and then a pair of methods
186 | `transform()` and `fit()` are used to construct a
187 | corresponding model matrix.
188 |
189 | We first describe this process for our simple regression model using a single predictor `lstat` in
190 | the `Boston` data frame, but will use it repeatedly in more
191 | complex tasks in this and other labs in this book.
192 | In our case the transform is created by the expression
193 | `design = MS(['lstat'])`.
194 |
195 | The `fit()` method takes the original array and may do some
196 | initial computations on it, as specified in the transform object.
197 | For example, it may compute means and standard deviations for centering and scaling.
198 | The `transform()`
199 | method applies the fitted transformation to the array of data, and produces the model matrix.
200 |
201 |
202 | ```{python}
203 | design = MS(['lstat'])
204 | design = design.fit(Boston)
205 | X = design.transform(Boston)
206 | X[:4]
207 | ```
208 | In this simple case, the `fit()` method does very little; it simply checks that the variable `'lstat'` specified in `design` exists in `Boston`. Then `transform()` constructs the model matrix with two columns: an `intercept` and the variable `lstat`.
209 |
210 | These two operations can be combined with the
211 | `fit_transform()` method.
212 |
213 | ```{python}
214 | design = MS(['lstat'])
215 | X = design.fit_transform(Boston)
216 | X[:4]
217 | ```
218 | Note that, as in the previous code chunk when the two steps were done separately, the `design` object is changed as a result of the `fit()` operation. The power of this pipeline will become clearer when we fit more complex models that involve interactions and transformations.
219 |
220 |
221 | Let's return to our fitted regression model.
222 | The object
223 | `results` has several methods that can be used for inference.
224 | We already presented a function `summarize()` for showing the essentials of the fit.
225 | For a full and somewhat exhaustive summary of the fit, we can use the `summary()`
226 | method.
227 |
228 | ```{python}
229 | results.summary()
230 |
231 | ```
232 |
233 | The fitted coefficients can also be retrieved as the
234 | `params` attribute of `results`.
235 |
236 | ```{python}
237 | results.params
238 |
239 | ```
240 |
241 |
242 | The `get_prediction()` method can be used to obtain predictions, and produce confidence intervals and
243 | prediction intervals for the prediction of `medv` for given values of `lstat`.
244 |
245 | We first create a new data frame, in this case containing only the variable `lstat`, with the values for this variable at which we wish to make predictions.
246 | We then use the `transform()` method of `design` to create the corresponding model matrix.
247 |
248 | ```{python}
249 | new_df = pd.DataFrame({'lstat':[5, 10, 15]})
250 | newX = design.transform(new_df)
251 | newX
252 |
253 | ```
254 |
255 | Next we compute the predictions at `newX`, and view them by extracting the `predicted_mean` attribute.
256 |
257 | ```{python}
258 | new_predictions = results.get_prediction(newX);
259 | new_predictions.predicted_mean
260 |
261 | ```
262 | We can produce confidence intervals for the predicted values.
263 |
264 | ```{python}
265 | new_predictions.conf_int(alpha=0.05)
266 |
267 | ```
268 | Prediction intervals are computed by setting `obs=True`:
269 |
270 | ```{python}
271 | new_predictions.conf_int(obs=True, alpha=0.05)
272 |
273 | ```
274 | For instance, the 95% confidence interval associated with an
275 | `lstat` value of 10 is (24.47, 25.63), and the 95% prediction
276 | interval is (12.82, 37.28). As expected, the confidence and
277 | prediction intervals are centered around the same point (a predicted
278 | value of 25.05 for `medv` when `lstat` equals
279 | 10), but the latter are substantially wider.
280 |
281 | Next we will plot `medv` and `lstat`
282 | using `DataFrame.plot.scatter()`, \definelongblankMR{plot.scatter()}{plot.slashslashscatter()}
283 | and wish to
284 | add the regression line to the resulting plot.
285 |
286 |
287 | ### Defining Functions
288 | While there is a function
289 | within the `ISLP` package that adds a line to an existing plot, we take this opportunity
290 | to define our first function to do so.
291 |
292 | ```{python}
293 | def abline(ax, b, m):
294 | "Add a line with slope m and intercept b to ax"
295 | xlim = ax.get_xlim()
296 | ylim = [m * xlim[0] + b, m * xlim[1] + b]
297 | ax.plot(xlim, ylim)
298 |
299 | ```
300 | A few things are illustrated above. First we see the syntax for defining a function:
301 | `def funcname(...)`. The function has arguments `ax, b, m`
302 | where `ax` is an axis object for an existing plot, `b` is the intercept and
303 | `m` is the slope of the desired line. Other plotting options can be passed on to
304 | `ax.plot` by including additional optional arguments as follows:
305 |
306 | ```{python}
307 | def abline(ax, b, m, *args, **kwargs):
308 | "Add a line with slope m and intercept b to ax"
309 | xlim = ax.get_xlim()
310 | ylim = [m * xlim[0] + b, m * xlim[1] + b]
311 | ax.plot(xlim, ylim, *args, **kwargs)
312 |
313 | ```
314 | The addition of `*args` allows any number of
315 | non-named arguments to `abline`, while `**kwargs` allows any
316 | number of named arguments (such as `linewidth=3`) to `abline`.
317 | In our function, we pass
318 | these arguments verbatim to `ax.plot` above. Readers
319 | interested in learning more about
320 | functions are referred to the section on
321 | defining functions in [docs.python.org/tutorial](https://docs.python.org/3/tutorial/controlflow.html#defining-functions).
322 |
323 | Let’s use our new function to add this regression line to a plot of
324 | `medv` vs. `lstat`.
325 |
326 | ```{python}
327 | ax = Boston.plot.scatter('lstat', 'medv')
328 | abline(ax,
329 | results.params[0],
330 | results.params[1],
331 | 'r--',
332 | linewidth=3)
333 |
334 | ```
335 | Thus, the final call to `ax.plot()` is `ax.plot(xlim, ylim, 'r--', linewidth=3)`.
336 | We have used the argument `'r--'` to produce a red dashed line, and added
337 | an argument to make it of width 3.
338 | There is some evidence for non-linearity in the relationship between `lstat` and `medv`. We will explore this issue later in this lab.
339 |
340 | As mentioned above, there is an existing function to add a line to a plot --- `ax.axline()` --- but knowing how to write such functions empowers us to create more expressive displays.
341 |
342 |
343 |
344 |
345 | Next we examine some diagnostic plots, several of which were discussed
346 | in Section 3.3.3.
347 | We can find the fitted values and residuals
348 | of the fit as attributes of the `results` object.
349 | Various influence measures describing the regression model
350 | are computed with the `get_influence()` method.
351 | As we will not use the `fig` component returned
352 | as the first value from `subplots()`, we simply
353 | capture the second returned value in `ax` below.
354 |
355 | ```{python}
356 | ax = subplots(figsize=(8,8))[1]
357 | ax.scatter(results.fittedvalues, results.resid)
358 | ax.set_xlabel('Fitted value')
359 | ax.set_ylabel('Residual')
360 | ax.axhline(0, c='k', ls='--');
361 |
362 | ```
363 | We add a horizontal line at 0 for reference using the
364 | `ax.axhline()` method, indicating
365 | it should be black (`c='k'`) and have a dashed linestyle (`ls='--'`).
366 |
367 | On the basis of the residual plot, there is some evidence of non-linearity.
368 | Leverage statistics can be computed for any number of predictors using the
369 | `hat_matrix_diag` attribute of the value returned by the
370 | `get_influence()` method.
371 |
372 | ```{python}
373 | infl = results.get_influence()
374 | ax = subplots(figsize=(8,8))[1]
375 | ax.scatter(np.arange(X.shape[0]), infl.hat_matrix_diag)
376 | ax.set_xlabel('Index')
377 | ax.set_ylabel('Leverage')
378 | np.argmax(infl.hat_matrix_diag)
379 |
380 | ```
381 | The `np.argmax()` function identifies the index of the largest element of an array, optionally computed over an axis of the array.
382 | In this case, we maximized over the entire array
383 | to determine which observation has the largest leverage statistic.
384 |
385 |
386 | ## Multiple Linear Regression
387 | In order to fit a multiple linear regression model using least squares, we again use
388 | the `ModelSpec()` transform to construct the required
389 | model matrix and response. The arguments
390 | to `ModelSpec()` can be quite general, but in this case
391 | a list of column names suffice. We consider a fit here with
392 | the two variables `lstat` and `age`.
393 |
394 | ```{python}
395 | X = MS(['lstat', 'age']).fit_transform(Boston)
396 | model1 = sm.OLS(y, X)
397 | results1 = model1.fit()
398 | summarize(results1)
399 | ```
400 | Notice how we have compacted the first line into a succinct expression describing the construction of `X`.
401 |
402 | The `Boston` data set contains 12 variables, and so it would be cumbersome
403 | to have to type all of these in order to perform a regression using all of the predictors.
404 | Instead, we can use the following short-hand:\definelongblankMR{columns.drop()}{columns.slashslashdrop()}
405 |
406 | ```{python}
407 | terms = Boston.columns.drop('medv')
408 | terms
409 |
410 | ```
411 |
412 | We can now fit the model with all the variables in `terms` using
413 | the same model matrix builder.
414 |
415 | ```{python}
416 | X = MS(terms).fit_transform(Boston)
417 | model = sm.OLS(y, X)
418 | results = model.fit()
419 | summarize(results)
420 |
421 | ```
422 |
423 | What if we would like to perform a regression using all of the variables but one? For
424 | example, in the above regression output, `age` has a high $p$-value.
425 | So we may wish to run a regression excluding this predictor.
426 | The following syntax results in a regression using all predictors except `age`.
427 |
428 | ```{python}
429 | minus_age = Boston.columns.drop(['medv', 'age'])
430 | Xma = MS(minus_age).fit_transform(Boston)
431 | model1 = sm.OLS(y, Xma)
432 | summarize(model1.fit())
433 |
434 | ```
435 |
436 | ## Multivariate Goodness of Fit
437 | We can access the individual components of `results` by name
438 | (`dir(results)` shows us what is available). Hence
439 | `results.rsquared` gives us the $R^2$,
440 | and
441 | `np.sqrt(results.scale)` gives us the RSE.
442 |
443 | Variance inflation factors (section 3.3.3) are sometimes useful
444 | to assess the effect of collinearity in the model matrix of a regression model.
445 | We will compute the VIFs in our multiple regression fit, and use the opportunity to introduce the idea of *list comprehension*.
446 |
447 | ### List Comprehension
448 | Often we encounter a sequence of objects which we would like to transform
449 | for some other task. Below, we compute the VIF for each
450 | feature in our `X` matrix and produce a data frame
451 | whose index agrees with the columns of `X`.
452 | The notion of list comprehension can often make such
453 | a task easier.
454 |
455 | List comprehensions are simple and powerful ways to form
456 | lists of `Python` objects. The language also supports
457 | dictionary and *generator* comprehension, though these are
458 | beyond our scope here. Let's look at an example. We compute the VIF for each of the variables
459 | in the model matrix `X`, using the function `variance_inflation_factor()`.
460 |
461 |
462 | ```{python}
463 | vals = [VIF(X, i)
464 | for i in range(1, X.shape[1])]
465 | vif = pd.DataFrame({'vif':vals},
466 | index=X.columns[1:])
467 | vif
468 |
469 | ```
470 | The function `VIF()` takes two arguments: a dataframe or array,
471 | and a variable column index. In the code above we call `VIF()` on the fly for all columns in `X`.
472 | We have excluded column 0 above (the intercept), which is not of interest. In this case the VIFs are not that exciting.
473 |
474 | The object `vals` above could have been constructed with the following for loop:
475 |
476 | ```{python}
477 | vals = []
478 | for i in range(1, X.values.shape[1]):
479 | vals.append(VIF(X.values, i))
480 |
481 | ```
482 | List comprehension allows us to perform such repetitive operations in a more straightforward way.
483 | ## Interaction Terms
484 | It is easy to include interaction terms in a linear model using `ModelSpec()`.
485 | Including a tuple `("lstat","age")` tells the model
486 | matrix builder to include an interaction term between
487 | `lstat` and `age`.
488 |
489 | ```{python}
490 | X = MS(['lstat',
491 | 'age',
492 | ('lstat', 'age')]).fit_transform(Boston)
493 | model2 = sm.OLS(y, X)
494 | summarize(model2.fit())
495 |
496 | ```
497 |
498 |
499 | ## Non-linear Transformations of the Predictors
500 | The model matrix builder can include terms beyond
501 | just column names and interactions. For instance,
502 | the `poly()` function supplied in `ISLP` specifies that
503 | columns representing polynomial functions
504 | of its first argument are added to the model matrix.
505 |
506 | ```{python}
507 | X = MS([poly('lstat', degree=2), 'age']).fit_transform(Boston)
508 | model3 = sm.OLS(y, X)
509 | results3 = model3.fit()
510 | summarize(results3)
511 |
512 | ```
513 | The effectively zero *p*-value associated with the quadratic term
514 | (i.e. the third row above) suggests that it leads to an improved model.
515 |
516 | By default, `poly()` creates a basis matrix for inclusion in the
517 | model matrix whose
518 | columns are *orthogonal polynomials*, which are designed for stable
519 | least squares computations. {Actually, `poly()` is a wrapper for the workhorse and standalone function `Poly()` that does the work in building the model matrix.}
520 | Alternatively, had we included an argument
521 | `raw=True` in the above call to `poly()`, the basis matrix would consist simply of
522 | `lstat` and `lstat**2`. Since either of these bases
523 | represent quadratic polynomials, the fitted values would not
524 | change in this case, just the polynomial coefficients. Also by default, the columns
525 | created by `poly()` do not include an intercept column as
526 | that is automatically added by `MS()`.
527 |
528 | We use the `anova_lm()` function to further quantify the extent to which the quadratic fit is
529 | superior to the linear fit.
530 |
531 | ```{python}
532 | anova_lm(results1, results3)
533 |
534 | ```
535 | Here `results1` represents the linear submodel containing
536 | predictors `lstat` and `age`,
537 | while `results3` corresponds to the larger model above with a quadratic
538 | term in `lstat`.
539 | The `anova_lm()` function performs a hypothesis test
540 | comparing the two models. The null hypothesis is that the quadratic
541 | term in the bigger model is not needed, and the alternative hypothesis is that the
542 | bigger model is superior. Here the *F*-statistic is 177.28 and
543 | the associated *p*-value is zero.
544 | In this case the *F*-statistic is the square of the
545 | *t*-statistic for the quadratic term in the linear model summary
546 | for `results3` --- a consequence of the fact that these nested
547 | models differ by one degree of freedom.
548 | This provides very clear evidence that the quadratic polynomial in
549 | `lstat` improves the linear model.
550 | This is not surprising, since earlier we saw evidence for non-linearity in the relationship between `medv`
551 | and `lstat`.
552 |
553 | The function `anova_lm()` can take more than two nested models
554 | as input, in which case it compares every successive pair of models.
555 | That also explains why there are `NaN`s in the first row above, since
556 | there is no previous model with which to compare the first.
557 |
558 |
559 | ```{python}
560 | ax = subplots(figsize=(8,8))[1]
561 | ax.scatter(results3.fittedvalues, results3.resid)
562 | ax.set_xlabel('Fitted value')
563 | ax.set_ylabel('Residual')
564 | ax.axhline(0, c='k', ls='--');
565 |
566 | ```
567 | We see that when the quadratic term is included in the model,
568 | there is little discernible pattern in the residuals.
569 | In order to create a cubic or higher-degree polynomial fit, we can simply change the degree argument
570 | to `poly()`.
571 |
572 |
573 |
574 | ## Qualitative Predictors
575 | Here we use the `Carseats` data, which is included in the
576 | `ISLP` package. We will attempt to predict `Sales`
577 | (child car seat sales) in 400 locations based on a number of
578 | predictors.
579 |
580 | ```{python}
581 | Carseats = load_data('Carseats')
582 | Carseats.columns
583 |
584 | ```
585 | The `Carseats`
586 | data includes qualitative predictors such as
587 | `ShelveLoc`, an indicator of the quality of the shelving
588 | location --- that is,
589 | the space within a store in which the car seat is displayed. The predictor
590 | `ShelveLoc` takes on three possible values, `Bad`, `Medium`, and `Good`.
591 | Given a qualitative variable such as `ShelveLoc`, `ModelSpec()` generates dummy
592 | variables automatically.
593 | These variables are often referred to as a *one-hot encoding* of the categorical
594 | feature. Their columns sum to one, so to avoid collinearity with an intercept, the first column is dropped. Below we see
595 | the column `ShelveLoc[Bad]` has been dropped, since `Bad` is the first level of `ShelveLoc`.
596 | Below we fit a multiple regression model that includes some interaction terms.
597 |
598 | ```{python}
599 | allvars = list(Carseats.columns.drop('Sales'))
600 | y = Carseats['Sales']
601 | final = allvars + [('Income', 'Advertising'),
602 | ('Price', 'Age')]
603 | X = MS(final).fit_transform(Carseats)
604 | model = sm.OLS(y, X)
605 | summarize(model.fit())
606 |
607 | ```
608 | In the first line above, we made `allvars` a list, so that we
609 | could add the interaction terms two lines down.
610 | Our model-matrix builder has created a `ShelveLoc[Good]`
611 | dummy variable that takes on a value of 1 if the
612 | shelving location is good, and 0 otherwise. It has also created a `ShelveLoc[Medium]`
613 | dummy variable that equals 1 if the shelving location is medium, and 0 otherwise.
614 | A bad shelving location corresponds to a zero for each of the two dummy variables.
615 | The fact that the coefficient for `ShelveLoc[Good]` in the regression output is
616 | positive indicates that a good shelving location is associated with high sales (relative to a bad location).
617 | And `ShelveLoc[Medium]` has a smaller positive coefficient,
618 | indicating that a medium shelving location leads to higher sales than a bad
619 | shelving location, but lower sales than a good shelving location.
620 |
621 |
622 |
--------------------------------------------------------------------------------
/Ch05-resample-lab.Rmd:
--------------------------------------------------------------------------------
1 | ---
2 | jupyter:
3 | jupytext:
4 | cell_metadata_filter: -all
5 | formats: ipynb,Rmd
6 | main_language: python
7 | text_representation:
8 | extension: .Rmd
9 | format_name: rmarkdown
10 | format_version: '1.2'
11 | jupytext_version: 1.16.7
12 | ---
13 |
14 | # Cross-Validation and the Bootstrap
15 |
16 |
17 |
18 |
19 |
20 | [](https://mybinder.org/v2/gh/intro-stat-learning/ISLP_labs/v2.2?labpath=Ch05-resample-lab.ipynb)
21 |
22 |
23 | In this lab, we explore the resampling techniques covered in this
24 | chapter. Some of the commands in this lab may take a while to run on
25 | your computer.
26 |
27 | We again begin by placing most of our imports at this top level.
28 |
29 | ```{python}
30 | import numpy as np
31 | import statsmodels.api as sm
32 | from ISLP import load_data
33 | from ISLP.models import (ModelSpec as MS,
34 | summarize,
35 | poly)
36 | from sklearn.model_selection import train_test_split
37 |
38 | ```
39 |
40 |
41 | There are several new imports needed for this lab.
42 |
43 | ```{python}
44 | from functools import partial
45 | from sklearn.model_selection import \
46 | (cross_validate,
47 | KFold,
48 | ShuffleSplit)
49 | from sklearn.base import clone
50 | from ISLP.models import sklearn_sm
51 |
52 | ```
53 |
54 |
55 | ## The Validation Set Approach
56 | We explore the use of the validation set approach in order to estimate
57 | the test error rates that result from fitting various linear models on
58 | the `Auto` data set.
59 |
60 | We use the function `train_test_split()` to split
61 | the data into training and validation sets. As there are 392 observations,
62 | we split into two equal sets of size 196 using the
63 | argument `test_size=196`. It is generally a good idea to set a random seed
64 | when performing operations like this that contain an
65 | element of randomness, so that the results obtained can be reproduced
66 | precisely at a later time. We set the random seed of the splitter
67 | with the argument `random_state=0`.
68 |
69 | ```{python}
70 | Auto = load_data('Auto')
71 | Auto_train, Auto_valid = train_test_split(Auto,
72 | test_size=196,
73 | random_state=0)
74 |
75 | ```
76 |
77 | Now we can fit a linear regression using only the observations corresponding to the training set `Auto_train`.
78 |
79 | ```{python}
80 | hp_mm = MS(['horsepower'])
81 | X_train = hp_mm.fit_transform(Auto_train)
82 | y_train = Auto_train['mpg']
83 | model = sm.OLS(y_train, X_train)
84 | results = model.fit()
85 |
86 | ```
87 |
88 | We now use the `predict()` method of `results` evaluated on the model matrix for this model
89 | created using the validation data set. We also calculate the validation MSE of our model.
90 |
91 | ```{python}
92 | X_valid = hp_mm.transform(Auto_valid)
93 | y_valid = Auto_valid['mpg']
94 | valid_pred = results.predict(X_valid)
95 | np.mean((y_valid - valid_pred)**2)
96 |
97 | ```
98 |
99 | Hence our estimate for the validation MSE of the linear regression
100 | fit is $23.62$.
101 |
102 | We can also estimate the validation error for
103 | higher-degree polynomial regressions. We first provide a function `evalMSE()` that takes a model string as well
104 | as training and test sets and returns the MSE on the test set.
105 |
106 | ```{python}
107 | def evalMSE(terms,
108 | response,
109 | train,
110 | test):
111 |
112 | mm = MS(terms)
113 | X_train = mm.fit_transform(train)
114 | y_train = train[response]
115 |
116 | X_test = mm.transform(test)
117 | y_test = test[response]
118 |
119 | results = sm.OLS(y_train, X_train).fit()
120 | test_pred = results.predict(X_test)
121 |
122 | return np.mean((y_test - test_pred)**2)
123 |
124 | ```
125 |
126 | Let’s use this function to estimate the validation MSE
127 | using linear, quadratic and cubic fits. We use the `enumerate()` function
128 | here, which gives both the values and indices of objects as one iterates
129 | over a for loop.
130 |
131 | ```{python}
132 | MSE = np.zeros(3)
133 | for idx, degree in enumerate(range(1, 4)):
134 | MSE[idx] = evalMSE([poly('horsepower', degree)],
135 | 'mpg',
136 | Auto_train,
137 | Auto_valid)
138 | MSE
139 |
140 | ```
141 |
142 | These error rates are $23.62, 18.76$, and $18.80$, respectively. If we
143 | choose a different training/validation split instead, then we
144 | can expect somewhat different errors on the validation set.
145 |
146 | ```{python}
147 | Auto_train, Auto_valid = train_test_split(Auto,
148 | test_size=196,
149 | random_state=3)
150 | MSE = np.zeros(3)
151 | for idx, degree in enumerate(range(1, 4)):
152 | MSE[idx] = evalMSE([poly('horsepower', degree)],
153 | 'mpg',
154 | Auto_train,
155 | Auto_valid)
156 | MSE
157 | ```
158 |
159 | Using this split of the observations into a training set and a validation set,
160 | we find that the validation set error rates for the models with linear, quadratic, and cubic terms are $20.76$, $16.95$, and $16.97$, respectively.
161 |
162 | These results are consistent with our previous findings: a model that
163 | predicts `mpg` using a quadratic function of `horsepower`
164 | performs better than a model that involves only a linear function of
165 | `horsepower`, and there is no evidence of an improvement in using a cubic function of `horsepower`.
166 |
167 |
168 | ## Cross-Validation
169 | In theory, the cross-validation estimate can be computed for any generalized
170 | linear model. {}
171 | In practice, however, the simplest way to cross-validate in
172 | Python is to use `sklearn`, which has a different interface or API
173 | than `statsmodels`, the code we have been using to fit GLMs.
174 |
175 | This is a problem which often confronts data scientists: "I have a function to do task $A$, and need to feed it into something that performs task $B$, so that I can compute $B(A(D))$, where $D$ is my data." When $A$ and $B$ don’t naturally speak to each other, this
176 | requires the use of a *wrapper*.
177 | In the `ISLP` package,
178 | we provide
179 | a wrapper, `sklearn_sm()`, that enables us to easily use the cross-validation tools of `sklearn` with
180 | models fit by `statsmodels`.
181 |
182 | The class `sklearn_sm()`
183 | has as its first argument
184 | a model from `statsmodels`. It can take two additional
185 | optional arguments: `model_str` which can be
186 | used to specify a formula, and `model_args` which should
187 | be a dictionary of additional arguments used when fitting
188 | the model. For example, to fit a logistic regression model
189 | we have to specify a `family` argument. This
190 | is passed as `model_args={'family':sm.families.Binomial()}`.
191 |
192 | Here is our wrapper in action:
193 |
194 | ```{python}
195 | hp_model = sklearn_sm(sm.OLS,
196 | MS(['horsepower']))
197 | X, Y = Auto.drop(columns=['mpg']), Auto['mpg']
198 | cv_results = cross_validate(hp_model,
199 | X,
200 | Y,
201 | cv=Auto.shape[0])
202 | cv_err = np.mean(cv_results['test_score'])
203 | cv_err
204 |
205 | ```
206 | The arguments to `cross_validate()` are as follows: an
207 | object with the appropriate `fit()`, `predict()`,
208 | and `score()` methods, an
209 | array of features `X` and a response `Y`.
210 | We also included an additional argument `cv` to `cross_validate()`; specifying an integer
211 | $k$ results in $k$-fold cross-validation. We have provided a value
212 | corresponding to the total number of observations, which results in
213 | leave-one-out cross-validation (LOOCV). The `cross_validate()` function produces a dictionary with several components;
214 | we simply want the cross-validated test score here (MSE), which is estimated to be 24.23.
215 |
216 |
217 | We can repeat this procedure for increasingly complex polynomial fits.
218 | To automate the process, we again
219 | use a for loop which iteratively fits polynomial
220 | regressions of degree 1 to 5, computes the
221 | associated cross-validation error, and stores it in the $i$th element
222 | of the vector `cv_error`. The variable `d` in the for loop
223 | corresponds to the degree of the polynomial. We begin by initializing the
224 | vector. This command may take a couple of seconds to run.
225 |
226 | ```{python}
227 | cv_error = np.zeros(5)
228 | H = np.array(Auto['horsepower'])
229 | M = sklearn_sm(sm.OLS)
230 | for i, d in enumerate(range(1,6)):
231 | X = np.power.outer(H, np.arange(d+1))
232 | M_CV = cross_validate(M,
233 | X,
234 | Y,
235 | cv=Auto.shape[0])
236 | cv_error[i] = np.mean(M_CV['test_score'])
237 | cv_error
238 |
239 | ```
240 | As in Figure 5.4, we see a sharp drop in the estimated test MSE between the linear and
241 | quadratic fits, but then no clear improvement from using higher-degree polynomials.
242 |
243 | Above we introduced the `outer()` method of the `np.power()`
244 | function. The `outer()` method is applied to an operation
245 | that has two arguments, such as `add()`, `min()`, or
246 | `power()`.
247 | It has two arrays as
248 | arguments, and then forms a larger
249 | array where the operation is applied to each pair of elements of the
250 | two arrays.
251 |
252 | ```{python}
253 | A = np.array([3, 5, 9])
254 | B = np.array([2, 4])
255 | np.add.outer(A, B)
256 |
257 | ```
258 |
259 | In the CV example above, we used $k=n$, but of course we can also use $k
18 |
19 |
20 | [](https://mybinder.org/v2/gh/intro-stat-learning/ISLP_labs/v2.2?labpath=Ch08-baggboost-lab.ipynb)
21 |
22 |
23 | We import some of our usual libraries at this top
24 | level.
25 |
26 | ```{python}
27 | import numpy as np
28 | import pandas as pd
29 | from matplotlib.pyplot import subplots
30 | import sklearn.model_selection as skm
31 | from ISLP import load_data, confusion_table
32 | from ISLP.models import ModelSpec as MS
33 |
34 | ```
35 | We also collect the new imports
36 | needed for this lab.
37 |
38 | ```{python}
39 | from sklearn.tree import (DecisionTreeClassifier as DTC,
40 | DecisionTreeRegressor as DTR,
41 | plot_tree,
42 | export_text)
43 | from sklearn.metrics import (accuracy_score,
44 | log_loss)
45 | from sklearn.ensemble import \
46 | (RandomForestRegressor as RF,
47 | GradientBoostingRegressor as GBR)
48 | from ISLP.bart import BART
49 |
50 | ```
51 |
52 |
53 | ## Fitting Classification Trees
54 |
55 |
56 | We first use classification trees to analyze the `Carseats` data set.
57 | In these data, `Sales` is a continuous variable, and so we begin
58 | by recoding it as a binary variable. We use the `where()`
59 | function to create a variable, called `High`, which takes on a
60 | value of `Yes` if the `Sales` variable exceeds 8, and takes
61 | on a value of `No` otherwise.
62 |
63 | ```{python}
64 | Carseats = load_data('Carseats')
65 | High = np.where(Carseats.Sales > 8,
66 | "Yes",
67 | "No")
68 |
69 | ```
70 |
71 | We now use `DecisionTreeClassifier()` to fit a classification tree in
72 | order to predict `High` using all variables but `Sales`.
73 | To do so, we must form a model matrix as we did when fitting regression
74 | models.
75 |
76 | ```{python}
77 | model = MS(Carseats.columns.drop('Sales'), intercept=False)
78 | D = model.fit_transform(Carseats)
79 | feature_names = list(D.columns)
80 | X = np.asarray(D)
81 |
82 | ```
83 | We have converted `D` from a data frame to an array `X`, which is needed in some of the analysis below. We also need the `feature_names` for annotating our plots later.
84 |
85 | There are several options needed to specify the classifier,
86 | such as `max_depth` (how deep to grow the tree), `min_samples_split`
87 | (minimum number of observations in a node to be eligible for splitting)
88 | and `criterion` (whether to use Gini or cross-entropy as the split criterion).
89 | We also set `random_state` for reproducibility; ties in the split criterion are broken at random.
90 |
91 | ```{python}
92 | clf = DTC(criterion='entropy',
93 | max_depth=3,
94 | random_state=0)
95 | clf.fit(X, High)
96 |
97 | ```
98 |
99 |
100 | In our discussion of qualitative features in Section 3.3,
101 | we noted that for a linear regression model such a feature could be
102 | represented by including a matrix of dummy variables (one-hot-encoding) in the model
103 | matrix, using the formula notation of `statsmodels`.
104 | As mentioned in Section 8.1, there is a more
105 | natural way to handle qualitative features when building a decision
106 | tree, that does not require such dummy variables; each split amounts to partitioning the levels into two groups.
107 | However,
108 | the `sklearn` implementation of decision trees does not take
109 | advantage of this approach; instead it simply treats the one-hot-encoded levels as separate variables.
110 |
111 | ```{python}
112 | accuracy_score(High, clf.predict(X))
113 |
114 | ```
115 |
116 |
117 | With only the default arguments, the training error rate is
118 | 21%.
119 | For classification trees, we can
120 | access the value of the deviance using `log_loss()`,
121 | \begin{equation*}
122 | \begin{split}
123 | -2 \sum_m \sum_k n_{mk} \log \hat{p}_{mk},
124 | \end{split}
125 | \end{equation*}
126 | where $n_{mk}$ is the number of observations in the $m$th terminal
127 | node that belong to the $k$th class.
128 |
129 | ```{python}
130 | resid_dev = np.sum(log_loss(High, clf.predict_proba(X)))
131 | resid_dev
132 |
133 | ```
134 |
135 | This is closely related to the *entropy*, defined in (8.7).
136 | A small deviance indicates a
137 | tree that provides a good fit to the (training) data.
138 |
139 | One of the most attractive properties of trees is that they can
140 | be graphically displayed. Here we use the `plot()` function
141 | to display the tree structure.
142 |
143 | ```{python}
144 | ax = subplots(figsize=(12,12))[1]
145 | plot_tree(clf,
146 | feature_names=feature_names,
147 | ax=ax);
148 |
149 | ```
150 | The most important indicator of `Sales` appears to be `ShelveLoc`.
151 |
152 | We can see a text representation of the tree using
153 | `export_text()`, which displays the split
154 | criterion (e.g. `Price <= 92.5`) for each branch.
155 | For leaf nodes it shows the overall prediction
156 | (`Yes` or `No`).
157 | We can also see the number of observations in that
158 | leaf that take on values of `Yes` and `No` by specifying `show_weights=True`.
159 |
160 | ```{python}
161 | print(export_text(clf,
162 | feature_names=feature_names,
163 | show_weights=True))
164 |
165 | ```
166 |
167 | In order to properly evaluate the performance of a classification tree
168 | on these data, we must estimate the test error rather than simply
169 | computing the training error. We split the observations into a
170 | training set and a test set, build the tree using the training set,
171 | and evaluate its performance on the test data. This pattern is
172 | similar to that in Chapter 6, with the linear models
173 | replaced here by decision trees --- the code for validation
174 | is almost identical. This approach leads to correct predictions
175 | for 68.5% of the locations in the test data set.
176 |
177 | ```{python}
178 | validation = skm.ShuffleSplit(n_splits=1,
179 | test_size=200,
180 | random_state=0)
181 | results = skm.cross_validate(clf,
182 | D,
183 | High,
184 | cv=validation)
185 | results['test_score']
186 |
187 | ```
188 |
189 |
190 |
191 | Next, we consider whether pruning the tree might lead to improved
192 | classification performance. We first split the data into a training and
193 | test set. We will use cross-validation to prune the tree on the training
194 | set, and then evaluate the performance of the pruned tree on the test
195 | set.
196 |
197 | ```{python}
198 | (X_train,
199 | X_test,
200 | High_train,
201 | High_test) = skm.train_test_split(X,
202 | High,
203 | test_size=0.5,
204 | random_state=0)
205 |
206 | ```
207 | We first refit the full tree on the training set; here we do not set a `max_depth` parameter, since we will learn that through cross-validation.
208 |
209 |
210 | ```{python}
211 | clf = DTC(criterion='entropy', random_state=0)
212 | clf.fit(X_train, High_train)
213 | accuracy_score(High_test, clf.predict(X_test))
214 |
215 | ```
216 | Next we use the `cost_complexity_pruning_path()` method of
217 | `clf` to extract cost-complexity values.
218 |
219 | ```{python}
220 | ccp_path = clf.cost_complexity_pruning_path(X_train, High_train)
221 | kfold = skm.KFold(10,
222 | random_state=1,
223 | shuffle=True)
224 |
225 | ```
226 | This yields a set of impurities and $\alpha$ values
227 | from which we can extract an optimal one by cross-validation.
228 |
229 | ```{python}
230 | grid = skm.GridSearchCV(clf,
231 | {'ccp_alpha': ccp_path.ccp_alphas},
232 | refit=True,
233 | cv=kfold,
234 | scoring='accuracy')
235 | grid.fit(X_train, High_train)
236 | grid.best_score_
237 |
238 | ```
239 | Let’s take a look at the pruned tree.
240 |
241 | ```{python}
242 | ax = subplots(figsize=(12, 12))[1]
243 | best_ = grid.best_estimator_
244 | plot_tree(best_,
245 | feature_names=feature_names,
246 | ax=ax);
247 |
248 | ```
249 | This is quite a bushy tree. We could count the leaves, or query
250 | `best_` instead.
251 |
252 | ```{python}
253 | best_.tree_.n_leaves
254 |
255 | ```
256 | The tree with 30 terminal
257 | nodes results in the lowest cross-validation error rate, with an accuracy of
258 | 68.5%. How well does this pruned tree perform on the test data set? Once
259 | again, we apply the `predict()` function.
260 |
261 | ```{python}
262 | print(accuracy_score(High_test,
263 | best_.predict(X_test)))
264 | confusion = confusion_table(best_.predict(X_test),
265 | High_test)
266 | confusion
267 |
268 | ```
269 |
270 |
271 | Now 72.0% of the test observations are correctly classified, which is slightly worse than the error for the full tree (with 35 leaves). So cross-validation has not helped us much here; it only pruned off 5 leaves, at a cost of a slightly worse error. These results would change if we were to change the random number seeds above; even though cross-validation gives an unbiased approach to model selection, it does have variance.
272 |
273 |
274 |
275 |
276 | ## Fitting Regression Trees
277 | Here we fit a regression tree to the `Boston` data set. The
278 | steps are similar to those for classification trees.
279 |
280 | ```{python}
281 | Boston = load_data("Boston")
282 | model = MS(Boston.columns.drop('medv'), intercept=False)
283 | D = model.fit_transform(Boston)
284 | feature_names = list(D.columns)
285 | X = np.asarray(D)
286 |
287 | ```
288 |
289 | First, we split the data into training and test sets, and fit the tree
290 | to the training data. Here we use 30% of the data for the test set.
291 |
292 |
293 | ```{python}
294 | (X_train,
295 | X_test,
296 | y_train,
297 | y_test) = skm.train_test_split(X,
298 | Boston['medv'],
299 | test_size=0.3,
300 | random_state=0)
301 |
302 | ```
303 |
304 | Having formed our training and test data sets, we fit the regression tree.
305 |
306 | ```{python}
307 | reg = DTR(max_depth=3)
308 | reg.fit(X_train, y_train)
309 | ax = subplots(figsize=(12,12))[1]
310 | plot_tree(reg,
311 | feature_names=feature_names,
312 | ax=ax);
313 |
314 | ```
315 |
316 | The variable `lstat` measures the percentage of individuals with
317 | lower socioeconomic status. The tree indicates that lower
318 | values of `lstat` correspond to more expensive houses.
319 | The tree predicts a median house price of $12,042 for small-sized homes (`rm < 6.8`), in
320 | suburbs in which residents have low socioeconomic status (`lstat > 14.4`) and the crime-rate is moderate (`crim > 5.8`).
321 |
322 |
323 | Now we use the cross-validation function to see whether pruning
324 | the tree will improve performance.
325 |
326 | ```{python}
327 | ccp_path = reg.cost_complexity_pruning_path(X_train, y_train)
328 | kfold = skm.KFold(5,
329 | shuffle=True,
330 | random_state=10)
331 | grid = skm.GridSearchCV(reg,
332 | {'ccp_alpha': ccp_path.ccp_alphas},
333 | refit=True,
334 | cv=kfold,
335 | scoring='neg_mean_squared_error')
336 | G = grid.fit(X_train, y_train)
337 |
338 | ```
339 |
340 | In keeping with the cross-validation results, we use the pruned tree
341 | to make predictions on the test set.
342 |
343 | ```{python}
344 | best_ = grid.best_estimator_
345 | np.mean((y_test - best_.predict(X_test))**2)
346 |
347 | ```
348 |
349 |
350 | In other words, the test set MSE associated with the regression tree
351 | is 28.07. The square root of
352 | the MSE is therefore around
353 | 5.30,
354 | indicating that this model leads to test predictions that are within around
355 | $5300
356 | of the true median home value for the suburb.
357 |
358 | Let’s plot the best tree to see how interpretable it is.
359 |
360 | ```{python}
361 | ax = subplots(figsize=(12,12))[1]
362 | plot_tree(G.best_estimator_,
363 | feature_names=feature_names,
364 | ax=ax);
365 |
366 | ```
367 |
368 |
369 |
370 |
371 | ## Bagging and Random Forests
372 |
373 |
374 | Here we apply bagging and random forests to the `Boston` data, using
375 | the `RandomForestRegressor()` from the `sklearn.ensemble` package. Recall
376 | that bagging is simply a special case of a random forest with
377 | $m=p$. Therefore, the `RandomForestRegressor()` function can be used to
378 | perform both bagging and random forests. We start with bagging.
379 |
380 | ```{python}
381 | bag_boston = RF(max_features=X_train.shape[1], random_state=0)
382 | bag_boston.fit(X_train, y_train)
383 |
384 | ```
385 |
386 |
387 | The argument `max_features` indicates that all 12 predictors should
388 | be considered for each split of the tree --- in other words, that
389 | bagging should be done. How well does this bagged model perform on
390 | the test set?
391 |
392 | ```{python}
393 | ax = subplots(figsize=(8,8))[1]
394 | y_hat_bag = bag_boston.predict(X_test)
395 | ax.scatter(y_hat_bag, y_test)
396 | np.mean((y_test - y_hat_bag)**2)
397 |
398 | ```
399 |
400 | The test set MSE associated with the bagged regression tree is
401 | 14.63, about half that obtained using an optimally-pruned single
402 | tree. We could change the number of trees grown from the default of
403 | 100 by
404 | using the `n_estimators` argument:
405 |
406 | ```{python}
407 | bag_boston = RF(max_features=X_train.shape[1],
408 | n_estimators=500,
409 | random_state=0).fit(X_train, y_train)
410 | y_hat_bag = bag_boston.predict(X_test)
411 | np.mean((y_test - y_hat_bag)**2)
412 | ```
413 | There is not much change. Bagging and random forests cannot overfit by
414 | increasing the number of trees, but can underfit if the number is too small.
415 |
416 | Growing a random forest proceeds in exactly the same way, except that
417 | we use a smaller value of the `max_features` argument. By default,
418 | `RandomForestRegressor()` uses $p$ variables when building a random
419 | forest of regression trees (i.e. it defaults to bagging), and `RandomForestClassifier()` uses
420 | $\sqrt{p}$ variables when building a
421 | random forest of classification trees. Here we use `max_features=6`.
422 |
423 | ```{python}
424 | RF_boston = RF(max_features=6,
425 | random_state=0).fit(X_train, y_train)
426 | y_hat_RF = RF_boston.predict(X_test)
427 | np.mean((y_test - y_hat_RF)**2)
428 |
429 | ```
430 |
431 |
432 | The test set MSE is 20.04;
433 | this indicates that random forests did somewhat worse than bagging
434 | in this case. Extracting the `feature_importances_` values from the fitted model, we can view the
435 | importance of each variable.
436 |
437 | ```{python}
438 | feature_imp = pd.DataFrame(
439 | {'importance':RF_boston.feature_importances_},
440 | index=feature_names)
441 | feature_imp.sort_values(by='importance', ascending=False)
442 | ```
443 | This
444 | is a relative measure of the total decrease in node impurity that results from
445 | splits over that variable, averaged over all trees (this was plotted in Figure 8.9 for a model fit to the `Heart` data).
446 |
447 | The results indicate that across all of the trees considered in the
448 | random forest, the wealth level of the community (`lstat`) and the
449 | house size (`rm`) are by far the two most important variables.
450 |
451 |
452 |
453 |
454 | ## Boosting
455 |
456 |
457 | Here we use `GradientBoostingRegressor()` from `sklearn.ensemble`
458 | to fit boosted regression trees to the `Boston` data
459 | set. For classification we would use `GradientBoostingClassifier()`.
460 | The argument `n_estimators=5000`
461 | indicates that we want 5000 trees, and the option
462 | `max_depth=3` limits the depth of each tree. The
463 | argument `learning_rate` is the $\lambda$
464 | mentioned earlier in the description of boosting.
465 |
466 | ```{python}
467 | boost_boston = GBR(n_estimators=5000,
468 | learning_rate=0.001,
469 | max_depth=3,
470 | random_state=0)
471 | boost_boston.fit(X_train, y_train)
472 |
473 | ```
474 |
475 | We can see how the training error decreases with the `train_score_` attribute.
476 | To get an idea of how the test error decreases we can use the
477 | `staged_predict()` method to get the predicted values along the path.
478 |
479 | ```{python}
480 | test_error = np.zeros_like(boost_boston.train_score_)
481 | for idx, y_ in enumerate(boost_boston.staged_predict(X_test)):
482 | test_error[idx] = np.mean((y_test - y_)**2)
483 |
484 | plot_idx = np.arange(boost_boston.train_score_.shape[0])
485 | ax = subplots(figsize=(8,8))[1]
486 | ax.plot(plot_idx,
487 | boost_boston.train_score_,
488 | 'b',
489 | label='Training')
490 | ax.plot(plot_idx,
491 | test_error,
492 | 'r',
493 | label='Test')
494 | ax.legend();
495 |
496 | ```
497 |
498 | We now use the boosted model to predict `medv` on the test set:
499 |
500 | ```{python}
501 | y_hat_boost = boost_boston.predict(X_test);
502 | np.mean((y_test - y_hat_boost)**2)
503 |
504 | ```
505 |
506 | The test MSE obtained is 14.48,
507 | similar to the test MSE for bagging. If we want to, we can
508 | perform boosting with a different value of the shrinkage parameter
509 | $\lambda$ in (8.10). The default value is 0.001, but
510 | this is easily modified. Here we take $\lambda=0.2$.
511 |
512 | ```{python}
513 | boost_boston = GBR(n_estimators=5000,
514 | learning_rate=0.2,
515 | max_depth=3,
516 | random_state=0)
517 | boost_boston.fit(X_train,
518 | y_train)
519 | y_hat_boost = boost_boston.predict(X_test);
520 | np.mean((y_test - y_hat_boost)**2)
521 |
522 | ```
523 |
524 |
525 | In this case, using $\lambda=0.2$ leads to almost the same test MSE
526 | as when using $\lambda=0.001$.
527 |
528 |
529 |
530 |
531 | ## Bayesian Additive Regression Trees
532 |
533 |
534 | In this section we demonstrate a `Python` implementation of BART found in the
535 | `ISLP.bart` package. We fit a model
536 | to the `Boston` housing data set. This `BART()` estimator is
537 | designed for quantitative outcome variables, though other implementations are available for
538 | fitting logistic and probit models to categorical outcomes.
539 |
540 | ```{python}
541 | bart_boston = BART(random_state=0, burnin=5, ndraw=15)
542 | bart_boston.fit(X_train, y_train)
543 |
544 | ```
545 |
546 |
547 | On this data set, with this split into test and training, we see that the test error of BART is similar to that of random forest.
548 |
549 | ```{python}
550 | yhat_test = bart_boston.predict(X_test.astype(np.float32))
551 | np.mean((y_test - yhat_test)**2)
552 |
553 | ```
554 |
555 |
556 | We can check how many times each variable appeared in the collection of trees.
557 | This gives a summary similar to the variable importance plot for boosting and random forests.
558 |
559 | ```{python}
560 | var_inclusion = pd.Series(bart_boston.variable_inclusion_.mean(0),
561 | index=D.columns)
562 | var_inclusion
563 |
564 | ```
565 |
566 |
567 |
568 |
569 |
570 |
--------------------------------------------------------------------------------
/Ch09-svm-lab.Rmd:
--------------------------------------------------------------------------------
1 | ---
2 | jupyter:
3 | jupytext:
4 | cell_metadata_filter: -all
5 | formats: ipynb,Rmd
6 | main_language: python
7 | text_representation:
8 | extension: .Rmd
9 | format_name: rmarkdown
10 | format_version: '1.2'
11 | jupytext_version: 1.16.7
12 | ---
13 |
14 | # Support Vector Machines
15 |
16 |
17 |
18 |
19 |
20 | [](https://mybinder.org/v2/gh/intro-stat-learning/ISLP_labs/v2.2?labpath=Ch09-svm-lab.ipynb)
21 |
22 |
23 | In this lab, we use the `sklearn.svm` library to demonstrate the support
24 | vector classifier and the support vector machine.
25 |
26 | We import some of our usual libraries.
27 |
28 | ```{python}
29 | import numpy as np
30 | from matplotlib.pyplot import subplots, cm
31 | import sklearn.model_selection as skm
32 | from ISLP import load_data, confusion_table
33 |
34 | ```
35 | We also collect the new imports
36 | needed for this lab.
37 |
38 | ```{python}
39 | from sklearn.svm import SVC
40 | from ISLP.svm import plot as plot_svm
41 | from sklearn.metrics import RocCurveDisplay
42 |
43 | ```
44 |
45 | We will use the function `RocCurveDisplay.from_estimator()` to
46 | produce several ROC plots, using a shorthand `roc_curve`.
47 |
48 | ```{python}
49 | roc_curve = RocCurveDisplay.from_estimator # shorthand
50 |
51 | ```
52 |
53 | ## Support Vector Classifier
54 |
55 | We now use the `SupportVectorClassifier()` function (abbreviated `SVC()`) from `sklearn` to fit the support vector
56 | classifier for a given value of the parameter `C`. The
57 | `C` argument allows us to specify the cost of a violation to
58 | the margin. When the `C` argument is small, then the margins
59 | will be wide and many support vectors will be on the margin or will
60 | violate the margin. When the `C` argument is large, then the
61 | margins will be narrow and there will be few support vectors on the
62 | margin or violating the margin.
63 |
64 | Here we demonstrate
65 | the use of `SVC()` on a two-dimensional example, so that we can
66 | plot the resulting decision boundary. We begin by generating the
67 | observations, which belong to two classes, and checking whether the
68 | classes are linearly separable.
69 |
70 | ```{python}
71 | rng = np.random.default_rng(1)
72 | X = rng.standard_normal((50, 2))
73 | y = np.array([-1]*25+[1]*25)
74 | X[y==1] += 1
75 | fig, ax = subplots(figsize=(8,8))
76 | ax.scatter(X[:,0],
77 | X[:,1],
78 | c=y,
79 | cmap=cm.coolwarm);
80 |
81 | ```
82 | They are not. We now fit the classifier.
83 |
84 | ```{python}
85 | svm_linear = SVC(C=10, kernel='linear')
86 | svm_linear.fit(X, y)
87 |
88 | ```
89 |
90 |
91 | The support vector classifier with two features can
92 | be visualized by plotting values of its *decision function*.
93 | We have included a function for this in the `ISLP` package (inspired by a similar
94 | example in the `sklearn` docs).
95 |
96 | ```{python}
97 | fig, ax = subplots(figsize=(8,8))
98 | plot_svm(X,
99 | y,
100 | svm_linear,
101 | ax=ax)
102 |
103 | ```
104 |
105 | The decision
106 | boundary between the two classes is linear (because we used the
107 | argument `kernel='linear'`). The support vectors are marked with `+`
108 | and the remaining observations are plotted as circles.
109 |
110 | What if we instead used a smaller value of the cost parameter?
111 |
112 | ```{python}
113 | svm_linear_small = SVC(C=0.1, kernel='linear')
114 | svm_linear_small.fit(X, y)
115 | fig, ax = subplots(figsize=(8,8))
116 | plot_svm(X,
117 | y,
118 | svm_linear_small,
119 | ax=ax)
120 |
121 | ```
122 | With a smaller value of the cost parameter, we
123 | obtain a larger number of support vectors, because the margin is now
124 | wider. For linear kernels, we can extract the
125 | coefficients of the linear decision boundary as follows:
126 |
127 | ```{python}
128 | svm_linear.coef_
129 |
130 | ```
131 |
132 |
133 | Since the support vector machine is an estimator in `sklearn`, we
134 | can use the usual machinery to tune it.
135 |
136 | ```{python}
137 | kfold = skm.KFold(5,
138 | random_state=0,
139 | shuffle=True)
140 | grid = skm.GridSearchCV(svm_linear,
141 | {'C':[0.001,0.01,0.1,1,5,10,100]},
142 | refit=True,
143 | cv=kfold,
144 | scoring='accuracy')
145 | grid.fit(X, y)
146 | grid.best_params_
147 |
148 | ```
149 |
150 |
151 | We can easily access the cross-validation errors for each of these models
152 | in `grid.cv_results_`. This prints out a lot of detail, so we
153 | extract the accuracy results only.
154 |
155 | ```{python}
156 | grid.cv_results_[('mean_test_score')]
157 |
158 | ```
159 | We see that `C=1` results in the highest cross-validation
160 | accuracy of 0.74, though
161 | the accuracy is the same for several values of `C`.
162 | The classifier `grid.best_estimator_` can be used to predict the class
163 | label on a set of test observations. Let’s generate a test data set.
164 |
165 | ```{python}
166 | X_test = rng.standard_normal((20, 2))
167 | y_test = np.array([-1]*10+[1]*10)
168 | X_test[y_test==1] += 1
169 |
170 | ```
171 |
172 | Now we predict the class labels of these test observations. Here we
173 | use the best model selected by cross-validation in order to make the
174 | predictions.
175 |
176 | ```{python}
177 | best_ = grid.best_estimator_
178 | y_test_hat = best_.predict(X_test)
179 | confusion_table(y_test_hat, y_test)
180 |
181 | ```
182 |
183 | Thus, with this value of `C`,
184 | 70% of the test
185 | observations are correctly classified. What if we had instead used
186 | `C=0.001`?
187 |
188 | ```{python}
189 | svm_ = SVC(C=0.001,
190 | kernel='linear').fit(X, y)
191 | y_test_hat = svm_.predict(X_test)
192 | confusion_table(y_test_hat, y_test)
193 |
194 | ```
195 |
196 | In this case 60% of test observations are correctly classified.
197 |
198 | We now consider a situation in which the two classes are linearly
199 | separable. Then we can find an optimal separating hyperplane using the
200 | `SVC()` estimator. We first
201 | further separate the two classes in our simulated data so that they
202 | are linearly separable:
203 |
204 | ```{python}
205 | X[y==1] += 1.9;
206 | fig, ax = subplots(figsize=(8,8))
207 | ax.scatter(X[:,0], X[:,1], c=y, cmap=cm.coolwarm);
208 |
209 | ```
210 |
211 | Now the observations are just barely linearly separable.
212 |
213 | ```{python}
214 | svm_ = SVC(C=1e5, kernel='linear').fit(X, y)
215 | y_hat = svm_.predict(X)
216 | confusion_table(y_hat, y)
217 |
218 | ```
219 |
220 | We fit the
221 | support vector classifier and plot the resulting hyperplane, using a
222 | very large value of `C` so that no observations are
223 | misclassified.
224 |
225 | ```{python}
226 | fig, ax = subplots(figsize=(8,8))
227 | plot_svm(X,
228 | y,
229 | svm_,
230 | ax=ax)
231 |
232 | ```
233 | Indeed no training errors were made and only three support vectors were used.
234 | In fact, the large value of `C` also means that these three support points are *on the margin*, and define it.
235 | One may wonder how good the classifier could be on test data that depends on only three data points!
236 | We now try a smaller
237 | value of `C`.
238 |
239 | ```{python}
240 | svm_ = SVC(C=0.1, kernel='linear').fit(X, y)
241 | y_hat = svm_.predict(X)
242 | confusion_table(y_hat, y)
243 |
244 | ```
245 |
246 | Using `C=0.1`, we again do not misclassify any training observations, but we
247 | also obtain a much wider margin and make use of twelve support
248 | vectors. These jointly define the orientation of the decision boundary, and since there are more of them, it is more stable. It seems possible that this model will perform better on test
249 | data than the model with `C=1e5` (and indeed, a simple experiment with a large test set would bear this out).
250 |
251 | ```{python}
252 | fig, ax = subplots(figsize=(8,8))
253 | plot_svm(X,
254 | y,
255 | svm_,
256 | ax=ax)
257 |
258 | ```
259 |
260 |
261 | ## Support Vector Machine
262 | In order to fit an SVM using a non-linear kernel, we once again use
263 | the `SVC()` estimator. However, now we use a different value
264 | of the parameter `kernel`. To fit an SVM with a polynomial
265 | kernel we use `kernel="poly"`, and to fit an SVM with a
266 | radial kernel we use
267 | `kernel="rbf"`. In the former case we also use the
268 | `degree` argument to specify a degree for the polynomial kernel
269 | (this is $d$ in (9.22)), and in the latter case we use
270 | `gamma` to specify a value of $\gamma$ for the radial basis
271 | kernel (9.24).
272 |
273 | We first generate some data with a non-linear class boundary, as follows:
274 |
275 | ```{python}
276 | X = rng.standard_normal((200, 2))
277 | X[:100] += 2
278 | X[100:150] -= 2
279 | y = np.array([1]*150+[2]*50)
280 |
281 | ```
282 |
283 | Plotting the data makes it clear that the class boundary is indeed non-linear.
284 |
285 | ```{python}
286 | fig, ax = subplots(figsize=(8,8))
287 | ax.scatter(X[:,0],
288 | X[:,1],
289 | c=y,
290 | cmap=cm.coolwarm);
291 |
292 | ```
293 |
294 |
295 | The data is randomly split into training and testing groups. We then
296 | fit the training data using the `SVC()` estimator with a
297 | radial kernel and $\gamma=1$:
298 |
299 | ```{python}
300 | (X_train,
301 | X_test,
302 | y_train,
303 | y_test) = skm.train_test_split(X,
304 | y,
305 | test_size=0.5,
306 | random_state=0)
307 | svm_rbf = SVC(kernel="rbf", gamma=1, C=1)
308 | svm_rbf.fit(X_train, y_train)
309 |
310 | ```
311 |
312 | The plot shows that the resulting SVM has a decidedly non-linear
313 | boundary.
314 |
315 | ```{python}
316 | fig, ax = subplots(figsize=(8,8))
317 | plot_svm(X_train,
318 | y_train,
319 | svm_rbf,
320 | ax=ax)
321 |
322 | ```
323 |
324 | We can see from the figure that there are a fair number of training
325 | errors in this SVM fit. If we increase the value of `C`, we
326 | can reduce the number of training errors. However, this comes at the
327 | price of a more irregular decision boundary that seems to be at risk
328 | of overfitting the data.
329 |
330 | ```{python}
331 | svm_rbf = SVC(kernel="rbf", gamma=1, C=1e5)
332 | svm_rbf.fit(X_train, y_train)
333 | fig, ax = subplots(figsize=(8,8))
334 | plot_svm(X_train,
335 | y_train,
336 | svm_rbf,
337 | ax=ax)
338 |
339 | ```
340 |
341 | We can perform cross-validation using `skm.GridSearchCV()` to select the
342 | best choice of $\gamma$ and `C` for an SVM with a radial
343 | kernel:
344 |
345 | ```{python}
346 | kfold = skm.KFold(5,
347 | random_state=0,
348 | shuffle=True)
349 | grid = skm.GridSearchCV(svm_rbf,
350 | {'C':[0.1,1,10,100,1000],
351 | 'gamma':[0.5,1,2,3,4]},
352 | refit=True,
353 | cv=kfold,
354 | scoring='accuracy');
355 | grid.fit(X_train, y_train)
356 | grid.best_params_
357 |
358 | ```
359 |
360 | The best choice of parameters under five-fold CV is achieved at `C=1`
361 | and `gamma=0.5`, though several other values also achieve the same
362 | value.
363 |
364 | ```{python}
365 | best_svm = grid.best_estimator_
366 | fig, ax = subplots(figsize=(8,8))
367 | plot_svm(X_train,
368 | y_train,
369 | best_svm,
370 | ax=ax)
371 |
372 | y_hat_test = best_svm.predict(X_test)
373 | confusion_table(y_hat_test, y_test)
374 |
375 | ```
376 |
377 | With these parameters, 12% of test
378 | observations are misclassified by this SVM.
379 |
380 |
381 | ## ROC Curves
382 |
383 | SVMs and support vector classifiers output class labels for each
384 | observation. However, it is also possible to obtain *fitted values*
385 | for each observation, which are the numerical scores used to
386 | obtain the class labels. For instance, in the case of a support vector
387 | classifier, the fitted value for an observation $X= (X_1, X_2, \ldots,
388 | X_p)^T$ takes the form $\hat{\beta}_0 + \hat{\beta}_1 X_1 +
389 | \hat{\beta}_2 X_2 + \ldots + \hat{\beta}_p X_p$. For an SVM with a
390 | non-linear kernel, the equation that yields the fitted value is given
391 | in (9.23). The sign of the fitted value
392 | determines on which side of the decision boundary the observation
393 | lies. Therefore, the relationship between the fitted value and the
394 | class prediction for a given observation is simple: if the fitted
395 | value exceeds zero then the observation is assigned to one class, and
396 | if it is less than zero then it is assigned to the other.
397 | By changing this threshold from zero to some positive value,
398 | we skew the classifications in favor of one class versus the other.
399 | By considering a range of these thresholds, positive and negative, we produce the ingredients for a ROC plot.
400 | We can access these values by calling the `decision_function()`
401 | method of a fitted SVM estimator.
402 |
403 | The function `ROCCurveDisplay.from_estimator()` (which we have abbreviated to `roc_curve()`) will produce a plot of a ROC curve. It takes a fitted estimator as its first argument, followed
404 | by a model matrix $X$ and labels $y$. The argument `name` is used in the legend,
405 | while `color` is used for the color of the line. Results are plotted
406 | on our axis object `ax`.
407 |
408 | ```{python}
409 | fig, ax = subplots(figsize=(8,8))
410 | roc_curve(best_svm,
411 | X_train,
412 | y_train,
413 | name='Training',
414 | color='r',
415 | ax=ax);
416 |
417 | ```
418 | In this example, the SVM appears to provide accurate predictions. By increasing
419 | $\gamma$ we can produce a more flexible fit and generate further
420 | improvements in accuracy.
421 |
422 | ```{python}
423 | svm_flex = SVC(kernel="rbf",
424 | gamma=50,
425 | C=1)
426 | svm_flex.fit(X_train, y_train)
427 | fig, ax = subplots(figsize=(8,8))
428 | roc_curve(svm_flex,
429 | X_train,
430 | y_train,
431 | name='Training $\gamma=50$',
432 | color='r',
433 | ax=ax);
434 |
435 | ```
436 |
437 | However, these ROC curves are all on the training data. We are really
438 | more interested in the level of prediction accuracy on the test
439 | data. When we compute the ROC curves on the test data, the model with
440 | $\gamma=0.5$ appears to provide the most accurate results.
441 |
442 | ```{python}
443 | roc_curve(svm_flex,
444 | X_test,
445 | y_test,
446 | name='Test $\gamma=50$',
447 | color='b',
448 | ax=ax)
449 | fig;
450 |
451 | ```
452 |
453 | Let’s look at our tuned SVM.
454 |
455 | ```{python}
456 | fig, ax = subplots(figsize=(8,8))
457 | for (X_, y_, c, name) in zip(
458 | (X_train, X_test),
459 | (y_train, y_test),
460 | ('r', 'b'),
461 | ('CV tuned on training',
462 | 'CV tuned on test')):
463 | roc_curve(best_svm,
464 | X_,
465 | y_,
466 | name=name,
467 | ax=ax,
468 | color=c)
469 |
470 | ```
471 |
472 | ## SVM with Multiple Classes
473 |
474 | If the response is a factor containing more than two levels, then the
475 | `SVC()` function will perform multi-class classification using
476 | either the one-versus-one approach (when `decision_function_shape=='ovo'`)
477 | or one-versus-rest {One-versus-rest is also known as one-versus-all.} (when `decision_function_shape=='ovr'`).
478 | We explore that setting briefly here by
479 | generating a third class of observations.
480 |
481 | ```{python}
482 | rng = np.random.default_rng(123)
483 | X = np.vstack([X, rng.standard_normal((50, 2))])
484 | y = np.hstack([y, [0]*50])
485 | X[y==0,1] += 2
486 | fig, ax = subplots(figsize=(8,8))
487 | ax.scatter(X[:,0], X[:,1], c=y, cmap=cm.coolwarm);
488 |
489 | ```
490 |
491 | We now fit an SVM to the data:
492 |
493 | ```{python}
494 | svm_rbf_3 = SVC(kernel="rbf",
495 | C=10,
496 | gamma=1,
497 | decision_function_shape='ovo');
498 | svm_rbf_3.fit(X, y)
499 | fig, ax = subplots(figsize=(8,8))
500 | plot_svm(X,
501 | y,
502 | svm_rbf_3,
503 | scatter_cmap=cm.tab10,
504 | ax=ax)
505 |
506 | ```
507 | The `sklearn.svm` library can also be used to perform support vector
508 | regression with a numerical response using the estimator `SupportVectorRegression()`.
509 |
510 |
511 | ## Application to Gene Expression Data
512 |
513 | We now examine the `Khan` data set, which consists of a number of
514 | tissue samples corresponding to four distinct types of small round
515 | blue cell tumors. For each tissue sample, gene expression measurements
516 | are available. The data set consists of training data, `xtrain`
517 | and `ytrain`, and testing data, `xtest` and `ytest`.
518 |
519 | We examine the dimension of the data:
520 |
521 | ```{python}
522 | Khan = load_data('Khan')
523 | Khan['xtrain'].shape, Khan['xtest'].shape
524 |
525 | ```
526 |
527 | This data set consists of expression measurements for 2,308
528 | genes. The training and test sets consist of 63 and 20
529 | observations, respectively.
530 |
531 | We will use a support vector approach to predict cancer subtype using
532 | gene expression measurements. In this data set, there is a very
533 | large number of features relative to the number of observations. This
534 | suggests that we should use a linear kernel, because the additional
535 | flexibility that will result from using a polynomial or radial kernel
536 | is unnecessary.
537 |
538 | ```{python}
539 | khan_linear = SVC(kernel='linear', C=10)
540 | khan_linear.fit(Khan['xtrain'], Khan['ytrain'])
541 | confusion_table(khan_linear.predict(Khan['xtrain']),
542 | Khan['ytrain'])
543 |
544 | ```
545 |
546 | We see that there are *no* training
547 | errors. In fact, this is not surprising, because the large number of
548 | variables relative to the number of observations implies that it is
549 | easy to find hyperplanes that fully separate the classes. We are more
550 | interested in the support vector classifier’s performance on the
551 | test observations.
552 |
553 | ```{python}
554 | confusion_table(khan_linear.predict(Khan['xtest']),
555 | Khan['ytest'])
556 |
557 | ```
558 |
559 | We see that using `C=10` yields two test set errors on these data.
560 |
561 |
562 |
--------------------------------------------------------------------------------
/Ch11-surv-lab.Rmd:
--------------------------------------------------------------------------------
1 | ---
2 | jupyter:
3 | jupytext:
4 | cell_metadata_filter: -all
5 | formats: ipynb,Rmd
6 | main_language: python
7 | text_representation:
8 | extension: .Rmd
9 | format_name: rmarkdown
10 | format_version: '1.2'
11 | jupytext_version: 1.16.7
12 | ---
13 |
14 | # Survival Analysis
15 |
16 |
17 |
18 |
19 |
20 | [](https://mybinder.org/v2/gh/intro-stat-learning/ISLP_labs/v2.2?labpath=Ch11-surv-lab.ipynb)
21 |
22 |
23 | In this lab, we perform survival analyses on three separate data
24 | sets. In Section 11.8.1 we analyze the `BrainCancer`
25 | data that was first described in Section 11.3. In Section 11.8.2, we examine the `Publication`
26 | data from Section 11.5.4. Finally, Section 11.8.3 explores
27 | a simulated call-center data set.
28 |
29 | We begin by importing some of our libraries at this top
30 | level. This makes the code more readable, as scanning the first few
31 | lines of the notebook tell us what libraries are used in this
32 | notebook.
33 |
34 | ```{python}
35 | from matplotlib.pyplot import subplots
36 | import numpy as np
37 | import pandas as pd
38 | from ISLP.models import ModelSpec as MS
39 | from ISLP import load_data
40 |
41 | ```
42 |
43 | We also collect the new imports
44 | needed for this lab.
45 |
46 | ```{python}
47 | from lifelines import \
48 | (KaplanMeierFitter,
49 | CoxPHFitter)
50 | from lifelines.statistics import \
51 | (logrank_test,
52 | multivariate_logrank_test)
53 | from ISLP.survival import sim_time
54 |
55 | ```
56 |
57 | ## Brain Cancer Data
58 |
59 | We begin with the `BrainCancer` data set, contained in the `ISLP` package.
60 |
61 | ```{python}
62 | BrainCancer = load_data('BrainCancer')
63 | BrainCancer.columns
64 |
65 | ```
66 |
67 | The rows index the 88 patients, while the 8 columns contain the predictors and outcome variables.
68 | We first briefly examine the data.
69 |
70 | ```{python}
71 | BrainCancer['sex'].value_counts()
72 |
73 | ```
74 |
75 |
76 | ```{python}
77 | BrainCancer['diagnosis'].value_counts()
78 |
79 | ```
80 |
81 |
82 | ```{python}
83 | BrainCancer['status'].value_counts()
84 |
85 | ```
86 |
87 |
88 | Before beginning an analysis, it is important to know how the
89 | `status` variable has been coded. Most software
90 | uses the convention that a `status` of 1 indicates an
91 | uncensored observation (often death), and a `status` of 0 indicates a censored
92 | observation. But some scientists might use the opposite coding. For
93 | the `BrainCancer` data set 35 patients died before the end of
94 | the study, so we are using the conventional coding.
95 |
96 | To begin the analysis, we re-create the Kaplan-Meier survival curve shown in Figure 11.2. The main
97 | package we will use for survival analysis
98 | is `lifelines`.
99 | The variable `time` corresponds to $y_i$, the time to the $i$th event (either censoring or
100 | death). The first argument to `km.fit` is the event time, and the
101 | second argument is the censoring variable, with a 1 indicating an observed
102 | failure time. The `plot()` method produces a survival curve with pointwise confidence
103 | intervals. By default, these are 90% confidence intervals, but this can be changed
104 | by setting the `alpha` argument to one minus the desired
105 | confidence level.
106 |
107 | ```{python}
108 | fig, ax = subplots(figsize=(8,8))
109 | km = KaplanMeierFitter()
110 | km_brain = km.fit(BrainCancer['time'], BrainCancer['status'])
111 | km_brain.plot(label='Kaplan Meier estimate', ax=ax)
112 |
113 | ```
114 |
115 | Next we create Kaplan-Meier survival curves that are stratified by
116 | `sex`, in order to reproduce Figure 11.3.
117 | We do this using the `groupby()` method of a dataframe.
118 | This method returns a generator that can
119 | be iterated over in the `for` loop. In this case,
120 | the items in the `for` loop are 2-tuples representing
121 | the groups: the first entry is the value
122 | of the grouping column `sex` while the second value
123 | is the dataframe consisting of all rows in the
124 | dataframe matching that value of `sex`.
125 | We will want to use this data below
126 | in the log-rank test, hence we store this
127 | information in the dictionary `by_sex`. Finally,
128 | we have also used the notion of
129 | *string interpolation* to automatically
130 | label the different lines in the plot. String
131 | interpolation is a powerful technique to format strings ---
132 | `Python` has many ways to facilitate such operations.
133 |
134 | ```{python}
135 | fig, ax = subplots(figsize=(8,8))
136 | by_sex = {}
137 | for sex, df in BrainCancer.groupby('sex'):
138 | by_sex[sex] = df
139 | km_sex = km.fit(df['time'], df['status'])
140 | km_sex.plot(label='Sex=%s' % sex, ax=ax)
141 |
142 | ```
143 |
144 | As discussed in Section 11.4, we can perform a
145 | log-rank test to compare the survival of males to females. We use
146 | the `logrank_test()` function from the `lifelines.statistics` module.
147 | The first two arguments are the event times, with the second
148 | denoting the corresponding (optional) censoring indicators.
149 |
150 | ```{python}
151 | logrank_test(by_sex['Male']['time'],
152 | by_sex['Female']['time'],
153 | by_sex['Male']['status'],
154 | by_sex['Female']['status'])
155 |
156 | ```
157 |
158 |
159 | The resulting $p$-value is $0.23$, indicating no evidence of a
160 | difference in survival between the two sexes.
161 |
162 | Next, we use the `CoxPHFitter()` estimator
163 | from `lifelines` to fit Cox proportional hazards models.
164 | To begin, we consider a model that uses `sex` as the only predictor.
165 |
166 | ```{python}
167 | coxph = CoxPHFitter # shorthand
168 | sex_df = BrainCancer[['time', 'status', 'sex']]
169 | model_df = MS(['time', 'status', 'sex'],
170 | intercept=False).fit_transform(sex_df)
171 | cox_fit = coxph().fit(model_df,
172 | 'time',
173 | 'status')
174 | cox_fit.summary[['coef', 'se(coef)', 'p']]
175 |
176 | ```
177 |
178 | The first argument to `fit` should be a data frame containing
179 | at least the event time (the second argument `time` in this case),
180 | as well as an optional censoring variable (the argument `status` in this case).
181 | Note also that the Cox model does not include an intercept, which is why
182 | we used the `intercept=False` argument to `ModelSpec` above.
183 | The `summary()` method delivers many columns; we chose to abbreviate its output here.
184 | It is possible to obtain the likelihood ratio test comparing this model to the one
185 | with no features as follows:
186 |
187 | ```{python}
188 | cox_fit.log_likelihood_ratio_test()
189 |
190 | ```
191 |
192 | Regardless of which test we use, we see that there is no clear
193 | evidence for a difference in survival between males and females. As
194 | we learned in this chapter, the score test from the Cox model is
195 | exactly equal to the log rank test statistic!
196 |
197 | Now we fit a model that makes use of additional predictors. We first note
198 | that one of our `diagnosis` values is missing, hence
199 | we drop that observation before continuing.
200 |
201 | ```{python}
202 | cleaned = BrainCancer.dropna()
203 | all_MS = MS(cleaned.columns, intercept=False)
204 | all_df = all_MS.fit_transform(cleaned)
205 | fit_all = coxph().fit(all_df,
206 | 'time',
207 | 'status')
208 | fit_all.summary[['coef', 'se(coef)', 'p']]
209 |
210 | ```
211 |
212 | The `diagnosis` variable has been coded so that the baseline
213 | corresponds to HG glioma. The results indicate that the risk associated with HG glioma
214 | is more than eight times (i.e. $e^{2.15}=8.62$) the risk associated
215 | with meningioma. In other words, after adjusting for the other
216 | predictors, patients with HG glioma have much worse survival compared
217 | to those with meningioma. In addition, larger values of the Karnofsky
218 | index, `ki`, are associated with lower risk, i.e. longer survival.
219 |
220 | Finally, we plot estimated survival curves for each diagnosis category,
221 | adjusting for the other predictors. To make these plots, we set the
222 | values of the other predictors equal to the mean for quantitative variables
223 | and equal to the mode for categorical. To do this, we use the
224 | `apply()` method across rows (i.e. `axis=0`) with a function
225 | `representative` that checks if a column is categorical
226 | or not.
227 |
228 | ```{python}
229 | levels = cleaned['diagnosis'].unique()
230 | def representative(series):
231 | if hasattr(series.dtype, 'categories'):
232 | return pd.Series.mode(series)
233 | else:
234 | return series.mean()
235 | modal_data = cleaned.apply(representative, axis=0)
236 |
237 | ```
238 |
239 | We make four
240 | copies of the column means and assign the `diagnosis` column to be the four different
241 | diagnoses.
242 |
243 | ```{python}
244 | modal_df = pd.DataFrame(
245 | [modal_data.iloc[0] for _ in range(len(levels))])
246 | modal_df['diagnosis'] = levels
247 | modal_df
248 |
249 | ```
250 |
251 | We then construct the model matrix based on the model specification `all_MS` used to fit
252 | the model, and name the rows according to the levels of `diagnosis`.
253 |
254 | ```{python}
255 | modal_X = all_MS.transform(modal_df)
256 | modal_X.index = levels
257 | modal_X
258 |
259 | ```
260 |
261 | We can use the `predict_survival_function()` method to obtain the estimated survival function.
262 |
263 | ```{python}
264 | predicted_survival = fit_all.predict_survival_function(modal_X)
265 | predicted_survival
266 |
267 | ```
268 | This returns a data frame,
269 | whose plot methods yields the different survival curves. To avoid clutter in
270 | the plots, we do not display confidence intervals.
271 |
272 | ```{python}
273 | fig, ax = subplots(figsize=(8, 8))
274 | predicted_survival.plot(ax=ax);
275 |
276 | ```
277 |
278 |
279 | ## Publication Data
280 | The `Publication` data presented in Section 11.5.4 can be
281 | found in the `ISLP` package.
282 | We first reproduce Figure 11.5 by plotting the Kaplan-Meier curves
283 | stratified on the `posres` variable, which records whether the
284 | study had a positive or negative result.
285 |
286 | ```{python}
287 | fig, ax = subplots(figsize=(8,8))
288 | Publication = load_data('Publication')
289 | by_result = {}
290 | for result, df in Publication.groupby('posres'):
291 | by_result[result] = df
292 | km_result = km.fit(df['time'], df['status'])
293 | km_result.plot(label='Result=%d' % result, ax=ax)
294 |
295 | ```
296 |
297 | As discussed previously, the $p$-values from fitting Cox’s
298 | proportional hazards model to the `posres` variable are quite
299 | large, providing no evidence of a difference in time-to-publication
300 | between studies with positive versus negative results.
301 |
302 | ```{python}
303 | posres_df = MS(['posres',
304 | 'time',
305 | 'status'],
306 | intercept=False).fit_transform(Publication)
307 | posres_fit = coxph().fit(posres_df,
308 | 'time',
309 | 'status')
310 | posres_fit.summary[['coef', 'se(coef)', 'p']]
311 |
312 | ```
313 |
314 |
315 | However, the results change dramatically when we include other
316 | predictors in the model. Here we exclude the funding mechanism
317 | variable.
318 |
319 | ```{python}
320 | model = MS(Publication.columns.drop('mech'),
321 | intercept=False)
322 | coxph().fit(model.fit_transform(Publication),
323 | 'time',
324 | 'status').summary[['coef', 'se(coef)', 'p']]
325 |
326 | ```
327 |
328 | We see that there are a number of statistically significant variables,
329 | including whether the trial focused on a clinical endpoint, the impact
330 | of the study, and whether the study had positive or negative results.
331 |
332 |
333 | ## Call Center Data
334 |
335 | In this section, we will simulate survival data using the relationship
336 | between cumulative hazard and
337 | the survival function explored in Exercise 8.
338 | Our simulated data will represent the observed
339 | wait times (in seconds) for 2,000 customers who have phoned a call
340 | center. In this context, censoring occurs if a customer hangs up
341 | before his or her call is answered.
342 |
343 | There are three covariates: `Operators` (the number of call
344 | center operators available at the time of the call, which can range
345 | from $5$ to $15$), `Center` (either A, B, or C), and
346 | `Time` of day (Morning, Afternoon, or Evening). We generate data
347 | for these covariates so that all possibilities are equally likely: for
348 | instance, morning, afternoon and evening calls are equally likely, and
349 | any number of operators from $5$ to $15$ is equally likely.
350 |
351 | ```{python}
352 | rng = np.random.default_rng(10)
353 | N = 2000
354 | Operators = rng.choice(np.arange(5, 16),
355 | N,
356 | replace=True)
357 | Center = rng.choice(['A', 'B', 'C'],
358 | N,
359 | replace=True)
360 | Time = rng.choice(['Morn.', 'After.', 'Even.'],
361 | N,
362 | replace=True)
363 | D = pd.DataFrame({'Operators': Operators,
364 | 'Center': pd.Categorical(Center),
365 | 'Time': pd.Categorical(Time)})
366 | ```
367 |
368 | We then build a model matrix (omitting the intercept)
369 |
370 | ```{python}
371 | model = MS(['Operators',
372 | 'Center',
373 | 'Time'],
374 | intercept=False)
375 | X = model.fit_transform(D)
376 | ```
377 |
378 | It is worthwhile to take a peek at the model matrix `X`, so
379 | that we can be sure that we understand how the variables have been coded. By default,
380 | the levels of categorical variables are sorted and, as usual, the first column of the one-hot encoding
381 | of the variable is dropped.
382 |
383 | ```{python}
384 | X[:5]
385 |
386 | ```
387 |
388 | Next, we specify the coefficients and the hazard function.
389 |
390 | ```{python}
391 | true_beta = np.array([0.04, -0.3, 0, 0.2, -0.2])
392 | true_linpred = X.dot(true_beta)
393 | hazard = lambda t: 1e-5 * t
394 |
395 | ```
396 |
397 | Here, we have set the coefficient associated with `Operators` to
398 | equal $0.04$; in other words, each additional operator leads to a
399 | $e^{0.04}=1.041$-fold increase in the “risk” that the call will be
400 | answered, given the `Center` and `Time` covariates. This
401 | makes sense: the greater the number of operators at hand, the shorter
402 | the wait time! The coefficient associated with `Center == B` is
403 | $-0.3$, and `Center == A` is treated as the baseline. This means
404 | that the risk of a call being answered at Center B is 0.74 times the
405 | risk that it will be answered at Center A; in other words, the wait
406 | times are a bit longer at Center B.
407 |
408 | Recall from Section 2.3.7 the use of `lambda`
409 | for creating short functions on the fly.
410 | We use the function
411 | `sim_time()` from the `ISLP.survival` package. This function
412 | uses the relationship between the survival function
413 | and cumulative hazard $S(t) = \exp(-H(t))$ and the specific
414 | form of the cumulative hazard function in the Cox model
415 | to simulate data based on values of the linear predictor
416 | `true_linpred` and the cumulative hazard.
417 | We need to provide the cumulative hazard function, which we do here.
418 |
419 | ```{python}
420 | cum_hazard = lambda t: 1e-5 * t**2 / 2
421 |
422 | ```
423 | We are now ready to generate data under the Cox proportional hazards
424 | model. We truncate the maximum time to 1000 seconds to keep
425 | simulated wait times reasonable. The function
426 | `sim_time()` takes a linear predictor,
427 | a cumulative hazard function and a
428 | random number generator.
429 |
430 | ```{python}
431 | W = np.array([sim_time(l, cum_hazard, rng)
432 | for l in true_linpred])
433 | D['Wait time'] = np.clip(W, 0, 1000)
434 |
435 | ```
436 |
437 | We now simulate our censoring variable, for which we assume
438 | 90% of calls were answered (`Failed==1`) before the
439 | customer hung up (`Failed==0`).
440 |
441 | ```{python}
442 | D['Failed'] = rng.choice([1, 0],
443 | N,
444 | p=[0.9, 0.1])
445 | D[:5]
446 |
447 | ```
448 |
449 |
450 | ```{python}
451 | D['Failed'].mean()
452 |
453 | ```
454 |
455 | We now plot Kaplan-Meier survival curves. First, we stratify by `Center`.
456 |
457 | ```{python}
458 | fig, ax = subplots(figsize=(8,8))
459 | by_center = {}
460 | for center, df in D.groupby('Center'):
461 | by_center[center] = df
462 | km_center = km.fit(df['Wait time'], df['Failed'])
463 | km_center.plot(label='Center=%s' % center, ax=ax)
464 | ax.set_title("Probability of Still Being on Hold")
465 |
466 | ```
467 |
468 | Next, we stratify by `Time`.
469 |
470 | ```{python}
471 | fig, ax = subplots(figsize=(8,8))
472 | by_time = {}
473 | for time, df in D.groupby('Time'):
474 | by_time[time] = df
475 | km_time = km.fit(df['Wait time'], df['Failed'])
476 | km_time.plot(label='Time=%s' % time, ax=ax)
477 | ax.set_title("Probability of Still Being on Hold")
478 |
479 | ```
480 |
481 | It seems that calls at Call Center B take longer to be answered than
482 | calls at Centers A and C. Similarly, it appears that wait times are
483 | longest in the morning and shortest in the evening hours. We can use a
484 | log-rank test to determine whether these differences are statistically
485 | significant using the function `multivariate_logrank_test()`.
486 |
487 | ```{python}
488 | multivariate_logrank_test(D['Wait time'],
489 | D['Center'],
490 | D['Failed'])
491 |
492 | ```
493 |
494 |
495 | Next, we consider the effect of `Time`.
496 |
497 | ```{python}
498 | multivariate_logrank_test(D['Wait time'],
499 | D['Time'],
500 | D['Failed'])
501 |
502 | ```
503 |
504 |
505 | As in the case of a categorical variable with 2 levels, these
506 | results are similar to the likelihood ratio test
507 | from the Cox proportional hazards model. First, we
508 | look at the results for `Center`.
509 |
510 | ```{python}
511 | X = MS(['Wait time',
512 | 'Failed',
513 | 'Center'],
514 | intercept=False).fit_transform(D)
515 | F = coxph().fit(X, 'Wait time', 'Failed')
516 | F.log_likelihood_ratio_test()
517 |
518 | ```
519 |
520 |
521 | Next, we look at the results for `Time`.
522 |
523 | ```{python}
524 | X = MS(['Wait time',
525 | 'Failed',
526 | 'Time'],
527 | intercept=False).fit_transform(D)
528 | F = coxph().fit(X, 'Wait time', 'Failed')
529 | F.log_likelihood_ratio_test()
530 |
531 | ```
532 |
533 |
534 | We find that differences between centers are highly significant, as
535 | are differences between times of day.
536 |
537 | Finally, we fit Cox's proportional hazards model to the data.
538 |
539 | ```{python}
540 | X = MS(D.columns,
541 | intercept=False).fit_transform(D)
542 | fit_queuing = coxph().fit(
543 | X,
544 | 'Wait time',
545 | 'Failed')
546 | fit_queuing.summary[['coef', 'se(coef)', 'p']]
547 |
548 | ```
549 |
550 |
551 | The $p$-values for Center B and evening time
552 | are very small. It is also clear that the
553 | hazard --- that is, the instantaneous risk that a call will be
554 | answered --- increases with the number of operators. Since we
555 | generated the data ourselves, we know that the true coefficients for
556 | `Operators`, `Center = B`, `Center = C`,
557 | `Time = Even.` and `Time = Morn.` are $0.04$, $-0.3$,
558 | $0$, $0.2$, and $-0.2$, respectively. The coefficient estimates
559 | from the fitted Cox model are fairly accurate.
560 |
561 |
562 |
--------------------------------------------------------------------------------
/Ch13-multiple-lab.Rmd:
--------------------------------------------------------------------------------
1 | ---
2 | jupyter:
3 | jupytext:
4 | cell_metadata_filter: -all
5 | formats: ipynb,Rmd
6 | main_language: python
7 | text_representation:
8 | extension: .Rmd
9 | format_name: rmarkdown
10 | format_version: '1.2'
11 | jupytext_version: 1.16.7
12 | ---
13 |
14 | # Multiple Testing
15 |
16 |
17 |
18 |
19 |
20 | [](https://mybinder.org/v2/gh/intro-stat-learning/ISLP_labs/v2.2?labpath=Ch13-multiple-lab.ipynb)
21 |
22 |
23 |
24 |
25 |
26 | We include our usual imports seen in earlier labs.
27 |
28 | ```{python}
29 | import numpy as np
30 | import pandas as pd
31 | import matplotlib.pyplot as plt
32 | import statsmodels.api as sm
33 | from ISLP import load_data
34 |
35 | ```
36 |
37 | We also collect the new imports
38 | needed for this lab.
39 |
40 | ```{python}
41 | from scipy.stats import \
42 | (ttest_1samp,
43 | ttest_rel,
44 | ttest_ind,
45 | t as t_dbn)
46 | from statsmodels.stats.multicomp import \
47 | pairwise_tukeyhsd
48 | from statsmodels.stats.multitest import \
49 | multipletests as mult_test
50 |
51 | ```
52 |
53 |
54 | ## Review of Hypothesis Tests
55 | We begin by performing some one-sample $t$-tests.
56 |
57 | First we create 100 variables, each consisting of 10 observations. The
58 | first 50 variables have mean $0.5$ and variance $1$, while the others
59 | have mean $0$ and variance $1$.
60 |
61 | ```{python}
62 | rng = np.random.default_rng(12)
63 | X = rng.standard_normal((10, 100))
64 | true_mean = np.array([0.5]*50 + [0]*50)
65 | X += true_mean[None,:]
66 |
67 | ```
68 |
69 | To begin, we use `ttest_1samp()` from the
70 | `scipy.stats` module to test $H_{0}: \mu_1=0$, the null
71 | hypothesis that the first variable has mean zero.
72 |
73 | ```{python}
74 | result = ttest_1samp(X[:,0], 0)
75 | result.pvalue
76 |
77 | ```
78 |
79 | The $p$-value comes out to 0.931, which is not low enough to
80 | reject the null hypothesis at level $\alpha=0.05$. In this case,
81 | $\mu_1=0.5$, so the null hypothesis is false. Therefore, we have made
82 | a Type II error by failing to reject the null hypothesis when the null
83 | hypothesis is false.
84 |
85 | We now test $H_{0,j}: \mu_j=0$ for $j=1,\ldots,100$. We compute the
86 | 100 $p$-values, and then construct a vector recording whether the
87 | $j$th $p$-value is less than or equal to 0.05, in which case we reject
88 | $H_{0j}$, or greater than 0.05, in which case we do not reject
89 | $H_{0j}$, for $j=1,\ldots,100$.
90 |
91 | ```{python}
92 | p_values = np.empty(100)
93 | for i in range(100):
94 | p_values[i] = ttest_1samp(X[:,i], 0).pvalue
95 | decision = pd.cut(p_values,
96 | [0, 0.05, 1],
97 | labels=['Reject H0',
98 | 'Do not reject H0'])
99 | truth = pd.Categorical(true_mean == 0,
100 | categories=[True, False],
101 | ordered=True)
102 |
103 | ```
104 | Since this is a simulated data set, we can create a $2 \times 2$ table
105 | similar to Table 13.2.
106 |
107 | ```{python}
108 | pd.crosstab(decision,
109 | truth,
110 | rownames=['Decision'],
111 | colnames=['H0'])
112 |
113 | ```
114 | Therefore, at level $\alpha=0.05$, we reject 15 of the 50 false
115 | null hypotheses, and we incorrectly reject 5 of the true null
116 | hypotheses. Using the notation from Section 13.3, we have
117 | $V=5$, $S=15$, $U=45$ and $W=35$.
118 | We have set $\alpha=0.05$, which means that we expect to reject around
119 | 5% of the true null hypotheses. This is in line with the $2 \times 2$
120 | table above, which indicates that we rejected $V=5$ of the $50$ true
121 | null hypotheses.
122 |
123 | In the simulation above, for the false null hypotheses, the ratio of
124 | the mean to the standard deviation was only $0.5/1 = 0.5$. This
125 | amounts to quite a weak signal, and it resulted in a high number of
126 | Type II errors. Let’s instead simulate data with a stronger signal,
127 | so that the ratio of the mean to the standard deviation for the false
128 | null hypotheses equals $1$. We make only 10 Type II errors.
129 |
130 |
131 | ```{python}
132 | true_mean = np.array([1]*50 + [0]*50)
133 | X = rng.standard_normal((10, 100))
134 | X += true_mean[None,:]
135 | for i in range(100):
136 | p_values[i] = ttest_1samp(X[:,i], 0).pvalue
137 | decision = pd.cut(p_values,
138 | [0, 0.05, 1],
139 | labels=['Reject H0',
140 | 'Do not reject H0'])
141 | truth = pd.Categorical(true_mean == 0,
142 | categories=[True, False],
143 | ordered=True)
144 | pd.crosstab(decision,
145 | truth,
146 | rownames=['Decision'],
147 | colnames=['H0'])
148 |
149 | ```
150 |
151 |
152 |
153 | ## Family-Wise Error Rate
154 | Recall from (13.5) that if the null hypothesis is true
155 | for each of $m$ independent hypothesis tests, then the FWER is equal
156 | to $1-(1-\alpha)^m$. We can use this expression to compute the FWER
157 | for $m=1,\ldots, 500$ and $\alpha=0.05$, $0.01$, and $0.001$.
158 | We plot the FWER for these values of $\alpha$ in order to
159 | reproduce Figure 13.2.
160 |
161 | ```{python}
162 | m = np.linspace(1, 501)
163 | fig, ax = plt.subplots()
164 | [ax.plot(m,
165 | 1 - (1 - alpha)**m,
166 | label=r'$\alpha=%s$' % str(alpha))
167 | for alpha in [0.05, 0.01, 0.001]]
168 | ax.set_xscale('log')
169 | ax.set_xlabel('Number of Hypotheses')
170 | ax.set_ylabel('Family-Wise Error Rate')
171 | ax.legend()
172 | ax.axhline(0.05, c='k', ls='--');
173 |
174 | ```
175 |
176 | As discussed previously, even for moderate values of $m$ such as $50$,
177 | the FWER exceeds $0.05$ unless $\alpha$ is set to a very low value,
178 | such as $0.001$. Of course, the problem with setting $\alpha$ to such
179 | a low value is that we are likely to make a number of Type II errors:
180 | in other words, our power is very low.
181 |
182 | We now conduct a one-sample $t$-test for each of the first five
183 | managers in the
184 | `Fund` dataset, in order to test the null
185 | hypothesis that the $j$th fund manager’s mean return equals zero,
186 | $H_{0,j}: \mu_j=0$.
187 |
188 | ```{python}
189 | Fund = load_data('Fund')
190 | fund_mini = Fund.iloc[:,:5]
191 | fund_mini_pvals = np.empty(5)
192 | for i in range(5):
193 | fund_mini_pvals[i] = ttest_1samp(fund_mini.iloc[:,i], 0).pvalue
194 | fund_mini_pvals
195 |
196 | ```
197 |
198 | The $p$-values are low for Managers One and Three, and high for the
199 | other three managers. However, we cannot simply reject $H_{0,1}$ and
200 | $H_{0,3}$, since this would fail to account for the multiple testing
201 | that we have performed. Instead, we will conduct Bonferroni’s method
202 | and Holm’s method to control the FWER.
203 |
204 | To do this, we use the `multipletests()` function from the
205 | `statsmodels` module (abbreviated to `mult_test()`). Given the $p$-values,
206 | for methods like Holm and Bonferroni the function outputs
207 | adjusted $p$-values, which
208 | can be thought of as a new set of $p$-values that have been corrected
209 | for multiple testing. If the adjusted $p$-value for a given hypothesis
210 | is less than or equal to $\alpha$, then that hypothesis can be
211 | rejected while maintaining a FWER of no more than $\alpha$. In other
212 | words, for such methods, the adjusted $p$-values resulting from the `multipletests()`
213 | function can simply be compared to the desired FWER in order to
214 | determine whether or not to reject each hypothesis. We will later
215 | see that we can use the same function to control FDR as well.
216 |
217 |
218 | The `mult_test()` function takes $p$-values and a `method` argument, as well as an optional
219 | `alpha` argument. It returns the decisions (`reject` below)
220 | as well as the adjusted $p$-values (`bonf`).
221 |
222 | ```{python}
223 | reject, bonf = mult_test(fund_mini_pvals, method = "bonferroni")[:2]
224 | reject
225 |
226 | ```
227 |
228 |
229 | The $p$-values `bonf` are simply the `fund_mini_pvalues` multiplied by 5 and truncated to be less than
230 | or equal to 1.
231 |
232 | ```{python}
233 | bonf, np.minimum(fund_mini_pvals * 5, 1)
234 |
235 | ```
236 |
237 | Therefore, using Bonferroni’s method, we are able to reject the null hypothesis only for Manager
238 | One while controlling FWER at $0.05$.
239 |
240 | By contrast, using Holm’s method, the adjusted $p$-values indicate
241 | that we can reject the null
242 | hypotheses for Managers One and Three at a FWER of $0.05$.
243 |
244 | ```{python}
245 | mult_test(fund_mini_pvals, method = "holm", alpha=0.05)[:2]
246 |
247 | ```
248 |
249 |
250 | As discussed previously, Manager One seems to perform particularly
251 | well, whereas Manager Two has poor performance.
252 |
253 |
254 | ```{python}
255 | fund_mini.mean()
256 |
257 | ```
258 |
259 |
260 | Is there evidence of a meaningful difference in performance between
261 | these two managers? We can check this by performing a paired $t$-test using the `ttest_rel()` function
262 | from `scipy.stats`:
263 |
264 | ```{python}
265 | ttest_rel(fund_mini['Manager1'],
266 | fund_mini['Manager2']).pvalue
267 |
268 | ```
269 |
270 | The test results in a $p$-value of 0.038,
271 | suggesting a statistically significant difference.
272 |
273 | However, we decided to perform this test only after examining the data
274 | and noting that Managers One and Two had the highest and lowest mean
275 | performances. In a sense, this means that we have implicitly
276 | performed ${5 \choose 2} = 5(5-1)/2=10$ hypothesis tests, rather than
277 | just one, as discussed in Section 13.3.2. Hence, we use the
278 | `pairwise_tukeyhsd()` function from
279 | `statsmodels.stats.multicomp` to apply Tukey’s method
280 | in order to adjust for multiple testing. This function takes
281 | as input a fitted *ANOVA* regression model, which is
282 | essentially just a linear regression in which all of the predictors
283 | are qualitative. In this case, the response consists of the monthly
284 | excess returns achieved by each manager, and the predictor indicates
285 | the manager to which each return corresponds.
286 |
287 | ```{python}
288 | returns = np.hstack([fund_mini.iloc[:,i] for i in range(5)])
289 | managers = np.hstack([[i+1]*50 for i in range(5)])
290 | tukey = pairwise_tukeyhsd(returns, managers)
291 | print(tukey.summary())
292 |
293 | ```
294 |
295 |
296 | The `pairwise_tukeyhsd()` function provides confidence intervals
297 | for the difference between each pair of managers (`lower` and
298 | `upper`), as well as a $p$-value. All of these quantities have
299 | been adjusted for multiple testing. Notice that the $p$-value for the
300 | difference between Managers One and Two has increased from $0.038$ to
301 | $0.186$, so there is no longer clear evidence of a difference between
302 | the managers’ performances. We can plot the confidence intervals for
303 | the pairwise comparisons using the `plot_simultaneous()` method
304 | of `tukey`. Any pair of intervals that don’t overlap indicates a significant difference at the nominal level of 0.05. In this case,
305 | no differences are considered significant as reported in the table above.
306 |
307 | ```{python}
308 | fig, ax = plt.subplots(figsize=(8,8))
309 | tukey.plot_simultaneous(ax=ax);
310 |
311 | ```
312 |
313 | ## False Discovery Rate
314 | Now we perform hypothesis tests for all 2,000 fund managers in the
315 | `Fund` dataset. We perform a one-sample $t$-test
316 | of $H_{0,j}: \mu_j=0$, which states that the
317 | $j$th fund manager’s mean return is zero.
318 |
319 | ```{python}
320 | fund_pvalues = np.empty(2000)
321 | for i, manager in enumerate(Fund.columns):
322 | fund_pvalues[i] = ttest_1samp(Fund[manager], 0).pvalue
323 |
324 | ```
325 |
326 | There are far too many managers to consider trying to control the FWER.
327 | Instead, we focus on controlling the FDR: that is, the expected fraction of rejected null hypotheses that are actually false positives.
328 | The `multipletests()` function (abbreviated `mult_test()`) can be used to carry out the Benjamini--Hochberg procedure.
329 |
330 | ```{python}
331 | fund_qvalues = mult_test(fund_pvalues, method = "fdr_bh")[1]
332 | fund_qvalues[:10]
333 |
334 | ```
335 |
336 | The *q-values* output by the
337 | Benjamini--Hochberg procedure can be interpreted as the smallest FDR
338 | threshold at which we would reject a particular null hypothesis. For
339 | instance, a $q$-value of $0.1$ indicates that we can reject the
340 | corresponding null hypothesis at an FDR of 10% or greater, but that
341 | we cannot reject the null hypothesis at an FDR below 10%.
342 |
343 | If we control the FDR at 10%, then for how many of the fund managers can we reject $H_{0,j}: \mu_j=0$?
344 |
345 | ```{python}
346 | (fund_qvalues <= 0.1).sum()
347 |
348 | ```
349 | We find that 146 of the 2,000 fund managers have a $q$-value below
350 | 0.1; therefore, we are able to conclude that 146 of the fund managers
351 | beat the market at an FDR of 10%. Only about 15 (10% of 146) of
352 | these fund managers are likely to be false discoveries.
353 |
354 | By contrast, if we had instead used Bonferroni’s method to control the
355 | FWER at level $\alpha=0.1$, then we would have failed to reject any
356 | null hypotheses!
357 |
358 | ```{python}
359 | (fund_pvalues <= 0.1 / 2000).sum()
360 |
361 | ```
362 |
363 |
364 | Figure 13.6 displays the ordered
365 | $p$-values, $p_{(1)} \leq p_{(2)} \leq \cdots \leq p_{(2000)}$, for
366 | the `Fund` dataset, as well as the threshold for rejection by the
367 | Benjamini--Hochberg procedure. Recall that the Benjamini--Hochberg
368 | procedure identifies the largest $p$-value such that $p_{(j)} tibshirani 💻 🖋 |
71 |  trevorhastie 💻 🖋 |
72 |  danielawitten 💻 🖋 |
73 |  Jonathan Taylor 💻 🖋 |
74 |  Thomas Schmelzer 💻 |
75 |