189 |
190 | The title field should be the name of your computer or some other
191 | way to identify this particular ssh key.
192 |
193 | In the key field, you'll need to copy and paste
194 | your *public* key. **Do not paste your private ssh key here.**
195 |
196 | When you hit "Add key", you should see the key name and some
197 | hexadecimal characters show up in the list. You're set.
198 |
199 | Now, whenever you clone a repository using this form:
200 |
201 | `$ git clone git@github.com:rdadolf/ac297r-git-demo.git`,
202 |
203 | you'll be connecting over ssh, and will not be asked for your github password
204 |
205 | You will need to repeat steps 2 and 3 of the setup for each computer you wish to use with github.
206 |
207 | ### 4. Setting global config for git
208 |
209 | Again, from the terminal, issue the command
210 |
211 | `git config --global user.name "YOUR NAME"`
212 |
213 | This sets up a name for you. Then do
214 |
215 | `git config --global user.email "YOUR EMAIL ADDRESS"`
216 |
217 | Use the **SAME** email address you used in setting up your github account.
218 |
219 | These commands set up your global configuration. On my Mac,
220 | these are stored in the text file `.gitconfig` in my home folder.
221 |
222 |
223 |
224 |
225 |
226 |
227 |
--------------------------------------------------------------------------------
/lectures/inference/association-tests.Rmd:
--------------------------------------------------------------------------------
1 | ## Association Tests
2 |
3 | ```{r,include=FALSE}
4 | set.seed(1)
5 | ```
6 |
7 | The statistical tests we have covered up to now leave out a
8 | substantial portion of data types. Specifically, we have not discussed inference for binary, categorical and ordinal data. To give a
9 | very specific example, consider the following case study.
10 |
11 |
12 | A [2014 PNAS paper](http://www.pnas.org/content/112/40/12349.abstract) analyzed success rates from funding agencies in the Netherlands and concluded that their
13 |
14 | > results reveal gender bias favoring male applicants over female applicants in the prioritization of their “quality of researcher” (but not "quality of proposal") evaluations and success rates, as well as in the language use in instructional and evaluation materials.
15 |
16 | The main evidence for this conclusion comes down to a comparison of the percentages. Table S1 in the paper includes the information we need:
17 |
18 | ```{r,echo=FALSE}
19 | data("research_funding_rates")
20 | research_funding_rates
21 | ```
22 |
23 | If we compute the difference in percentage
24 |
25 |
26 | We can compute the totals that were successful and the totals that were not like this:
27 |
28 | ```{r}
29 | totals <- research_funding_rates %>%
30 | select(-discipline) %>%
31 | summarize_all(funs(sum)) %>%
32 | summarize(yes_men = awards_men,
33 | no_men = applications_men - awards_men,
34 | yes_women = awards_women,
35 | no_women = applications_women - awards_women)
36 | ```
37 |
38 | So we see that a larger percent of men received awards than women:
39 |
40 | ```{r}
41 | totals %>% summarize(percent_men = yes_men/(yes_men+no_men),
42 | percent_women = yes_women/(yes_women+no_women))
43 | ```
44 |
45 | But could this be due just to random variability?
46 | Here we learn how to perform inference for this type of data.
47 |
48 |
49 | #### Lady Tasting Tea
50 |
51 |
52 | [R.A. Fisher](https://en.wikipedia.org/wiki/Ronald_Fisher) was one of the first to formalize hypothesis testing. The "Lady Testing Tea" is one of the most famous examples.
53 |
54 | The story goes like this. An acquaintance of Fisher's claimed that she could tell if milk was added
55 | before or after tea was poured. Fisher was skeptical. He designed an experiment to test this claim. He gave her four pairs of
56 | cups of tea: one with milk poured first, the other after. The order
57 | was randomized. The null hypothesis here is that she is guessing. Fisher derived the distribution for the number of correct picks on the assumption that the choice were random and independent.
58 |
59 | As an example, suppose she picked 3 out of 4 correctly, do we believe
60 | she has a special ability? The basic question we ask is: if the tester is actually guessing, what
61 | are the chances that she gets 3 or more correct? Just as we have done
62 | before, we can compute a probability under the null hypothesis that she
63 | is guessing 4 of each. Under this null hypothesis, we can
64 | think of this particular example as picking 4 balls out of an urn
65 | with 4 blue (correct answer) and 4 red (incorrect answer) balls. Remember she knows that there are four before tea and four after.
66 |
67 | Under the null hypothesis that she is simply guessing, each ball
68 | has the same chance of being picked. We can then use combinations to
69 | figure out each probability. The probability of picking 3 is
70 | ${4 \choose 3} {4 \choose 1} / {8 \choose 4} = 16/70$. The probability of
71 | picking all 4 correct is
72 | ${4 \choose 4} {4 \choose 0}/{8 \choose 4}= 1/70$.
73 | Thus, the chance of observing a 3 or something more extreme,
74 | under the null hypothesis, is $\approx 0.24$. This is the p-value. The
75 | procedure that produced this p-value is called _Fisher's exact test_ and
76 | it uses the *hypergeometric distribution*.
77 |
78 | #### Two By Two Tables
79 |
80 | The data from the experiment is usually summarized by a table like this:
81 |
82 | ```{r}
83 | tab <- matrix(c(3,1,1,3),2,2)
84 | rownames(tab)<-c("Poured Before","Poured After")
85 | colnames(tab)<-c("Guessed before","Guessed after")
86 | tab
87 | ```
88 |
89 | These are referred to two-by-two table. They show, for each of the four combinations on can get with a pair of binary variables, the observed counts for each occurrence.
90 |
91 | The function `fisher.test` performs the inference calculations above and can be obtained like this:
92 |
93 | ```{r}
94 | fisher.test(tab, alternative="greater")
95 | ```
96 |
97 | #### Chi-square Test
98 |
99 | Note that,in a way, our funding rates example is similar to the Lady Tasting Tea. However, in the Tasting Tea example the
100 | number of blue and red beads is experimentally fixed and the number
101 | of answers given for each category is also fixed. This is because Fisher made sure there were four before tea and four after tea and the lady knew this, so the answers would also have four and four. If this is the case the sum of the rows and the sum of the columns are
102 | fixed. This defines constraints on the possible ways we can fill the two
103 | by two table and also permits us to use the hypergeometric
104 | distribution. In general, this is not the case. Nonetheless, there is
105 | another approach, the Chi-squared test, which is described below.
106 |
107 |
108 | Imagine we have `r sum(totals)`, some are men and some are women and some get funded other don't. We saw that the success rates for men and woman were:
109 |
110 | ```{r}
111 | totals %>% summarize(percent_men = yes_men/(yes_men+no_men),
112 | percent_women = yes_women/(yes_women+no_women))
113 | ```
114 |
115 | respectively. Would we see this again if we randomly assign funding at the rate:
116 |
117 | ```{r}
118 | funding_rate <- totals %>%
119 | summarize(percent_total =
120 | (yes_men + yes_women)/
121 | (yes_men + no_men +yes_women + no_women)) %>%
122 | .$percent_total
123 | funding_rate
124 | ```
125 |
126 | The Chi-square test answers this question. The first step is to create the two-by-two data:
127 |
128 | ```{r}
129 | two_by_two <- tibble(awarded = c("no", "yes"),
130 | men = c(totals$no_men, totals$yes_men),
131 | women = c(totals$no_women, totals$yes_women))
132 | two_by_two
133 | ```
134 |
135 | The general idea of the Chi-square test is to compare this two-by-two table to what you expect to see which would be:
136 |
137 | ```{r}
138 | tibble(awarded = c("no", "yes"),
139 | men = (totals$no_men + totals$yes_men) *
140 | c(1 - funding_rate, funding_rate),
141 | women = (totals$no_women + totals$yes_women)*
142 | c(1 - funding_rate, funding_rate))
143 |
144 | ```
145 |
146 | We can see that more men than expected and less women than expected received funding. However, under the null hypothesis these observations are random variable. The Chi-square test tells us how likely it is to see
147 | a deviation this large of larger. This test uses an asymptotic result, similar to the CLT, related to the sums of independent binary outcomes.
148 | The R function `chisq.test` takes a two by two table and returns the results from the test:
149 |
150 | ```{r}
151 | two_by_two %>%
152 | select(-awarded) %>%
153 | chisq.test()
154 | ```
155 |
156 | We see that the p-value is 0.051.
157 |
158 |
159 | ### The Odds Ratio
160 |
161 | An informative summary statistic associated with two-by-two tables is the odds ratio. Define the two variables as $X = 1$ if you are a male and 0 otherwise and $Y=1$ if you are funded and 0 otherwise. The odds of getting funded if you are a man is defined
162 |
163 | $$\mbox{Pr}(Y=1 \mid X=1) / \mbox{Pr}(Y=0 \mid X=1)$$
164 |
165 | and can be computed like this:
166 | ```{r}
167 | odds_men <- (two_by_two$men[2] / sum(two_by_two$men)) /
168 | (two_by_two$men[1] / sum(two_by_two$men))
169 | ```
170 |
171 | And the odds of being funded if you are a women is
172 |
173 |
174 | $$\mbox{Pr}(Y=1 \mid X=0) / \mbox{Pr}(Y=0 \mid X=0)$$
175 |
176 |
177 | and can be computed like this:
178 | ```{r}
179 | odds_women <- (two_by_two$women[2] / sum(two_by_two$women)) /
180 | (two_by_two$women[1] / sum(two_by_two$women))
181 | ```
182 |
183 | The odds ratio is the ratio for these two odds: how many times larger are the odds for men than women:
184 |
185 | ```{r}
186 | odds_men / odds_women
187 | ```
188 |
189 | #### Large Samples, Small p-values
190 |
191 | As mentioned earlier, reporting only p-values is not an appropriate
192 | way to report the results of data analysis. In scientific journals, for example
193 | some studies seem to overemphasize p-values. Some of these studies large sample sizes
194 | and report impressively small p-values. Yet when one looks closely at
195 | the results, we realize odds ratios are quite modest: barely bigger
196 | than 1. In this case the difference may not be *practically significant* or *scientifically significant*.
197 |
198 | Note that the relationship between odds ratio and p-value is not a one-to-one. It depends on the sample size. So a very small p-values does not necessarily mean very large odds ratio.
199 |
200 | Notice what happens to the p-value if we multiply our two by two table by 10:
201 |
202 | ```{r}
203 | two_by_two %>%
204 | select(-awarded) %>%
205 | mutate(men = men*10, women = women*10) %>%
206 | chisq.test()
207 | ```
208 |
209 | Yet the odds ratio is unchanged.
210 |
211 | #### Confidence Intervals for the Odds Ratio
212 |
213 | Computing confidence intervals for the odds ratio is not mathematically
214 | straightforward. Unlike other statistics, for which we can derive
215 | useful approximations of their distributions, the odd ratio is not only a
216 | ratio, but a ratio of ratios. Therefore, there is no simple way of
217 | using, for example, the CLT.
218 |
219 | One approach is to use the theory of *generalized linear models*. You can learn more about these in this book:
220 | [McCullagh and Nelder, 1989](https://books.google.com/books?hl=en&lr=&id=h9kFH2_FfBkC)):
221 |
222 |
--------------------------------------------------------------------------------
/lectures/inference/clt.Rmd:
--------------------------------------------------------------------------------
1 | ## Central Limit Theorem in Practice
2 |
3 | ```{r, echo=FALSE, message=FALSE}
4 | library(tidyverse)
5 | library(dslabs)
6 | ds_theme_set()
7 | ```
8 |
9 | The CLT tells us that the distribution function for a sum of draws is approximately normal. We also learned that dividing a normally distributed random variable by a constant results in a normally distributed variable. This implies that the distribution of $\bar{X}$ is approximately normal.
10 |
11 | So in summary we have the $\bar{X}$ has an approximately normal distribution with expected value $p$ and standard error $\sqrt{p(1-p)/N}$.
12 |
13 | Now how does this help us? Suppose we want to know what is the probability that we are withing 1% from $p$. We are basically asking if
14 |
15 | $$
16 | \mbox{Pr}(| \bar{X} - p| \leq .01)
17 | $$
18 | which is the same as:
19 |
20 | $$
21 | \mbox{Pr}(\bar{X}\leq p + .01) - \mbox{Pr}(\bar{X} \leq p - .01)
22 | $$
23 |
24 | Can we answer this question? Note that we can use the mathematical trick we learned in the previous chapter. Subtract the expected value and divide by the standard error to get a standard normal random variable, call it $Z$, on the left. Since $p$ is the expected value and $\mbox{SE}(\bar{X}) = \sqrt{p(1-p)/N}$ is the standard error we get:
25 |
26 | $$
27 | \mbox{Pr}\left(Z \leq \,.01 / \mbox{SE}(\bar{X}) \right) -
28 | \mbox{Pr}\left(Z \leq - \,.01 / \mbox{SE}(\bar{X}) \right)
29 | $$
30 |
31 | A problem is that we don't know $p$, so we don't know $\mbox{SE}(\bar{X})$. But it turns out that the CLT still works if we use estimate the standard error by using $\bar{X}$ in place of $p$. We say that we _plug-in_ the estimate. Our estimate of the standard error is therefore:
32 |
33 | $$
34 | \hat{\mbox{SE}}(\bar{X})=\sqrt{\bar{X}(1-\bar{X})/N}
35 | $$
36 | In statistics textbooks, we use a little hat to denote estimates. Note that the estimate can be constructed using the observed data and $N$.
37 |
38 | Now we continue with our calculation but dividing by $\hat{\mbox{SE}}(\bar{X})=\sqrt{\bar{X}(1-\bar{X})/N})$ instead. In our first sample we had 12 blue and 13 red so $\bar{X} = 0.48$ and so our estimate of standard error is
39 |
40 | ```{r}
41 | X_hat <- 0.48
42 | se <- sqrt(X_hat*(1-X_hat)/25)
43 | se
44 | ```
45 |
46 | And now we can answer the question of the probability of being close to $p$. The answer is
47 |
48 | ```{r}
49 | pnorm(0.01/se) - pnorm(-0.01/se)
50 | ```
51 |
52 | So there is a small chance that we will be close. A poll of only $N=25$ people is not really very useful. At least for a close elections.
53 |
54 | Earlier we mentioned the _margin of error_. Now we can define it because it is simply two times the standard error which we can now estimate and in our case it is:
55 |
56 | ```{r}
57 | 2*se
58 | ```
59 |
60 | Why do we multiply by 2? Because if you ask what is the probability that we are withing two standard error from $p$ we get:
61 |
62 | $$
63 | \mbox{Pr}\left(Z \leq \, 2\mbox{SE}(\bar{X}) / \mbox{SE}(\bar{X}) \right) -
64 | \mbox{Pr}\left(Z \leq - 2 \mbox{SE}(\bar{X}) / \mbox{SE}(\bar{X}) \right)
65 | $$
66 | which is
67 |
68 | $$
69 | \mbox{Pr}\left(Z \leq 2 \right) -
70 | \mbox{Pr}\left(Z \leq - 2\right)
71 | $$
72 |
73 | which we know is about 95\%:
74 |
75 | ```{r}
76 | pnorm(2)-pnorm(-2)
77 | ```
78 |
79 | So there is a 95% that $\bar{X}$ will be within $2\times \hat{SE}(\bar{X})$, in our case `r round(2*se)`, to $p$. Note that 95% is somewhat of an arbitrary choice and sometimes other percentages are used, but it is the most commonly used value to define _margin of error_.
80 |
81 | In summary, the CLT tells us that our poll based on a sample size of $25$ is not a very useful. We don't really learn much when the margin of error is this large. All we can really say is that the popular vote will not be won by a large margin. This is why pollsters tend to use larger sample sizes.
82 |
83 | From the table above we see that typical sample sizes from 700 to 3500. To see how this gives us a much more practical result, note that if we had obtained a $\bar{X}$=0.48 with a sample size of 2,000 our standard error $\hat{\mbox{SE}}(\bar{X})$ would have been `r n<-2000;se<-sqrt(0.48*(1-0.48)/n);se`. So our result is an estimate of `48`% with a margin of error is `r round(2*se*100)`%. In this case, the result is much more informative and would make us think that there are more red balls than blue. But keep in mind, we this is hypothetical. We did not take a poll of 2,000 since we don't want ruin the competition.
84 |
85 |
86 | ### A Monte Carlo simulation
87 |
88 |
89 | Suppose we want to use a Monte Carlo simulation to corroborate the tools we have built using probability theory. To create a the simulation we would write code like this:
90 |
91 | ```{r, eval=FALSE}
92 | B <- 10000
93 | N <- 1000
94 | Xhat <- replicate(B, {
95 | X <- sample(c(0,1), size=N, replace=TRUE, prob=c(1-p, p))
96 | mean(X)
97 | })
98 | ```
99 |
100 | The problem is, of course, we don't know `p`. We could
101 | construct an urn like the one pictured above and run an analog (without a computer) simulation. It would take a long time, but you could take 10,000 samples, count the beads and keep track of the
102 | proportions of blue. We can use the function `take_poll(n=1000)` instead of drawing from an actual urn, but it would still take time to count the beads and enter the results.
103 |
104 | So, one thing we do to corroborate theoretical results is to pick one, or several values of `p`, and run the simulations. Let's set `p=0.45`. We can then simulate a poll
105 |
106 | ```{r}
107 | p <- 0.45
108 | N <- 1000
109 |
110 | X <- sample(c(0,1), size=N, replace=TRUE, prob=c(1-p, p))
111 | Xhat <- mean(X)
112 | ```
113 |
114 | In this particular sample our estimate is `Xhat`. We can use that code to a Monte Carlo simulation:
115 |
116 | ```{r}
117 | B <- 10000
118 | Xhat <- replicate(B, {
119 | X <- sample(c(0,1), size=N, replace=TRUE, prob=c(1-p, p))
120 | mean(X)
121 | })
122 | ```
123 |
124 | To review, the theory tells us that $\bar{X}$ has is approximately normally distributed, has expected value $p=$`r p` and standard error $\sqrt{p(1-p)/N}$ = `r sqrt(p*(1-p)/N)`. The simulation confirms this
125 |
126 | ```{r}
127 | mean(Xhat)
128 | sd(Xhat)
129 | ```
130 |
131 | A histogram and qq-plot confirm that the the normal approximation is accurate as well:
132 |
133 | ```{r,echo=FALSE, warning=FALSE, message=FALSE}
134 | library(gridExtra)
135 | p1 <- data.frame(Xhat=Xhat) %>%
136 | ggplot(aes(Xhat)) +
137 | geom_histogram(binwidth = 0.005, color="black")
138 | p2 <- data.frame(Xhat=Xhat) %>%
139 | ggplot(aes(sample=Xhat)) +
140 | stat_qq(dparams = list(mean=mean(Xhat), sd=sd(Xhat))) + geom_abline() + ylab("Xhat") + xlab("Theoretical normal")
141 | grid.arrange(p1,p2, nrow=1)
142 | ```
143 |
144 | Again, note that in a real life we would never be able to run such an experiment because we don't know $p$. But we could run it for various values of $p$ and $N$ and see that the theory does indeed work well for most values. You can easily do this by re-running the code above after changing `p` and `N`.
145 |
146 | ### The spread
147 |
148 | The competition is to predict the spread not the proportion $p$. However, because we are assuming there are only two parties, we know that the spread if $p - (1-p) = 2p - 1$. So everything we have done can easily be adapted to an estimate of $2p - 1$. Once we have our estimate $\bar{X}$ and $\hat{\mbox{SE}}(\bar{X})$ we estimate the spread with $2\bar{X} - 1$ and, since we are multiplying by 2, the standard error is $2\hat{\mbox{SE}}(\bar{X})$. Note that subtracting 1 does not add any variability so it does not affect the standard error.
149 |
150 | So for our 25 sample above, our estimate $p$ is `.48` with margin of error `.20` so our estimate of the spread is `0.04` with margin of error `.40`. Again, not a very useful sample size. But the point is that once we have an estimate and standard error for $p$, we have it for the spread $2p-1$.
151 |
152 |
153 | ### Bias: Why not run a very large poll?
154 |
155 | Note that for realistic value of $p$, say from 0.35 to 0.65, if we run a very large poll with 100,000 people, theory would tell us that we would predict the election perfectly since the largest possible margin of error is around 0.3\%. Here are the calculations:
156 |
157 | ```{r}
158 | N <- 100000
159 | p <- seq(0.35, 0.65, length = 100)
160 | SE <- sapply(p, function(x) 2*sqrt(x*(1-x)/N))
161 | data.frame(p=p, SE = SE) %>% ggplot(aes(p, SE)) +
162 | geom_line()
163 | ```
164 |
165 | One reason is that running such a poll is very expensive. But perhaps a more important reason is that theory has it's limitations. Polling is much more complicated than picking beads from an urn. People might lie to you and other might not have phones. But perhaps the most different way an actual poll is from an urn model is that we actually don't know for sure who is in our population and who is not. How do we know who is going to vote? Are we reaching all possible voters? So even if our margin of error is very small it might not be exactly right that our expected value is $p$. We call this bias. Historically, we observe that polls are indeed biased although not by that much. The typical bias appears to be about 1-2%. This makes election forecasting a bit more interesting and we will talk about how to model this in a later chapter.
166 |
167 |
168 |
169 |
170 |
171 |
172 |
173 |
--------------------------------------------------------------------------------
/lectures/inference/confidence-intervals-p-values-assessment.Rmd:
--------------------------------------------------------------------------------
1 | ### Confidence Intervals and p-values
2 |
3 | For these exercises we will use actual polls for the 2016 election. You can load the data from the `dslabs` package
4 |
5 | ```{r}
6 | library(dslabs)
7 | data("polls_us_election_2016")
8 | ```
9 |
10 | Specifically, we will use all the national polls that ended within two weeks before the election.
11 |
12 | ```{r, message=FALSE, comment=FALSE}
13 | library(tidyverse)
14 | polls <- polls_us_election_2016 %>% filter(enddate >= "2016-10-31" & state == "U.S.")
15 | ```
16 |
17 | 1. For the first poll you can obtain the samples size and estimated Clinton percentage with
18 |
19 | ```{r}
20 | N <- polls$samplesize[1]
21 | X_hat <- polls$rawpoll_clinton[1]/100
22 | ```
23 |
24 | Assume there are only two candidates and construct a 95% confidence interval for the election night proportion $p$.
25 |
26 |
27 | 2. Now use `dplyr` to add a confidence intravel as two columns, called them `lower` and `upper`, of the object `poll` then show a Pollster. Show the enddate, pollster, the estimated proportion, and confidence interval columns. Hint: define temporaty columns `X_hat` and `se_hat`.
28 |
29 |
30 | 3. The final tally for the popular vote was Clinton 48.2% and Trump 46.1%. Add column, call it `hit`, to the previous table stating if the confidence interval included the true proporiton $p=0.482$ or not.
31 |
32 |
33 |
34 | 4. For the table you just created, what proportion of confidence intervals included $p$?
35 |
36 |
37 |
38 | 5. If these confidence intervals are constructed correctly, and the theory holds up, what proportion should include $p$?
39 |
40 |
41 | 6. Note a much smaller proportion of the polls than expected produce confidence intervals containing $p$. If you look closely at the table, you will see that most polls that fail to include $p$ are underestimating. The reason for this is undecided voters, individuals polled that do not yet know who they will vote for or do not want to say. Because, historically, undecideds divide evenly between the two main candidates on election day, it is more informative to estimate the spread or the difference beteween the proportion of two candidates $d$, which in this election was $0.482 - 0.461 = 0.021$. Assume that there are only two parties and that $d = 2p - 1$, define
42 |
43 | ```{r, message=FALSE, comment=FALSE}
44 | polls <- polls_us_election_2016 %>% filter(enddate >= "2016-10-31" & state == "U.S.") %>%
45 | mutate(d_hat = rawpoll_clinton/100 - rawpoll_trump/100)
46 | ```
47 |
48 | and re-do exercise 1 but for the difference.
49 |
50 |
51 | 7. Now repeat exercise 3, but for the difference.
52 |
53 |
54 | 8. Now repeat exercise 4, but for the difference
55 |
56 |
57 | 9. Note that althought the proportion of confidence intervals goes up substantially, it is still lower that 0.95. In the next lecture we learn why this is. To motivate this make a plot of the error, the difference between each poll's estimate and the actual $d=0.021$. Stratify by pollster. Hint: use `theme(axis.text.x = element_text(angle = 90, hjust = 1))`
58 |
59 |
60 | 10. Re-do the plot you made for 9 but only for pollsters that took five or more polls.
61 |
62 |
--------------------------------------------------------------------------------
/lectures/inference/confidence-intervals-p-values.Rmd:
--------------------------------------------------------------------------------
1 | ## Confidence Intervals
2 |
3 | ```{r, echo=FALSE, message=FALSE}
4 | library(tidyverse)
5 | library(dslabs)
6 | ds_theme_set()
7 | ```
8 |
9 | Confidence intervals are a very useful concept that is widely used by data scientists. A version of these that very commonly seen come from the `ggplot` geometry `geom_smooth`. Here is an example using a temperature dataset available in R:
10 |
11 | ```{r, message=FALSE}
12 | data("nhtemp")
13 | data.frame(year = as.numeric(time(nhtemp)), temperature=as.numeric(nhtemp)) %>%
14 | ggplot(aes(year, temperature)) +
15 | geom_point() +
16 | geom_smooth() +
17 | ggtitle("Average Yearly Temperatures in New Haven")
18 | ```
19 |
20 |
21 | We will later learn how the curve is formed, but note the shaded area around the curve. The shaded area around the curve is created using the concept of confidence intervals.
22 |
23 | In our competition we were asked to give an interval. If the interval you submit includes the $p$ you get half the money you spent on your "poll" back and pass to the next stage of the competition. One way to pass to the second round is to report a very large interval. For example, the interval $[0,1]$ is guaranteed to include $p$. However, with an interval this big, we have no chance of winning the competition. Similarly, if you are an election forecaster and predict the spread will be between -100% and 100% you will be ridiculed for stating the obvious. Even a smaller interval such as saying the spread will be between -10 and 10% will not be considered serious.
24 |
25 | On the other-hand the smaller the interval we report, the smaller our chances of winning the prize. Similarly, a bold pollster that reports very small intervals and misses the mark most of the time will not be considered a good pollster. We want to be somewhere in between.
26 |
27 | We can use the statistical theory we have learned to compute the probability of any given interval including $p$. Similarly,if we are asked to create an interval with, say, a 95\% chance of including $p$, we can do that as well. These are called 95\% confidence intervals.
28 |
29 | Note, that when pollster report an estimate and a margin of error, they are, in a way, reporting a 95\% confidence interval. Let's show how this works mathematically.
30 |
31 | We want to know that probability that the interval $[\bar{X} - 2\hat{\mbox{SE}}(\bar{X}), \bar{X} + 2\hat{\mbox{SE}}(\bar{X})]$ contains the true proportion $p$. First, note that the start and end of this intervals are random variables: every time they we take a sample they change. To illustrate this run Monte Carlo simulation above twice. We use the same parameters as above
32 |
33 | ```{r}
34 | p <- 0.45
35 | N <- 1000
36 | ```
37 |
38 | And note that the interval here
39 |
40 | ```{r}
41 | X <- sample(c(0,1), size=N, replace=TRUE, prob=c(1-p, p))
42 | X_hat <- mean(X)
43 | SE_hat <- sqrt(X_hat*(1-X_hat)/N)
44 | c(X_hat - 2*SE_hat, X_hat + 2*SE_hat)
45 | ```
46 |
47 | is different from this one:
48 |
49 | ```{r}
50 | X <- sample(c(0,1), size=N, replace=TRUE, prob=c(1-p, p))
51 | X_hat <- mean(X)
52 | SE_hat <- sqrt(X_hat*(1-X_hat)/N)
53 | c(X_hat - 2*SE_hat, X_hat + 2*SE_hat)
54 | ```
55 |
56 | Keep sampling and creating intervals and you will see the random variation.
57 |
58 | To determine the probability that the interval includes $p$ we need to compute this:
59 | $$
60 | \mbox{Pr}\left(\bar{X} - 2\hat{\mbox{SE}}(\bar{X}) \leq p \leq \bar{X} + 2\hat{\mbox{SE}}(\bar{X})\right)
61 | $$
62 |
63 | By subtracting and dividing the same quantities in all parts of the equation we
64 | get that the above is equivalent to:
65 |
66 | $$
67 | \mbox{Pr}\left(-2 \leq \frac{\bar{X}- p}{\hat{\mbox{SE}}(\bar{X})} \leq 2\right)
68 | $$
69 |
70 |
71 | The term in the middle is an approximately normal random variable with expected value 0 and standard error 1, which we have been denoting with $Z$, so we have
72 |
73 | $$
74 | \mbox{Pr}\left(-2 \leq Z \leq 2\right)
75 | $$
76 |
77 | which we can quickly compute using
78 |
79 | ```{r}
80 | pnorm(2) - pnorm(-2)
81 | ```
82 |
83 | proving the we have a 95\% probability.
84 |
85 | Note that if we want to have a larger probability, say 99\%, we need to multiply by whatever `z` satisfies the following:
86 |
87 |
88 | $$
89 | \mbox{Pr}\left(-z \leq Z \leq z\right) = 0.99
90 | $$
91 |
92 | Note that by using
93 |
94 | ```{r}
95 | z <- qnorm(0.995)
96 | z
97 | ```
98 |
99 | will do it because by definition `pnorm(qnorm(0.995)` is 0.995 and by symmetry `pnorm(1-qnorm(0.995))` is 1 - 0.995, we have that
100 |
101 |
102 | ```{r}
103 | pnorm(z)-pnorm(-z)
104 | ```
105 |
106 | is `0.995 - 0.005 = 0.99`. We can use this approach for any percentile $q$: we use $1 - (1 - q)/2$. Why this number? Because $1 - (1 - q)/2 + (1 - q)/2 = q$.
107 |
108 | Note that to get exactly 0.95 confidence interval, we actually use a slightly smaller number than 2:
109 |
110 | ```{r}
111 | qnorm(0.975)
112 | ```
113 |
114 | ### A Monte Carlo Simulation
115 |
116 | We can run a Monte Carlo simulation to confirm that in fact that a 95\% confidence interval includes $p$ 95\% of the time.
117 |
118 | ```{r, eval=FALSE}
119 | set.seed(1)
120 | ```
121 |
122 |
123 | ```{r}
124 | B <- 10000
125 | inside <- replicate(B, {
126 | X <- sample(c(0,1), size=N, replace=TRUE, prob=c(1-p, p))
127 | X_hat <- mean(X)
128 | SE_hat <- sqrt(X_hat*(1-X_hat)/N)
129 | between(p, X_hat - 2*SE_hat, X_hat + 2*SE_hat)
130 | })
131 | mean(inside)
132 | ```
133 |
134 |
135 | The following plot shows the first 100 confidence intervals. In this case we created the simulation so The black line
136 |
137 | ```{r, message=FALSE, echo=FALSE}
138 | set.seed(1)
139 | tab <- replicate(100, {
140 | X <- sample(c(0,1), size=N, replace=TRUE, prob=c(1-p, p))
141 | X_hat <- mean(X)
142 | SE_hat <- sqrt(X_hat*(1-X_hat)/N)
143 | hit <- between(p, X_hat - 2.58*SE_hat, X_hat + 2.58*SE_hat)
144 | c(X_hat, X_hat - 2.58*SE_hat, X_hat + 2.58*SE_hat, hit)
145 | })
146 |
147 | tab <- data.frame(poll=1:ncol(tab), t(tab))
148 | names(tab)<-c("poll", "estimate", "low", "high", "hit")
149 | tab <- mutate(tab, p_inside = ifelse(hit, "Yes", "No") )
150 | ggplot(tab, aes(poll, estimate, ymin=low, ymax=high, col = p_inside)) +
151 | geom_point()+
152 | geom_errorbar() +
153 | coord_flip() +
154 | geom_hline(yintercept = p)
155 | ```
156 |
157 |
158 | ### The Correct Language
159 |
160 | When using the theory we described above it is important to remember that it is the intervals that are random not $p$. In the plot above we can see the random intervals moving around and $p$, represented with the vertical line, staying in the same place. The proportion of blue in the urn $p$ is not. So the 95\% relates to the probability that this random interval falls on top of $p$. Saying the $p$ has a 95\% of being between this and that is technically an incorrect statement. Again, because $p$ is not random.
161 |
162 |
163 | ### Power
164 |
165 | Pollsters are not successful for providing correct confidence intervals but rather for predicting who will win. When we took a 25 bead sample size, the confidence interval for the spread:
166 |
167 | ```{r}
168 | N <- 25
169 | X_hat <- 0.48
170 | (2*X_hat - 1) + c(-2,2)*2*sqrt(X_hat*(1-X_hat)/sqrt(N))
171 | ```
172 |
173 | includes 0. If this were a poll and we were forced to make a declaration, we would have to say it was a "toss-up".
174 |
175 | A problem with our poll results is that, given the sample size and the value of $p$, we would have to sacrifice on the probability of an incorrect call to create an interval that does not include 0.
176 |
177 | This does not mean that the election is close. It only means that we have a small sample size. In statistical textbooks this is called lack of _power_. In the context of polls, _power_ is the probability of detecting spreads different from 0.
178 |
179 | By increasing our sample size, we lower our standard error and therefore have a much better chance of detecting the direction of the spread.
180 |
181 |
182 | ## p-values
183 |
184 | p-values are ubiquitous in the scientific literature. They are related to confidence interval so we introduce the concept here.
185 |
186 | Let's consider the blue and red beads. Suppose that rather than wanting an estimate of the spread or the proportion of blue, I am interested only in the question: are there more blue beads or red beads? I want to know if the spread $2p-1 > 0$.
187 |
188 | Suppose we take a random sample of $N=100$ and we observe $52$ blue beads which gives us $2\bar{X}-1=0.04$. This seems to be pointing to their being more blue than red since 0.04 is larger than 0. However, as data scientists we need to be skeptical. We know there is chance involved in this process and we could get a 52 even when the actual spread is 0. We call this a _null hypothesis_. The null hypothesis is the skeptics hypothesis: the spread is $2p-1=0$. We have observed a random variable $2*\bar{X}-1 = 0.52$ and the p-value is the answer to the question how likely is it to see a value this large, when the null hypothesis is true. So we write
189 |
190 | $$\mbox{Pr}(\mid \bar{X} - 0.5 \mid > 0.02 ) $$
191 |
192 | assuming the $2p-1=0$ or $p=0.5$. Under the null hypothesis we know that
193 |
194 | $$
195 | \sqrt{N}\frac{\bar{X} - 0.5}{\sqrt{0.5(1-0.5)}}
196 | $$
197 |
198 | is standard normal. So we can compute the probability above, which is the p-value.
199 |
200 | $$\mbox{Pr}\left(\sqrt{N}\frac{\mid \bar{X} - 0.5\mid}{\sqrt{0.5(1-0.5)}} > \sqrt{N} \frac{0.02}{ \sqrt{0.5(1-0.5)}}\right)$$
201 |
202 |
203 | ```{r}
204 | N=100
205 | z <- sqrt(N)*0.02/0.5
206 | 1 - (pnorm(z) - pnorm(-z))
207 | ```
208 |
209 | This is the p-value. In this case there is actually a large chance of seeing 52 or larger under the null hypothesis.
210 |
211 | Note that there is a close connection between p-values and confidence intervals. If a 95% confidence interval of the spread does not include 0, we know that the p-value must be smaller than 0.05.
212 |
213 | To learn more about p-values you can consult any statistcs textbook. However, in general we prefer reporting confidence intervals over p-values since it gives us an idea of the size of the estimate. The p-value simply reports a probability and says nothing about the significance of the finding in the context of the problem.
214 |
215 |
216 |
217 |
--------------------------------------------------------------------------------
/lectures/inference/img/pollster-2016-predictions.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/datasciencelabs/2017/20caca62a7ac20f1cdee9012b64ff19680c37137/lectures/inference/img/pollster-2016-predictions.png
--------------------------------------------------------------------------------
/lectures/inference/img/popular-vote-538.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/datasciencelabs/2017/20caca62a7ac20f1cdee9012b64ff19680c37137/lectures/inference/img/popular-vote-538.png
--------------------------------------------------------------------------------
/lectures/inference/img/rcp-polls.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/datasciencelabs/2017/20caca62a7ac20f1cdee9012b64ff19680c37137/lectures/inference/img/rcp-polls.png
--------------------------------------------------------------------------------
/lectures/inference/img/urn.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/datasciencelabs/2017/20caca62a7ac20f1cdee9012b64ff19680c37137/lectures/inference/img/urn.jpg
--------------------------------------------------------------------------------
/lectures/inference/intro-to-inference.Rmd:
--------------------------------------------------------------------------------
1 | # Inference and Modeling
2 |
3 | The day before the 2008 presidential election Nate Silver's Fivethirtyeight stated that "Barack Obama appears poised for a decisive electoral victroy". They went further and predicted that Obama would win the election 349 electoral votes to 189 and the popular vote by a margin of 6.1%. Fivethirtyeight also attached a probabilistic statement to their prediction claiming that Obama had a 91% chance of winning the election. The predictions were quite accurate since in the final tally actual results were Obama won the electoral college 365 to 173 and the popular vote by 7.2% difference. Their performance in the 2008 election brought Fivethirtyeight to the attention of political pundints and TV personalities. The week before the 2012, Fivethirtyeight Nate Silver was giving a Obama a 90% chance of winning despite many of the experts thinking they were close, political commentator Joe Scarborough said [during his show](https://www.youtube.com/watch?v=TbKkjm-gheY)
4 |
5 | >>Anybody that thinks that this race is anything but a tossup right now is such an ideologue ... they're jokes.".
6 |
7 | To which Nate Silver responded via Twitter:
8 |
9 | >> If you think it's a toss-up, let's bet. If Obama wins, you donate $1,000 to the American Red Cross. If Romney wins, I do. Deal?
10 |
11 | How was Mr. Silver so confident. We will demonstrate how _poll aggregators_, such as Fivethirtyeight, collected and combined data reported by different experts to produce imrpoved predictions. The two main statistical tools used by the aggregators are the topic of this chapter: inference and modeling. To begin to understand how election forecasting works we need to understand the basic data point they use: poll results.
12 |
13 |
14 | ## Polls
15 |
16 | Opinion polling have been conducted since the 19-th century. The general goal of these is to describe the opinions held by a specific population on a given set of topics. In recent times, these polls have been pervasive during presidential elections. Polls are useful when asking everybody in the population is logistically impossible. The general strategy is to ask a smaller group, chosen at random, and then infer the opinions of the entire population from the opinions of the smaller group. Statistical theory is used to justify the process. This theory is referred to as _inference_ and is the main topic of this section.
17 |
18 | Perhaps the best know opinion polls are those conducted to determining which candidate is preferred by voters in a given election. Political strategists make extensive use of polls to determine, for example, how to invest resources. For example, they may want to know which geographical locations to focus get out the vote efforts.
19 |
20 | Elections are a particularly interesting case of opinion polls because the actual opinion of the entire population is revealed on election day. Of course, it costs millions of dollars to run an actual election which makes polling a cost effective strategy for those that want to forecast the results.
21 |
22 | Although typically the results of these polls are kept private, similar polls are conducted by news organizations because results tend to be of interest to the general public and are often made public. We will eventually be looking at such data.
23 |
24 | [Real Clear Politics](http://www.realclearpolitics.com) is an example of a news aggregation that organizers and publishes poll results. For example, [here](http://www.realclearpolitics.com/epolls/2016/president/us/general_election_trump_vs_clinton-5491.html) are examples of polls reporting estimate of the popular vote for the 2016 presidential election:
25 |
26 | ```{r, echo=FALSE}
27 | knitr::include_graphics("img/rcp-polls.png")
28 | ```
29 |
30 | Although, in the United States, the popular vote does not determine the result of the election, we will use it here as an illustrative and simple example of how well polls work. Forecasting the election is a more complex process since it involves combining results from 50 states and DC.
31 |
32 | Let's make some observation about the table above. First note that different polls, all taken days before the election, report a different _spread_: the estimated difference between support for the two candidates. Note also that the reported spreads hover around what ended up being the actual result: Clinton won the popular vote by 2.1%.We also see a column titled **MoE** which stands for _margin of error_.
33 |
34 | In this section we show how the probability concepts we learned in the previous chapter can be applied to develop the statistical approaches that make polls an effective tool. We will learn the statistical concepts necessary to define _estimates_ and _margins of errors_, and show how we can use these to forecast final results relatively well and also provide an estimate of the precision of our forecast. Once we learn this we will be able to understand two concepts that are
35 | ubiquitous in data science: _confidence intervals_, and _p-values_. Finally, to understand probabilistic statetmets about the probability of a candidate winning we will have to learn about Bayesian modelling. In the final sections we put it all together to recreate the simplified version of the Fivethirtyeight model and apply it to 2016 election.
36 |
37 | We start by connecting probability theory to the task of using polls to learn about a population.
38 |
39 |
--------------------------------------------------------------------------------
/lectures/inference/models-assessment.Rmd:
--------------------------------------------------------------------------------
1 | ## Models
2 |
3 | 1. We have been using urn models to motivate the use of probability models. Most data science applications are not related to data obtained from urns. More common are data that come from individuals. The reason probability plays a role here is because the data come from a random sample. The random sample is taken from a population. The urn serves as an analogy for the population.
4 |
5 | Let's revisit the heights dataset. Suppose we consider the males in our course the population.
6 |
7 | ```{r}
8 | library(dplyr)
9 | library(dslabs)
10 | data(heights)
11 | x <- heights %>% filter(sex == "Male") %>%
12 | .$height
13 | ```
14 |
15 | Mathematically speaking, `x` is our population. Using the urn analogy, we have an urn with the values of `x` in it. What are the population average and standard deviation of our population?
16 |
17 |
18 | 2. Call the population average computed above $\mu$ and the standard deviation $\sigma$. Now take a sample of size 50, with replacement, and construct an estimate for $\mu$ and $\sigma$. Set the seed at 1 based on what has been described in this section.
19 |
20 |
21 | 3. What does the theory tell us about the sample average $\bar{X}$ and how it related to $\mu$?
22 |
23 | A. It is practically identical to $\mu$.
24 | B. It is a random variable with expected value $\mu$ and standard error $\sigma/\sqrt{N}$
25 | C. It is a random variable with expected value $\mu$ and standard error $\sigma$.
26 | D. Contains no information
27 |
28 |
29 | 4. So how is this useful? We are going to use an over-simplified yet illustrative example. Suppose we want to know the average height of our male students but we only get to measure 50 of the 708. We will use $\bar{X}$ as our estimate. We know from the answer to 3 that the standard estimate of our error $\bar{X}-\mu$ is $\sigma/sqrt{N}$. If want to know what this is, but we don't know $\sigma$. Based on what is described in thie section, show your estimate of $\sigma$
30 |
31 |
32 | 5. Now that we have an estimate of $\sigma$, let's call our estimate $s$. Construct a 95% confidence interval for $\mu$.
33 |
34 |
35 |
36 | 6. Now run a Monte Carlo simulation in which you
37 | compute 10,000 confidence intervals as you have just done. What proportion of these intervals include $\mu$? Set the seed to 1.
38 |
39 |
40 | 7. In this section we talked about pollster bias. We used visualization to motivate the presence of such bias. Here we will give it a more rigorous treatment. We will consider two pollsters that conducted daily polls. We will look at national polls for the month before the election.
41 |
42 | ```{r}
43 | data(polls_us_election_2016)
44 | polls <- polls_us_election_2016 %>%
45 | filter(pollster %in% c("Rasmussen Reports/Pulse Opinion Research","The Times-Picayune/Lucid") &
46 | enddate >= "2016-10-15" &
47 | state == "U.S.") %>%
48 | mutate(spread = rawpoll_clinton/100 - rawpoll_trump/100)
49 | ```
50 |
51 | We want to answer the quetion "Is there a poll bias?". Make a plot showing the spreads for each poll.
52 |
53 |
54 |
55 | 8. The data does seem to suggest there is a difference. But these
56 | data are subject to variability. Maybe the differences we observe are due to chance.
57 |
58 | The urn model theory tells says nothing about pollster effect. Under the urn model both pollsters have the same expected value: the election day difference, that we call $d$.
59 |
60 | To answer this question, is there an urn model, we will model the observed data $Y_ij$ in the following way
61 |
62 | $$
63 | Y_{ij} = d + b_i + \varepsilon_{ij}
64 | $$
65 | with $i=1,2$ indexing the two pollsters, $b_i$ the bias for pollster $i$ and $\varepsinon_ij$ poll to poll chance variability. We assume the $\varepsilon$ are independent from each other, have expected value $0$ and standard deviation $\sigma_i$ regardless of $j$.
66 |
67 | Which of the following best represents our question?
68 |
69 | A. Is $\varepsilon_ij$ = 0
70 | B. How close are the $Y_ij$ to $d$
71 | C. Is $b_1 \neq b_2$?
72 | D. Are $b_1 = 0$ and $b_2 = 0$ ?
73 |
74 | 9. Note thet in the right side of this model only $\varepsilon_ij$ is a random variable. The other two are constants. What is the expected value of $Y_{1j}$?
75 |
76 |
77 | 10. Suppose we define $\bar{Y}_1$ as the average of poll results from the first poll, $Y_{11},\dots,Y_{1N_1}$ with $N_1$ the number of polls conducted by the first pollster:
78 |
79 | ```{r}
80 | polls %>%
81 | filter(pollster=="Rasmussen Reports/Pulse Opinion Research") %>%
82 | summarize(N_1 = n())
83 | ```
84 |
85 | What is the expected values $\bar{Y}_1$?
86 |
87 |
88 | 11. What is the standard error of $\bar_{Y}_1$ ?
89 |
90 |
91 | 12. What is the expected values $\bar{Y}_2$?
92 |
93 |
94 | 13. What is the standard error of $\bar_{Y}_2$ ?
95 |
96 |
97 | 14. Using what we learned in answering the questions above what is the expected value of $\bar{Y}_{2} - \bar{Y}_1$?
98 |
99 |
100 | 15. Using what we learned in answering the questions above what is the standard error of $\bar{Y}_{2} - \bar{Y}_1$?
101 |
102 |
--------------------------------------------------------------------------------
/lectures/inference/t-distribution.Rmd:
--------------------------------------------------------------------------------
1 | ## The t-distribution
2 |
3 | ```{r, echo=FALSE}
4 | library(tidyverse)
5 | library(dslabs)
6 | ds_theme_set()
7 | polls <- polls_us_election_2016 %>%
8 | filter(state == "U.S." & enddate >= "2016-10-31" &
9 | (grade %in% c("A+","A","A-","B+") | is.na(grade))) %>%
10 | mutate(spread = rawpoll_clinton/100 - rawpoll_trump/100)
11 |
12 | one_poll_per_pollster <- polls %>% group_by(pollster) %>%
13 | filter(enddate == max(enddate)) %>%
14 | ungroup()
15 | ```
16 |
17 | Above we made use of the CLT with a sample size of 15. Because we are estimating a second parameters $\sigma$, further variability is introduced into our confidence interval which result in intervals that are too small. For very large sample sizes this extra variability is negligible, but, in general, for values smaller than 30 we need to be cautions about using CLT.
18 |
19 | However, if the data in the urn is known to follow a normal distribution, then we actually have mathematical theory that tell us how much bigger we need to make the intervals to account for the estimation of $\sigma$. Using this theory, we can construct confidence intervals for any $N$. But again only if **the data in the urn is known to follow a normal distribution**. So for the 0,1 data of our previous urn model, this theory definitely does not apply.
20 |
21 | The statistic on which confidence intervals for $d$ are based is
22 |
23 | $$
24 | Z = \frac{\bar{X} - d}{\sigma/\sqrt{N}}
25 | $$
26 |
27 | CLT tells us that Z is approximately normally distributed with expected value 0 and standard error 1. But in practice we don't know $\sigma$ so we use:
28 |
29 | $$
30 | Z = \frac{\bar{X} - d}{s\sqrt{N}}
31 | $$
32 |
33 |
34 | By substituting $\sigma$ with $s$ we introduce some variability. The theory tells us that $Z$ follows a t-distribution with $N-1$ _degrees of freedom_. The degrees of freedom is a parameter that controls the variability via fatter tails:
35 |
36 | ```{r, echo=FALSE}
37 | x <- seq(-5,5, len=100)
38 | data.frame(x=x, Normal = dnorm(x, 0, 1), t_03 = dt(x,3), t_05 = dt(x,5), t_15=dt(x,15)) %>% gather(distribution, f, -x) %>% ggplot(aes(x,f, color = distribution)) + geom_line() +ylab("f(x)")
39 | ```
40 |
41 | If we are willing to assume the pollster effect data is normally distributed, based on the sample data $X_1, \dots, X_N$,
42 | ```{r}
43 | one_poll_per_pollster %>%
44 | ggplot(aes(sample=spread)) + stat_qq()
45 | ```
46 | then $Z$ follows a t-distribution with $N-1$ degrees of freedom. So perhaps a better confidence interval for $d$ is:
47 |
48 |
49 | ```{r}
50 | z <- qt(0.975, nrow(one_poll_per_pollster)-1)
51 | one_poll_per_pollster %>%
52 | summarize(avg = mean(spread), moe = z*sd(spread)/sqrt(length(spread))) %>%
53 | mutate(start = avg - moe, end = avg + moe)
54 | ```
55 |
56 | A bit larger than the one using normal is
57 |
58 | ```{r}
59 | qt(0.975, 14)
60 | ```
61 |
62 | is bigger than
63 |
64 | ```{r}
65 | qnorm(0.975)
66 | ```
67 |
68 | Fivethirtyeight uses the t-distribution to generate errors that better model the deviations we see in election data. For example the deviation we saw in Wisconsin between the polls and actual result, where Trump won by 0.7%, is more in line with t-distributed data than the normal distribution.
69 |
70 | ```{r}
71 | results %>% filter(state == "Wisconsin")
72 | ```
73 |
74 |
75 |
--------------------------------------------------------------------------------
/lectures/ml/cross-validation-slides.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/datasciencelabs/2017/20caca62a7ac20f1cdee9012b64ff19680c37137/lectures/ml/cross-validation-slides.pdf
--------------------------------------------------------------------------------
/lectures/ml/cross-validation.Rmd:
--------------------------------------------------------------------------------
1 | ## Cross validation
2 | ```{r, message=FALSE, warning=FALSE}
3 | library(readr)
4 | library(dplyr)
5 | library(tidyr)
6 | library(ggplot2)
7 | library(dslabs)
8 | ds_theme_set()
9 | ```
10 |
11 | ```{r}
12 | plotit <- function(dat, i, n=sqrt(ncol(dat)-1)){
13 | dat <- slice(dat,i)
14 | tmp <- expand.grid(Row=1:n, Column=1:n) %>%
15 | mutate(id=i, label=dat$label,
16 | value = unlist(dat[,-1]))
17 | tmp%>%ggplot(aes(Row, Column, fill=value)) +
18 | geom_raster() +
19 | scale_y_reverse() +
20 | scale_fill_gradient(low="white", high="black") +
21 | ggtitle(tmp$label[1])
22 | }
23 | ```
24 |
25 |
26 | ```{r}
27 | url <- "https://raw.githubusercontent.com/datasciencelabs/data/master/hand-written-digits-train.csv"
28 | original_dat <- read_csv(url)
29 | original_dat <- mutate(original_dat, label = as.factor(label))
30 | ```
31 |
32 | There a test set with no labels given:
33 |
34 | ```{r}
35 | url <- "https://raw.githubusercontent.com/datasciencelabs/data/master/hand-written-digits-test.csv"
36 | original_test<- read_csv(url)
37 | #View(original_test)
38 | ```
39 |
40 | ## Data Exploration
41 |
42 | ```{r}
43 | X <- sample_n(original_dat,200) %>%
44 | arrange(label)
45 |
46 | d <- dist(as.matrix(X[,-1]))
47 | image(as.matrix(d))
48 |
49 | plot(hclust(d),labels=as.character(X$label))
50 | ```
51 |
52 | 784 is too much to handle in a demo. So we will compress the predictors by combining groups of 16 pixels.
53 |
54 | ```{r}
55 | tmp <- slice(original_dat,1:100)
56 | names(tmp) <- gsub("pixel","",names(tmp))
57 | tmp <- tmp %>% mutate(obs = 1:nrow(tmp))
58 | tmp <- tmp %>% gather(feature, value, `0`:`783`)
59 | tmp <- tmp %>% mutate(feature = as.numeric(feature))
60 | tmp <- tmp %>% mutate(row = feature%%28, col =floor(feature/28))
61 | tmp <- tmp %>% mutate(row = floor(row/4), col = floor(col/4))
62 | tmp <- tmp %>% group_by(obs, row, col)
63 | tmp <- tmp %>% summarize(label = label[1], value = mean(value))
64 | tmp <- tmp %>% ungroup
65 | tmp <- tmp %>% mutate(feature = sprintf("X_%02d_%02d",col,row))
66 | tmp <- tmp %>% select(-row, -col)
67 | tmp <- tmp %>% group_by(obs) %>% spread(feature, value) %>% ungroup %>% select(-obs)
68 | ```
69 |
70 | Let's write a function to perform the compression.
71 |
72 | ```{r}
73 | compress <- function(tbl, n=4){
74 | names(tbl) <- gsub("pixel","",names(tbl))
75 | tbl %>% mutate(obs = 1:nrow(tbl)) %>%
76 | gather(feature, value, `0`:`783`) %>%
77 | mutate(feature = as.numeric(feature)) %>%
78 | mutate(row = feature%%28, col =floor(feature/28)) %>%
79 | mutate(row = floor(row/n), col = floor(col/n)) %>%
80 | group_by(obs, row, col) %>%
81 | summarize(label = label[1], value = mean(value)) %>%
82 | ungroup %>%
83 | mutate(feature = sprintf("X_%02d_%02d",col,row)) %>%
84 | select(-row, -col) %>%
85 | group_by(obs) %>% spread(feature, value) %>%
86 | ungroup %>%
87 | select(-obs)
88 | }
89 | ```
90 |
91 | Now, we compress the entire dataset. This will take a bit of time:
92 |
93 | ```{r}
94 | dat <- compress(original_dat)
95 | ```
96 |
97 | Note that some features are almost always 0:
98 |
99 | ```{r}
100 | library(caret)
101 | set.seed(1)
102 | inTrain <- createDataPartition(y = dat$label,
103 | p=0.9)$Resample
104 | X <- dat %>% select(-label) %>% slice(inTrain) %>% as.matrix
105 | column_means <- colMeans(X)
106 | plot(table(round(column_means)))
107 | ```
108 |
109 | Let's remove these "low information" features:
110 |
111 | ```{r}
112 | keep_columns <- which(column_means>10)
113 | ```
114 |
115 | Let's next define the training data and test data:
116 |
117 | ```{r}
118 | train_set <- slice(dat, inTrain) %>%
119 | select(label, keep_columns+1)
120 | test_set <- slice(dat, -inTrain) %>%
121 | select(label, keep_columns+1)
122 | ```
123 |
124 | Note that the distances look a bit cleaner:
125 |
126 | ```{r}
127 | X <- sample_n(train_set,200) %>%
128 | arrange(label)
129 |
130 | d <- dist(as.matrix(X[,-1]))
131 | image(as.matrix(d))
132 | plot(hclust(d),labels=as.character(X$label))
133 | ```
134 |
135 |
136 | ```{r}
137 | tmp = sample_n(train_set,5000)
138 |
139 | control <- trainControl(method='cv', number=20)
140 | res <- train(label ~ .,
141 | data = tmp,
142 | method = "knn",
143 | trControl = control,
144 | tuneGrid=data.frame(k=seq(1,15,2)),
145 | metric="Accuracy")
146 |
147 | plot(res)
148 |
149 | fit <- knn3(label~., train_set, k=3)
150 | pred <- predict(fit, newdata = test_set, type="class")
151 |
152 | tab <- table(pred, test_set$label)
153 | confusionMatrix(tab)
154 | ```
155 |
156 | Compete in Kaggle?
157 |
158 | ```{r}
159 | original_test <- mutate(original_test, label=NA)
160 | test <- compress(original_test)
161 | test <- test %>% select(label, keep_columns+1)
162 | pred <- predict(fit, newdata = test, type="class")
163 |
164 | i=11
165 | pred[i]
166 | plotit(original_test,i)
167 |
168 | res <- data.frame(ImageId=1:nrow(test),Label=as.character(pred))
169 | write.csv(res, file="test.csv", row.names=FALSE)
170 | ```
171 |
172 |
--------------------------------------------------------------------------------
/lectures/ml/img/binsmoother1.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/datasciencelabs/2017/20caca62a7ac20f1cdee9012b64ff19680c37137/lectures/ml/img/binsmoother1.gif
--------------------------------------------------------------------------------
/lectures/ml/img/binsmoother2.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/datasciencelabs/2017/20caca62a7ac20f1cdee9012b64ff19680c37137/lectures/ml/img/binsmoother2.gif
--------------------------------------------------------------------------------
/lectures/ml/img/loess.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/datasciencelabs/2017/20caca62a7ac20f1cdee9012b64ff19680c37137/lectures/ml/img/loess.gif
--------------------------------------------------------------------------------
/lectures/ml/img/loesses.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/datasciencelabs/2017/20caca62a7ac20f1cdee9012b64ff19680c37137/lectures/ml/img/loesses.gif
--------------------------------------------------------------------------------
/lectures/ml/intro-ml-assessment.Rmd:
--------------------------------------------------------------------------------
1 | #### Assessment
2 |
3 | 1. For each of the following determine if the outcome is continuos or categorical
4 |
5 | a. Digit reader
6 | b. Movie recommendations
7 | c. Spam filter
8 | d. Hospitalizations
9 | e. Siri
10 |
11 |
12 | 2. How many features are availalbe to us for prediction in the digits dataset?
13 |
14 |
15 | 3. Create a predictor by rounding the heights to the nearest inch. What is the conditional probability of being Male if you are 70 inches tall?
16 |
17 | ```{r}
18 | heights %>% mutate(height = round(height)) %>%
19 | filter(height==70) %>%
20 | summarize(mean(sex=="Male"))
21 |
22 | ```
23 |
24 | 4. Define the following predictor
25 |
26 | ```{r}
27 | X = round(heights$height)
28 | ```
29 |
30 | Estimate $p(x) = \mbox{Pr}( y = 1 | X=x)$ for each $x$ and plot it against $x$.
31 |
32 | ```{r}
33 | heights %>% mutate(X = round(height)) %>%
34 | group_by(X) %>%
35 | summarize(pr = mean(sex=="Male")) %>%
36 | ggplot(aes(X, pr)) +
37 | geom_point()
38 | ```
39 |
40 |
--------------------------------------------------------------------------------
/lectures/ml/rf.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/datasciencelabs/2017/20caca62a7ac20f1cdee9012b64ff19680c37137/lectures/ml/rf.gif
--------------------------------------------------------------------------------
/lectures/shiny/introToDataScience2017_shiny.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/datasciencelabs/2017/20caca62a7ac20f1cdee9012b64ff19680c37137/lectures/shiny/introToDataScience2017_shiny.pdf
--------------------------------------------------------------------------------
/lectures/shiny/introToDataScience2017_shiny.pptx:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/datasciencelabs/2017/20caca62a7ac20f1cdee9012b64ff19680c37137/lectures/shiny/introToDataScience2017_shiny.pptx
--------------------------------------------------------------------------------
/lectures/shiny/shiny_assessments.R:
--------------------------------------------------------------------------------
1 | # Here are the assessment templates for each of the assessments we will perform today
2 |
3 | # Assessment 1
4 | library(shiny)
5 | ui <- fluidPage(
6 | # Put code here for a slider with n as the id,
7 | # 'This is a slider' as the label, and {value, min, max} = {1, 0, 100}
8 | )
9 | server <- function(input,output){ }
10 | shinyApp(ui=ui,server=server)
11 |
12 |
13 | # Assessment 2
14 | library(shiny)
15 | ui <- fluidPage(
16 | # Place code here
17 | # hint you need a 'numericInput' with labelId = n
18 | # hint you also need to 'plot' the 'Output'
19 | )
20 | server <- function(input,output){
21 |
22 | output$hist <- renderPlot({
23 | hist(rnorm(input$n))
24 | })
25 |
26 | }
27 | shinyApp(ui=ui,server=server)
28 |
29 | # Assessment 3
30 | library(shiny)
31 | ui <- fluidPage(
32 | plotOutput("plot")
33 | )
34 | server <- function(input,output){
35 | # place code here to render a plot of the iris dataset
36 | # hint: plot(iris) and ?renderPlot
37 | }
38 |
39 | shinyApp(ui=ui,server=server)
40 |
41 | # Assessment 4
42 | library(shiny)
43 | ui <- fluidPage(
44 | selectInput("dataset","Which dataset?",choices=c("iris","cars")),
45 | plotOutput("plot")
46 | )
47 | server <- function(input,output){
48 | # place code here to create a 'dat' reactive function
49 | output$plot <- renderPlot({
50 | plot(dat())
51 | })
52 | }
53 | shinyApp(ui=ui,server=server)
54 |
55 |
56 | # Assessment 5
57 | library(shiny)
58 | ui <- fluidPage(
59 | tabsetPanel(
60 | # put code here
61 | )
62 | )
63 | server <- function(input,output){
64 |
65 | output$hist <- renderPlot({
66 | hist(rnorm(input$n))
67 | })
68 | }
69 |
70 | shinyApp(ui=ui,server=server)
71 |
--------------------------------------------------------------------------------
/lectures/wrangling/combining-tables.Rmd:
--------------------------------------------------------------------------------
1 | ## Combining tables
2 |
3 | ```{r, message=FALSE, warning=FALSE}
4 | library(tidyverse)
5 | library(ggrepel)
6 | library(dslabs)
7 | ds_theme_set()
8 | ```
9 |
10 | The information we need for a given analysis may not be just on table. For example, when forecasting elections we used the function `left_join` to combine the information from two tables. Here we use a simpler example to illustrate the general challenge of combining tables.
11 |
12 | Suppose we want to explore the relationship between population size for US states, which we have in this table
13 |
14 | ```{r}
15 | data(murders)
16 | head(murders)
17 | ```
18 |
19 | and electoral votes, which we have in this one:
20 |
21 | ```{r}
22 | data(polls_us_election_2016)
23 | head(results_us_election_2016)
24 | ```
25 |
26 | Notice that just joining these two tables together will not work since the order of the states is not quite the same:
27 |
28 | ```{r}
29 | identical(results_us_election_2016$state, murders$state)
30 | ```
31 |
32 | The _join_ functions described below, are designed to handle this challenge.
33 |
34 | ### Joins
35 |
36 | The `join` functions in the `dplyr` package, which are based SQL joins, make sure that the tables are combined so that matching rows are together.
37 | The general idea is that one needs to identify one or more columns that will serve to match the two tables. Then a new table with the combined information is returned. Note what happens if we join the two tables above by state using `left_join`:
38 |
39 | ```{r}
40 | tab <- left_join(murders, results_us_election_2016, by = "state")
41 | tab %>% select(state, population, electoral_votes) %>% head()
42 | ```
43 |
44 | The data has been successfully joins and we can now, for example, make a plot to explore the relationship
45 |
46 | ```{r}
47 | tab %>% ggplot(aes(population/10^6, electoral_votes, label = abb)) +
48 | geom_point() +
49 | geom_text_repel() +
50 | scale_x_continuous(trans = "log2") +
51 | scale_y_continuous(trans = "log2") +
52 | geom_smooth(method = "lm", se = FALSE)
53 | ```
54 |
55 | We see the relationship is close to linear with about 2 electoral votes for every million persons, but with smaller states getting a higher ratios.
56 |
57 |
58 | In practice, it is not always the case that each row in one table has a matching row in the other. For this reason we have several different ways to join. To illustrate this challenge, take subsets of the matrices above
59 |
60 | ```{r}
61 | tab1 <- slice(murders, 1:6) %>% select(state, population)
62 | tab1
63 | ```
64 |
65 | so that we no longer have the same states in the two tables:
66 | ```{r}
67 | tab2 <- slice(results_us_election_2016, c(1:3, 5, 7:8)) %>% select(state, electoral_votes)
68 | tab2
69 | ```
70 |
71 | We will use these two tables as examples.
72 |
73 | #### Left join
74 |
75 | Suppose we want a table like `tab1` but adding electoral votes to whatever states we have available. For this we use left join with `tab1` as the first argument.
76 |
77 | ```{r}
78 | left_join(tab1, tab2)
79 | ```
80 |
81 | Note that `NA`s are added to the two states not appearing in `tab2`. Also note that this function, as well as all the other joins, can receive the first arguments through the pipe:
82 |
83 | ```{r}
84 | tab1 %>% left_join(tab2)
85 | ```
86 |
87 |
88 | #### Right join
89 |
90 | If instead of a table like `tab1` we want one like `tab2` we can use `right_join`:
91 |
92 | ```{r}
93 | tab1 %>% right_join(tab2)
94 | ```
95 |
96 | Notice that now the NAs are in the column coming from `tab1`.
97 |
98 | #### inner join
99 |
100 | If we want to keep only the rows that have information in both tables we use inner join. You can think of this an an intersection:
101 |
102 | ```{r}
103 | inner_join(tab1, tab2)
104 | ```
105 |
106 | #### full join
107 |
108 | And if we want to keep all the rows, and fill the missing parts with NAs, we can use a full join. You can think of this as a union:
109 |
110 | ```{r}
111 | full_join(tab1, tab2)
112 | ```
113 |
114 | #### semi join
115 |
116 | The `semi_join` let's us keep the part of first table for which we have information in the second. It does not add the columns of the second:
117 |
118 | ```{r}
119 | semi_join(tab1, tab2)
120 | ```
121 |
122 |
123 | #### anti joun
124 |
125 | The function `anti_join` is the opposite of `semi_join`. It keeps the elements of the first table for which there is no information in the second:
126 |
127 | ```{r}
128 | anti_join(tab1, tab2)
129 | ```
130 |
131 | The cheat sheet contains a diagram that summarizes the above joins
132 |
133 | ### Binding
134 |
135 | Although we have yet to use it in this book, another common way in which datasets are combined is by _binding_ them. Unlike the join function, the binding function do no try to match by a variable but rather just combine datasets. If the datasets don't match by the appropriate dimensions one obtains error.
136 |
137 | #### Columns
138 |
139 | The `dplyr` function _bind_cols_ binds two objects by making them columns in a tibble. For example we quickly want to make data frame consisting of numbers we can use.
140 |
141 | ```{r}
142 | bind_cols(a = 1:3, b = 4:6)
143 | ```
144 |
145 | This function requires that we assign names to the columns. Here we chose `a` and `b`.
146 |
147 | Note there is an R-base function `cbind` that performs function but creates objects other that tibbles.
148 |
149 | `bind_cols` can also bind data frames. For example here we break up the `tab` data frame and then bind them back together:
150 |
151 | ```{r}
152 | tab1 <- tab[, 1:3]
153 | tab2 <- tab[, 4:6]
154 | tab3 <- tab[, 7:9]
155 | new_tab <- bind_cols(tab1, tab2, tab3)
156 | head(new_tab)
157 | ```
158 |
159 |
160 | #### rows
161 |
162 | The `bind_rows` is similar but binds rows instead of columns
163 |
164 | ```{r}
165 | tab1 <- tab[1:2,]
166 | tab2 <- tab[3:4,]
167 | bind_rows(tab1, tab2)
168 | ```
169 |
170 | This is based on an R-base function `rbind`.
171 |
172 | ### Set Operators
173 |
174 | Another set of commands useful for combing are the set operators. When applied to vectors, these behave as their names suggest. However, if the `tidyvers`, or more specifically, `dplyr` is loaded, these function can be used on data frames as opposed to just on vectors.
175 |
176 | #### Intersect
177 |
178 | You can take intersections of vectors:
179 |
180 | ```{r}
181 | intersect(1:10, 6:15)
182 | ```
183 |
184 | ```{r}
185 | intersect(c("a","b","c"), c("b","c","d"))
186 | ```
187 |
188 | But with `dplyr` loaded we can also do this for tables having the same column names:
189 |
190 | ```{r}
191 | tab1 <- tab[1:5,]
192 | tab2 <- tab[3:7,]
193 | intersect(tab1, tab2)
194 | ```
195 |
196 |
197 | #### Union
198 |
199 | Similarly _union_ takes the union:
200 |
201 | ```{r}
202 | union(1:10, 6:15)
203 | ```
204 |
205 | ```{r}
206 | union(c("a","b","c"), c("b","c","d"))
207 | ```
208 |
209 | But with `dplyr` loaded we can also do this for tables having the same column names:
210 |
211 | ```{r}
212 | tab1 <- tab[1:5,]
213 | tab2 <- tab[3:7,]
214 | union(tab1, tab2)
215 | ```
216 |
217 |
218 | #### Set differrence
219 |
220 | The set difference between a first and second argument can be obtained with `setdiff`. Not unlike `instersect` and `union`, this function is not symmetric:
221 |
222 |
223 | ```{r}
224 | setdiff(1:10, 6:15)
225 | setdiff(6:15, 1:10)
226 | ```
227 |
228 | As the others we can apply it to data frames:
229 | ```{r}
230 | tab1 <- tab[1:5,]
231 | tab2 <- tab[3:7,]
232 | setdiff(tab1, tab2)
233 | ```
234 |
235 | #### `setequal`
236 |
237 | Finally, the function `set_equal` tells us if two sets are the same, regardless of order. So
238 |
239 | ```{r}
240 | setequal(1:5, 1:6)
241 | ```
242 |
243 | but
244 |
245 | ```{r}
246 | setequal(1:5, 5:1)
247 | ```
248 |
249 | When applied to data frames that are not equal regardless of order, it provides a useful message letting us know how the sets are different:
250 |
251 | ```{r}
252 | setequal(tab1, tab2)
253 | ```
254 |
255 |
--------------------------------------------------------------------------------
/lectures/wrangling/dates-and-times.Rmd:
--------------------------------------------------------------------------------
1 | ## Parsing Dates and Times
2 |
3 | We have desribed three main types of vectors: numeric, character, and logical. In data science projects we very often encouter varialbles that are dates. Although we can represent a date with a string, for example, `November 2, 2017`, once we pick a reference day, referred to as the _epoch_, they can be converted to numbers. Computer languages usually use January 1, 1970 as the epoch. So November 2, 2017 is day 17,204.
4 |
5 | Now how should we represents dates and times when analyzing data in R? We could just use days since the epoch, but then it is almost impossible to interpret. If I tell you its November 2, 2017 you know what this means immediately. If I tell you it's day 17,204 you will be quite confused. Similar problems arise with times. In this case it gets even more complicated due to time zones.
6 |
7 | For this reason in R defines a data type just for dataes and times. We saw an exmaple in the polls data:
8 |
9 | ```{r}
10 | library(dslabs)
11 | data("polls_us_election_2016")
12 | polls_us_election_2016$startdate %>% head
13 | ```
14 |
15 | These look like strings. But they are not:
16 |
17 | ```{r}
18 | class(polls_us_election_2016$startdate)
19 | ```
20 |
21 | Look at what happens when we convert them to numbers:
22 |
23 | ```{r}
24 | as.numeric(polls_us_election_2016$startdate) %>% head
25 | ```
26 |
27 | It turns them into dates since the epoch.
28 |
29 | Plotting functions, such as those in ggplot, are aware of dates. This means that, for example, a scatter plot can use the numeric representation to decide on the poisition of the point, but include the string in the labels:
30 |
31 | ```{r}
32 | polls_us_election_2016 %>% filter(pollster == "Ipsos" & state =="U.S.") %>%
33 | ggplot(aes(startdate, rawpoll_trump)) +
34 | geom_line()
35 | ```
36 |
37 | Note in particular that the months are displayed. The tidyverse includes a functionality for dealing with dates through the `lubridate` package.
38 |
39 | ```{r}
40 | library(lubridate)
41 | ```
42 |
43 | We will take a random sample of dates to show some of the useful things one can do:
44 | ```{r}
45 | set.seed(2)
46 | dates <- sample(polls_us_election_2016$startdate, 10) %>% sort
47 | dates
48 | ```
49 |
50 | The functions `year`, `month` and `day` extract those values:
51 |
52 | ```{r}
53 | data.frame(date = days,
54 | month = month(dates),
55 | day = day(dates),
56 | year = year(dates))
57 | ```
58 |
59 | We can also extract the month labels:
60 |
61 | ```{r}
62 | month(dates, label = TRUE)
63 | ```
64 |
65 |
66 | Another useful set of functions are the _parsers_ that converts strings into dates.
67 |
68 | ```{r}
69 | x <- c(20090101, "2009-01-02", "2009 01 03", "2009-1-4",
70 | "2009-1, 5", "Created on 2009 1 6", "200901 !!! 07")
71 | ymd(x)
72 | ```
73 |
74 |
75 | A further complication comes from the fact that dates often come in different formats in which the order of year month and day are different. The preferred format is to show year (with all four digits), month (two digits) and then day or what is called the ISO 8601. Specifically we use YYYY-MM-DD so that if we order the string it will be ordered by date. You can see the function `ymd` returns them in this format.
76 |
77 | What if you encouter dates such as "09/01/02"? This could be September 1, 2002 or Janary 2, 2009 or January 9, 2002.
78 | In these cases examining the entire vector of dates will help you determine what format it is by process of elimination. Once you know you can use of the many parses provided by lubridate.
79 |
80 | For example if the string is
81 |
82 | ```{r}
83 | x <- "09/01/02"
84 | ```
85 |
86 | The `ymd` function assumes the first entry is the year the second the month and the third the day so it coverts it to:
87 |
88 | ```{r}
89 | ymd(x)
90 | ```
91 |
92 | The `mdy` function assumes the first entry is the month then the day then the year:
93 |
94 | ```{r}
95 | mdy(x)
96 | ```
97 |
98 | Lubridate provides a function for every possibility:
99 | ```{r}
100 | ydm(x)
101 | myd(x)
102 | dmy(x)
103 | dym(x)
104 | ```
105 |
106 |
107 |
108 | Lubridate is also useful for dealing with times. In R, you can get the current time tpying `Sys.time()`. Lubridate provides a slightly more advanced function, `now`, that permits you define the time zone
109 |
110 | ```{r}
111 | now()
112 | now("GMT")
113 | ```
114 |
115 | You can see all the available times zones with `OlsonNames()` function.
116 |
117 | Lubridate also has function to extract hours, minutes as seconds:
118 |
119 | ```{r}
120 | now() %>% hour()
121 | now() %>% minute()
122 | now() %>% second()
123 | ```
124 |
125 | as as function to conver parse strings into times:
126 |
127 | ```{r}
128 | x <- c("12:34:56")
129 | hms(x)
130 | ```
131 |
132 | as well as parses for time objects that include dates:
133 |
134 | ```{r}
135 | x <- "Nov/2/2012 12:34:56"
136 | mdy_hms(x)
137 | ```
138 |
139 |
140 |
141 |
142 |
143 |
144 |
145 |
146 |
147 |
148 |
149 |
150 |
151 |
152 |
153 |
154 |
155 |
156 |
--------------------------------------------------------------------------------
/lectures/wrangling/intro-to-wrangling.Rmd:
--------------------------------------------------------------------------------
1 | # Data Wrangling
2 |
3 | The datasets used in this book have been made available to you as R objects, specifically as data frames. The US murders data, the reported heights data, the Gapminder data, and the poll data are all examples. These datasets come included in the `dslabs` package and we loaded them using the the `data` function. Furthermore, we have made the data available in what is referred to as `tidy` form, a concept we define later in this chapter. The _tidyverse_ packages and functions assume that the data is `tidy` and this assumption is a big part of the reason these packages work so well together.
4 |
5 | However, very rarely in a data science project is data easily available as part of a package. We did quite a bit of work "behind the scenes" to get the original raw data into the _tidy_ tables you worked with. Much more typical is for the data to be in a file, a database, or extracted from a document including web pages, tweets, or PDFs. In these cases, the the first step is to import the data into R and, when using the _tidyverse_, tidy the data. The first step in the data analysis process usually involves several, often complicated, steps to covert data from its raw form to the _tidy_ form that greatly facilitates the rest of the analysis. We refer to this process as `data wrangling`.
6 |
7 | Here we cover several common steps of the data wrangling process including importing data into R from files, tidying data, string processing, html parsing, working with dates and times, and text mining. Rarely are all these wrangling steps necessary in a single analysis, but data scientist will likely face them all at some point. Some of the example we use to demonstrate data wrangling techniques are based on the work we did to convert raw data into the the tidy datasets provided by the `dslabs` package and used in the book as examples.
8 |
9 |
10 |
--------------------------------------------------------------------------------
/lectures/wrangling/reshaping-data.Rmd:
--------------------------------------------------------------------------------
1 | ## Reshaping data
2 |
3 | ```{r, echo=FALSE, message=FALSE}
4 | library(tidyverse)
5 | path <- system.file("extdata", package="dslabs")
6 | filename <- file.path(path, "fertility-two-countries-example.csv")
7 | wide_data <- read_csv(filename)
8 | ```
9 |
10 | As we have seen, having data in `tidy` format is what makes the `tidyverse` flow. After the first step in the data analysis process, importing data,
11 | a common next step is to reshape the data into a form that facilitates the rest of the analysis. The `tidyr` package includes several functions that are useful for tyding data.
12 |
13 | ### gather
14 |
15 | One of the most used functions in this package is `gather`, which converts wide data into tidy data.
16 |
17 | In the third argument of the `gather` function you specify the columns that will be _gathered_. The default is to gather all function, so in most cases we have to specify the columns. Here we want columns `1960`, `1961` up to `2015`. The first argument sets the column/variable name that will hold the variable that is currently kept in the wide data column names. In our case it makes sense to set name to `year`, but we can name it anything. The second argument sets the column/variable name that will hold the values in the column cells. In this case we call it `fertility` since this what is stored in this file. Note that nowhere in this file does it tell us this is fertility data. Instead, this information was kept in the file name.
18 |
19 | The gathering code looks like this:
20 |
21 | ```{r}
22 | new_tidy_data <- wide_data %>%
23 | gather(my_year, my_fertility, `1960`:`2015`)
24 | ```
25 |
26 | We can see that the data have been converted to tidy format with columns `year` and `fertility`:
27 |
28 | ```{r}
29 | head(new_tidy_data)
30 | ```
31 |
32 | However, each year resulted in two rows since we have two countries and this column was not gathered.
33 |
34 | A somewhat quicker way to write this code is to specific which column will **not** be gathered rather than all the columns that will be gathered:
35 |
36 | ```{r}
37 | new_tidy_data <- wide_data %>%
38 | gather(year, fertility, -country)
39 | ```
40 |
41 | This data looks a lot like the original `tidy_data` we used. There is just one minor difference. Can you spot it? Look at the data type of the year column:
42 |
43 | ```{r}
44 | library(dslabs)
45 | data("gapminder")
46 | tidy_data <- gapminder %>%
47 | filter(country %in% c("South Korea", "Germany")) %>%
48 | select(country, year, fertility)
49 |
50 | class(tidy_data$year)
51 | class(new_tidy_data$year)
52 | ```
53 |
54 | The `gather` function assumes that column names are characters. So we need a bit more wrangling before we are ready to make a plot. We need to convert the column to numbers. The `gather` function has an argument for that, the `convert` argument:
55 |
56 | ```{r}
57 | new_tidy_data <- wide_data %>%
58 | gather(year, fertility, -country, convert = TRUE)
59 | class(new_tidy_data$year)
60 | ```
61 |
62 | We could have also used the `mutate` and `as.numeric`.
63 |
64 | Now that the data is tidy we can use the same ggplot as before
65 |
66 | ```{r}
67 | new_tidy_data %>% ggplot(aes(year, fertility, color = country)) +
68 | geom_point()
69 | ```
70 |
71 | ### spread
72 |
73 | As we will see in later examples it is sometimes useful for data wrangling purposes to convert tidy data into wide data. We often use this as intermediate step in tidying up data. The `spread` function is basically the inverse of `gather`. The first argument tells `spread` which variable will be used as the column names. The second argument specifies which variable to use to fill out the cells:
74 |
75 | ```{r}
76 | new_wide_data <- new_tidy_data %>% spread(year, fertility)
77 | select(new_wide_data, country, `1960`:`1967`)
78 | ```
79 |
80 |
81 | The diagrams in the cheat sheet can help remind you how these two functions work.
82 |
83 |
84 | ### `separate`
85 |
86 | The data wrangling shown above was simple compared to what is usually required. In our example spreadsheet files we include an example that is slightly more complicated. It includes two variables: life expectancy as well as fertility. However, the way it is stored is not tidy and, as we will explain, not optimal.
87 |
88 | ```{r}
89 | path <- system.file("extdata", package = "dslabs")
90 | filename <- file.path(path, "life-expectancy-and-fertility-two-countries-example.csv")
91 |
92 | raw_dat <- read_csv(filename)
93 | select(raw_dat, 1:5)
94 | ```
95 |
96 | First note that the data is in wide format. Second, note that now there are values for two variables with the column names encoding which column represents which variable. We can start the data wrangling with the `gather` function, but we should no longer use the column name `year` for the new column since since it also contains the variable type. We will call it `key`, the default, for now:
97 |
98 | ```{r}
99 | dat <- raw_dat %>% gather(key, value, -country)
100 | head(dat)
101 | ```
102 |
103 | The result is not exactly what we refer to as tidy since each observation is associated with two, not one, row. We want to have the values from the two variables, fertility and life expectancy, in two separate columns. The first challenge to achieve this is to separate the `key` column into the year and the variable type. Note that the entries in this column separate the year from the variable name with an underscore
104 |
105 | ```{r}
106 | dat$key[1:5]
107 | ```
108 |
109 | Encoding multiple variables in a column name is such a common problem that the `readr` package includes a function to separate these columns into two or more. Apart from the data, the `separate` function takes three arguments: the name of the column to be separated, the names to be used for the new columns and the character that separates the variables. So a first attempt at this is:
110 |
111 | ```{r, eval=FALSE}
112 | dat %>% separate(key, c("year", "variable_name"), "_")
113 | ```
114 |
115 | Because "_" is the default separator we actually can simply write:
116 |
117 | ```{r}
118 | dat %>% separate(key, c("year", "variable_name"))
119 | ```
120 |
121 | However, we run into a problem. Note that we receive the warning `Too many values at 112 locations:` and the that `life_exepectancy` variable is truncated to `life`. This is because the `_` is used to separate `life` and `expectancy` not just year and variable name. We could add a third column to catch this and let the `separate` function know which column to _fill in_ with missing values, `NA`, when there is no third value. Here we tell it to fill the column on the right:
122 |
123 | ```{r}
124 | dat %>% separate(key,
125 | c("year", "first_variable_name", "second_variable_name"),
126 | fill = "right")
127 | ```
128 |
129 | However, if we read the `separate` help file we find that a better approach is to merge the last two variables when there is an extra separation:
130 |
131 | ```{r}
132 | dat %>% separate(key, c("year", "variable_name"), sep = "_", extra = "merge")
133 | ```
134 |
135 | This achieves the separation we wanted. However, we are not done yet. We need to create a column for each variable. As we learned, the `spread` function can do this:
136 |
137 | ```{r}
138 | dat %>% separate(key, c("year", "variable_name"), sep = "_", extra = "merge") %>%
139 | spread(variable_name, value)
140 | ```
141 |
142 | The data is now in tidy format with one row for each observation with three variable: year, fertility and life expectancy.
143 |
144 | ### `unite`
145 |
146 | It is sometimes useful to do the inverse of `separate`, unite two columns into one. So, although this is *not* an optimal approach, had we used this command to separate
147 |
148 | ```{r}
149 | dat %>%
150 | separate(key, c("year", "first_variable_name", "second_variable_name"), fill = "right")
151 | ```
152 |
153 | We can achieve the same final result by uniting the second and third column like this:
154 |
155 | ```{r}
156 | dat %>%
157 | separate(key, c("year", "first_variable_name", "second_variable_name"), fill = "right") %>%
158 | unite(variable_name, first_variable_name, second_variable_name, sep="_")
159 | ```
160 |
161 | Then spreading the columns:
162 |
163 | ```{r}
164 | dat %>%
165 | separate(key, c("year", "first_variable_name", "second_variable_name"), fill = "right") %>%
166 | unite(variable_name, first_variable_name, second_variable_name, sep="_") %>%
167 | spread(variable_name, value) %>%
168 | rename(fertlity = fertility_NA)
169 | ```
170 |
171 |
172 |
173 |
174 |
175 |
--------------------------------------------------------------------------------
/lectures/wrangling/tidy-data.Rmd:
--------------------------------------------------------------------------------
1 | ## Tidy data
2 |
3 | ```{r, message=FALSE}
4 | library(tidyverse)
5 | library(dslabs)
6 | ds_theme_set()
7 | ```
8 |
9 | To help define tidy data we go back to an example we showed in the data visualization chapter in which we plotted fertility data across time for two countries: South Korea and Germany. To make the plot we used this subset of the data
10 |
11 | ```{r}
12 | data("gapminder")
13 | tidy_data <- gapminder %>%
14 | filter(country %in% c("South Korea", "Germany")) %>%
15 | select(country, year, fertility)
16 | head(tidy_data)
17 | ```
18 |
19 | With the data in this format we could quickly make the desired plot:
20 |
21 | ```{r}
22 | tidy_data %>%
23 | ggplot(aes(year, fertility, color = country)) +
24 | geom_point()
25 | ```
26 |
27 | One reason this code works seamlessly is because the data is _tidy_: each point is represented in a row. This brings us to the definition of _tidy data_: each row represents one observation and the column represent the different variable that we have data on for those observations.
28 |
29 | If we go back to the original data provided by GapMinder we see that it does not start out _tidy_. We include an example file with the he data shown in this graph mimicking the way it was originally saved in a spreadsheet:
30 |
31 | ```{r}
32 | path <- system.file("extdata", package="dslabs")
33 | filename <- file.path(path, "fertility-two-countries-example.csv")
34 | wide_data <- read_csv(filename)
35 | ```
36 |
37 | The object `wide_data` includes the same information as the object `tidy_data` except it is in a different format: a `wide` format. Here are the first nine columns:
38 |
39 | ```{r}
40 | select(wide_data, country, `1960`:`1967`)
41 | ```
42 |
43 | Two important differences between the wide and tidy formats. First, in the wide format each row includes several observations. Second, one of the variables, year, is stored in the header.
44 |
45 | The `ggplot` code we introduced earlier no longer works here. For one there is no `year` variable. So to use the `tidyverse` we need to wrangle this data into `tidy` format.
46 |
47 |
--------------------------------------------------------------------------------
/lectures/wrangling/web-scraping.Rmd:
--------------------------------------------------------------------------------
1 | ## Web Scraping
2 |
3 | The data we need to answer a question is not always in a spreadsheet, ready for us to read. For example the US murders dataset we used in the R Basics chapter originally comes from this Wikipedia page: [https://en.wikipedia.org/wiki/Murder_in_the_United_States_by_state](https://en.wikipedia.org/wiki/Murder_in_the_United_States_by_state). You can see the data table when you visit the web page.
4 |
5 | But, unfortunately, there is no link to a data file. To make the data frame we loaded using `data(murders)`, or reading the csv file made available through `dslabs`, we had to do some _web scrapping_.
6 |
7 | _Web scraping_, or _web harvesting_, are the terms we use to describe the process of extracting data from a website. The reason we can do this is because the information used by a browser to render web pages is received as **text** from a server. The text is computer code written in hyper text markup language (HTML). To see the code for a web page you can visit the page on your browser, then you can use the _View Source_ tool to see it.
8 |
9 |
10 | Because this code is accessible, we can download the HTML files, import it into R, and then write programs to extract the information we need from the page. However, once we look at HTML code this might seem like a daunting task. But we will show you some convenient tools to facilitate the process. To get an idea of how it works, here we show a few lines of code from the Wikipedia page that provides the US murders data:
11 |
12 | ```{r, eval = FALSE}
13 | p>The 2015 U.S. population total was 320.9 million. The 2015 U.S. overall murder rate per 100,000 inhabitants was 4.89.
14 | | State | 18 |Population 19 | (total inhabitants) 20 | (2015) [1] |
21 | Murders and Nonnegligent
22 | Manslaughter |
26 | Murder and Nonnegligent
27 | Manslaughter Rate |
31 |
|---|---|---|---|
| Alabama | 34 |4,853,875 | 35 |348 | 36 |7.2 | 37 |
| Alaska | 40 |737,709 | 41 |59 | 42 |8.0 | 43 |