├── LICENSE.md └── LICENSE.md │ └── LICENSE.md ├── assumptions.png ├── model.png ├── simulationtutorial.Rmd ├── simulationtutorial.html └── simulationtutorial.pdf /LICENSE.md/LICENSE.md/LICENSE.md: -------------------------------------------------------------------------------- 1 |

This work is licensed under a Creative Commons Attribution 4.0 International License. 2 | -------------------------------------------------------------------------------- /assumptions.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jessiesunpsych/power-simulations/d4af6769978d0ed5817b04b7a548e2650c54b0bf/assumptions.png -------------------------------------------------------------------------------- /model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jessiesunpsych/power-simulations/d4af6769978d0ed5817b04b7a548e2650c54b0bf/model.png -------------------------------------------------------------------------------- /simulationtutorial.Rmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Intro to Power Analysis Using Simulation Methods" 3 | author: "Jessie Sun" 4 | date: "15/11/2018" 5 | output: pdf_document 6 | --- 7 | 8 | ```{r setup, include=FALSE} 9 | knitr::opts_chunk$set(echo = TRUE) 10 | ``` 11 | 12 | This is a very basic introduction to conducting power analyses based on simulations, using the *lavaan* package. 13 | 14 | ## Types of Power Analysis 15 | 16 | 1. *A priori power analysis:* What is the smallest sample size we need to have 80% power to detect an effect size of interest (e.g. $\beta$ = 0.20, $\beta$ = 0.50), at an alpha level of .05? 17 | 2. *Sensitivity power analysis:* What is the smallest effect size we can detect with 80% power, given our sample size, at an alpha level of .05? 18 | 3. *Post-hoc power analysis:* What power did we have to detect the observed effect size, given the sample size actually used, at an alpha level of .05? 19 | 20 | As Daniel Lakens has [explained elsewhere](http://daniellakens.blogspot.com/2014/12/observed-power-and-what-to-do-if-your.html), observed power (from a post-hoc power analysis) is a useless statistical concept. Thus, this tutorial will focus on a priori and sensitivity power analyses. 21 | 22 | (Note: You can also set different alpha thresholds and power goals, but to keep things simple, let's assume the standard .05 alpha threshold and the goal of 80% power.) 23 | 24 | ## The Model 25 | 26 | As shown in Figure 1 below, we will be considering a fairly simple model, with two predictor variables $X_1$ and $X_2$, and one outcome variable, Y. All variables are observed. 27 | 28 | ```{r, out.width = "300px", echo = FALSE} 29 | knitr::include_graphics("model.png") 30 | ``` 31 | 32 | *Figure 1.* Labelled multiple regression model. 33 | 34 | ## Our Goal 35 | 36 | Our goal in this tutorial is to conduct a priori and sensitivity power analyses for the regression path for Y on $X_2$ ($\beta_{2}$). \pagebreak 37 | 38 | ## Model Assumptions 39 | 40 | This model includes the following parameters: 41 | 42 | -Variance of $X_1$ ($s^2_{X1}$) 43 | 44 | -Variance of $X_2$ ($s^2_{X2}$) 45 | 46 | -Covariance between X1 and X2 ($cov_{X1,X2}$) 47 | 48 | -Regression path for Y on X1 ($\beta_{1}$) 49 | 50 | -Regression path for Y on X2 ($\beta_{2}$) 51 | 52 | -Residual variance of Y ($\epsilon_{Y}$) 53 | 54 | In this case, to conduct power analysis based on standardized effect sizes, we will: 55 | 56 | 1. Fix the variance of $X_1$ and $X_2$ to 1. 57 | 2. Assume that Y has a variance of 1, such that the residual variance $\epsilon{Y}$ (i.e., the variance not explained by $X_1$ and $X_2$) will be equal to 1 - ($\beta_{1}$^2^ + $\beta_{2}$^2^). 58 | 59 | Note that since we are using a standardized metric, $cov_{X1,X2}$ is now simply the correlation between $X_1$ and $X_2$ (i.e., $r_{X1,X2}$). 60 | 61 | To simulate data based on a **population model**, we need to make assumptions about each of the other parameters. Some of these assumptions might be based on existing data (e.g., you might already know how strongly $X_1$ and $X_2$ tend to be correlated in the literature), but at other times, they might seem somewhat arbitrary. Because of the potential arbitrariness of our assumptions, it is useful to simulate power under a range of different assumptions (e.g., what if the correlation between $X_1$ and $X_2$ was stronger, or if the regression path for $\beta_{1}$ was larger or smaller?). 62 | 63 | However, for the purposes of this tutorial, let's assume that $X_1$ and $X_2$ are moderately positively correlated (*r* = .30), and that $X_1$ positively predicts Y to a small extent ($\beta_{1}$ = 0.10). 64 | 65 | Figure 2 illustrates these assumptions. 66 | 67 | ```{r, out.width = "300px", echo = FALSE} 68 | knitr::include_graphics("assumptions.png") 69 | ``` 70 | 71 | *Figure 2.* Labelled multiple regression model with model assumptions. 72 | 73 | ## A Priori Power Analysis 74 | 75 | Let's get started with an a priori power analysis. What is the smallest sample size we need to have 80% power to detect an effect size of $\beta_2$ = 0.20, at an alpha level of .05? 76 | 77 | First, we need to load the *lavaan* package 78 | 79 | ```{r} 80 | library(lavaan) 81 | ``` 82 | 83 | Next, we need to specify the population model, based on the assumptions in Figure 2, plus our effect size of interest ($\beta_2$ = 0.20). This is the model that, at the population level, we assume is generating the data that we might see in any given dataset. 84 | 85 | Basic *lavaan* notation: a double ~~ denotes variances and covariances, whereas a single ~ denotes a regression path. 86 | 87 | ```{r} 88 | 89 | popmod1 <- ' 90 | # variances of X1 and X2 are fixed at 1 91 | x1~~1*x1 92 | x2~~1*x2 93 | 94 | # correlation between X1 and X2 is assumed to be .30 95 | x1~~.3*x2 96 | 97 | # regression path for Y on X1 is assumed to be .10 98 | y~.10*x1 99 | 100 | # regression path of interest, Y on X2, is assumed to be .20 101 | y~.20*x2 102 | 103 | # residual variance of Y is 1 - (.1^2 + .2^2) = .95 104 | y~~.95*y 105 | ' 106 | 107 | ``` 108 | 109 | We also need to create another *lavaan* model, without those population-level assumptions. 110 | 111 | ```{r} 112 | 113 | fitmod <- ' 114 | # variances of X1 and X2 115 | x1~~x1 116 | x2~~x2 117 | 118 | # correlation between X1 and X2 119 | x1~~x2 120 | 121 | # regression path for Y on X1 122 | y~x1 123 | 124 | # regression path of interest, Y on X2 125 | y~x2 126 | 127 | # residual variance of Y 128 | y~~y 129 | ' 130 | 131 | ``` 132 | 133 | To see the logic of the simulation process, let's first just simulate one dataset based on the population model, popmod1. 134 | 135 | ```{r} 136 | set.seed(20181102) # setting a seed for reproducibility of the example 137 | data <- simulateData(popmod1, sample.nobs = 500) # assume 500 participants for now 138 | ``` 139 | 140 | Now, we're going to fit our model (fitmod) to this dataset. 141 | 142 | ```{r} 143 | fit <- sem(model = fitmod, data=data, fixed.x=F) 144 | ``` 145 | 146 | Here are the parameter estimates. The parameter of interest, y ~ x2, is in row 5. As you can see, this parameter was statistically significant (*p* < .005) in this simulation based on one dataset. 147 | 148 | ```{r} 149 | parameterEstimates(fit) # see all parameters 150 | parameterEstimates(fit)[5,] # isolating the row with the parameter of interest 151 | ``` 152 | 153 | However, to estimate power, we need to simulate many datasets. Then, we can obtain the % of datasets in which the parameter of interest is statistically significant. This is our power estimate. 154 | 155 | So, let's go ahead and simulate 1000 datasets, still assuming a sample size of 500. 156 | 157 | ```{r} 158 | results <- NULL # create empty object to store results 159 | 160 | for (i in 1:1000){ # simulating 1000 datasets 161 | data <- simulateData(popmod1, sample.nobs = 500) # each dataset contains 500 participants 162 | fit <- sem(model = fitmod, data=data, fixed.x=F) # fit the model to each dataset 163 | results <- rbind(results,parameterEstimates(fit)[5,]) # save the row for y ~ x2 for each dataset 164 | } 165 | 166 | # Count the proportion of significant parameter estimates out of 1000 datasets 167 | paste0('Estimated power = ',mean(results$pvalue < .05)) 168 | ``` 169 | 170 | As you can see, power in this example was excellent; we were able to detect the effect of interest in 99.4% of those 1,000 simulations. 171 | 172 | Now, let's simulate some results based on different sample sizes. It's a good idea to start with rough increments (e.g., *N* = 100, 200, 300, 400) and fewer datasets (e.g., 100) to save computing time. Then once you know which ballpark to aim for, re-run the simulations based on narrower increments (e.g., increments of *N* = 10) and more datasets (e.g., 1000) to get a more exact estimate of sample requirements. 173 | 174 | ```{r} 175 | # now that we are trying a few different sample sizes, we need to create a list to store these results 176 | powerlist <- list() 177 | 178 | # extending the for() loop with a loop for different sample sizes 179 | for(j in seq(100,400,by=100)){ # trying sample sizes of 100, 200, 300, 400 180 | results <- NULL 181 | for (i in 1:100){ # starting with 100 datasets for each sample size 182 | data <- simulateData(popmod1, sample.nobs = j) # j corresponds to the sample size 183 | fit <- sem(model = fitmod, data=data, fixed.x=F) 184 | results <- rbind(results,parameterEstimates(fit)[5,]) # row for y ~ x2 185 | powerlist[[j]] <- mean(results$pvalue < .05) 186 | } 187 | } 188 | 189 | # Convert the power list into a table 190 | library(plyr) 191 | powertable <- ldply(powerlist) 192 | names(powertable)[1] <- c('power') 193 | 194 | # Add a column for the sample size 195 | powertable$N <- seq(100,400,by=100) 196 | 197 | # Here are all the power estimates: 198 | powertable 199 | 200 | # Conclusion: 201 | paste0('The smallest sample size that provided at least 80% power was N = ', 202 | powertable[which(powertable$power>.80),'N'][1]) 203 | 204 | ``` 205 | 206 | Based on 200 simulations, it seems like the ballpark for 80% power is between 150 and 250 participants. So now let's re-run the simulations, but in increments of *N* = 10, and with 1000 simulations each. 207 | 208 | ```{r} 209 | # create a list to store these results for different sample sizes 210 | powerlist <- list() 211 | 212 | # extending the for() loop with a loop for different sample sizes 213 | for(j in seq(150,250,by=10)){ # trying sample sizes between 150 to 250, in increments of 10 214 | results <- NULL 215 | for (i in 1:1000){ # now simulating 1000 datasets for each sample size 216 | data <- simulateData(popmod1, sample.nobs = j) # j corresponds to the sample size 217 | fit <- sem(model = fitmod, data=data, fixed.x=F) 218 | results <- rbind(results,parameterEstimates(fit)[5,]) # row for y ~ x2 219 | powerlist[[j]] <- mean(results$pvalue < .05) 220 | } 221 | } 222 | 223 | # Convert the power list into a table 224 | powertable <- ldply(powerlist) 225 | names(powertable)[1] <- c('power') 226 | 227 | # Add a column for the sample size 228 | powertable$N <- seq(150,250,by=10) 229 | 230 | # Here are all the power estimates: 231 | powertable 232 | 233 | # Conclusion: 234 | paste0('The smallest sample size that provided at least 80% power was N = ', 235 | powertable[which(powertable$power>.80),'N'][1]) 236 | 237 | ``` 238 | 239 | ### Try it Out: Alternative Effect Sizes 240 | 241 | In this example, we have assumed that we are interested in detecting an effect size at least as great as $\beta_2$ = 0.20. But of course, you might be interested in detecting smaller or larger effect sizes. Can you try adapting the population model, and re-run the simulations for different effect sizes? 242 | 243 | ```{r} 244 | 245 | 246 | ``` 247 | 248 | ## Sensitivity Power Analysis 249 | 250 | Instead of specifying an **effect size of interest**, you might already have a **predefined sample size** (e.g., due to time and financial constraints, or already-collected data). In this case, you can ask the question: What is the smallest effect size we can detect with at least 80% power, given our sample size? Let's assume that we have a sample size of *N* = 500. 251 | 252 | This time, we'll need to generate a series of population models with different effect sizes. Just as before, it's a good idea to start with larger increments (e.g., effect sizes from .10 to .50, in increments of .10) and fewer simulations, then refine the simulations in a second round. 253 | 254 | ```{r} 255 | 256 | # First, we need to create a template for the population model 257 | popmodtemplate <- ' 258 | # variances of X1 and X2 are fixed at 1 259 | x1~~1*x1 260 | x2~~1*x2 261 | 262 | # correlation between X1 and X2 is assumed to be .30 263 | x1~~.3*x2 264 | 265 | # regression path for Y on X1 is assumed to be .10 266 | y~.10*x1 267 | 268 | # regression path of interest, Y on X2, will be varied 269 | y~beta2*x2 # we will be substituting different effect sizes into beta2 270 | 271 | # residual variance of Y 272 | y~~resy*y # we will be substituting different effect sizes into res, depending on beta2 273 | ' 274 | 275 | # Use this template to generate a series of population model 276 | # syntaxes with varying sizes of beta2 277 | 278 | popmodlist <- list() 279 | 280 | for(i in seq(0.10,0.50,by=0.10)){ 281 | popmodlist[[paste0(i)]] <- gsub('beta2',i,popmodtemplate) 282 | popmodlist[[paste0(i)]] <- gsub('resy',paste0(1-(.1^2+i^2)),popmodlist[[paste0(i)]]) 283 | } 284 | 285 | ``` 286 | 287 | Now that we have our initial list of population models, we can run the first round of simulations to get our ballpark estimates. 288 | 289 | ```{r} 290 | # create a list to store power estimates for each effect size 291 | powerlist <- list() 292 | 293 | # this time, the outside for() loop is for each of the different population models/effect sizes 294 | for(j in names(popmodlist)){ 295 | results <- NULL 296 | for (i in 1:100){ # starting with 100 datasets for each effect size 297 | data <- simulateData(popmodlist[[j]], # for a given population model 298 | sample.nobs = 500) # assuming a sample size of 500 299 | fit <- sem(model = fitmod, data=data, fixed.x=F) # fitmod is the same as for the a priori power analysis 300 | results <- rbind(results,parameterEstimates(fit)[5,]) # row for y ~ x2 301 | powerlist[[j]] <- mean(results$pvalue < .05) 302 | } 303 | } 304 | 305 | # Convert the power list into a table 306 | powertable <- ldply(powerlist) 307 | names(powertable) <- c('beta2','power') 308 | 309 | # Here are all the power estimates: 310 | powertable 311 | 312 | # Conclusion: 313 | paste0('The smallest effect size that could be detected with at least 80% power was beta = ', 314 | powertable[which(powertable$power>.80),'beta2'][1]) 315 | 316 | ``` 317 | 318 | Now that we know the ballpark is between $\beta_{2}$ = 0.10 and $\beta_{2}$ = 0.20, we can repeat the process with narrower increments, and more simulations: 319 | 320 | ```{r} 321 | 322 | # create population models 323 | popmodlist <- list() 324 | 325 | for(i in seq(0.10,0.20,by=0.01)){ 326 | popmodlist[[paste0(i)]] <- gsub('beta2',i,popmodtemplate) 327 | popmodlist[[paste0(i)]] <- gsub('resy',paste0(1-(.1^2+i^2)),popmodlist[[paste0(i)]]) 328 | } 329 | 330 | # create a list to store power estimates for each effect size 331 | powerlist <- list() 332 | 333 | # this time, the outside for() loop is for each of the different population models/effect sizes 334 | for(j in names(popmodlist)){ 335 | results <- NULL 336 | for (i in 1:1000){ # 1000 simulations for each model 337 | data <- simulateData(popmodlist[[j]], # for a given population model 338 | sample.nobs = 500) # assuming a sample size of 500 339 | fit <- sem(model = fitmod, data=data, fixed.x=F) # fitmod is the same as for the a priori power analysis 340 | results <- rbind(results,parameterEstimates(fit)[5,]) # row for y ~ x2 341 | powerlist[[j]] <- mean(results$pvalue < .05) 342 | } 343 | } 344 | 345 | # Convert the power list into a table 346 | powertable <- ldply(powerlist) 347 | names(powertable) <- c('beta2','power') 348 | 349 | # Here are all the power estimates: 350 | powertable 351 | 352 | # Conclusion: 353 | paste0('The smallest effect size that could be detected with at least 80% power was beta = ', 354 | powertable[which(powertable$power>.80),'beta2'][1]) 355 | 356 | ``` 357 | 358 | ### Try it Out: Alternative Sample Sizes 359 | 360 | In this example, we have assumed that we only have 500 participants. Can you try adapting the code to run sensitivity power analyses assuming a different sample size (e.g., 300 participants)? 361 | 362 | ```{r} 363 | 364 | 365 | ``` 366 | 367 | ## Contact 368 | 369 | Feel free to use and adapt for teaching and research purposes (with attribution), and please get in touch (jesun@ucdavis.edu) if you spot any errors! 370 | 371 | -------------------------------------------------------------------------------- /simulationtutorial.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jessiesunpsych/power-simulations/d4af6769978d0ed5817b04b7a548e2650c54b0bf/simulationtutorial.pdf --------------------------------------------------------------------------------