├── .gitignore ├── plots ├── samples_1.png ├── samples_2.png ├── samples_3.png ├── samples_4.png ├── samples_5.png ├── gelman_rubin.png ├── three_way_mean.png ├── autocorrelation_1.png ├── autocorrelation_2.png ├── autocorrelation_3.png └── parameter_estimates.png ├── TODO.org ├── data └── data.tsv ├── README.R └── README.org /.gitignore: -------------------------------------------------------------------------------- 1 | /.Rhistory 2 | /README.html 3 | -------------------------------------------------------------------------------- /plots/samples_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmalsburg/MCMCglmm-intro/HEAD/plots/samples_1.png -------------------------------------------------------------------------------- /plots/samples_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmalsburg/MCMCglmm-intro/HEAD/plots/samples_2.png -------------------------------------------------------------------------------- /plots/samples_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmalsburg/MCMCglmm-intro/HEAD/plots/samples_3.png -------------------------------------------------------------------------------- /plots/samples_4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmalsburg/MCMCglmm-intro/HEAD/plots/samples_4.png -------------------------------------------------------------------------------- /plots/samples_5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmalsburg/MCMCglmm-intro/HEAD/plots/samples_5.png -------------------------------------------------------------------------------- /plots/gelman_rubin.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmalsburg/MCMCglmm-intro/HEAD/plots/gelman_rubin.png -------------------------------------------------------------------------------- /plots/three_way_mean.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmalsburg/MCMCglmm-intro/HEAD/plots/three_way_mean.png -------------------------------------------------------------------------------- /plots/autocorrelation_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmalsburg/MCMCglmm-intro/HEAD/plots/autocorrelation_1.png -------------------------------------------------------------------------------- /plots/autocorrelation_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmalsburg/MCMCglmm-intro/HEAD/plots/autocorrelation_2.png -------------------------------------------------------------------------------- /plots/autocorrelation_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmalsburg/MCMCglmm-intro/HEAD/plots/autocorrelation_3.png -------------------------------------------------------------------------------- /plots/parameter_estimates.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmalsburg/MCMCglmm-intro/HEAD/plots/parameter_estimates.png -------------------------------------------------------------------------------- /TODO.org: -------------------------------------------------------------------------------- 1 | * TODO Use ~effectiveSize~ from the coda package. 2 | * TODO Mention or use other criteria for determining convergence. 3 | - Heidelberg & Welch convergence diagnostic 4 | - Geweke’s convergence diagnostic 5 | - Raftery and Lewis diagnostic 6 | 7 | All available in the coda package. 8 | 9 | -------------------------------------------------------------------------------- /data/data.tsv: -------------------------------------------------------------------------------- 1 | subject item a b c pronoun 2 | 1 1 -1 1 -1 0 3 | 1 3 1 -1 -1 1 4 | 1 5 -1 1 -1 1 5 | 1 7 1 -1 -1 1 6 | 1 9 -1 1 -1 0 7 | 1 11 -1 -1 -1 0 8 | 1 13 -1 1 -1 0 9 | 1 15 1 -1 -1 1 10 | 1 17 -1 1 -1 0 11 | 1 19 1 -1 -1 1 12 | 1 21 1 1 1 1 13 | 1 23 -1 -1 1 0 14 | 1 25 -1 1 1 0 15 | 1 27 -1 -1 1 0 16 | 1 29 1 1 1 1 17 | 1 31 -1 -1 1 0 18 | 1 33 1 1 1 1 19 | 1 35 -1 -1 1 0 20 | 2 2 1 -1 -1 1 21 | 2 4 -1 1 -1 1 22 | 2 6 1 -1 -1 1 23 | 2 8 -1 1 -1 0 24 | 2 12 -1 1 -1 1 25 | 2 14 1 -1 -1 1 26 | 2 16 -1 1 -1 1 27 | 2 18 1 -1 -1 1 28 | 2 20 -1 1 -1 1 29 | 2 22 -1 -1 1 1 30 | 2 24 -1 1 1 1 31 | 2 26 1 -1 1 1 32 | 2 28 -1 1 1 0 33 | 2 30 1 -1 1 1 34 | 2 32 -1 1 1 1 35 | 2 34 1 -1 1 1 36 | 2 36 1 1 1 1 37 | 3 2 1 -1 -1 1 38 | 3 4 -1 1 -1 0 39 | 3 6 1 -1 -1 0 40 | 3 8 -1 1 -1 0 41 | 3 10 1 -1 -1 1 42 | 3 12 -1 1 -1 0 43 | 3 14 -1 -1 -1 0 44 | 3 16 -1 1 -1 0 45 | 3 18 1 -1 -1 1 46 | 3 20 -1 1 -1 0 47 | 3 22 1 -1 1 1 48 | 3 24 1 1 1 0 49 | 3 26 1 -1 1 1 50 | 3 28 -1 1 1 0 51 | 3 30 1 -1 1 1 52 | 3 32 -1 1 1 0 53 | 3 34 1 -1 1 1 54 | 3 36 1 1 1 1 55 | 4 2 -1 1 -1 0 56 | 4 4 -1 -1 -1 0 57 | 4 6 -1 1 -1 0 58 | 4 8 -1 -1 -1 0 59 | 4 12 -1 -1 -1 0 60 | 4 14 -1 1 -1 0 61 | 4 16 1 -1 -1 1 62 | 4 18 -1 1 -1 0 63 | 4 20 -1 -1 -1 0 64 | 4 22 -1 1 1 0 65 | 4 24 1 -1 1 1 66 | 4 26 -1 1 1 0 67 | 4 28 -1 -1 1 0 68 | 4 30 -1 1 1 0 69 | 4 32 1 -1 1 1 70 | 4 34 -1 1 1 0 71 | 4 36 -1 -1 1 0 72 | 5 2 -1 1 -1 0 73 | 5 4 1 -1 -1 1 74 | 5 6 -1 1 -1 0 75 | 5 8 1 -1 -1 1 76 | 5 10 1 1 -1 1 77 | 5 12 1 -1 -1 1 78 | 5 14 -1 1 -1 0 79 | 5 16 1 -1 -1 1 80 | 5 18 -1 1 -1 0 81 | 5 20 1 -1 -1 1 82 | 5 22 -1 1 1 0 83 | 5 24 -1 -1 1 0 84 | 5 26 1 1 1 1 85 | 5 28 1 -1 1 1 86 | 5 30 -1 1 1 0 87 | 5 32 1 -1 1 1 88 | 5 34 1 1 1 1 89 | 5 36 -1 -1 1 0 90 | 6 2 1 -1 -1 1 91 | 6 4 -1 1 -1 1 92 | 6 6 1 -1 -1 1 93 | 6 10 1 -1 -1 0 94 | 6 12 -1 1 -1 1 95 | 6 14 1 -1 -1 1 96 | 6 16 -1 1 -1 0 97 | 6 18 1 -1 -1 1 98 | 6 20 -1 1 -1 0 99 | 6 22 1 -1 1 1 100 | 6 24 -1 1 1 1 101 | 6 26 1 -1 1 1 102 | 6 28 -1 1 1 0 103 | 6 30 1 -1 1 1 104 | 6 34 1 -1 1 1 105 | 6 36 -1 1 1 0 106 | 7 1 1 -1 -1 0 107 | 7 3 -1 1 -1 0 108 | 7 5 1 -1 -1 1 109 | 7 7 -1 1 -1 0 110 | 7 9 1 -1 -1 0 111 | 7 11 -1 1 -1 0 112 | 7 13 1 -1 -1 1 113 | 7 15 -1 1 -1 0 114 | 7 17 1 -1 -1 1 115 | 7 23 1 1 1 1 116 | 7 27 -1 1 1 0 117 | 7 29 1 -1 1 1 118 | 7 31 -1 1 1 0 119 | 7 33 1 -1 1 1 120 | 8 1 1 -1 -1 1 121 | 8 3 1 1 -1 1 122 | 8 9 1 -1 -1 0 123 | 8 25 1 -1 1 1 124 | 8 29 1 -1 1 1 125 | 8 33 1 -1 1 1 126 | 8 35 1 1 1 1 127 | 9 2 -1 1 -1 0 128 | 9 4 1 -1 -1 0 129 | 9 6 -1 1 -1 0 130 | 9 8 1 -1 -1 0 131 | 9 10 -1 1 -1 0 132 | 9 12 1 -1 -1 0 133 | 9 14 -1 1 -1 0 134 | 9 16 1 -1 -1 0 135 | 9 18 -1 1 -1 0 136 | 9 22 -1 1 1 0 137 | 9 24 -1 -1 1 0 138 | 9 26 -1 1 1 0 139 | 9 28 -1 -1 1 0 140 | 9 30 -1 1 1 0 141 | 9 34 -1 1 1 0 142 | 9 36 1 -1 1 1 143 | 10 1 -1 1 -1 0 144 | 10 3 1 -1 -1 1 145 | 10 5 -1 1 -1 0 146 | 10 7 1 -1 -1 0 147 | 10 9 1 1 -1 1 148 | 10 11 1 -1 -1 1 149 | 10 13 -1 1 -1 0 150 | 10 15 1 -1 -1 1 151 | 10 17 1 1 -1 1 152 | 10 19 -1 -1 -1 0 153 | 10 21 -1 1 1 0 154 | 10 23 1 -1 1 1 155 | 10 25 1 1 1 1 156 | 10 27 1 -1 1 1 157 | 10 29 1 1 1 0 158 | 10 31 -1 -1 1 0 159 | 10 33 1 1 1 1 160 | 10 35 -1 -1 1 0 161 | 11 2 -1 -1 -1 0 162 | 11 4 1 1 -1 1 163 | 11 6 -1 -1 -1 1 164 | 11 8 -1 1 -1 0 165 | 11 14 1 -1 -1 1 166 | 11 16 -1 1 -1 0 167 | 11 18 1 -1 -1 0 168 | 11 20 -1 1 -1 0 169 | 11 22 -1 -1 1 1 170 | 11 24 -1 1 1 0 171 | 11 30 1 -1 1 1 172 | 11 34 1 -1 1 1 173 | 11 36 1 1 1 1 174 | 12 2 1 1 -1 0 175 | 12 4 1 -1 -1 0 176 | 12 6 -1 1 -1 0 177 | 12 8 1 -1 -1 1 178 | 12 12 -1 -1 -1 0 179 | 12 14 -1 1 -1 0 180 | 12 16 1 -1 -1 1 181 | 12 18 -1 1 -1 0 182 | 12 20 -1 -1 -1 0 183 | 12 22 1 1 1 1 184 | 12 24 -1 -1 1 1 185 | 12 26 1 1 1 1 186 | 12 28 1 -1 1 0 187 | 12 30 -1 1 1 0 188 | 12 34 -1 1 1 1 189 | 12 36 1 -1 1 0 190 | 13 1 -1 1 -1 0 191 | 13 3 1 -1 -1 1 192 | 13 5 -1 1 -1 0 193 | 13 7 1 -1 -1 1 194 | 13 11 1 -1 -1 1 195 | 13 13 1 1 -1 1 196 | 13 15 1 -1 -1 1 197 | 13 17 -1 1 -1 0 198 | 13 21 -1 1 1 1 199 | 13 23 1 -1 1 1 200 | 13 27 1 -1 1 1 201 | 13 29 -1 1 1 0 202 | 13 31 1 -1 1 1 203 | 13 33 1 1 1 1 204 | 13 35 -1 -1 1 0 205 | 14 1 -1 1 -1 0 206 | 14 3 -1 -1 -1 0 207 | 14 5 -1 1 -1 0 208 | 14 7 1 -1 -1 0 209 | 14 9 -1 1 -1 0 210 | 14 11 1 -1 -1 0 211 | 14 13 1 1 -1 1 212 | 14 15 1 -1 -1 1 213 | 14 17 1 1 -1 1 214 | 14 19 -1 -1 -1 0 215 | 14 21 1 1 1 0 216 | 14 23 -1 -1 1 0 217 | 14 25 1 1 1 0 218 | 14 27 -1 -1 1 0 219 | 14 29 1 1 1 0 220 | 14 31 -1 -1 1 0 221 | 14 33 1 1 1 1 222 | 14 35 -1 -1 1 0 223 | 15 2 1 -1 -1 1 224 | 15 4 -1 1 -1 0 225 | 15 6 1 -1 -1 0 226 | 15 8 -1 1 -1 0 227 | 15 10 1 -1 -1 0 228 | 15 12 -1 1 -1 1 229 | 15 14 -1 -1 -1 0 230 | 15 18 1 -1 -1 1 231 | 15 22 -1 -1 1 1 232 | 15 24 1 1 1 1 233 | 15 28 -1 1 1 0 234 | 15 32 -1 1 1 1 235 | 15 34 1 -1 1 0 236 | 15 36 -1 1 1 0 237 | 16 2 1 -1 -1 0 238 | 16 4 -1 1 -1 0 239 | 16 6 1 -1 -1 1 240 | 16 8 -1 1 -1 0 241 | 16 16 -1 1 -1 0 242 | 16 18 1 -1 -1 0 243 | 16 20 -1 1 -1 0 244 | 16 22 1 -1 1 0 245 | 16 24 1 1 1 1 246 | 16 26 1 -1 1 1 247 | 16 32 1 1 1 1 248 | 17 2 1 -1 -1 1 249 | 17 6 1 -1 -1 1 250 | 17 12 -1 1 -1 1 251 | 17 16 -1 1 -1 0 252 | 17 18 1 -1 -1 1 253 | 17 22 1 -1 1 1 254 | 17 26 1 -1 1 1 255 | 17 28 -1 1 1 0 256 | 17 30 1 -1 1 1 257 | 17 34 1 -1 1 1 258 | 17 36 1 1 1 1 259 | 18 1 1 -1 -1 0 260 | 18 3 -1 1 -1 0 261 | 18 5 1 -1 -1 0 262 | 18 7 -1 1 -1 0 263 | 18 9 1 -1 -1 1 264 | 18 11 -1 1 -1 0 265 | 18 13 -1 -1 -1 0 266 | 18 15 1 1 -1 0 267 | 18 17 1 -1 -1 0 268 | 18 19 -1 1 -1 0 269 | 18 21 1 -1 1 0 270 | 18 23 -1 1 1 0 271 | 18 25 -1 -1 1 0 272 | 18 27 1 1 1 0 273 | 18 29 -1 -1 1 0 274 | 18 31 1 1 1 0 275 | 18 33 -1 -1 1 0 276 | 18 35 1 1 1 0 277 | 19 2 -1 1 -1 0 278 | 19 4 1 -1 -1 1 279 | 19 6 -1 1 -1 0 280 | 19 8 1 -1 -1 1 281 | 19 10 -1 1 -1 0 282 | 19 12 1 -1 -1 1 283 | 19 16 1 -1 -1 1 284 | 19 20 1 -1 -1 0 285 | 19 22 -1 1 1 0 286 | 19 24 1 -1 1 1 287 | 19 26 -1 1 1 0 288 | 19 28 1 -1 1 1 289 | 19 30 -1 1 1 0 290 | 19 32 1 -1 1 1 291 | 19 34 -1 1 1 0 292 | 20 1 -1 1 -1 0 293 | 20 3 -1 -1 -1 1 294 | 20 5 -1 1 -1 0 295 | 20 7 1 -1 -1 1 296 | 20 9 1 1 -1 1 297 | 20 11 1 -1 -1 1 298 | 20 15 1 -1 -1 1 299 | 20 17 -1 1 -1 0 300 | 20 19 1 -1 -1 1 301 | 20 21 1 1 1 1 302 | 20 23 1 -1 1 1 303 | 20 25 1 1 1 1 304 | 20 27 -1 -1 1 1 305 | 20 29 -1 1 1 1 306 | 20 33 1 1 1 1 307 | 20 35 1 -1 1 1 308 | 21 2 1 -1 -1 1 309 | 21 4 -1 1 -1 0 310 | 21 6 1 -1 -1 1 311 | 21 8 -1 1 -1 0 312 | 21 10 1 -1 -1 1 313 | 21 12 -1 1 -1 0 314 | 21 14 1 -1 -1 1 315 | 21 16 -1 1 -1 0 316 | 21 18 1 -1 -1 1 317 | 21 20 -1 1 -1 0 318 | 21 22 -1 -1 1 0 319 | 21 24 1 1 1 1 320 | 21 26 1 -1 1 1 321 | 21 28 1 1 1 1 322 | 21 30 1 -1 1 1 323 | 21 32 -1 1 1 0 324 | 21 34 1 -1 1 1 325 | 21 36 1 1 1 1 326 | 22 2 1 -1 -1 0 327 | 22 4 -1 1 -1 0 328 | 22 6 1 -1 -1 0 329 | 22 8 -1 1 -1 0 330 | 22 12 -1 1 -1 0 331 | 22 14 1 -1 -1 0 332 | 22 16 -1 1 -1 0 333 | 22 18 1 -1 -1 0 334 | 22 20 -1 1 -1 0 335 | 22 22 1 -1 1 0 336 | 22 24 1 1 1 0 337 | 22 26 1 -1 1 0 338 | 22 28 -1 1 1 0 339 | 22 30 -1 -1 1 0 340 | 22 32 1 1 1 1 341 | 22 34 1 -1 1 0 342 | 22 36 -1 1 1 0 343 | 23 2 -1 1 -1 1 344 | 23 4 1 -1 -1 1 345 | 23 6 -1 1 -1 1 346 | 23 8 1 -1 -1 0 347 | 23 10 -1 1 -1 0 348 | 23 12 1 -1 -1 0 349 | 23 14 -1 1 -1 0 350 | 23 16 1 -1 -1 1 351 | 23 18 -1 1 -1 0 352 | 23 20 1 -1 -1 1 353 | 23 22 1 1 1 0 354 | 23 24 -1 -1 1 0 355 | 23 26 1 1 1 1 356 | 23 28 -1 -1 1 0 357 | 23 30 1 1 1 0 358 | 23 32 -1 -1 1 1 359 | 23 34 1 1 1 1 360 | 23 36 -1 -1 1 1 361 | 24 2 -1 1 -1 0 362 | 24 4 1 -1 -1 1 363 | 24 8 1 -1 -1 1 364 | 24 10 1 1 -1 1 365 | 24 12 1 -1 -1 1 366 | 24 14 1 1 -1 1 367 | 24 16 1 -1 -1 1 368 | 24 18 1 1 -1 1 369 | 24 20 1 -1 -1 0 370 | 24 22 1 1 1 1 371 | 24 24 1 -1 1 1 372 | 24 26 1 1 1 1 373 | 24 28 1 -1 1 1 374 | 24 30 -1 1 1 0 375 | 24 32 -1 -1 1 1 376 | 24 36 -1 -1 1 1 377 | 25 2 -1 1 -1 0 378 | 25 4 1 -1 -1 0 379 | 25 6 -1 1 -1 0 380 | 25 8 1 -1 -1 0 381 | 25 10 1 1 -1 1 382 | 25 12 1 -1 -1 0 383 | 25 14 1 1 -1 1 384 | 25 16 1 -1 -1 0 385 | 25 18 -1 1 -1 0 386 | 25 22 -1 1 1 0 387 | 25 24 1 -1 1 1 388 | 25 26 -1 1 1 0 389 | 25 28 1 -1 1 1 390 | 25 30 -1 1 1 0 391 | 25 32 1 -1 1 1 392 | 25 34 -1 1 1 0 393 | 25 36 -1 -1 1 0 394 | 26 1 -1 1 -1 1 395 | 26 3 1 -1 -1 1 396 | 26 7 -1 -1 -1 1 397 | 26 9 -1 1 -1 0 398 | 26 11 1 -1 -1 1 399 | 26 13 -1 1 -1 0 400 | 26 15 -1 -1 -1 1 401 | 26 19 1 -1 -1 0 402 | 26 23 1 -1 1 1 403 | 26 27 -1 -1 1 1 404 | 26 31 -1 -1 1 1 405 | 26 33 1 1 1 1 406 | 26 35 -1 -1 1 1 407 | 27 2 1 -1 -1 1 408 | 27 4 1 1 -1 1 409 | 27 6 1 -1 -1 1 410 | 27 8 1 1 -1 1 411 | 27 12 1 1 -1 1 412 | 27 16 -1 1 -1 0 413 | 27 18 -1 -1 -1 1 414 | 27 20 1 1 -1 1 415 | 27 22 1 -1 1 1 416 | 27 24 1 1 1 1 417 | 27 26 1 -1 1 1 418 | 27 28 -1 1 1 0 419 | 27 30 -1 -1 1 1 420 | 27 32 -1 1 1 1 421 | 27 34 1 -1 1 1 422 | 27 36 1 1 1 1 423 | 28 2 -1 1 -1 0 424 | 28 4 1 -1 -1 1 425 | 28 6 -1 1 -1 0 426 | 28 8 1 -1 -1 1 427 | 28 12 1 -1 -1 0 428 | 28 14 1 1 -1 0 429 | 28 16 1 -1 -1 0 430 | 28 18 -1 1 -1 0 431 | 28 20 1 -1 -1 1 432 | 28 22 -1 1 1 0 433 | 28 24 1 -1 1 1 434 | 28 28 -1 -1 1 0 435 | 28 30 -1 1 1 0 436 | 28 32 1 -1 1 1 437 | 28 34 -1 1 1 0 438 | 28 36 1 -1 1 0 439 | 29 1 1 1 -1 1 440 | 29 3 1 -1 -1 1 441 | 29 5 1 1 -1 1 442 | 29 7 1 -1 -1 1 443 | 29 9 1 1 -1 1 444 | 29 11 1 -1 -1 1 445 | 29 13 1 1 -1 1 446 | 29 15 1 -1 -1 1 447 | 29 17 1 1 -1 1 448 | 29 19 1 -1 -1 1 449 | 29 23 1 -1 1 1 450 | 29 25 1 1 1 1 451 | 29 27 1 -1 1 1 452 | 29 29 1 1 1 1 453 | 29 31 -1 -1 1 1 454 | 29 33 1 1 1 1 455 | 29 35 -1 -1 1 0 456 | 30 2 1 1 -1 1 457 | 30 6 1 1 -1 1 458 | 30 8 1 -1 -1 0 459 | 30 10 1 1 -1 1 460 | 30 14 1 1 -1 1 461 | 30 16 1 -1 -1 1 462 | 30 18 1 1 -1 1 463 | 30 22 1 1 1 1 464 | 30 24 1 -1 1 1 465 | 30 28 1 -1 1 1 466 | 30 32 1 -1 1 1 467 | 30 34 1 1 1 1 468 | 30 36 1 -1 1 1 469 | 31 2 1 -1 -1 1 470 | 31 4 -1 1 -1 1 471 | 31 6 1 -1 -1 1 472 | 31 8 1 1 -1 1 473 | 31 10 1 -1 -1 1 474 | 31 12 1 1 -1 1 475 | 31 16 1 1 -1 1 476 | 31 18 1 -1 -1 1 477 | 31 20 -1 1 -1 0 478 | 31 24 1 1 1 1 479 | 31 26 1 -1 1 1 480 | 31 30 1 -1 1 1 481 | 31 32 1 1 1 0 482 | 31 34 1 -1 1 1 483 | 32 1 -1 1 -1 0 484 | 32 3 1 -1 -1 0 485 | 32 5 1 1 -1 1 486 | 32 7 1 -1 -1 1 487 | 32 9 -1 1 -1 0 488 | 32 11 1 -1 -1 1 489 | 32 13 -1 1 -1 0 490 | 32 15 -1 -1 -1 0 491 | 32 17 -1 1 -1 0 492 | 32 19 1 -1 -1 0 493 | 32 21 1 1 1 0 494 | 32 23 -1 -1 1 1 495 | 32 25 1 1 1 1 496 | 32 27 -1 -1 1 0 497 | 32 29 1 1 1 0 498 | 32 31 1 -1 1 1 499 | 32 33 1 1 1 1 500 | 32 35 -1 -1 1 0 501 | 33 2 1 -1 -1 1 502 | 33 4 -1 1 -1 1 503 | 33 6 1 -1 -1 1 504 | 33 8 -1 1 -1 1 505 | 33 14 1 -1 -1 1 506 | 33 16 -1 1 -1 1 507 | 33 18 1 -1 -1 1 508 | 33 20 -1 1 -1 1 509 | 33 22 -1 -1 1 1 510 | 33 24 -1 1 1 1 511 | 33 26 -1 -1 1 1 512 | 33 28 1 1 1 1 513 | 33 30 1 -1 1 1 514 | 33 32 1 1 1 1 515 | 33 34 -1 -1 1 1 516 | 33 36 -1 1 1 1 517 | 34 2 -1 1 -1 1 518 | 34 4 1 -1 -1 1 519 | 34 6 -1 1 -1 1 520 | 34 8 1 -1 -1 1 521 | 34 10 1 1 -1 1 522 | 34 12 1 -1 -1 1 523 | 34 14 -1 1 -1 0 524 | 34 16 1 -1 -1 1 525 | 34 18 -1 1 -1 0 526 | 34 20 1 -1 -1 1 527 | 34 22 1 1 1 1 528 | 34 24 -1 -1 1 1 529 | 34 26 1 1 1 1 530 | 34 28 -1 -1 1 0 531 | 34 32 1 -1 1 1 532 | 34 34 -1 1 1 0 533 | 34 36 1 -1 1 1 534 | 35 1 -1 1 -1 0 535 | 35 7 1 -1 -1 1 536 | 35 9 -1 1 -1 0 537 | 35 11 -1 -1 -1 1 538 | 35 13 1 1 -1 1 539 | 35 15 1 -1 -1 1 540 | 35 17 -1 1 -1 1 541 | 35 19 1 -1 -1 1 542 | 35 21 -1 1 1 0 543 | 35 23 1 -1 1 1 544 | 35 25 1 1 1 1 545 | 35 27 -1 -1 1 1 546 | 35 29 1 1 1 1 547 | 35 31 1 -1 1 1 548 | 35 33 1 1 1 1 549 | 35 35 -1 -1 1 1 550 | 36 1 -1 1 -1 0 551 | 36 3 1 -1 -1 1 552 | 36 5 -1 1 -1 0 553 | 36 7 1 -1 -1 1 554 | 36 9 -1 1 -1 0 555 | 36 11 1 -1 -1 1 556 | 36 13 -1 1 -1 0 557 | 36 15 1 -1 -1 1 558 | 36 17 1 1 -1 1 559 | 36 19 -1 -1 -1 1 560 | 36 21 -1 1 1 0 561 | 36 23 -1 -1 1 1 562 | 36 27 1 -1 1 1 563 | 36 29 -1 1 1 0 564 | 36 31 1 -1 1 1 565 | 36 33 1 1 1 1 566 | 36 35 -1 -1 1 1 567 | 37 1 1 -1 -1 1 568 | 37 3 1 1 -1 1 569 | 37 7 -1 1 -1 0 570 | 37 9 -1 -1 -1 0 571 | 37 13 1 -1 -1 1 572 | 37 15 -1 1 -1 1 573 | 37 17 1 -1 -1 1 574 | 37 19 1 1 -1 1 575 | 37 21 1 -1 1 1 576 | 37 23 -1 1 1 0 577 | 37 25 1 -1 1 1 578 | 37 27 -1 1 1 1 579 | 37 29 -1 -1 1 0 580 | 37 31 -1 1 1 0 581 | 37 33 1 -1 1 1 582 | 38 1 1 1 -1 1 583 | 38 3 1 -1 -1 1 584 | 38 5 -1 1 -1 1 585 | 38 7 1 -1 -1 1 586 | 38 9 1 1 -1 1 587 | 38 11 1 -1 -1 1 588 | 38 13 -1 1 -1 1 589 | 38 15 1 -1 -1 0 590 | 38 17 -1 1 -1 0 591 | 38 21 -1 1 1 1 592 | 38 23 1 -1 1 1 593 | 38 25 -1 1 1 0 594 | 38 27 -1 -1 1 1 595 | 38 29 1 1 1 1 596 | 38 33 1 1 1 1 597 | 38 35 -1 -1 1 1 598 | 39 2 1 -1 -1 1 599 | 39 4 -1 1 -1 0 600 | 39 6 1 -1 -1 0 601 | 39 8 -1 1 -1 0 602 | 39 12 -1 1 -1 0 603 | 39 14 1 -1 -1 0 604 | 39 18 1 -1 -1 1 605 | 39 20 -1 1 -1 0 606 | 39 22 1 -1 1 1 607 | 39 26 1 -1 1 0 608 | 39 28 -1 1 1 0 609 | 39 32 1 1 1 1 610 | 39 34 1 -1 1 1 611 | 39 36 1 1 1 0 612 | 40 1 1 -1 -1 1 613 | 40 3 -1 1 -1 1 614 | 40 5 1 -1 -1 1 615 | 40 7 -1 1 -1 1 616 | 40 9 1 -1 -1 1 617 | 40 11 -1 1 -1 1 618 | 40 13 1 -1 -1 1 619 | 40 15 1 1 -1 1 620 | 40 17 1 -1 -1 1 621 | 40 19 -1 1 -1 1 622 | 40 21 -1 -1 1 1 623 | 40 23 1 1 1 1 624 | 40 27 1 1 1 1 625 | 40 29 -1 -1 1 1 626 | 40 33 -1 -1 1 1 627 | 40 35 -1 1 1 1 628 | 41 1 1 -1 -1 0 629 | 41 3 -1 1 -1 0 630 | 41 5 1 -1 -1 0 631 | 41 7 -1 1 -1 0 632 | 41 9 1 -1 -1 0 633 | 41 11 -1 1 -1 0 634 | 41 13 1 -1 -1 0 635 | 41 15 -1 1 -1 0 636 | 41 17 1 -1 -1 1 637 | 41 19 -1 1 -1 1 638 | 41 21 -1 -1 1 0 639 | 41 23 1 1 1 0 640 | 41 25 -1 -1 1 0 641 | 41 27 1 1 1 1 642 | 41 29 -1 -1 1 0 643 | 41 31 1 1 1 0 644 | 41 33 -1 -1 1 0 645 | 41 35 1 1 1 0 646 | 42 2 1 -1 -1 1 647 | 42 4 -1 1 -1 0 648 | 42 6 1 -1 -1 0 649 | 42 8 1 1 -1 1 650 | 42 10 1 -1 -1 0 651 | 42 12 -1 1 -1 0 652 | 42 14 1 -1 -1 1 653 | 42 16 -1 1 -1 0 654 | 42 18 1 -1 -1 1 655 | 42 20 -1 1 -1 0 656 | 42 22 1 -1 1 0 657 | 42 24 1 1 1 1 658 | 42 26 1 -1 1 1 659 | 42 28 1 1 1 1 660 | 42 30 1 -1 1 1 661 | 42 32 1 1 1 0 662 | 42 34 1 -1 1 0 663 | 42 36 -1 1 1 0 664 | 43 1 1 -1 -1 1 665 | 43 3 -1 1 -1 1 666 | 43 5 1 -1 -1 1 667 | 43 7 -1 1 -1 1 668 | 43 9 1 -1 -1 1 669 | 43 11 -1 1 -1 1 670 | 43 13 1 -1 -1 1 671 | 43 15 -1 1 -1 1 672 | 43 17 1 -1 -1 1 673 | 43 19 -1 1 -1 1 674 | 43 21 1 -1 1 1 675 | 43 25 1 -1 1 1 676 | 43 27 1 1 1 1 677 | 43 29 1 -1 1 1 678 | 43 31 1 1 1 1 679 | 43 33 1 -1 1 1 680 | 43 35 1 1 1 1 681 | 44 2 -1 1 -1 0 682 | 44 4 1 -1 -1 0 683 | 44 6 -1 1 -1 0 684 | 44 8 1 -1 -1 0 685 | 44 10 -1 1 -1 0 686 | 44 12 1 -1 -1 0 687 | 44 14 -1 1 -1 0 688 | 44 16 -1 -1 -1 0 689 | 44 18 -1 1 -1 0 690 | 44 20 1 -1 -1 1 691 | 44 22 -1 1 1 0 692 | 44 24 1 -1 1 1 693 | 44 26 -1 1 1 0 694 | 44 28 -1 -1 1 0 695 | 44 30 -1 1 1 0 696 | 44 32 -1 -1 1 0 697 | 44 34 1 1 1 0 698 | 44 36 -1 -1 1 0 699 | 45 2 -1 1 -1 0 700 | 45 4 1 -1 -1 1 701 | 45 6 -1 1 -1 0 702 | 45 8 1 -1 -1 1 703 | 45 12 1 -1 -1 0 704 | 45 14 -1 1 -1 1 705 | 45 16 1 -1 -1 1 706 | 45 18 -1 1 -1 1 707 | 45 20 1 -1 -1 1 708 | 45 24 -1 -1 1 0 709 | 45 26 1 1 1 0 710 | 45 28 -1 -1 1 1 711 | 45 30 1 1 1 1 712 | 45 32 -1 -1 1 1 713 | 45 34 -1 1 1 1 714 | 45 36 1 -1 1 1 715 | 46 1 -1 -1 -1 1 716 | 46 5 1 -1 -1 1 717 | 46 7 -1 1 -1 0 718 | 46 9 -1 -1 -1 0 719 | 46 13 1 -1 -1 1 720 | 46 15 -1 1 -1 0 721 | 46 17 1 -1 -1 0 722 | 46 19 -1 1 -1 1 723 | 46 21 1 -1 1 1 724 | 46 23 -1 1 1 0 725 | 46 25 1 -1 1 1 726 | 46 27 1 1 1 0 727 | 46 29 -1 -1 1 0 728 | 46 31 1 1 1 0 729 | 46 33 1 -1 1 1 730 | 46 35 1 1 1 0 731 | 47 1 1 -1 -1 1 732 | 47 3 -1 1 -1 1 733 | 47 5 1 -1 -1 1 734 | 47 11 -1 1 -1 1 735 | 47 15 -1 1 -1 1 736 | 47 17 1 -1 -1 1 737 | 47 21 1 -1 1 1 738 | 47 25 -1 -1 1 1 739 | 47 27 -1 1 1 1 740 | 47 31 1 1 1 1 741 | 47 33 1 -1 1 1 742 | 48 1 1 -1 -1 1 743 | 48 3 1 1 -1 1 744 | 48 5 1 -1 -1 1 745 | 48 7 -1 1 -1 0 746 | 48 9 1 -1 -1 1 747 | 48 11 -1 1 -1 0 748 | 48 13 1 -1 -1 0 749 | 48 15 -1 1 -1 0 750 | 48 17 1 -1 -1 1 751 | 48 21 -1 -1 1 1 752 | 48 23 -1 1 1 0 753 | 48 25 1 -1 1 1 754 | 48 29 1 -1 1 1 755 | 48 31 -1 1 1 1 756 | 48 33 1 -1 1 1 757 | 48 35 1 1 1 0 758 | 49 1 1 -1 -1 1 759 | 49 3 -1 1 -1 0 760 | 49 5 1 -1 -1 1 761 | 49 7 -1 1 -1 0 762 | 49 9 1 -1 -1 0 763 | 49 11 -1 1 -1 0 764 | 49 13 1 -1 -1 1 765 | 49 17 1 -1 -1 0 766 | 49 19 -1 1 -1 0 767 | 49 21 -1 -1 1 0 768 | 49 23 -1 1 1 0 769 | 49 25 -1 -1 1 1 770 | 49 27 1 1 1 1 771 | 49 29 1 -1 1 1 772 | 49 31 1 1 1 0 773 | 49 33 1 -1 1 1 774 | 49 35 1 1 1 1 775 | 50 1 1 -1 -1 1 776 | 50 3 1 1 -1 1 777 | 50 5 1 -1 -1 1 778 | 50 9 1 -1 -1 1 779 | 50 11 -1 1 -1 1 780 | 50 13 1 -1 -1 1 781 | 50 15 -1 1 -1 0 782 | 50 17 1 -1 -1 1 783 | 50 19 -1 1 -1 1 784 | 50 21 -1 -1 1 1 785 | 50 25 -1 -1 1 1 786 | 50 27 1 1 1 1 787 | 50 29 1 -1 1 1 788 | 50 33 1 -1 1 1 789 | 50 35 1 1 1 1 790 | 51 1 1 -1 -1 1 791 | 51 3 -1 1 -1 1 792 | 51 7 -1 1 -1 1 793 | 51 9 1 -1 -1 1 794 | 51 11 1 1 -1 1 795 | 51 13 1 -1 -1 1 796 | 51 15 1 1 -1 1 797 | 51 17 1 -1 -1 0 798 | 51 19 1 1 -1 1 799 | 51 21 1 -1 1 1 800 | 51 29 1 -1 1 1 801 | 51 31 1 1 1 1 802 | 51 33 1 -1 1 0 803 | 52 1 1 -1 -1 0 804 | 52 3 -1 1 -1 0 805 | 52 5 1 -1 -1 0 806 | 52 7 -1 1 -1 0 807 | 52 9 1 -1 -1 0 808 | 52 11 -1 1 -1 0 809 | 52 13 -1 -1 -1 0 810 | 52 15 -1 1 -1 0 811 | 52 17 1 -1 -1 0 812 | 52 19 -1 1 -1 0 813 | 52 21 1 -1 1 1 814 | 52 23 -1 1 1 0 815 | 52 25 1 -1 1 1 816 | 52 27 1 1 1 1 817 | 52 29 -1 -1 1 0 818 | 52 31 -1 1 1 0 819 | 52 33 1 -1 1 1 820 | 52 35 -1 1 1 0 821 | 53 1 -1 1 -1 1 822 | 53 3 1 -1 -1 1 823 | 53 5 1 1 -1 1 824 | 53 9 1 1 -1 1 825 | 53 11 1 -1 -1 1 826 | 53 13 1 1 -1 1 827 | 53 19 1 -1 -1 1 828 | 53 21 -1 1 1 1 829 | 53 25 -1 1 1 1 830 | 53 27 1 -1 1 1 831 | 53 31 1 -1 1 1 832 | 53 33 1 1 1 1 833 | 53 35 1 -1 1 1 834 | 54 2 1 -1 -1 1 835 | 54 4 -1 1 -1 0 836 | 54 6 1 -1 -1 1 837 | 54 8 1 1 -1 1 838 | 54 10 -1 -1 -1 0 839 | 54 12 -1 1 -1 0 840 | 54 16 -1 1 -1 0 841 | 54 18 1 -1 -1 1 842 | 54 20 -1 1 -1 0 843 | 54 22 1 -1 1 0 844 | 54 24 1 1 1 1 845 | 54 26 -1 -1 1 0 846 | 54 28 1 1 1 1 847 | 54 30 -1 -1 1 0 848 | 54 32 -1 1 1 0 849 | 54 34 -1 -1 1 0 850 | 54 36 1 1 1 1 851 | 55 1 1 -1 -1 1 852 | 55 3 -1 1 -1 1 853 | 55 5 1 -1 -1 1 854 | 55 7 -1 1 -1 0 855 | 55 9 1 -1 -1 1 856 | 55 11 1 1 -1 1 857 | 55 13 1 -1 -1 1 858 | 55 15 -1 1 -1 0 859 | 55 17 1 -1 -1 1 860 | 55 19 -1 1 -1 0 861 | 55 21 1 -1 1 1 862 | 55 23 1 1 1 1 863 | 55 27 1 1 1 1 864 | 55 29 -1 -1 1 0 865 | 55 31 -1 1 1 0 866 | 55 33 -1 -1 1 0 867 | 55 35 1 1 1 1 868 | 56 1 1 -1 -1 0 869 | 56 3 -1 1 -1 0 870 | 56 5 1 -1 -1 0 871 | 56 7 -1 1 -1 0 872 | 56 9 1 -1 -1 0 873 | 56 11 -1 1 -1 0 874 | 56 13 1 -1 -1 1 875 | 56 15 -1 1 -1 0 876 | 56 17 -1 -1 -1 0 877 | 56 19 -1 1 -1 0 878 | 56 21 -1 -1 1 0 879 | 56 23 -1 1 1 1 880 | 56 27 -1 1 1 0 881 | 56 29 1 -1 1 0 882 | 56 31 1 1 1 1 883 | 56 35 1 1 1 1 884 | 57 1 1 -1 -1 1 885 | 57 3 -1 1 -1 0 886 | 57 5 1 -1 -1 1 887 | 57 7 1 1 -1 1 888 | 57 9 1 -1 -1 1 889 | 57 11 -1 1 -1 0 890 | 57 13 1 -1 -1 1 891 | 57 15 -1 1 -1 0 892 | 57 17 1 -1 -1 0 893 | 57 19 1 1 -1 1 894 | 57 21 1 -1 1 1 895 | 57 23 -1 1 1 0 896 | 57 25 1 -1 1 1 897 | 57 27 1 1 1 1 898 | 57 29 1 -1 1 1 899 | 57 33 1 -1 1 1 900 | 57 35 1 1 1 1 901 | 58 2 -1 1 -1 0 902 | 58 4 1 -1 -1 1 903 | 58 6 -1 1 -1 0 904 | 58 8 1 -1 -1 1 905 | 58 10 1 1 -1 1 906 | 58 12 1 -1 -1 1 907 | 58 14 -1 1 -1 0 908 | 58 16 1 -1 -1 1 909 | 58 18 1 1 -1 1 910 | 58 20 1 -1 -1 1 911 | 58 22 -1 1 1 0 912 | 58 24 1 -1 1 1 913 | 58 28 1 -1 1 1 914 | 58 30 -1 1 1 0 915 | 58 32 -1 -1 1 1 916 | 58 34 -1 1 1 0 917 | 58 36 1 -1 1 1 918 | 59 2 1 -1 -1 1 919 | 59 4 -1 1 -1 0 920 | 59 10 -1 -1 -1 0 921 | 59 12 -1 1 -1 0 922 | 59 14 1 -1 -1 1 923 | 59 16 -1 1 -1 0 924 | 59 18 1 -1 -1 0 925 | 59 24 1 1 1 1 926 | 59 26 -1 -1 1 0 927 | 59 28 -1 1 1 0 928 | 59 30 -1 -1 1 1 929 | 59 32 1 1 1 0 930 | 59 34 -1 -1 1 0 931 | 59 36 1 1 1 1 932 | 60 1 1 1 -1 1 933 | 60 3 1 -1 -1 1 934 | 60 7 1 -1 -1 1 935 | 60 9 1 1 -1 1 936 | 60 11 1 -1 -1 1 937 | 60 13 1 1 -1 1 938 | 60 15 1 -1 -1 1 939 | 60 17 1 1 -1 1 940 | 60 19 1 -1 -1 1 941 | 60 23 1 -1 1 1 942 | 60 27 1 -1 1 1 943 | 60 31 1 -1 1 1 944 | 60 33 1 1 1 1 945 | 60 35 1 -1 1 1 946 | 61 2 1 -1 -1 1 947 | 61 4 -1 1 -1 0 948 | 61 6 1 -1 -1 1 949 | 61 8 -1 1 -1 0 950 | 61 12 -1 1 -1 0 951 | 61 14 -1 -1 -1 0 952 | 61 16 -1 1 -1 0 953 | 61 18 1 -1 -1 0 954 | 61 20 -1 1 -1 0 955 | 61 22 -1 -1 1 0 956 | 61 24 -1 1 1 0 957 | 61 26 1 -1 1 0 958 | 61 28 -1 1 1 0 959 | 61 30 1 -1 1 1 960 | 61 32 -1 1 1 0 961 | 61 34 1 -1 1 1 962 | 61 36 -1 1 1 0 963 | 62 1 1 -1 -1 1 964 | 62 3 -1 1 -1 0 965 | 62 5 1 -1 -1 1 966 | 62 7 -1 1 -1 0 967 | 62 9 1 -1 -1 1 968 | 62 11 -1 1 -1 0 969 | 62 13 1 -1 -1 1 970 | 62 15 -1 1 -1 0 971 | 62 17 1 -1 -1 1 972 | 62 19 1 1 -1 1 973 | 62 21 1 -1 1 1 974 | 62 23 -1 1 1 0 975 | 62 25 -1 -1 1 0 976 | 62 27 -1 1 1 1 977 | 62 29 1 -1 1 1 978 | 62 31 1 1 1 1 979 | 62 33 1 -1 1 1 980 | 63 1 1 -1 -1 1 981 | 63 3 -1 1 -1 0 982 | 63 5 1 -1 -1 1 983 | 63 7 -1 1 -1 0 984 | 63 9 1 -1 -1 0 985 | 63 11 -1 1 -1 0 986 | 63 13 1 -1 -1 1 987 | 63 15 -1 1 -1 0 988 | 63 17 1 -1 -1 1 989 | 63 19 -1 1 -1 0 990 | 63 21 1 -1 1 1 991 | 63 23 1 1 1 1 992 | 63 25 -1 -1 1 0 993 | 63 27 1 1 1 1 994 | 63 29 1 -1 1 1 995 | 63 31 -1 1 1 0 996 | 63 33 -1 -1 1 0 997 | 63 35 1 1 1 1 998 | 64 2 -1 1 -1 1 999 | 64 6 -1 1 -1 1 1000 | 64 12 1 -1 -1 1 1001 | 64 14 1 1 -1 1 1002 | 64 16 1 -1 -1 1 1003 | 64 18 -1 1 -1 1 1004 | 64 20 1 -1 -1 1 1005 | 64 30 1 1 1 1 1006 | 64 32 1 -1 1 1 1007 | 64 34 -1 1 1 1 1008 | 64 36 1 -1 1 1 1009 | 65 1 1 -1 -1 0 1010 | 65 3 -1 1 -1 0 1011 | 65 5 1 -1 -1 0 1012 | 65 7 -1 1 -1 0 1013 | 65 9 1 -1 -1 0 1014 | 65 13 -1 -1 -1 1 1015 | 65 15 -1 1 -1 0 1016 | 65 17 -1 -1 -1 0 1017 | 65 19 1 1 -1 1 1018 | 65 21 1 -1 1 1 1019 | 65 23 1 1 1 1 1020 | 65 25 1 -1 1 1 1021 | 65 27 1 1 1 1 1022 | 65 29 1 -1 1 0 1023 | 65 31 -1 1 1 0 1024 | 65 33 1 -1 1 0 1025 | 65 35 -1 1 1 0 1026 | 66 3 1 -1 -1 1 1027 | 66 7 1 -1 -1 1 1028 | 66 9 1 1 -1 1 1029 | 66 11 1 -1 -1 1 1030 | 66 13 1 1 -1 1 1031 | 66 15 1 -1 -1 1 1032 | 66 17 1 1 -1 1 1033 | 66 19 1 -1 -1 1 1034 | 66 23 1 -1 1 1 1035 | 66 25 -1 1 1 0 1036 | 66 27 1 -1 1 1 1037 | 66 31 1 -1 1 1 1038 | 66 33 1 1 1 1 1039 | 67 2 1 -1 -1 0 1040 | 67 4 -1 1 -1 1 1041 | 67 6 1 -1 -1 1 1042 | 67 8 -1 1 -1 1 1043 | 67 12 -1 1 -1 1 1044 | 67 16 -1 1 -1 1 1045 | 67 18 1 -1 -1 0 1046 | 67 20 -1 1 -1 0 1047 | 67 22 -1 -1 1 0 1048 | 67 24 -1 1 1 0 1049 | 67 28 -1 1 1 0 1050 | 67 30 -1 -1 1 0 1051 | 67 34 1 -1 1 1 1052 | 67 36 1 1 1 1 1053 | 68 2 1 -1 -1 0 1054 | 68 4 -1 1 -1 0 1055 | 68 6 1 -1 -1 0 1056 | 68 8 1 1 -1 1 1057 | 68 10 1 -1 -1 1 1058 | 68 12 -1 1 -1 0 1059 | 68 14 1 -1 -1 0 1060 | 68 16 -1 1 -1 0 1061 | 68 20 -1 1 -1 0 1062 | 68 22 -1 -1 1 1 1063 | 68 24 1 1 1 1 1064 | 68 26 1 -1 1 1 1065 | 68 28 1 1 1 1 1066 | 68 30 -1 -1 1 0 1067 | 68 32 1 1 1 1 1068 | 68 36 1 1 1 0 1069 | 69 2 -1 1 -1 0 1070 | 69 6 -1 1 -1 0 1071 | 69 8 -1 -1 -1 0 1072 | 69 12 1 -1 -1 1 1073 | 69 16 1 -1 -1 0 1074 | 69 18 -1 1 -1 0 1075 | 69 22 1 1 1 1 1076 | 69 24 1 -1 1 1 1077 | 69 26 1 1 1 1 1078 | 69 28 -1 -1 1 0 1079 | 69 34 -1 1 1 0 1080 | 70 2 -1 1 -1 0 1081 | 70 4 1 -1 -1 1 1082 | 70 6 1 1 -1 1 1083 | 70 8 1 -1 -1 1 1084 | 70 12 1 -1 -1 1 1085 | 70 14 1 1 -1 1 1086 | 70 16 1 -1 -1 1 1087 | 70 18 -1 1 -1 0 1088 | 70 20 1 -1 -1 0 1089 | 70 22 -1 1 1 0 1090 | 70 24 -1 -1 1 1 1091 | 70 26 1 1 1 1 1092 | 70 28 1 -1 1 0 1093 | 70 30 -1 1 1 0 1094 | 70 32 1 -1 1 1 1095 | 70 34 -1 1 1 0 1096 | 70 36 1 -1 1 1 1097 | 71 2 1 -1 -1 1 1098 | 71 4 -1 1 -1 0 1099 | 71 6 1 -1 -1 1 1100 | 71 8 -1 1 -1 1 1101 | 71 10 1 -1 -1 1 1102 | 71 12 -1 1 -1 1 1103 | 71 14 1 -1 -1 1 1104 | 71 16 -1 1 -1 1 1105 | 71 18 1 -1 -1 1 1106 | 71 20 -1 1 -1 1 1107 | 71 22 -1 -1 1 1 1108 | 71 24 1 1 1 1 1109 | 71 26 -1 -1 1 1 1110 | 71 28 -1 1 1 0 1111 | 71 30 -1 -1 1 1 1112 | 71 32 1 1 1 1 1113 | 71 36 1 1 1 1 1114 | -------------------------------------------------------------------------------- /README.R: -------------------------------------------------------------------------------- 1 | ## Design of the study and data 2 | 3 | ## The experiment had a 2×2×2 design and the factors were ~a~, ~b~, and ~c~. (The description of the experiment is censored because the study is not yet published. We might add more details once that has happened.) The factors ~a~ and ~b~ were within-item and ~c~ was between-items. All three factors are coded using -1 and 1. There were 71 subjects and 36 items, and each subject saw one version of each item. The dependent variable indicates for each trial whether or not the participant used a pronoun in his description of a visual scene (1 if a pronoun was used, 0 otherwise). 4 | 5 | ## We first load the data and whip it into shape: 6 | 7 | 8 | ## [[file:README.org::*Design%20of%20the%20study%20and%20data][Design\ of\ the\ study\ and\ data:1]] 9 | d <- read.table("data/data.tsv", sep="\t", head=T) 10 | d$pronoun <- as.logical(d$pronoun) 11 | head(d) 12 | ## Design\ of\ the\ study\ and\ data:1 ends here 13 | 14 | 15 | 16 | ## #+RESULTS: 17 | ## | subject | item | a | b | c | pronoun | 18 | ## |---------+------+----+----+----+---------| 19 | ## | 1 | 1 | -1 | 1 | -1 | FALSE | 20 | ## | 1 | 3 | 1 | -1 | -1 | TRUE | 21 | ## | 1 | 5 | -1 | 1 | -1 | TRUE | 22 | ## | 1 | 7 | 1 | -1 | -1 | TRUE | 23 | ## | 1 | 9 | -1 | 1 | -1 | FALSE | 24 | ## | 1 | 11 | -1 | -1 | -1 | FALSE | 25 | 26 | 27 | ## [[file:README.org::*Design%20of%20the%20study%20and%20data][Design\ of\ the\ study\ and\ data:2]] 28 | summary(d) 29 | ## Design\ of\ the\ study\ and\ data:2 ends here 30 | 31 | 32 | 33 | ## #+RESULTS: 34 | ## : subject item a b c pronoun 35 | ## : Min. : 1.00 Min. : 1.00 Min. :-1.0000 Min. :-1.00000 Min. :-1.0000 Mode :logical 36 | ## : 1st Qu.:19.00 1st Qu.: 9.00 1st Qu.:-1.0000 1st Qu.:-1.00000 1st Qu.:-1.0000 FALSE:477 37 | ## : Median :36.00 Median :18.00 Median : 1.0000 Median :-1.00000 Median :-1.0000 TRUE :635 38 | ## : Mean :35.89 Mean :18.35 Mean : 0.1421 Mean :-0.02338 Mean :-0.1241 NA's :0 39 | ## : 3rd Qu.:54.00 3rd Qu.:27.25 3rd Qu.: 1.0000 3rd Qu.: 1.00000 3rd Qu.: 1.0000 40 | ## : Max. :71.00 Max. :36.00 Max. : 1.0000 Max. : 1.00000 Max. : 1.0000 41 | 42 | 43 | ## Proportions of pronoun responses per cell of the design: 44 | 45 | 46 | ## [[file:README.org::*Design%20of%20the%20study%20and%20data][Design\ of\ the\ study\ and\ data:3]] 47 | x <- with(d, tapply(pronoun, list(a, b, c), mean)) 48 | dimnames(x) <- list(c("not-a", "a"), c("not-b", "b"), c("not-c", "c")) 49 | x 50 | ## Design\ of\ the\ study\ and\ data:3 ends here 51 | 52 | 53 | 54 | ## #+RESULTS: 55 | ## | | not.b.not.c | b.not.c | not.b.c | b.c | 56 | ## |-------+-------------------+-------------------+-------------------+-------------------| 57 | ## | not-a | 0.264705882352941 | 0.235294117647059 | 0.42 | 0.2 | 58 | ## | a | 0.723404255319149 | 0.957746478873239 | 0.875816993464052 | 0.782945736434108 | 59 | 60 | 61 | ## Looking at the contingency table below, we see that some cells of the design had very few measurements. In fact, only nine subjects contributed measurements to all cells of the design and nine contributed only to 4 of the 8 cells (second table below). The reason for these strong unbalances is that the factors ~a~ and ~b~ were not experimentally controlled but features of the utterances that the participants produced. 62 | 63 | 64 | ## [[file:README.org::*Design%20of%20the%20study%20and%20data][Design\ of\ the\ study\ and\ data:4]] 65 | with(d, table(a, b, c)) 66 | ## Design\ of\ the\ study\ and\ data:4 ends here 67 | 68 | 69 | 70 | ## #+RESULTS: 71 | ## | | a | b | c | Freq | 72 | ## |---+----+----+----+------| 73 | ## | 1 | -1 | -1 | -1 | 34 | 74 | ## | 2 | 1 | -1 | -1 | 282 | 75 | ## | 3 | -1 | 1 | -1 | 238 | 76 | ## | 4 | 1 | 1 | -1 | 71 | 77 | ## | 5 | -1 | -1 | 1 | 100 | 78 | ## | 6 | 1 | -1 | 1 | 153 | 79 | ## | 7 | -1 | 1 | 1 | 105 | 80 | ## | 8 | 1 | 1 | 1 | 129 | 81 | 82 | 83 | 84 | ## [[file:README.org::*Design%20of%20the%20study%20and%20data][Design\ of\ the\ study\ and\ data:5]] 85 | library(dplyr) 86 | 87 | x <- d %>% 88 | group_by(subject) %>% 89 | summarize(nc = length(unique(paste(a,b,c)))) 90 | 91 | table(x$nc) 92 | ## Design\ of\ the\ study\ and\ data:5 ends here 93 | 94 | 95 | 96 | ## #+RESULTS: 97 | ## | Var1 | Freq | 98 | ## |------+------| 99 | ## | 4 | 9 | 100 | ## | 5 | 9 | 101 | ## | 6 | 26 | 102 | ## | 7 | 18 | 103 | ## | 8 | 9 | 104 | 105 | ## The following plot shows the proportion of pronoun responses in all cells of the design. 106 | 107 | 108 | ## [[file:README.org::*Design%20of%20the%20study%20and%20data][Design\ of\ the\ study\ and\ data:6]] 109 | subject.means <- d %>% 110 | group_by(subject, c, a, b) %>% 111 | summarize(prop = mean(pronoun)) 112 | 113 | condition.means <- subject.means %>% 114 | group_by(c, a, b) %>% 115 | summarize(mean = mean(prop), 116 | se = sd(prop)/sqrt(n())) 117 | 118 | library(ggplot2) 119 | 120 | ggplot(condition.means, aes(x=interaction(c, b), fill=factor(a), y=mean)) + 121 | geom_bar(stat="identity", pos="dodge", colour="black") + 122 | geom_errorbar(aes(ymin=mean-se, ymax=mean+se), size=.5, width=.2, position=position_dodge(.9)) + 123 | ylim(c(0,1)) + 124 | theme_bw(base_size=12) + 125 | ylab("Proportion of pronouns") 126 | ## Design\ of\ the\ study\ and\ data:6 ends here 127 | 128 | ## Attempt to model the data with lme4 129 | 130 | ## In principle, lme4 can deal with unbalanced data sets but the low number of data points in some cells of the design means that it is hard to estimate some of the effects. One of these effects is the three-way interaction which depends on the proportions of pronouns in all cells of the design. Unfortunately, this three-way interaction was precisely the effect of interest in the study. 131 | 132 | ## We start with the maximal model justified by the design: 133 | 134 | 135 | ## [[file:README.org::*Attempt%20to%20model%20the%20data%20with%20lme4][Attempt\ to\ model\ the\ data\ with\ lme4:1]] 136 | library(lme4) 137 | ## Attempt\ to\ model\ the\ data\ with\ lme4:1 ends here 138 | 139 | ## [[file:README.org::*Attempt%20to%20model%20the%20data%20with%20lme4][Attempt\ to\ model\ the\ data\ with\ lme4:2]] 140 | m1 <- glmer(pronoun ~ (a + b + c)^3 + 141 | ((a + b + c)^3 | subject) + 142 | ((a + b )^2 | item), 143 | data=d, family="binomial") 144 | ## Attempt\ to\ model\ the\ data\ with\ lme4:2 ends here 145 | 146 | 147 | 148 | ## A side note on the formula notation used above: ~(a + b + c)^3~ is a little known alternative notation for ~a * b * c~. So it gives us parameters for the main effects, the two-way interactions, and the three-way interaction. The benefit of this notation is that it is more convenient during the design stage of the model when we often change the structure of the model. For example if we want to exclude the three-way interaction, we can simply replace the 3 by a 2 (~(a + b + c)^2~) because what the exponent says is up to which level we want to include interactions. 149 | 150 | ## The model above is the most complex model that can be fit given the design. The model has fixed effects terms for all three factors and all their interactions. Following Barr, Levy, Scheepers, Tily (2013), there are also random slopes for all these factors. The exception is ~c~ which was manipulated between items, so there can’t be by-item random-slopes for that factor or any interaction in which this factor is involved. 151 | 152 | ## The attempt to fit this model takes about 30 minutes on my machine and ultimately fails with one of the most colorful collections of warning messages I have ever seen from lme4: 153 | 154 | ## #+BEGIN_EXAMPLE 155 | ## Warning messages: 156 | ## 1: In commonArgs(par, fn, control, environment()) : 157 | ## maxfun < 10 * length(par)^2 is not recommended. 158 | ## 2: In optwrap(optimizer, devfun, start, rho$lower, control = control, : 159 | ## convergence code 1 from bobyqa: bobyqa -- maximum number of function evaluations exceeded 160 | ## 3: In (function (fn, par, lower = rep.int(-Inf, n), upper = rep.int(Inf, : 161 | ## failure to converge in 10000 evaluations 162 | ## Warning messages: 163 | ## 1: In checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv, : 164 | ## unable to evaluate scaled gradient 165 | ## 2: In checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv, : 166 | ## Model failed to converge: degenerate Hessian with 4 negative eigenvalues 167 | ## #+END_EXAMPLE 168 | 169 | ## Ben Bolker, the current maintainer of the lme4 package, somewhere pointed out that the occurrence of a warning does not strictly imply that the model is degenerate, however, one of the above messages explicitly says that convergence failed and examining the model gives us good reasons to belief that: 170 | 171 | 172 | ## [[file:README.org::*Attempt%20to%20model%20the%20data%20with%20lme4][Attempt\ to\ model\ the\ data\ with\ lme4:3]] 173 | summary(m1) 174 | ## Attempt\ to\ model\ the\ data\ with\ lme4:3 ends here 175 | 176 | 177 | 178 | ## #+RESULTS: 179 | ## #+begin_example 180 | ## Generalized linear mixed model fit by maximum likelihood (Laplace Approximation) ['glmerMod'] 181 | ## Family: binomial ( logit ) 182 | ## Formula: pronoun ~ (a + b + c)^3 + ((a + b + c)^3 | subject) + ((a + b)^2 | item) 183 | ## Data: d 184 | 185 | ## AIC BIC logLik deviance df.resid 186 | ## 1015.3 1286.0 -453.6 907.3 1058 187 | 188 | ## Scaled residuals: 189 | ## Min 1Q Median 3Q Max 190 | ## -2.8041 -0.2491 0.0653 0.3344 3.3503 191 | 192 | ## Random effects: 193 | ## Groups Name Variance Std.Dev. Corr 194 | ## subject (Intercept) 18.90141 4.3476 195 | ## a 5.92954 2.4351 0.75 196 | ## b 3.78364 1.9452 0.93 0.93 197 | ## c 7.29737 2.7014 -0.95 -0.89 -0.99 198 | ## a:b 7.02041 2.6496 0.94 0.86 0.96 -0.97 199 | ## a:c 4.46273 2.1125 -0.91 -0.93 -0.99 0.99 -0.99 200 | ## b:c 6.65586 2.5799 -0.90 -0.93 -0.99 0.99 -0.95 0.98 201 | ## a:b:c 8.12665 2.8507 -0.90 -0.93 -0.99 0.99 -0.96 0.99 1.00 202 | ## item (Intercept) 0.07434 0.2726 203 | ## a 0.11726 0.3424 -1.00 204 | ## b 0.01363 0.1168 -1.00 1.00 205 | ## a:b 0.02852 0.1689 -1.00 1.00 0.99 206 | ## Number of obs: 1112, groups: subject, 71; item, 36 207 | 208 | ## Fixed effects: 209 | ## Estimate Std. Error z value Pr(>|z|) 210 | ## (Intercept) 2.7264 1.0488 2.599 0.00934 ** 211 | ## a 4.2605 0.9660 4.410 1.03e-05 *** 212 | ## b 1.9254 0.9223 2.088 0.03684 * 213 | ## c -1.9351 0.9454 -2.047 0.04068 * 214 | ## a:b 2.6403 0.9417 2.804 0.00505 ** 215 | ## a:c -2.2455 0.9285 -2.418 0.01559 * 216 | ## b:c -2.6537 0.9632 -2.755 0.00587 ** 217 | ## a:b:c -2.5660 0.9717 -2.641 0.00827 ** 218 | ## --- 219 | ## Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 220 | 221 | ## Correlation of Fixed Effects: 222 | ## (Intr) a b c a:b a:c b:c 223 | ## a 0.899 224 | ## b 0.920 0.977 225 | ## c -0.959 -0.948 -0.962 226 | ## a:b 0.959 0.948 0.956 -0.985 227 | ## a:c -0.923 -0.972 -0.986 0.957 -0.962 228 | ## b:c -0.937 -0.979 -0.980 0.964 -0.959 0.984 229 | ## a:b:c -0.958 -0.957 -0.958 0.983 -0.978 0.962 0.967 230 | ## convergence code: 0 231 | ## unable to evaluate scaled gradient 232 | ## Model failed to converge: degenerate Hessian with 2 negative eigenvalues 233 | ## failure to converge in 10000 evaluations 234 | 235 | ## Warning messages: 236 | ## 1: In vcov.merMod(object, use.hessian = use.hessian) : 237 | ## variance-covariance matrix computed from finite-difference Hessian is 238 | ## not positive definite or contains NA values: falling back to var-cov estimated from RX 239 | ## 2: In vcov.merMod(object, correlation = correlation, sigm = sig) : 240 | ## variance-covariance matrix computed from finite-difference Hessian is 241 | ## not positive definite or contains NA values: falling back to var-cov estimated from RX 242 | ## #+end_example 243 | 244 | ## Almost all estimates of the correlations of random effects are close to -1 or 1 and all fixed effects are significant. Both is fairly implausible. The standard thing to do in this situation is to simplify the model until it converges without warnings. However, according to Barr et al., the only hard constraint is that the random slopes for the effect of interest (the effect about which we want to make inferences) need to be in the model. This is often overlooked because the title of the paper – /Random effects structure for confirmatory hypothesis testing: Keep it maximal/ – leads many people to think that Barr et al. mandate maximal random effect structures no matter what. 245 | 246 | ## In our case, the effect of interest is the three-way interaction and the simplest possible model is therefore the following: 247 | 248 | 249 | ## [[file:README.org::*Attempt%20to%20model%20the%20data%20with%20lme4][Attempt\ to\ model\ the\ data\ with\ lme4:4]] 250 | m2 <- glmer(pronoun ~ (a + b + c)^3 + 251 | (0 + a : b : c|subject) + 252 | (0 + a : b : c|item), 253 | data=d, family="binomial") 254 | ## Attempt\ to\ model\ the\ data\ with\ lme4:4 ends here 255 | 256 | 257 | 258 | ## #+BEGIN_EXAMPLE 259 | ## Warning messages: 260 | ## 1: In commonArgs(par, fn, control, environment()) : 261 | ## maxfun < 10 * length(par)^2 is not recommended. 262 | ## 2: In (function (fn, par, lower = rep.int(-Inf, n), upper = rep.int(Inf, : 263 | ## failure to converge in 10000 evaluations 264 | ## Warning messages: 265 | ## 1: In checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv, : 266 | ## unable to evaluate scaled gradient 267 | ## 2: In checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv, : 268 | ## Model failed to converge: degenerate Hessian with 2 negative eigenvalues 269 | ## #+END_EXAMPLE 270 | 271 | ## Unfortunately, this model also fails to converge as do all other variations that we tried, including the intercepts-only model. The model fit (see below) looks more reasonable this time but we clearly can’t rely on this model. Since we are already using the simplest permissible model, we reached the end of the line of what we can do with lme4. 272 | 273 | 274 | ## [[file:README.org::*Attempt%20to%20model%20the%20data%20with%20lme4][Attempt\ to\ model\ the\ data\ with\ lme4:5]] 275 | summary(m2) 276 | ## Attempt\ to\ model\ the\ data\ with\ lme4:5 ends here 277 | 278 | ## Using MCMCglmm 279 | 280 | ## The way models are specified with MCMCglmm is similar to lme4. There are two main differences, though. First, we need to specify prior distributions for some parameters. These priors help to keep the model fitting process in the plausible areas of the parameter space. Specifically, this helps to avoid the pathological correlations between random effects found in the first lme4 model. Second, we have to take control of some aspects of the model fitting process which lme4 handles automatically. 281 | 282 | ## Below is the definition of the maximal model corresponding to the first lme4 model (~m1~). 283 | 284 | 285 | ## [[file:README.org::*Using%20MCMCglmm][Using\ MCMCglmm:1]] 286 | library(MCMCglmm) 287 | ## Using\ MCMCglmm:1 ends here 288 | 289 | ## [[file:README.org::*Using%20MCMCglmm][Using\ MCMCglmm:2]] 290 | set.seed(14) 291 | prior.m3 <- list( 292 | R=list(V=1, n=1, fix=1), 293 | G=list(G1=list(V = diag(8), 294 | n = 8, 295 | alpha.mu = rep(0, 8), 296 | alpha.V = diag(8)*25^2), 297 | G2=list(V = diag(4), 298 | n = 4, 299 | alpha.mu = rep(0, 4), 300 | alpha.V = diag(4)*25^2))) 301 | 302 | m3 <- MCMCglmm(pronoun ~ (a + b + c)^3, 303 | ~ us(1 + (a + b + c)^3):subject + 304 | us(1 + (a + b )^2):item, 305 | data = d, 306 | family = "categorical", 307 | prior = prior.m3, 308 | thin = 1, 309 | burnin = 3000, 310 | nitt = 4000) 311 | ## Using\ MCMCglmm:2 ends here 312 | 313 | 314 | 315 | ## The variable ~prior.m3~ contains the specification of the priors. Priors can be defined for the residuals, the fixed effects, and the random effects. Here, we only specify priors for the residuals (~R~) and the random effects (~G~). The distribution used for the priors is the inverse-Wishart distribution, a probability distribution on covariance matrices. The univariate special case of the inverse-Wishart distribution is the inverse-gamma distribution. This form is used as the prior for the variance of the residuals. ~V~ is the scale matrix of the inverse-Wishart and equals 1 because we want the univariate case. ~n~ is the degrees of freedom parameter and is set to 1 which gives us the weakest possible prior. 316 | 317 | ## ~G1~ is the prior definition for the eight subject random effects. ~V~ is set to 8 because we have eight random effects for subjects (intercept, the three factors, their three two-way interactions, and one three-way interaction) and the covariance matrix therefore needs 8×8 entries. Again, ~n~ is set to give us the weakest prior (the lower bound for ~n~ is the number of dimensions). Further, we have parameters ~alpha.mu~ and ~alpha.V~. These specify an additional prior which is used for parameter expansion, basically a trick to improve the rate of convergence. All we care about is that the ~alpha.mu~ is a vector of as many zeros as there are random effects and that ~alpha.V~ is a n×n matrix with large numbers on the diagonal and n being the number of random effects. See [[https://cran.r-project.org/web/packages/MCMCglmm/vignettes/Overview.pdf][Hadfield (2010)]] and Hadfield’s [[https://cran.r-project.org/web/packages/MCMCglmm/vignettes/CourseNotes.pdf][course notes]] on MCMCglmm for details. 318 | 319 | ## ~G2~ defines the prior for the by-item random effects and follows the same scheme. The only differences is that we have only four item random effects instead of the eight for subjects (because ~c~ is constant within item). In sum, these definitions give us mild priors for the residuals and random effects. 320 | 321 | ## The specification of the model structure is split into two parts. The fixed-effects part looks exactly as in lme4 (=pronoun~(a+b+c)^3=). The random-effects part is a little different. lme4 by default assumes that we want a completely parameterized covariance matrix, that is that we want to estimate the variances of the random effects and all covariances. MCMCglmm wants us to make this explicit. The notation ~us(…)~ can be used to specify parameters for all variances and covariances, in other words it gives us the same random-effects parameters that lme4 would give us by default. One alternative is to use ~idh(…)~ which tells MCMCglmm to estimate parameters for the variances but not for the covariances. 322 | 323 | ## Next, we need to specify the distribution of the residuals and link function to be used in the model. For the glmer model this is ~binomial~, but MCMCglmm uses ~categorical~ which can also be used for dependent variables with more than two levels. 324 | 325 | ## Finally, we need to set some parameters that control the MCMC sampling process. This process uses the data and the model specification to draw samples from the posterior distribution of the parameters and as we collect more and more samples the shape of this distribution emerges more and more clearly. Inferences are then made based on this approximation of the true distribution. The sequence of samples is called a chain (the second /C/ in /MCMC/). 326 | 327 | ## There are three parameters that we need to set to control the sampling process: ~nitt~, ~burnin~, and ~thin~. ~nitt~ is set to 4000 and defines how many samples we want to produce overall. ~burnin~ is set to 3000 and defines the length (in samples) of the so-called burn-in period after which we start collecting samples. The idea behind this is that the first samples may be influenced by the random starting point of the sampling process and may therefore distort our view on the true distribution. Ideally, consecutive samples would be statistically independent, but that is rarely the case in practice. Thinning can be used to reduce the resulting autocorrelation and is controlled by the parameter ~thin~ (more details about thinning below). ~thin=n~ means that we want to keep every n-th sample. Here we set ~thin~ to 1. Effectively, these parameter settings give us 1000 usable samples (4000 - 3000). 328 | 329 | ## Below we see the posterior means and quantiles obtained with the above model. The pattern of results looks qualitatively similar to that in the glmer model but there are considerable numerical differences. However, as mentioned earlier, MCMCglmm does not check convergence and these results may be unreliable. Below we will examine the results more closely to determine whether we can trust the results of this model and the sampling process. 330 | 331 | 332 | ## [[file:README.org::*Using%20MCMCglmm][Using\ MCMCglmm:3]] 333 | summary(m3$Sol) 334 | ## Using\ MCMCglmm:3 ends here 335 | 336 | ## Diagnosing the results using plots 337 | 338 | ## One way to get a sense of whether the samples drawn by MCMCglmm could be an accurate representation of the true posterior is to plot them. In the panels on the left, we see the traces of the parameters showing which values the parameters assumed throughout the sampling process; the index of the sample is on the x-axis (starting with 3000 because we discarded the first 3000 samples) and the value of the parameter in that sample is on the y-axis. In the panels on the right, we see the distribution of the values that the parameters assumed over the course of the sampling process (again ignoring burn-in samples), i.e. the posterior distribution. 339 | 340 | 341 | ## [[file:README.org::*Diagnosing%20the%20results%20using%20plots][Diagnosing\ the\ results\ using\ plots:1]] 342 | par(mfrow=c(8,2), mar=c(2,2,1,0)) 343 | plot(m3$Sol, auto.layout=F) 344 | ## Diagnosing\ the\ results\ using\ plots:1 ends here 345 | 346 | 347 | 348 | ## #+RESULTS: 349 | ## [[file:plots/samples_1.png]] 350 | 351 | ## There are signals in these plots suggesting that our sample may not be good. In general, there is high autocorrelation, which means that samples tend to have similar values as the directly preceding samples. Also the traces are not /stationary/, which means that the sampling process dwells in one part of the parameter space and then visits other parts of the parameter space. This can be observed at around 3900 samples where the trace of ~c~ suddenly moves to more positive values not visited before and the trace of ~a:b~ moves to more negative values. Think about it this way: looking at these plots, is it likely that the density plots on the right would change if we would continue taking samples? Yes, it is because there may be more sudden moves to other parts of the parameter space like that at around 3900. Or the sampling process might dwell in the position reached at 4000 for a longer time leading to a shift in the distributions. For example the density plot of ~a:b~ has a long tail coming from the last ~100 samples and this tail might have gotten fatter if we hadn’t ended the sampling process at 4000 (later we will see that this is exactly what happens). As long as these density plots keep changing, the sampling process has not converged and we don’t have a stable posterior. 352 | 353 | ## Ideally, we would like to have something like the following: 354 | 355 | 356 | ## [[file:README.org::*Diagnosing%20the%20results%20using%20plots][Diagnosing\ the\ results\ using\ plots:2]] 357 | set.seed(1) 358 | par(mfrow=c(1,2), mar=c(2,2,1,0)) 359 | x <- rnorm(1000) 360 | plot(3001:4000, x, t="l", main="Trace of x") 361 | plot(density(x), main="Density of x") 362 | ## Diagnosing\ the\ results\ using\ plots:2 ends here 363 | 364 | 365 | 366 | ## #+RESULTS: 367 | ## [[file:plots/samples_2.png]] 368 | 369 | ## In this trace plot of random data, there is no autocorrelation of consecutive samples and the distribution of samples is stationary. It is very likely that taking more samples wouldn’t shift the distribution substantially. Hence, if we see a plot like this, we would be more confident that our posterior is a good approximation of the true posterior. 370 | 371 | ## # Wiping the floor metaphor useful or not? 372 | 373 | ## How can we reduce autocorrelation? One simple way is thinning. Autocorrelation decays over time, meaning that the correlation of samples tends to be lower the further apart two samples are. Therefore we can lower the autocorrelation by keeping only every n-th sample and discarding the samples in between. The thinning factor is then n. Of course, thinning also requires that we run the sampling process longer to obtain a large-enough set of usable samples. 374 | 375 | ## Let’s have a look at the autocorrelation of samples obtained with the model above. The plots below show for each parameter the autocorrelation as a function of the distance between samples. If the distance is 0, the autocorrelation is one because the correlation of a variable with itself is one. However, as the distance between samples increases the autocorrelation diminishes. We also see that the parameter for the intercept has much lower autocorrelation than the other parameters. 376 | 377 | 378 | ## [[file:README.org::*Diagnosing%20the%20results%20using%20plots][Diagnosing\ the\ results\ using\ plots:3]] 379 | plot.acfs <- function(x) { 380 | n <- dim(x)[2] 381 | par(mfrow=c(ceiling(n/2),2), mar=c(3,2,3,0)) 382 | for (i in 1:n) { 383 | acf(x[,i], lag.max=100, main=colnames(x)[i]) 384 | grid() 385 | } 386 | } 387 | plot.acfs(m3$Sol) 388 | ## Diagnosing\ the\ results\ using\ plots:3 ends here 389 | 390 | 391 | 392 | ## #+RESULTS: 393 | ## [[file:plots/autocorrelation_1.png]] 394 | 395 | ## Now let’s see what happens when we increase the thinning factor from 1 to 20 (~thin=20~). To compensate for the samples that we lose by doing so, we also increase ~nitt~ from 4000 to 23000 (3000 burn-in samples plus 20000 samples of which we keep every twentieth). 396 | 397 | 398 | ## [[file:README.org::*Diagnosing%20the%20results%20using%20plots][Diagnosing\ the\ results\ using\ plots:4]] 399 | set.seed(1) 400 | m4 <- MCMCglmm(pronoun ~ (a + b + c)^3, 401 | ~ us(1 + (a + b + c)^3):subject + 402 | us(1 + (a + b )^2):item, 403 | data = d, 404 | family = "categorical", 405 | prior = prior.m3, 406 | thin = 20, 407 | burnin = 3000, 408 | nitt = 23000) 409 | ## Diagnosing\ the\ results\ using\ plots:4 ends here 410 | 411 | 412 | 413 | ## Examining the plots of the traces, we see that the autocorrelation is indeed much lower and the traces also look much more stationary than before. Inferences, based on this sample are therefore more trustworthy than inferences based on our earlier sample. However, the plots of the autocorrelation shows that there is still a great deal of it. 414 | 415 | 416 | ## [[file:README.org::*Diagnosing%20the%20results%20using%20plots][Diagnosing\ the\ results\ using\ plots:5]] 417 | trace.plots <- function(x) { 418 | n <- dim(x)[2] 419 | par(mfrow=c(ceiling(n/2),2), mar=c(0,0.5,1,0.5)) 420 | for (i in 1:n) { 421 | plot(as.numeric(x[,i]), t="l", main=colnames(x)[i], xaxt="n", yaxt="n") 422 | } 423 | } 424 | trace.plots(m4$Sol) 425 | ## Diagnosing\ the\ results\ using\ plots:5 ends here 426 | 427 | 428 | 429 | ## #+RESULTS: 430 | ## [[file:plots/samples_3.png]] 431 | 432 | 433 | ## [[file:README.org::*Diagnosing%20the%20results%20using%20plots][Diagnosing\ the\ results\ using\ plots:6]] 434 | plot.acfs(m4$Sol) 435 | ## Diagnosing\ the\ results\ using\ plots:6 ends here 436 | 437 | 438 | 439 | ## #+RESULTS: 440 | ## [[file:plots/autocorrelation_2.png]] 441 | 442 | ## At this point, we have to ask: What is the cause of the high autocorrelation? The most likely explanation is that the data is not constraining enough to inform us about the relatively large number of parameters in the model. If that’s the case, one thing we can do is to reduce the number of parameters. Below, we fit a model that has only random intercepts and the random slopes for the effects of interest (the three-way interaction) but no random slopes for the main effects and their two-way interactions. 443 | 444 | 445 | ## [[file:README.org::*Diagnosing%20the%20results%20using%20plots][Diagnosing\ the\ results\ using\ plots:7]] 446 | prior.m5 <- list( 447 | R=list(V=1, n=1, fix=1), 448 | G=list(G1=list(V = diag(2), 449 | n = 2, 450 | alpha.mu = rep(0, 2), 451 | alpha.V = diag(2)*25^2), 452 | G2=list(V = diag(2), 453 | n = 2, 454 | alpha.mu = rep(0, 2), 455 | alpha.V = diag(2)*25^2))) 456 | 457 | m5 <- MCMCglmm(pronoun ~ (a + b + c)^3, 458 | ~ us(1 + a : b : c):subject + 459 | us(1 + a : b ):item, 460 | data = d, 461 | family = "categorical", 462 | prior = prior.m5, 463 | thin = 20, 464 | burnin = 3000, 465 | nitt = 23000) 466 | ## Diagnosing\ the\ results\ using\ plots:7 ends here 467 | 468 | ## [[file:README.org::*Diagnosing%20the%20results%20using%20plots][Diagnosing\ the\ results\ using\ plots:8]] 469 | trace.plots(m5$Sol) 470 | ## Diagnosing\ the\ results\ using\ plots:8 ends here 471 | 472 | 473 | 474 | ## #+RESULTS: 475 | ## [[file:plots/samples_4.png]] 476 | 477 | 478 | ## [[file:README.org::*Diagnosing%20the%20results%20using%20plots][Diagnosing\ the\ results\ using\ plots:9]] 479 | plot.acfs(m5$Sol) 480 | ## Diagnosing\ the\ results\ using\ plots:9 ends here 481 | 482 | ## Gelman-Rubin criterion 483 | 484 | ## The idea is to run multiple chains and to check whether they converged to the same posterior distribution. Since the sampling process is stochastic this is not expected to happen by chance but only when the data was constraining enough to actually tell us something about likely parameter values. 485 | 486 | ## Below we use the package ~parallel~ to run four chains concurrently. This is faster than running one after the other because modern CPUs have several cores that can carry out computations in parallel. The chains are collected in the list ~m6~. 487 | 488 | 489 | ## [[file:README.org::*Gelman-Rubin%20criterion][Gelman-Rubin\ criterion:1]] 490 | library(parallel) 491 | 492 | set.seed(1) 493 | m6 <- mclapply(1:4, function(i) { 494 | MCMCglmm(pronoun ~ (a + b + c)^3, 495 | ~us(1 + a : b : c):subject + 496 | us(1 + a : b) :item, 497 | data = d, 498 | family = "categorical", 499 | prior = prior.m5, 500 | thin = 20, 501 | burnin = 3000, 502 | nitt = 23000) 503 | }, mc.cores=4) 504 | 505 | m6 <- lapply(m6, function(m) m$Sol) 506 | m6 <- do.call(mcmc.list, m6) 507 | ## Gelman-Rubin\ criterion:1 ends here 508 | 509 | 510 | 511 | ## The ~coda~ package provides a lot of functions that are useful for dealing with Markov chains and it also contains an implementation of the Gelman-Rubin criterion (along with a number of other criteria). For those who are interested, the documentation of ~gelman.diag~ contains a formal description of the criterion. 512 | 513 | ## The test statistic is called the scale reduction factor. The closer this factor is to 1, the better the convergence of our chains. In practice, values below 1.1 can be acceptable and values below 1.02 are good. In the plots below, the scale reduction is shown for bins of increasing size (1 to 50, 1 to 60, etc.), thus showing how the scale reduction factor develops over time. 97.5% confidence intervals are indicated by the red dashed line. Note that the x-axis shows the original indices of the samples before thinning. 514 | 515 | 516 | ## [[file:README.org::*Gelman-Rubin%20criterion][Gelman-Rubin\ criterion:2]] 517 | library(coda) 518 | 519 | par(mfrow=c(4,2), mar=c(2,2,1,2)) 520 | gelman.plot(m6, auto.layout=F) 521 | ## Gelman-Rubin\ criterion:2 ends here 522 | 523 | 524 | 525 | ## #+RESULTS: 526 | ## [[file:plots/gelman_rubin.png]] 527 | 528 | ## The plots suggest that the chains converged well enough after roughly half of the samples (after thinning), we say that the chains are mixing at that point. The function ~gelman.diag~ computes the scale reduction factors for each parameter and an overall (multivariate) scale reduction factor. All values suggest that our chains are good to be interpreted. 529 | 530 | 531 | ## [[file:README.org::*Gelman-Rubin%20criterion][Gelman-Rubin\ criterion:3]] 532 | gelman.diag(m6) 533 | ## Gelman-Rubin\ criterion:3 ends here 534 | 535 | 536 | 537 | ## #+RESULTS: 538 | ## #+begin_example 539 | ## Potential scale reduction factors: 540 | 541 | ## Point est. Upper C.I. 542 | ## (Intercept) 1.00 1.01 543 | ## a 1.01 1.03 544 | ## b 1.01 1.02 545 | ## c 1.01 1.02 546 | ## a:b 1.01 1.05 547 | ## a:c 1.02 1.05 548 | ## b:c 1.00 1.00 549 | ## a:b:c 1.01 1.02 550 | 551 | ## Multivariate psrf 552 | 553 | ## 1.04 554 | ## #+end_example 555 | 556 | ## We can also visually confirm that the chains are mixing. Below each chain is plotted in a different color and we see that all chains visit the same parts of the parameter space. 557 | 558 | 559 | ## [[file:README.org::*Gelman-Rubin%20criterion][Gelman-Rubin\ criterion:4]] 560 | par(mfrow=c(8,2), mar=c(2, 1, 1, 1)) 561 | plot(m6, ask=F, auto.layout=F) 562 | ## Gelman-Rubin\ criterion:4 ends here 563 | 564 | ## Results 565 | 566 | ## Having established that our sample is a good approximation of the posterior distribution, we can now move on and examine the results. We first look at the posterior means and the quantiles for each parameter. 567 | 568 | 569 | ## [[file:README.org::*Results][Results:1]] 570 | summary(m6) 571 | ## Results:1 ends here 572 | 573 | 574 | 575 | ## #+RESULTS: 576 | ## #+begin_example 577 | 578 | ## Iterations = 3001:22981 579 | ## Thinning interval = 20 580 | ## Number of chains = 4 581 | ## Sample size per chain = 1000 582 | 583 | ## 1. Empirical mean and standard deviation for each variable, 584 | ## plus standard error of the mean: 585 | 586 | ## Mean SD Naive SE Time-series SE 587 | ## (Intercept) 0.88924 0.3213 0.005081 0.008347 588 | ## a 2.15382 0.1762 0.002786 0.009356 589 | ## b -0.13308 0.1589 0.002513 0.007279 590 | ## c -0.07015 0.1693 0.002676 0.006484 591 | ## a:b 0.63598 0.1649 0.002608 0.006734 592 | ## a:c -0.08589 0.1541 0.002436 0.006564 593 | ## b:c -0.54825 0.1651 0.002610 0.008497 594 | ## a:b:c -0.39332 0.1684 0.002663 0.006357 595 | 596 | ## 2. Quantiles for each variable: 597 | 598 | ## 2.5% 25% 50% 75% 97.5% 599 | ## (Intercept) 0.2844 0.6706 0.87936 1.09855 1.54454 600 | ## a 1.8300 2.0299 2.14763 2.26796 2.51651 601 | ## b -0.4322 -0.2408 -0.13649 -0.03014 0.19503 602 | ## c -0.4219 -0.1855 -0.06182 0.04427 0.25302 603 | ## a:b 0.3280 0.5246 0.63149 0.74883 0.96597 604 | ## a:c -0.3960 -0.1856 -0.08108 0.01903 0.20818 605 | ## b:c -0.8833 -0.6597 -0.54194 -0.43252 -0.24247 606 | ## a:b:c -0.7308 -0.5079 -0.38878 -0.28100 -0.07177 607 | ## #+end_example 608 | 609 | ## And here is a plot of the posterior means along with 95% credible intervals: 610 | 611 | 612 | ## [[file:README.org::*Results][Results:2]] 613 | plot.estimates <- function(x) { 614 | if (class(x) != "summary.mcmc") 615 | x <- summary(x) 616 | n <- dim(x$statistics)[1] 617 | par(mar=c(2, 7, 4, 1)) 618 | plot(x$statistics[,1], n:1, 619 | yaxt="n", ylab="", 620 | xlim=range(x$quantiles)*1.2, 621 | pch=19, 622 | main="Posterior means and 95% credible intervals") 623 | grid() 624 | axis(2, at=n:1, rownames(x$statistics), las=2) 625 | arrows(x$quantiles[,1], n:1, x$quantiles[,5], n:1, code=0) 626 | abline(v=0, lty=2) 627 | } 628 | 629 | plot.estimates(m6) 630 | ## Results:2 ends here 631 | -------------------------------------------------------------------------------- /README.org: -------------------------------------------------------------------------------- 1 | #+TITLE: Using MCMCglmm to implement lme4-like Bayesian mixed-effects models (DRAFT) 2 | #+AUTHOR: Titus von der Malsburg, Meilin Zhan 3 | #+EMAIL: malsburg@ucsd.edu, mezhan@mail.ucsd.edu 4 | #+DATE: 11/06/2015 5 | #+HTML_HEAD: 8 | #+PROPERTY: header-args:R :session *R* :tangle yes :comments both :eval no-export 9 | 10 | # TODO About this document. 11 | # - disclaimer: Many questions will remain open but at least this tutorial will put you in the position to ask those questions. 12 | # - feel free to use issue tracker and PRs are welcome 13 | # - reference to tangled code 14 | 15 | *Note: I wrote this tutorial in 2015 when MCMCglmm was perhaps the best option for fitting Bayesian LMMs in R. In the meantime, much better alternatives have been developed, first and foremost the excellent [[https://github.com/paul-buerkner/brms][brms]] package. Brms is much more powerful and has brilliant documentation. For this reason, I consider this tutorial largely obsolete.* 16 | 17 | The lme4 package for fitting (generalized) linear mixed-effects models has proven to be a powerful tool for analyzing data from experimental studies. However, most people encounter situations in which lme4 does not provide answers because the model-fitting process fails to converge. This is can be the case when the model is too complex to be supported by the available data. Bayesian implementations of mixed-effects models can help in some of these situations because mild priors on the random effects parameters can be used to constrain the search space. In non-technical terms, the Bayesian framework allows us to nudge the model fitting process in the right direction. 18 | 19 | [[https://cran.r-project.org/web/packages/MCMCglmm/index.html][MCMCglmm]] is a package for fitting Bayesian mixed models in R and was written by [[http://jarrod.bio.ed.ac.uk/jarrod.html][Jarrod Hadfield]]. Its use is roughly similar to lme4’s but there are some additional complexities that the user has to deal with. This tutorial aims to get you started with MCMCglmm and shows how the Bayesian analogue of an lme4 model can be implemented with MCMCglmm. 20 | 21 | However, there are more reasons for considering MCMCglmm than just converge issues in lme4. For example, MCMCglmm supports multi-variate dependent variables and a wide variety of distributions, for instance, multinominal, exponential, zero-inflated, and censored distributions. Further, MCMCglmm can be used to implement Hurdle models and models with heterogenous residual variances. In this tutorial, however, we will not make use of these capabilities but simply translate an lme4 model to MCMCglmm. 22 | 23 | Note that this text is not supposed to be a self-contained introduction to Bayesian mixed models or MCMC sampling more generally. At some points we assume that the reader is at least vaguely familiar with the basic ideas behind these concepts. If you are not, give it a shot anyway, and google concepts that you are not familiar with. You can also consult the documentation of MCMCglmm: 24 | 25 | - [[https://cran.r-project.org/web/packages/MCMCglmm/MCMCglmm.pdf][Reference manual]] 26 | - [[https://cran.r-project.org/web/packages/MCMCglmm/vignettes/Overview.pdf][Overview]] 27 | - [[https://cran.r-project.org/web/packages/MCMCglmm/vignettes/CourseNotes.pdf][Course notes]] 28 | 29 | Also feel free to use GitHub’s issue tracker to ask questions or to point out mistakes. If you want to submit a pull request with improvements, that would be great, too. 30 | 31 | Finally, note that the complete code used in this tutorial is also available in a separate R file ([[file:README.R]]). You can download this file and execute the code in your favorite editor for R. The data can be found [[file:data/data.tsv][here]]. 32 | 33 | * Overview 34 | 35 | To illustrate the use of MCMCglmm, we will analyze a data set from a language production experiment. In this experiment, the participants had to describe visual scenes. The aim was to determine the factors governing the participant’s use of pronouns (/he/, /she/, /it/, …) in their descriptions. 36 | 37 | We first attempt an analysis using lme4 and we will see how and why this fails. Then we implement an analogous model using MCMCglmm. MCMCglmm does not perform automatic convergence checks like lme4 does, which means that it is our job to determine whether model fitting was successful. Since there are no definitive criteria for deciding whether the model has successfully converged, we have to make use of some heuristics. (Lme4 also uses heuristics for this purpose but this happens behind the scenes.) We first use some subjective heuristics and then a somewhat more principled criterion for determining convergence (Gelman-Rubin criterion). Finally, we evaluate the model with respect to the research question. 38 | 39 | * Design of the study and data 40 | 41 | The experiment had a 2×2×2 design and the factors were ~a~, ~b~, and ~c~. (The description of the experiment is censored because the study is not yet published. We might add more details once that has happened.) The factors ~a~ and ~b~ were within-item and ~c~ was between-items. All three factors are coded using -1 and 1. There were 71 subjects and 36 items, and each subject saw one version of each item. The dependent variable indicates for each trial whether or not the participant used a pronoun in his description of a visual scene (1 if a pronoun was used, 0 otherwise). 42 | 43 | We first load the data and whip it into shape: 44 | 45 | #+BEGIN_SRC R :exports both :results table :colnames yes 46 | d <- read.table("data/data.tsv", sep="\t", head=T) 47 | d$pronoun <- as.logical(d$pronoun) 48 | head(d) 49 | #+END_SRC 50 | 51 | #+RESULTS: 52 | | subject | item | a | b | c | pronoun | 53 | |---------+------+----+----+----+---------| 54 | | 1 | 1 | -1 | 1 | -1 | FALSE | 55 | | 1 | 3 | 1 | -1 | -1 | TRUE | 56 | | 1 | 5 | -1 | 1 | -1 | TRUE | 57 | | 1 | 7 | 1 | -1 | -1 | TRUE | 58 | | 1 | 9 | -1 | 1 | -1 | FALSE | 59 | | 1 | 11 | -1 | -1 | -1 | FALSE | 60 | 61 | #+BEGIN_SRC R :exports both :results output 62 | summary(d) 63 | #+END_SRC 64 | 65 | #+RESULTS: 66 | : subject item a b c pronoun 67 | : Min. : 1.00 Min. : 1.00 Min. :-1.0000 Min. :-1.00000 Min. :-1.0000 Mode :logical 68 | : 1st Qu.:19.00 1st Qu.: 9.00 1st Qu.:-1.0000 1st Qu.:-1.00000 1st Qu.:-1.0000 FALSE:477 69 | : Median :36.00 Median :18.00 Median : 1.0000 Median :-1.00000 Median :-1.0000 TRUE :635 70 | : Mean :35.89 Mean :18.35 Mean : 0.1421 Mean :-0.02338 Mean :-0.1241 NA's :0 71 | : 3rd Qu.:54.00 3rd Qu.:27.25 3rd Qu.: 1.0000 3rd Qu.: 1.00000 3rd Qu.: 1.0000 72 | : Max. :71.00 Max. :36.00 Max. : 1.0000 Max. : 1.00000 Max. : 1.0000 73 | 74 | 75 | Proportions of pronoun responses per cell of the design: 76 | 77 | #+BEGIN_SRC R :exports both :results table :rownames yes :colnames yes 78 | x <- with(d, tapply(pronoun, list(a, b, c), mean)) 79 | dimnames(x) <- list(c("not-a", "a"), c("not-b", "b"), c("not-c", "c")) 80 | x 81 | #+END_SRC 82 | 83 | #+RESULTS: 84 | | | not.b.not.c | b.not.c | not.b.c | b.c | 85 | |-------+-------------------+-------------------+-------------------+-------------------| 86 | | not-a | 0.264705882352941 | 0.235294117647059 | 0.42 | 0.2 | 87 | | a | 0.723404255319149 | 0.957746478873239 | 0.875816993464052 | 0.782945736434108 | 88 | 89 | 90 | Looking at the contingency table below, we see that some cells of the design had very few measurements. In fact, only nine subjects contributed measurements to all cells of the design and nine contributed only to 4 of the 8 cells (second table below). The reason for these strong unbalances is that the factors ~a~ and ~b~ were not experimentally controlled but features of the utterances that the participants produced. 91 | 92 | #+BEGIN_SRC R :exports both :results table :rownames yes :colnames yes 93 | with(d, table(a, b, c)) 94 | #+END_SRC 95 | 96 | #+RESULTS: 97 | | | a | b | c | Freq | 98 | |---+----+----+----+------| 99 | | 1 | -1 | -1 | -1 | 34 | 100 | | 2 | 1 | -1 | -1 | 282 | 101 | | 3 | -1 | 1 | -1 | 238 | 102 | | 4 | 1 | 1 | -1 | 71 | 103 | | 5 | -1 | -1 | 1 | 100 | 104 | | 6 | 1 | -1 | 1 | 153 | 105 | | 7 | -1 | 1 | 1 | 105 | 106 | | 8 | 1 | 1 | 1 | 129 | 107 | 108 | 109 | #+BEGIN_SRC R :exports both :results table :colnames yes 110 | library(dplyr) 111 | 112 | x <- d %>% 113 | group_by(subject) %>% 114 | summarize(nc = length(unique(paste(a,b,c)))) 115 | 116 | table(x$nc) 117 | #+END_SRC 118 | 119 | #+RESULTS: 120 | | Var1 | Freq | 121 | |------+------| 122 | | 4 | 9 | 123 | | 5 | 9 | 124 | | 6 | 26 | 125 | | 7 | 18 | 126 | | 8 | 9 | 127 | 128 | The following plot shows the proportion of pronoun responses in all cells of the design. 129 | 130 | #+BEGIN_SRC R :results graphics :exports both :file plots/three_way_mean.png :width 400 :height 400 :res 100 131 | subject.means <- d %>% 132 | group_by(subject, c, a, b) %>% 133 | summarize(prop = mean(pronoun)) 134 | 135 | condition.means <- subject.means %>% 136 | group_by(c, a, b) %>% 137 | summarize(mean = mean(prop), 138 | se = sd(prop)/sqrt(n())) 139 | 140 | library(ggplot2) 141 | 142 | ggplot(condition.means, aes(x=interaction(c, b), fill=factor(a), y=mean)) + 143 | geom_bar(stat="identity", pos="dodge", colour="black") + 144 | geom_errorbar(aes(ymin=mean-se, ymax=mean+se), size=.5, width=.2, position=position_dodge(.9)) + 145 | ylim(c(0,1)) + 146 | theme_bw(base_size=12) + 147 | ylab("Proportion of pronouns") 148 | #+END_SRC 149 | 150 | #+RESULTS: 151 | [[file:plots/three_way_mean.png]] 152 | 153 | * Attempt to model the data with lme4 154 | 155 | In principle, lme4 can deal with unbalanced data sets but the low number of data points in some cells of the design means that it is hard to estimate some of the effects. One of these effects is the three-way interaction which depends on the proportions of pronouns in all cells of the design. Unfortunately, this three-way interaction was precisely the effect of interest in the study. 156 | 157 | We start with the maximal model justified by the design: 158 | 159 | #+BEGIN_SRC R :export code :results none 160 | library(lme4) 161 | #+END_SRC 162 | 163 | #+BEGIN_SRC R :export code :results none 164 | m1 <- glmer(pronoun ~ (a + b + c)^3 + 165 | ((a + b + c)^3 | subject) + 166 | ((a + b )^2 | item), 167 | data=d, family="binomial") 168 | #+END_SRC 169 | 170 | A side note on the formula notation used above: ~(a + b + c)^3~ is a little known alternative notation for ~a * b * c~. So it gives us parameters for the main effects, the two-way interactions, and the three-way interaction. The benefit of this notation is that it is more convenient during the design stage of the model when we often change the structure of the model. For example if we want to exclude the three-way interaction, we can simply replace the 3 by a 2 (~(a + b + c)^2~) because what the exponent says is up to which level we want to include interactions. 171 | 172 | The model above is the most complex model that can be fit given the design. The model has fixed effects terms for all three factors and all their interactions. Following Barr, Levy, Scheepers, Tily (2013), there are also random slopes for all these factors. The exception is ~c~ which was manipulated between items, so there can’t be by-item random-slopes for that factor or any interaction in which this factor is involved. 173 | 174 | The attempt to fit this model takes about 30 minutes on my machine and ultimately fails with one of the most colorful collections of warning messages I have ever seen from lme4: 175 | 176 | #+BEGIN_EXAMPLE 177 | Warning messages: 178 | 1: In commonArgs(par, fn, control, environment()) : 179 | maxfun < 10 * length(par)^2 is not recommended. 180 | 2: In optwrap(optimizer, devfun, start, rho$lower, control = control, : 181 | convergence code 1 from bobyqa: bobyqa -- maximum number of function evaluations exceeded 182 | 3: In (function (fn, par, lower = rep.int(-Inf, n), upper = rep.int(Inf, : 183 | failure to converge in 10000 evaluations 184 | Warning messages: 185 | 1: In checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv, : 186 | unable to evaluate scaled gradient 187 | 2: In checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv, : 188 | Model failed to converge: degenerate Hessian with 4 negative eigenvalues 189 | #+END_EXAMPLE 190 | 191 | Ben Bolker, the current maintainer of the lme4 package, somewhere pointed out that the occurrence of a warning does not strictly imply that the model is degenerate, however, one of the above messages explicitly says that convergence failed and examining the model gives us good reasons to belief that: 192 | 193 | #+BEGIN_SRC R :exports both :results output 194 | summary(m1) 195 | #+END_SRC 196 | 197 | #+RESULTS: 198 | #+begin_example 199 | Generalized linear mixed model fit by maximum likelihood (Laplace Approximation) ['glmerMod'] 200 | Family: binomial ( logit ) 201 | Formula: pronoun ~ (a + b + c)^3 + ((a + b + c)^3 | subject) + ((a + b)^2 | item) 202 | Data: d 203 | 204 | AIC BIC logLik deviance df.resid 205 | 1015.3 1286.0 -453.6 907.3 1058 206 | 207 | Scaled residuals: 208 | Min 1Q Median 3Q Max 209 | -2.8041 -0.2491 0.0653 0.3344 3.3503 210 | 211 | Random effects: 212 | Groups Name Variance Std.Dev. Corr 213 | subject (Intercept) 18.90141 4.3476 214 | a 5.92954 2.4351 0.75 215 | b 3.78364 1.9452 0.93 0.93 216 | c 7.29737 2.7014 -0.95 -0.89 -0.99 217 | a:b 7.02041 2.6496 0.94 0.86 0.96 -0.97 218 | a:c 4.46273 2.1125 -0.91 -0.93 -0.99 0.99 -0.99 219 | b:c 6.65586 2.5799 -0.90 -0.93 -0.99 0.99 -0.95 0.98 220 | a:b:c 8.12665 2.8507 -0.90 -0.93 -0.99 0.99 -0.96 0.99 1.00 221 | item (Intercept) 0.07434 0.2726 222 | a 0.11726 0.3424 -1.00 223 | b 0.01363 0.1168 -1.00 1.00 224 | a:b 0.02852 0.1689 -1.00 1.00 0.99 225 | Number of obs: 1112, groups: subject, 71; item, 36 226 | 227 | Fixed effects: 228 | Estimate Std. Error z value Pr(>|z|) 229 | (Intercept) 2.7264 1.0488 2.599 0.00934 ** 230 | a 4.2605 0.9660 4.410 1.03e-05 *** 231 | b 1.9254 0.9223 2.088 0.03684 * 232 | c -1.9351 0.9454 -2.047 0.04068 * 233 | a:b 2.6403 0.9417 2.804 0.00505 ** 234 | a:c -2.2455 0.9285 -2.418 0.01559 * 235 | b:c -2.6537 0.9632 -2.755 0.00587 ** 236 | a:b:c -2.5660 0.9717 -2.641 0.00827 ** 237 | --- 238 | Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 239 | 240 | Correlation of Fixed Effects: 241 | (Intr) a b c a:b a:c b:c 242 | a 0.899 243 | b 0.920 0.977 244 | c -0.959 -0.948 -0.962 245 | a:b 0.959 0.948 0.956 -0.985 246 | a:c -0.923 -0.972 -0.986 0.957 -0.962 247 | b:c -0.937 -0.979 -0.980 0.964 -0.959 0.984 248 | a:b:c -0.958 -0.957 -0.958 0.983 -0.978 0.962 0.967 249 | convergence code: 0 250 | unable to evaluate scaled gradient 251 | Model failed to converge: degenerate Hessian with 2 negative eigenvalues 252 | failure to converge in 10000 evaluations 253 | 254 | Warning messages: 255 | 1: In vcov.merMod(object, use.hessian = use.hessian) : 256 | variance-covariance matrix computed from finite-difference Hessian is 257 | not positive definite or contains NA values: falling back to var-cov estimated from RX 258 | 2: In vcov.merMod(object, correlation = correlation, sigm = sig) : 259 | variance-covariance matrix computed from finite-difference Hessian is 260 | not positive definite or contains NA values: falling back to var-cov estimated from RX 261 | #+end_example 262 | 263 | Almost all estimates of the correlations of random effects are close to -1 or 1 and all fixed effects are significant. Both is fairly implausible. The standard thing to do in this situation is to simplify the model until it converges without warnings. However, according to Barr et al., the only hard constraint is that the random slopes for the effect of interest (the effect about which we want to make inferences) need to be in the model. This is often overlooked because the title of the paper – /Random effects structure for confirmatory hypothesis testing: Keep it maximal/ – leads many people to think that Barr et al. mandate maximal random effect structures no matter what. 264 | 265 | In our case, the effect of interest is the three-way interaction and the simplest possible model is therefore the following: 266 | 267 | #+BEGIN_SRC R :export code :results none 268 | m2 <- glmer(pronoun ~ (a + b + c)^3 + 269 | (0 + a : b : c|subject) + 270 | (0 + a : b : c|item), 271 | data=d, family="binomial") 272 | #+END_SRC 273 | 274 | #+BEGIN_EXAMPLE 275 | Warning messages: 276 | 1: In commonArgs(par, fn, control, environment()) : 277 | maxfun < 10 * length(par)^2 is not recommended. 278 | 2: In (function (fn, par, lower = rep.int(-Inf, n), upper = rep.int(Inf, : 279 | failure to converge in 10000 evaluations 280 | Warning messages: 281 | 1: In checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv, : 282 | unable to evaluate scaled gradient 283 | 2: In checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv, : 284 | Model failed to converge: degenerate Hessian with 2 negative eigenvalues 285 | #+END_EXAMPLE 286 | 287 | Unfortunately, this model also fails to converge as do all other variations that we tried, including the intercepts-only model. The model fit (see below) looks more reasonable this time but we clearly can’t rely on this model. Since we are already using the simplest permissible model, we reached the end of the line of what we can do with lme4. 288 | 289 | #+BEGIN_SRC R :exports results :results output 290 | summary(m2) 291 | #+END_SRC 292 | 293 | #+RESULTS: 294 | #+begin_example 295 | Generalized linear mixed model fit by maximum likelihood (Laplace Approximation) ['glmerMod'] 296 | Family: binomial ( logit ) 297 | Formula: pronoun ~ (a + b + c)^3 + (0 + a:b:c | subject) + (0 + a:b:c | item) 298 | Data: d 299 | 300 | AIC BIC logLik deviance df.resid 301 | 1133.9 1184.0 -556.9 1113.9 1102 302 | 303 | Scaled residuals: 304 | Min 1Q Median 3Q Max 305 | -8.4530 -0.5253 0.2503 0.5369 4.1687 306 | 307 | Random effects: 308 | Groups Name Variance Std.Dev. 309 | subject a:b:c 5.498e-01 0.7415049 310 | item a:b:c 2.524e-07 0.0005024 311 | Number of obs: 1112, groups: subject, 71; item, 36 312 | 313 | Fixed effects: 314 | Estimate Std. Error z value Pr(>|z|) 315 | (Intercept) 0.444294 0.113699 3.908 9.32e-05 *** 316 | a 1.576301 0.118933 13.254 < 2e-16 *** 317 | b 0.062480 0.112741 0.554 0.57945 318 | c -0.008851 0.113678 -0.078 0.93794 319 | a:b 0.360923 0.111885 3.226 0.00126 ** 320 | a:c -0.196345 0.112047 -1.752 0.07972 . 321 | b:c -0.537264 0.114899 -4.676 2.93e-06 *** 322 | a:b:c -0.209187 0.142544 -1.468 0.14223 323 | --- 324 | Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 325 | 326 | Correlation of Fixed Effects: 327 | (Intr) a b c a:b a:c b:c 328 | a 0.235 329 | b 0.253 0.545 330 | c -0.411 -0.194 -0.232 331 | a:b 0.563 0.256 0.234 -0.631 332 | a:c -0.231 -0.428 -0.641 0.222 -0.246 333 | b:c -0.248 -0.640 -0.431 0.237 -0.234 0.565 334 | a:b:c -0.492 -0.166 -0.176 0.443 -0.338 0.192 0.170 335 | convergence code: 0 336 | unable to evaluate scaled gradient 337 | Model failed to converge: degenerate Hessian with 1 negative eigenvalues 338 | #+end_example 339 | 340 | As indicated above, Bayesian mixed models may help in this situation. However, before we embark on an Bayesian adventure, we should consider a much simpler solution: the t-test! The t-test can be used to test whether the difference between two sets of data is significant. Since a three-way interaction is nothing else but a difference of differences of differences, the t-test is perfectly appropriate. The appeal of this is of course that the t-test is simple and relatively fool-proof, plus there is no risk of convergence errors. The approach would be to calculate the differences of differences on a by-subject basis, and to conduct a paired t-test with these values. However, there is one catch. Our data are so sparse that the vast majority of subjects (62 out of 71) do not have measurements in all eight cells of the design. Hence we can calculate the necessary difference values only for a tiny subset of the subjects. 341 | 342 | * Using MCMCglmm 343 | 344 | The way models are specified with MCMCglmm is similar to lme4. There are two main differences, though. First, we need to specify prior distributions for some parameters. These priors help to keep the model fitting process in the plausible areas of the parameter space. Specifically, this helps to avoid the pathological correlations between random effects found in the first lme4 model. Second, we have to take control of some aspects of the model fitting process which lme4 handles automatically. 345 | 346 | Below is the definition of the maximal model corresponding to the first lme4 model (~m1~). 347 | 348 | #+BEGIN_SRC R :export code :results none 349 | library(MCMCglmm) 350 | #+END_SRC 351 | 352 | #+BEGIN_SRC R :export code :results none 353 | set.seed(14) 354 | prior.m3 <- list( 355 | R=list(V=1, n=1, fix=1), 356 | G=list(G1=list(V = diag(8), 357 | n = 8, 358 | alpha.mu = rep(0, 8), 359 | alpha.V = diag(8)*25^2), 360 | G2=list(V = diag(4), 361 | n = 4, 362 | alpha.mu = rep(0, 4), 363 | alpha.V = diag(4)*25^2))) 364 | 365 | m3 <- MCMCglmm(pronoun ~ (a + b + c)^3, 366 | ~ us(1 + (a + b + c)^3):subject + 367 | us(1 + (a + b )^2):item, 368 | data = d, 369 | family = "categorical", 370 | prior = prior.m3, 371 | thin = 1, 372 | burnin = 3000, 373 | nitt = 4000) 374 | #+END_SRC 375 | 376 | The variable ~prior.m3~ contains the specification of the priors. Priors can be defined for the residuals, the fixed effects, and the random effects. Here, we only specify priors for the residuals (~R~) and the random effects (~G~). The distribution used for the priors is the inverse-Wishart distribution, a probability distribution on covariance matrices. The univariate special case of the inverse-Wishart distribution is the inverse-gamma distribution. This form is used as the prior for the variance of the residuals. ~V~ is the scale matrix of the inverse-Wishart and equals 1 because we want the univariate case. ~n~ is the degrees of freedom parameter and is set to 1 which gives us the weakest possible prior. 377 | 378 | ~G1~ is the prior definition for the eight subject random effects. ~V~ is set to 8 because we have eight random effects for subjects (intercept, the three factors, their three two-way interactions, and one three-way interaction) and the covariance matrix therefore needs 8×8 entries. Again, ~n~ is set to give us the weakest prior (the lower bound for ~n~ is the number of dimensions). Further, we have parameters ~alpha.mu~ and ~alpha.V~. These specify an additional prior which is used for parameter expansion, basically a trick to improve the rate of convergence. All we care about is that the ~alpha.mu~ is a vector of as many zeros as there are random effects and that ~alpha.V~ is a n×n matrix with large numbers on the diagonal and n being the number of random effects. See [[https://cran.r-project.org/web/packages/MCMCglmm/vignettes/Overview.pdf][Hadfield (2010)]] and Hadfield’s [[https://cran.r-project.org/web/packages/MCMCglmm/vignettes/CourseNotes.pdf][course notes]] on MCMCglmm for details. 379 | 380 | ~G2~ defines the prior for the by-item random effects and follows the same scheme. The only differences is that we have only four item random effects instead of the eight for subjects (because ~c~ is constant within item). In sum, these definitions give us mild priors for the residuals and random effects. 381 | 382 | The specification of the model structure is split into two parts. The fixed-effects part looks exactly as in lme4 (=pronoun~(a+b+c)^3=). The random-effects part is a little different. lme4 by default assumes that we want a completely parameterized covariance matrix, that is that we want to estimate the variances of the random effects and all covariances. MCMCglmm wants us to make this explicit. The notation ~us(…)~ can be used to specify parameters for all variances and covariances, in other words it gives us the same random-effects parameters that lme4 would give us by default. One alternative is to use ~idh(…)~ which tells MCMCglmm to estimate parameters for the variances but not for the covariances. 383 | 384 | Next, we need to specify the distribution of the residuals and link function to be used in the model. For the glmer model this is ~binomial~, but MCMCglmm uses ~categorical~ which can also be used for dependent variables with more than two levels. 385 | 386 | Finally, we need to set some parameters that control the MCMC sampling process. This process uses the data and the model specification to draw samples from the posterior distribution of the parameters and as we collect more and more samples the shape of this distribution emerges more and more clearly. Inferences are then made based on this approximation of the true distribution. The sequence of samples is called a chain (the second /C/ in /MCMC/). 387 | 388 | There are three parameters that we need to set to control the sampling process: ~nitt~, ~burnin~, and ~thin~. ~nitt~ is set to 4000 and defines how many samples we want to produce overall. ~burnin~ is set to 3000 and defines the length (in samples) of the so-called burn-in period after which we start collecting samples. The idea behind this is that the first samples may be influenced by the random starting point of the sampling process and may therefore distort our view on the true distribution. Ideally, consecutive samples would be statistically independent, but that is rarely the case in practice. Thinning can be used to reduce the resulting autocorrelation and is controlled by the parameter ~thin~ (more details about thinning below). ~thin=n~ means that we want to keep every n-th sample. Here we set ~thin~ to 1. Effectively, these parameter settings give us 1000 usable samples (4000 - 3000). 389 | 390 | Below we see the posterior means and quantiles obtained with the above model. The pattern of results looks qualitatively similar to that in the glmer model but there are considerable numerical differences. However, as mentioned earlier, MCMCglmm does not check convergence and these results may be unreliable. Below we will examine the results more closely to determine whether we can trust the results of this model and the sampling process. 391 | 392 | #+BEGIN_SRC R :exports both :results output 393 | summary(m3$Sol) 394 | #+END_SRC 395 | 396 | #+RESULTS: 397 | #+begin_example 398 | 399 | Iterations = 3001:4000 400 | Thinning interval = 1 401 | Number of chains = 1 402 | Sample size per chain = 1000 403 | 404 | 1. Empirical mean and standard deviation for each variable, 405 | plus standard error of the mean: 406 | 407 | Mean SD Naive SE Time-series SE 408 | (Intercept) 1.3475 0.4189 0.013246 0.06731 409 | a 3.1882 0.2967 0.009382 0.06020 410 | b -0.2202 0.2300 0.007275 0.06802 411 | c 0.0577 0.2299 0.007271 0.05356 412 | a:b 0.8467 0.3243 0.010257 0.13246 413 | a:c -0.2605 0.2454 0.007759 0.09630 414 | b:c -1.1221 0.2007 0.006348 0.03561 415 | a:b:c -0.9962 0.2921 0.009238 0.10529 416 | 417 | 2. Quantiles for each variable: 418 | 419 | 2.5% 25% 50% 75% 97.5% 420 | (Intercept) 0.52905 1.0558 1.35092 1.63646 2.2106 421 | a 2.61218 2.9793 3.19866 3.40216 3.7413 422 | b -0.61128 -0.3816 -0.24456 -0.06253 0.2465 423 | c -0.33693 -0.1002 0.02712 0.19129 0.5865 424 | a:b 0.01218 0.6840 0.88057 1.06400 1.3636 425 | a:c -0.71437 -0.4479 -0.25036 -0.07384 0.1743 426 | b:c -1.52459 -1.2596 -1.10782 -0.98058 -0.7350 427 | a:b:c -1.50290 -1.2142 -1.01716 -0.78711 -0.4160 428 | #+end_example 429 | 430 | * Diagnosing the results using plots 431 | 432 | One way to get a sense of whether the samples drawn by MCMCglmm could be an accurate representation of the true posterior is to plot them. In the panels on the left, we see the traces of the parameters showing which values the parameters assumed throughout the sampling process; the index of the sample is on the x-axis (starting with 3000 because we discarded the first 3000 samples) and the value of the parameter in that sample is on the y-axis. In the panels on the right, we see the distribution of the values that the parameters assumed over the course of the sampling process (again ignoring burn-in samples), i.e. the posterior distribution. 433 | 434 | #+BEGIN_SRC R :exports both :results graphics :file plots/samples_1.png :width 800 :height 1000 :res 100 435 | par(mfrow=c(8,2), mar=c(2,2,1,0)) 436 | plot(m3$Sol, auto.layout=F) 437 | #+END_SRC 438 | 439 | #+RESULTS: 440 | [[file:plots/samples_1.png]] 441 | 442 | There are signals in these plots suggesting that our sample may not be good. In general, there is high autocorrelation, which means that samples tend to have similar values as the directly preceding samples. Also the traces are not /stationary/, which means that the sampling process dwells in one part of the parameter space and then visits other parts of the parameter space. This can be observed at around 3900 samples where the trace of ~c~ suddenly moves to more positive values not visited before and the trace of ~a:b~ moves to more negative values. Think about it this way: looking at these plots, is it likely that the density plots on the right would change if we would continue taking samples? Yes, it is because there may be more sudden moves to other parts of the parameter space like that at around 3900. Or the sampling process might dwell in the position reached at 4000 for a longer time leading to a shift in the distributions. For example the density plot of ~a:b~ has a long tail coming from the last ~100 samples and this tail might have gotten fatter if we hadn’t ended the sampling process at 4000 (later we will see that this is exactly what happens). As long as these density plots keep changing, the sampling process has not converged and we don’t have a stable posterior. 443 | 444 | Ideally, we would like to have something like the following: 445 | 446 | #+BEGIN_SRC R :exports both :results graphics :file plots/samples_2.png :width 800 :height 125 :res 60 447 | set.seed(1) 448 | par(mfrow=c(1,2), mar=c(2,2,1,0)) 449 | x <- rnorm(1000) 450 | plot(3001:4000, x, t="l", main="Trace of x") 451 | plot(density(x), main="Density of x") 452 | #+END_SRC 453 | 454 | #+RESULTS: 455 | [[file:plots/samples_2.png]] 456 | 457 | In this trace plot of random data, there is no autocorrelation of consecutive samples and the distribution of samples is stationary. It is very likely that taking more samples wouldn’t shift the distribution substantially. Hence, if we see a plot like this, we would be more confident that our posterior is a good approximation of the true posterior. 458 | 459 | # Wiping the floor metaphor useful or not? 460 | 461 | How can we reduce autocorrelation? One simple way is thinning. Autocorrelation decays over time, meaning that the correlation of samples tends to be lower the further apart two samples are. Therefore we can lower the autocorrelation by keeping only every n-th sample and discarding the samples in between. The thinning factor is then n. Of course, thinning also requires that we run the sampling process longer to obtain a large-enough set of usable samples. 462 | 463 | Let’s have a look at the autocorrelation of samples obtained with the model above. The plots below show for each parameter the autocorrelation as a function of the distance between samples. If the distance is 0, the autocorrelation is one because the correlation of a variable with itself is one. However, as the distance between samples increases the autocorrelation diminishes. We also see that the parameter for the intercept has much lower autocorrelation than the other parameters. 464 | 465 | #+BEGIN_SRC R :exports both :results graphics :file plots/autocorrelation_1.png :width 800 :height 600 :res 100 466 | plot.acfs <- function(x) { 467 | n <- dim(x)[2] 468 | par(mfrow=c(ceiling(n/2),2), mar=c(3,2,3,0)) 469 | for (i in 1:n) { 470 | acf(x[,i], lag.max=100, main=colnames(x)[i]) 471 | grid() 472 | } 473 | } 474 | plot.acfs(m3$Sol) 475 | #+END_SRC 476 | 477 | #+RESULTS: 478 | [[file:plots/autocorrelation_1.png]] 479 | 480 | Now let’s see what happens when we increase the thinning factor from 1 to 20 (~thin=20~). To compensate for the samples that we lose by doing so, we also increase ~nitt~ from 4000 to 23000 (3000 burn-in samples plus 20000 samples of which we keep every twentieth). 481 | 482 | #+BEGIN_SRC R :export code :results none 483 | set.seed(1) 484 | m4 <- MCMCglmm(pronoun ~ (a + b + c)^3, 485 | ~ us(1 + (a + b + c)^3):subject + 486 | us(1 + (a + b )^2):item, 487 | data = d, 488 | family = "categorical", 489 | prior = prior.m3, 490 | thin = 20, 491 | burnin = 3000, 492 | nitt = 23000) 493 | #+END_SRC 494 | 495 | Examining the plots of the traces, we see that the autocorrelation is indeed much lower and the traces also look much more stationary than before. Inferences, based on this sample are therefore more trustworthy than inferences based on our earlier sample. However, the plots of the autocorrelation shows that there is still a great deal of it. 496 | 497 | #+BEGIN_SRC R :exports both :results graphics :file plots/samples_3.png :width 800 :height 400 :res 100 498 | trace.plots <- function(x) { 499 | n <- dim(x)[2] 500 | par(mfrow=c(ceiling(n/2),2), mar=c(0,0.5,1,0.5)) 501 | for (i in 1:n) { 502 | plot(as.numeric(x[,i]), t="l", main=colnames(x)[i], xaxt="n", yaxt="n") 503 | } 504 | } 505 | trace.plots(m4$Sol) 506 | #+END_SRC 507 | 508 | #+RESULTS: 509 | [[file:plots/samples_3.png]] 510 | 511 | #+BEGIN_SRC R :exports both :results graphics :file plots/autocorrelation_2.png :width 800 :height 600 :res 100 512 | plot.acfs(m4$Sol) 513 | #+END_SRC 514 | 515 | #+RESULTS: 516 | [[file:plots/autocorrelation_2.png]] 517 | 518 | At this point, we have to ask: What is the cause of the high autocorrelation? The most likely explanation is that the data is not constraining enough to inform us about the relatively large number of parameters in the model. If that’s the case, one thing we can do is to reduce the number of parameters. Below, we fit a model that has only random intercepts and the random slopes for the effects of interest (the three-way interaction) but no random slopes for the main effects and their two-way interactions. 519 | 520 | #+BEGIN_SRC R :export code :results none 521 | prior.m5 <- list( 522 | R=list(V=1, n=1, fix=1), 523 | G=list(G1=list(V = diag(2), 524 | n = 2, 525 | alpha.mu = rep(0, 2), 526 | alpha.V = diag(2)*25^2), 527 | G2=list(V = diag(2), 528 | n = 2, 529 | alpha.mu = rep(0, 2), 530 | alpha.V = diag(2)*25^2))) 531 | 532 | m5 <- MCMCglmm(pronoun ~ (a + b + c)^3, 533 | ~ us(1 + a : b : c):subject + 534 | us(1 + a : b ):item, 535 | data = d, 536 | family = "categorical", 537 | prior = prior.m5, 538 | thin = 20, 539 | burnin = 3000, 540 | nitt = 23000) 541 | #+END_SRC 542 | 543 | #+BEGIN_SRC R :exports both :results graphics :file plots/samples_4.png :width 800 :height 400 :res 100 544 | trace.plots(m5$Sol) 545 | #+END_SRC 546 | 547 | #+RESULTS: 548 | [[file:plots/samples_4.png]] 549 | 550 | #+BEGIN_SRC R :exports both :results graphics :file plots/autocorrelation_3.png :width 800 :height 600 :res 100 551 | plot.acfs(m5$Sol) 552 | #+END_SRC 553 | 554 | #+RESULTS: 555 | [[file:plots/autocorrelation_3.png]] 556 | 557 | This looks much better than what we had before but the situation is still somewhat unsatisfying because so far we have no objective test for determining whether the obtained samples are good enough. The Gelman-Rubin criterion is such an objective test. 558 | 559 | * Gelman-Rubin criterion 560 | 561 | The idea is to run multiple chains and to check whether they converged to the same posterior distribution. Since the sampling process is stochastic this is not expected to happen by chance but only when the data was constraining enough to actually tell us something about likely parameter values. 562 | 563 | Below we use the package ~parallel~ to run four chains concurrently. This is faster than running one after the other because modern CPUs have several cores that can carry out computations in parallel. The chains are collected in the list ~m6~. 564 | 565 | #+BEGIN_SRC R :export code :results none 566 | library(parallel) 567 | 568 | set.seed(1) 569 | m6 <- mclapply(1:4, function(i) { 570 | MCMCglmm(pronoun ~ (a + b + c)^3, 571 | ~us(1 + a : b : c):subject + 572 | us(1 + a : b) :item, 573 | data = d, 574 | family = "categorical", 575 | prior = prior.m5, 576 | thin = 20, 577 | burnin = 3000, 578 | nitt = 23000) 579 | }, mc.cores=4) 580 | 581 | m6 <- lapply(m6, function(m) m$Sol) 582 | m6 <- do.call(mcmc.list, m6) 583 | #+END_SRC 584 | 585 | The ~coda~ package provides a lot of functions that are useful for dealing with Markov chains and it also contains an implementation of the Gelman-Rubin criterion (along with a number of other criteria). For those who are interested, the documentation of ~gelman.diag~ contains a formal description of the criterion. 586 | 587 | The test statistic is called the scale reduction factor. The closer this factor is to 1, the better the convergence of our chains. In practice, values below 1.1 can be acceptable and values below 1.02 are good. In the plots below, the scale reduction is shown for bins of increasing size (1 to 50, 1 to 60, etc.), thus showing how the scale reduction factor develops over time. 97.5% confidence intervals are indicated by the red dashed line. Note that the x-axis shows the original indices of the samples before thinning. 588 | 589 | #+BEGIN_SRC R :exports both :results graphics :file plots/gelman_rubin.png :width 800 :height 600 :res 100 590 | library(coda) 591 | 592 | par(mfrow=c(4,2), mar=c(2,2,1,2)) 593 | gelman.plot(m6, auto.layout=F) 594 | #+END_SRC 595 | 596 | #+RESULTS: 597 | [[file:plots/gelman_rubin.png]] 598 | 599 | The plots suggest that the chains converged well enough after roughly half of the samples (after thinning), we say that the chains are mixing at that point. The function ~gelman.diag~ computes the scale reduction factors for each parameter and an overall (multivariate) scale reduction factor. All values suggest that our chains are good to be interpreted. 600 | 601 | #+BEGIN_SRC R :exports both :results output 602 | gelman.diag(m6) 603 | #+END_SRC 604 | 605 | #+RESULTS: 606 | #+begin_example 607 | Potential scale reduction factors: 608 | 609 | Point est. Upper C.I. 610 | (Intercept) 1.00 1.01 611 | a 1.01 1.03 612 | b 1.01 1.02 613 | c 1.01 1.02 614 | a:b 1.01 1.05 615 | a:c 1.02 1.05 616 | b:c 1.00 1.00 617 | a:b:c 1.01 1.02 618 | 619 | Multivariate psrf 620 | 621 | 1.04 622 | #+end_example 623 | 624 | We can also visually confirm that the chains are mixing. Below each chain is plotted in a different color and we see that all chains visit the same parts of the parameter space. 625 | 626 | #+BEGIN_SRC R :exports both :results graphics :file plots/samples_5.png :width 800 :height 1000 :res 100 627 | par(mfrow=c(8,2), mar=c(2, 1, 1, 1)) 628 | plot(m6, ask=F, auto.layout=F) 629 | #+END_SRC 630 | 631 | #+RESULTS: 632 | [[file:plots/samples_5.png]] 633 | 634 | 635 | * Results 636 | 637 | Having established that our sample is a good approximation of the posterior distribution, we can now move on and examine the results. We first look at the posterior means and the quantiles for each parameter. 638 | 639 | #+BEGIN_SRC R :exports both :results output 640 | summary(m6) 641 | #+END_SRC 642 | 643 | #+RESULTS: 644 | #+begin_example 645 | 646 | Iterations = 3001:22981 647 | Thinning interval = 20 648 | Number of chains = 4 649 | Sample size per chain = 1000 650 | 651 | 1. Empirical mean and standard deviation for each variable, 652 | plus standard error of the mean: 653 | 654 | Mean SD Naive SE Time-series SE 655 | (Intercept) 0.88924 0.3213 0.005081 0.008347 656 | a 2.15382 0.1762 0.002786 0.009356 657 | b -0.13308 0.1589 0.002513 0.007279 658 | c -0.07015 0.1693 0.002676 0.006484 659 | a:b 0.63598 0.1649 0.002608 0.006734 660 | a:c -0.08589 0.1541 0.002436 0.006564 661 | b:c -0.54825 0.1651 0.002610 0.008497 662 | a:b:c -0.39332 0.1684 0.002663 0.006357 663 | 664 | 2. Quantiles for each variable: 665 | 666 | 2.5% 25% 50% 75% 97.5% 667 | (Intercept) 0.2844 0.6706 0.87936 1.09855 1.54454 668 | a 1.8300 2.0299 2.14763 2.26796 2.51651 669 | b -0.4322 -0.2408 -0.13649 -0.03014 0.19503 670 | c -0.4219 -0.1855 -0.06182 0.04427 0.25302 671 | a:b 0.3280 0.5246 0.63149 0.74883 0.96597 672 | a:c -0.3960 -0.1856 -0.08108 0.01903 0.20818 673 | b:c -0.8833 -0.6597 -0.54194 -0.43252 -0.24247 674 | a:b:c -0.7308 -0.5079 -0.38878 -0.28100 -0.07177 675 | #+end_example 676 | 677 | And here is a plot of the posterior means along with 95% credible intervals: 678 | 679 | #+BEGIN_SRC R :exports both :results graphics :file plots/parameter_estimates.png :width 600 :height 300 :res 80 680 | plot.estimates <- function(x) { 681 | if (class(x) != "summary.mcmc") 682 | x <- summary(x) 683 | n <- dim(x$statistics)[1] 684 | par(mar=c(2, 7, 4, 1)) 685 | plot(x$statistics[,1], n:1, 686 | yaxt="n", ylab="", 687 | xlim=range(x$quantiles)*1.2, 688 | pch=19, 689 | main="Posterior means and 95% credible intervals") 690 | grid() 691 | axis(2, at=n:1, rownames(x$statistics), las=2) 692 | arrows(x$quantiles[,1], n:1, x$quantiles[,5], n:1, code=0) 693 | abline(v=0, lty=2) 694 | } 695 | 696 | plot.estimates(m6) 697 | #+END_SRC 698 | 699 | #+RESULTS: 700 | [[file:plots/parameter_estimates.png]] 701 | 702 | As we can see, the three-way interaction is significant. Some other effects are significant, too, but note that we should be careful in interpreting these effects because the model does not include the relevant random slopes and may therefore overestimate the reliability of these effects (see Barr et al., 2013, for details). 703 | 704 | * Summary 705 | 706 | Our attempts to analyze the data with lme4 failed due to convergence errors. Even a radical simplification of the random effects structure did not help. In contrast to that the Bayesian mixed-effects model converged fine after a simplification of the model and a few adjustments to the sampling process. The price for this was that we had to take more control of the model fitting process than we have to when working with lme4 which tries to handle all that automatically. In sum, MCMCglmm is a powerful tool that can be used when lme4 has convergence problems or when models are desired that are outside the scope of what lme4 can do. 707 | 708 | * References 709 | 710 | - Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random 711 | effects structure for confirmatory hypothesis testing: Keep it 712 | maximal. Journal of Memory and Language, 68(3), 713 | 255–278. http://dx.doi.org/10.1016/j.jml.2012.11.001 714 | - Bates, D., Kliegl, R., Vasishth, S., & Baayen, 715 | H. (2015). Parsimonious mixed models. Manuscript published on arXiv. 716 | http://arxiv.org/abs/1506.04967 717 | - Gelman, A., & Rubin, D. B. (1992). Inference from iterative 718 | simulation using multiple sequences. Statistical Science, 7(4), 719 | 457–472. 720 | - Hadfield, J. (2010). MCMC methods for multi-response generalized 721 | linear mixed models: the MCMCglmm R package. Journal of Statistical 722 | Software, 33(1), 1–22. 723 | https://cran.r-project.org/web/packages/MCMCglmm/vignettes/Overview.pdf 724 | - Hadfield, J. (2015). MCMCglmm Course Notes. 725 | https://cran.r-project.org/web/packages/MCMCglmm/vignettes/CourseNotes.pdf 726 | --------------------------------------------------------------------------------